Vmamba-based unmanned aerial vehicle position and attitude regression method

Abstract

The invention discloses an unmanned aerial vehicle position and attitude regression method based on Vmamba, and the method comprises the following steps: obtaining image data, and dividing the image data into a training set, a verification set and a test set; preprocessing the image data; constructing an unmanned aerial vehicle self-positioning model; pre-processing the training set, inputting the pre-processed training set into an unmanned aerial vehicle self-positioning model for training, pre-processing the verification set, inputting the pre-processed verification set into the unmanned aerial vehicle self-positioning model, and adjusting hyper-parameters of the unmanned aerial vehicle self-positioning model; preprocessing the images of the test set, inputting the images into an unmanned aerial vehicle self-positioning model, obtaining self-positioning data of the unmanned aerial vehicle, and outputting position information and direction information; according to the Vmamba-based unmanned aerial vehicle self-positioning model, the position attitude of the unmanned aerial vehicle is directly regressed through image data, dependence on external conditions which are prone to interference such as satellite signals and geomagnetic fields is eliminated, stable positioning can still be achieved in some complex scenes such as extreme weather or environments such as urban canyons, and the environmental adaptability of the unmanned aerial vehicle is enhanced.

Description

Technical Field

[0001] This invention relates to a drone, and more particularly to a method for retrieving the position and attitude of a drone. Background Technology

[0002] With the advancement of technology, drones have been widely used in various fields such as agricultural operations, ground reconnaissance, advertising filming, and power line inspection. To ensure that drones can perform operations efficiently and stably in various tasks, accurate self-positioning has become crucial. Typically, drones rely on global satellite navigation systems (such as GPS, GLONASS, and BeiDou) for precise positioning and navigation. However, in extreme weather conditions such as strong winds and thunderstorms, electromagnetic interference can occur, affecting the drone's navigation system and impacting its normal operation. Furthermore, in scenarios such as deep mountain valleys or tall buildings in cities where signal transmission is interfered with, or when outside satellite signal coverage or experiencing signal interruptions, drones may face reduced positioning accuracy or even complete loss of positioning capability. This situation not only limits the drone's autonomous flight capability but also seriously hinders its mission execution efficiency and quality.

[0003] To address the aforementioned issues, traditional solutions primarily employ technologies such as inertial measurement units (IMUs), visual simultaneous localization and mapping (V-SLAM), and geomagnetic navigation. These methods can provide additional positional references when satellite signals are weak or unavailable, thereby enhancing the positioning accuracy and stability of the UAV. For example, the inertial measurement unit (IMU) calculates the UAV's attitude changes by measuring acceleration and angular velocity; visual simultaneous localization and mapping (V-SLAM) uses image information captured by cameras to build an environmental map in real time and estimate its own position; and geomagnetic navigation technology relies on the unique distribution of the Earth's magnetic field to determine its location.

[0004] The methods mentioned above can indeed effectively alleviate the problem of drone positioning accuracy decline or even complete loss of positioning capability, but they still have certain limitations. For example, the error of the inertial measurement unit (IMU) will accumulate over time, leading to positioning deviation after long-term use; visual simultaneous localization and mapping (V-SLAM) has high requirements for ambient light and texture, and may not work properly under certain conditions; geomagnetic navigation is easily affected by surrounding metal objects or electromagnetic interference.

[0005] Therefore, it is necessary to design a method that enables drones to directly reconstruct their position and attitude information from video images, so as to accurately return their own position and attitude information even when the drone is in a situation without satellite signal, thereby improving the drone's adaptability in complex environments. Summary of the Invention

[0006] To enable drones to accurately return their position and attitude information even when there is no satellite signal, and to improve the adaptability of drones in complex environments, this invention provides a drone position and attitude regression method based on Vmamba.

[0007] The technical solution adopted by this invention to solve its technical problem is: A method for UAV position and attitude regression based on Vmamba includes the following steps: Step 1: Acquire video data by shooting with the drone's camera, extract the video data frame by frame into image data, and divide the image data into training set, validation set and test set; Step 2: Preprocess the image data, including interest point cropping, Gaussian blurring, and normalization. Interest point cropping involves detecting and evaluating interest points using the SIFT feature extraction algorithm before cropping the image data. The SIFT feature extraction algorithm constructs a scale-space detection mechanism for interest points and removes points with low contrast or unstable points located at image edges. The specific formula for constructing the scale-space mechanism using the SIFT feature extraction algorithm is as follows: ; in It is a variable-scale Gaussian function. The input image data; Convolving the image data using a Gaussian function preserves important structural features and effectively removes high-frequency noise. The formula for the Gaussian function is as follows: ; in, and It is the coordinate offset relative to the center point of the pixel. The standard deviation is denoted as .

[0008] The image data is normalized to reduce image differences caused by factors such as lighting, contrast, or acquisition equipment, thereby improving the stability of model training. Step 3: Construct a UAV self-localization model. The UAV self-localization model is based on the Vmamba model, with the addition of a multi-head attention mechanism. A fully connected layer is added at the end of the Vmamba model to enable it to output multi-scale feature vectors, making it more suitable for UAV self-localization applications. Step 4: After the training set has undergone the above preprocessing, input it into the UAV self-localization model for training, optimize the model weight values, and after the validation set has undergone the above preprocessing, input it into the UAV self-localization model to adjust the hyperparameters of the UAV self-localization model to improve the prediction accuracy. Step 5: After performing the above preprocessing on the images of the test set, input them into the UAV self-localization model to obtain the UAV's self-localization data and output position and orientation information.

[0009] The Vmamba-based UAV self-localization model in this invention directly regresses the UAV's position and attitude through image data, eliminating dependence on external conditions that are easily interfered with, such as satellite signals and geomagnetic fields. It can still achieve stable positioning in some complex scenarios, such as extreme weather or urban canyons, thus enhancing the UAV's environmental adaptability.

[0010] The self-localization process of the UAV self-localization model is as follows: (1) After the image data is input into the UAV self-localization model, it is divided into graphic blocks by the stem module, and then the graphic blocks are input into the first VSS module composed of two VSS blocks. The first multi-head attention module receives the image data from the first VSS module. (2) The image data is input to the first downsampling module, and the output image data is then input to the second VSS module composed of two VSS blocks and the second multi-head attention module to output new image data; (3) The image data output in step (2) is received by the second downsampling module and input into the third VSS module and the third multi-head attention module, which consists of nine VSS blocks; (4) The image data output by the third multi-head attention module is received by the third downsampling module and input into the fourth VSS module and the fourth multi-head attention module composed of two VSS blocks to generate image data. The image data outputs position information and direction information through two fully connected layers respectively.

[0011] A Dropout layer is also set between the fully connected layer and the fourth multi-head attention module, which can reduce co-adaptation between neurons and improve the generalization ability of the model.

[0012] The normalization process specifically involves calculating the maximum value (max) and minimum value (min) of all pixel values ​​in the image, then scaling the pixel values ​​of the image so that the maximum value (max) and minimum value (min) of the scaled pixel values ​​fall within the range of [0,1], thereby generating a normalized image and reducing image differences.

[0013] The data processing flow of the VSS block is as follows: the input image data is divided into two branches after layer normalization. In one branch, the input image data passes through the first linear layer and the first activation function. In the other branch, the input image data passes through the second linear layer, the depthwise separable convolution, and the second activation function before entering the SS2D module for feature extraction. Finally, the image data from the two branches undergoes element-wise product operation after layer normalization, and then passes through the third linear layer and is combined with the residual connection to form the output of the VSS block.

[0014] The first, second, third, and fourth multi-head attention modules divide the input image data into multiple heads, perform independent query, key-value linear transformation on each head, calculate attention weights for each head, and finally sum the weighted outputs of all heads to obtain the final output. The specific calculation formula for the query, key, and value matrix is ​​as follows: ; in For image data, , , These are the query, key, and value parameter matrices, respectively. The formula for calculating the attention weight of each attention head is: ; in It is the dimension of each attention head.

[0015] By introducing a multi-head attention mechanism into the UAV self-localization model, multiple heads independently learn different local details of the input, improving the model's analytical depth of the data. Furthermore, different heads learn complementary features, reducing the risk of overfitting to samples and enhancing the adaptability of the UAV self-localization model in complex scenarios.

[0016] The standard deviation Set it to 1.5, and the Gaussian kernel size to 9×9.

[0017] The beneficial effects of this invention are: 1. The UAV self-localization model based on Vmamba in this invention directly regresses the position and attitude of the UAV through image data, eliminating the dependence on external conditions that are easily interfered with, such as satellite signals and geomagnetic fields. It can still achieve stable positioning in some complex scenarios, such as extreme weather or urban canyons, thus enhancing the environmental adaptability of the UAV.

[0018] 2. The UAV self-localization model of the present invention combines the linear complexity of convolutional neural networks with the advantages of the global receptive field and dynamic weights of visual transformers, which greatly improves the computational efficiency. When the input image size increases, its computational complexity increases linearly, making it very suitable for UAV self-localization application scenarios and providing accurate and real-time position estimation.

[0019] 3. This invention also introduces a multi-head attention mechanism into the UAV self-localization model. By having multiple heads independently learn different local details of the input, the model's analytical depth of the data is improved. Furthermore, different heads learn complementary features, reducing the risk of overfitting to samples and enhancing the adaptability of the UAV self-localization model in complex scenarios.

[0020] 4. The UAV self-localization model performs cross-scanning on the input image, scanning the sequence from four directions: from top left to bottom right, from bottom right to top left, from top right to bottom left, and from bottom left to top right. This preserves the original two-dimensional structure of the image, enabling the model to more effectively capture and understand more complex image data and enhancing the model's robustness in dealing with changes in direction. Attached Figure Description

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Figure 1 This is a schematic diagram illustrating the principle of the present invention.

[0023] Figure 2 This is a schematic diagram illustrating the principle of the UAV self-localization model of the present invention.

[0024] Figure 3 This is a schematic diagram of the VSS block.

[0025] Figure 4 This is a flowchart of the SIFT feature extraction algorithm. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0027] It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the invention.

[0028] Reference Figures 1 to 4 A method for UAV position and attitude regression based on Vmamba includes the following steps: Step 1: Acquire video data using the camera of a DJI drone, and extract image data frame by frame from the video data. Divide the image data into training set, validation set, and test set; divide the image data into training set, validation set, and test set according to a 7:1:2 ratio. Step 2: Preprocess the image data, including interest point cropping, Gaussian blurring, and normalization. Interest point cropping involves using the SIFT feature extraction algorithm to detect and evaluate interest points before cropping the image data, cutting the most important part for UAV attitude estimation into an image of 455×256 pixels. By analyzing the original image, identifying and cropping sub-regions containing important information can effectively reduce the impact of background noise on model training and improve the model's learning efficiency and accuracy. The SIFT feature extraction algorithm constructs scale-space detection of interest points and removes unstable points with low contrast or located on image edges. For interest point evaluation, SIFT assigns a principal direction to each interest point, making it rotation-invariant. It also generates a 128-dimensional feature descriptor for each interest point, which can be used to measure the importance of the interest point. The SIFT feature extraction algorithm's scale-space construction formula is as follows: ; in It is a variable-scale Gaussian function. The input image data; The image data is convolved using a Gaussian function, i.e., Gaussian blurring is applied to the image to preserve important structural features and effectively remove high-frequency noise from the image data; the formula for the Gaussian function is as follows: ; in, and It is the coordinate offset relative to the center point of the pixel. The standard deviation is denoted as .

[0029] The standard deviation Set it to 1.5, and the Gaussian kernel size to 9×9.

[0030] The specific steps of Gaussian blurring are based on the size of the Gaussian kernel and... A Gaussian kernel matrix is ​​created, and then this Gaussian kernel is convolved with the image. This process covers every pixel of the image with the Gaussian kernel, and a new pixel value is calculated by weighting the surrounding pixel values ​​according to the weights in the Gaussian kernel. When processing image boundaries, a zero-padding strategy is used to address the issue that boundary pixels cannot be completely covered by the Gaussian kernel, ultimately reducing image noise and sharpness.

[0031] The image data is normalized to reduce image differences caused by factors such as lighting, contrast, or acquisition equipment, thereby improving the stability of model training. Step 3: Construct a UAV self-localization model. The UAV self-localization model is based on the Vmamba model, with the addition of a multi-head attention mechanism. A fully connected layer is added at the end of the Vmamba model to enable it to output multi-scale feature vectors, making it more suitable for UAV self-localization applications. Step 4: After the training set has undergone the above preprocessing, input it into the UAV self-localization model for training, optimize the model weights, and input the validation set after the above preprocessing into the UAV self-localization model to adjust the hyperparameters of the UAV self-localization model to improve the prediction accuracy. Before model training, determine the training configuration. Set the batch size to 45, the number of iterations to 35,000, the position information loss function weight to 1, the orientation information loss function weight to 50, and the Dropout probability to 0.2. Finally, all experiments were completed in an NVIDIA GPU environment, using the AdamW optimizer with the parameters set to learning rate lr=0.0001, betas=(0.9,0.999), and eps=1.

[0032] Step 5: After performing the above preprocessing on the images of the test set, input them into the UAV self-localization model to obtain the UAV's self-localization data and output position and orientation information.

[0033] The Vmamba-based UAV self-localization model in this invention directly regresses the UAV's position and attitude through image data, eliminating dependence on external conditions that are easily interfered with, such as satellite signals and geomagnetic fields. It can still achieve stable positioning in some complex scenarios, such as extreme weather or urban canyons, thus enhancing the UAV's environmental adaptability.

[0034] The self-localization process of the UAV self-localization model is as follows: (1) After the image data is input into the UAV self-localization model, it is divided into graphic blocks by the stem module, and then the graphic blocks are input into the first VSS module composed of two VSS blocks. The first multi-head attention module receives the image data from the first VSS module. (2) The image data is input into the first downsampling module, and the output image data is then input into the second VSS module composed of two VSS blocks and the second multi-head attention module to output new image data; (3) The image data output in step (2) is received by the second downsampling module and input into the third VSS module and the third multi-head attention module, which consists of nine VSS blocks; (4) The image data output by the third multi-head attention module is received by the third downsampling module and input into the fourth VSS module and the fourth multi-head attention module, which are composed of two VSS blocks, to generate image data. The image data outputs position information and orientation information through two fully connected layers respectively; the goal of directly regressing the UAV attitude from the RGB image is realized, providing reliable technical support for the autonomous flight of the UAV in GNSS-denied environments; the position information p(x, y, z) and orientation information q(w, x, y, z) are regressed, where the orientation information is represented by the quaternion q(w, x, y, z); the above-mentioned VSS block, downsampling module, multi-head attention module, and fully connected layer correspond to Figure 2 VSS Block, DownSampling, MultiHeadAttention, and FC are mentioned.

[0035] A Dropout layer is also set between the fully connected layer and the fourth multi-head attention module, which can reduce co-adaptation between neurons and improve the generalization ability of the model. The present invention introduces a Dropout layer before the fully connected layer to reduce co-adaptation between neurons, effectively prevent overfitting, and ensure the stable performance of the model in different environments.

[0036] The normalization process specifically involves calculating the maximum value (max) and minimum value (min) of all pixel values ​​in the image, then scaling the pixel values ​​of the image so that the maximum value (max) and minimum value (min) of the scaled pixel values ​​fall within the range of [0,1], thereby generating a normalized image and reducing image differences.

[0037] like Figure 3 As shown, the data processing flow of the VSS block is as follows: the input image data is divided into two branches after being normalized by LayerNorm. In one branch, the input image data passes through a first linear layer and a first activation function. In the other branch, the input image data passes through a second linear layer, a depthwise separable convolution (DW-Conv), and a second activation function before entering the SS2D module for feature extraction. Finally, the image data from the two branches undergoes element-wise product operation after being normalized by LayerNorm, and then passes through a third linear layer and is combined with residual connections to form the output of the VSS block.

[0038] The Cross-Scan Module (CSM), the core component of the SS2D module, is a key technical component in the VMamba model, designed to address the orientation sensitivity issue in visual data processing. By employing a four-way scanning strategy, the CSM ensures that each element can integrate information from different directions, thereby achieving more comprehensive global receptive field coverage. This design not only enhances the model's robustness to orientation changes but also improves its overall performance. Specifically, the CSM operates without disrupting the two-dimensional structure of the image, enabling the model to more effectively capture and understand complex patterns in visual data.

[0039] The first, second, third, and fourth multi-head attention modules divide the input image data into multiple heads, perform independent query, key-value linear transformation on each head, calculate attention weights for each head, and finally sum the weighted outputs of all heads to obtain the final output. The specific calculation formula for the query, key, and value matrix is ​​as follows: ; in For image data, , , These are the query, key, and value parameter matrices, respectively. The formula for calculating the attention weight of each attention head is: ; in It is the dimension of each attention head.

[0040] By introducing a multi-head attention mechanism into the UAV self-localization model, multiple heads independently learn different local details of the input, improving the model's analytical depth of the data. Furthermore, different heads learn complementary features, reducing the risk of overfitting to samples and enhancing the adaptability of the UAV self-localization model in complex scenarios.

[0041] The multi-head attention module mentioned in this embodiment is an attention method used in deep learning models. It allows the model to simultaneously focus on information from different parts of the input sequence and capture complex relationships between features from multiple representation subspaces. It uses multiple attention heads in parallel, each independently calculating the correlation between its query, key, and value, focusing on different information segments. The outputs of each attention head are concatenated, and then integrated through a linear transformation, thereby achieving a deep understanding and representation of the input sequence. This method effectively enhances the model's expressive power and its ability to handle long-distance dependencies, thus improving the model's robustness and generalization ability.

[0042] In summary, this invention provides an efficient and accurate solution for attitude estimation of UAVs in complex environments by integrating advanced image preprocessing techniques, deep learning architecture, and multi-dimensional feature fusion strategies. This method not only overcomes the limitations of traditional positioning technologies in the absence of satellite signals but also demonstrates significant practical value and broad engineering application prospects. It is applicable to various fields such as agricultural operations, ground reconnaissance, advertising photography, and power line inspection, greatly improving the mission execution efficiency and autonomous flight capabilities of UAVs.

[0043] In this invention, the term "a plurality of" refers to two or more unless otherwise expressly defined. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. The terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; "linking" can be a direct connection or an indirect connection via an intermediate medium. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0044] It should be noted that when a component is referred to as being "assembled on," "mounted on," "fixed to," or "set on" another component, it can be directly on the other component or there may be an intermediate component. When a component is considered to be "connected to" another component, it can be directly connected to the other component or there may be an intermediate component present. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0045] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0046] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for UAV position and attitude regression based on Vmamba, characterized in that... Includes the following steps: Step 1: Acquire video data by shooting with the drone's camera, extract the video data frame by frame into image data, and divide the image data into training set, validation set and test set; Step 2: Preprocess the image data. The preprocessing includes interest point cropping, Gaussian blurring, and normalization. Interest point cropping involves detecting and evaluating interest points using the SIFT feature extraction algorithm before cropping the image data. The SIFT feature extraction algorithm constructs a scale space to detect interest points and removes unstable points with low contrast or located at image edges. The specific formula for constructing the scale space using the SIFT feature extraction algorithm is as follows: ； in It is a variable-scale Gaussian function. The input image data; Convolving the image data using a Gaussian function preserves important structural features and effectively removes high-frequency noise. The formula for the Gaussian function is as follows: ； in, and It is the coordinate offset relative to the center point of the pixel. Standard deviation; The image data is normalized to reduce image differences caused by factors such as lighting, contrast, or acquisition equipment, thereby improving the stability of model training. Step 3: Construct a UAV self-localization model. The UAV self-localization model is based on the Vmamba model, with the addition of a multi-head attention mechanism. A fully connected layer is added at the end of the Vmamba model to enable it to output multi-scale feature vectors, making it more suitable for UAV self-localization applications. Step 4: After the training set has undergone the above preprocessing, input it into the UAV self-localization model for training, optimize the model weight values, and after the validation set has undergone the above preprocessing, input it into the UAV self-localization model to adjust the hyperparameters of the UAV self-localization model to improve the prediction accuracy. Step 5: After performing the above preprocessing on the images of the test set, input them into the UAV self-localization model to obtain the UAV's self-localization data and output position and orientation information.

2. The method for UAV position and attitude regression based on VMamba according to claim 1, characterized in that... The self-localization process of the UAV self-localization model is as follows: (1) After the image data is input into the UAV self-localization model, it is divided into graphic blocks by the stem module, and then the graphic blocks are input into the first VSS module composed of two VSS blocks. The first multi-head attention module receives the image data from the first VSS module. (2) The image data is input to the first downsampling module, and the output image data is then input to the second VSS module composed of two VSS blocks and the second multi-head attention module to output new image data; (3) The image data output in step (2) is received by the second downsampling module and input into the third VSS module and the third multi-head attention module, which consists of nine VSS blocks; (4) The image data output by the third multi-head attention module is received by the third downsampling module and input into the fourth VSS module and the fourth multi-head attention module composed of two VSS blocks to generate image data. The image data outputs position information and direction information through two fully connected layers respectively.

3. The method for UAV position and attitude regression based on VMamba according to claim 2, characterized in that... A Dropout layer is also set between the fully connected layer and the fourth multi-head attention module, which can reduce co-adaptation between neurons and improve the generalization ability of the model.

4. The method for UAV position and attitude regression based on VMamba according to claim 1, characterized in that... The normalization process specifically involves calculating the maximum value (max) and minimum value (min) of all pixel values ​​in the image, then scaling the pixel values ​​of the image so that the maximum value (max) and minimum value (min) of the scaled pixel values ​​fall within the range of [0,1], thereby generating a normalized image and reducing image differences.

5. The method for UAV position and attitude regression based on Vmamba according to claim 2, characterized in that... The data processing flow of the VSS block is as follows: the input image data is divided into two branches after layer normalization. In one branch, the input image data passes through the first linear layer and the first activation function. In the other branch, the input image data passes through the second linear layer, the depthwise separable convolution, and the second activation function before entering the SS2D module for feature extraction. Finally, the image data from the two branches undergoes element-wise product operation after layer normalization, and then passes through the third linear layer and is combined with the residual connection to form the output of the VSS block.

6. The method for UAV position and attitude regression based on Vmamba according to claim 2, characterized in that... The first, second, third, and fourth multi-head attention modules divide the input image data into multiple heads, perform independent query, key-value linear transformation on each head, calculate attention weights for each head, and finally sum the weighted outputs of all heads to obtain the final output. The specific calculation formula for the query, key, and value matrix is ​​as follows: ； in For image data, These are the query, key, and value parameter matrices, respectively. The formula for calculating the attention weight of each attention head is: ； in It is the dimension of each attention head.

7. The method for UAV position and attitude regression based on Vmamba according to claim 1, characterized in that... The standard deviation Set it to 1.5, and the Gaussian kernel size to 9×9.

Images