Visual inertia autonomous positioning method

Abstract

The invention provides a visual inertia autonomous positioning method. Performing feature extraction and tracking on the environment image acquired from the camera to identify feature points; carrying out offset correction on inertial data acquired by an inertial sensor by adopting a neural network, and then carrying out integral processing so as to accurately estimate the state change of the current frame; and then visual data and inertial data are fused and utilized, and pose optimization is carried out by adjusting a visual weight, so that accurate pose information is obtained. According to the method, the precision and robustness of visual inertia autonomous positioning can be improved, and the autonomous positioning requirements of high precision and high reliability are met.

Description

Technical Field

[0001] The invention belongs to the technical field of autonomous positioning of unmanned aerial vehicles, and in particular relates to a visual inertial autonomous positioning method. Background Art

[0002] Micro UAVs are widely used in disaster relief, industrial inspection, environmental exploration, and counter-terrorism reconnaissance. Therefore, micro UAVs need to accurately estimate their position and state information to ensure that they can complete these tasks effectively. However, many mission scenarios may be located in Global Navigation Satellite System (GNSS) denied environments, such as indoors, tunnels, or urban canyons, which makes it far from enough to rely solely on GNSS for positioning and navigation.

[0003] With the development of technology, the reduction of sensor size and the decrease of cost, Visual Inertial Odometry (VIO) technology has gradually been widely used in the field of drones. VIO technology has unique advantages. It can autonomously estimate its own posture information without relying on base stations and can operate effectively in complex environments. In addition, VIO can also be combined with mapping methods to build a 3D map of the environment, thereby achieving autonomous obstacle avoidance and path planning. This has made VIO widely used in many fields such as drones, robots, smart homes, etc., improving the intelligence and autonomy of equipment.

[0004] VIO technology achieves positioning by fusing camera and inertial measurement unit (IMU) data. First, the camera generates an image frame and obtains image features through feature extraction and matching to help identify feature points in the environment. Secondly, the IMU measures the acceleration and angular velocity information of the device and calculates the position, direction and speed information of the device in a short period of time through data integration. Finally, the camera and IMU data are fused and optimized to obtain the exact posture information of the device. This fusion method improves the accuracy of positioning and enhances the system's adaptability in dynamic and complex environments.

[0005] The development of VIO technology has provided more possibilities for the field of drones. However, VIO technology still faces many challenges in micro drones. First, although the IMU is not affected by external visual information and can provide information about its own motion at a high frequency, it is more robust to fast motion, but the IMU's measurement value itself has random errors. Long-term integration will cause the system to accumulate errors quickly. When the visual information is unreliable, VIO, which only relies on the IMU's raw measurement value for integration, will completely collapse within a few seconds due to the violent accumulation of noise. Secondly, current visual inertial positioning methods always rely on continuous and reliable visual tracking, but in some cases, such as when there are weak or repetitive textures, flickering light, dynamically changing scenes in the environment, or when the camera moves quickly and causes severe motion blur, visual tracking often fails.

[0006] VIO technology integrates visual and inertial data, but the quality of inertial and visual data directly affects the positioning accuracy and robustness of the system. For example, the bias in IMU data and low-quality visual information will weaken the positioning accuracy and robustness of the system. That is, there are two main problems with current VIO technology: one is the bias of IMU measurements, and the other is the fixedness of visual weights in back-end optimization. These problems lead to insufficient robustness of the positioning system in complex visual environments, affecting its accuracy and reliability. Summary of the invention

[0007] In order to overcome the shortcomings of the prior art, the present invention provides a visual inertial autonomous positioning method. Feature extraction and tracking are performed on the environmental image obtained from the camera to identify feature points; a neural network is used to perform bias correction on the inertial data obtained by the inertial sensor, and then an integral process is performed to accurately estimate the state change of the current frame; then the visual data and the inertial data are fused, and the posture is optimized by adjusting the visual weight to obtain accurate posture information. The present invention can improve the accuracy and robustness of visual inertial autonomous positioning and meet the needs of high-precision and high-reliability autonomous positioning.

[0008] A visual inertial autonomous positioning method, characterized by the following steps:

[0009] Step 1: Get the environment image from the camera and the inertial data from the inertial sensor;

[0010] Step 2: Perform Shi-Tomasi feature extraction and Lucas-Kanade optical flow tracking on the acquired environment image to obtain the pixel coordinates of the feature points in the image and the corresponding relationship between the feature points between images;

[0011] Step 3: Triangulate the feature point information processed in step 2 to obtain the three-dimensional spatial coordinates of the feature points, and use the PnP algorithm to solve and calculate the feature point data of two adjacent frames of images to obtain the camera pose of the adjacent frames of images;

[0012] Step 4: Use the neural network model to perform bias correction processing on the acquired inertial data;

[0013] Step 5: Integrate the inertial data processed in step 4 to obtain the IMU measurement state quantity between adjacent image frames;

[0014] Step 6: Subtract the IMU measured state from the estimated state solved by PnP to obtain the IMU residual; calculate the feature point reprojection error and use it as the visual residual; use the IMU residual and visual residual to calculate the Jacobian matrix of the camera pose data, where a fixed value is used As the visual information weight, f represents the focal length of the camera; the camera pose data is optimized by least squares;

[0015] Step 7: Calculate the feature point reprojection error based on the optimized camera pose, use the obtained reprojection error to update the visual information weight, repeat step 6, where the fixed value is replaced by the updated visual information weight, and the pose data after least squares optimization is output as the final camera pose data.

[0016] Specifically, the neural network model described in step 4 is composed of a masked IMU module, an IMU bias correction module and an IMU prediction module. The IMU data is input, the masked IMU module randomly masks part of its data and uses context information to predict the masked part, and outputs the prediction result of the masked position; the IMU bias correction module corrects its bias by analyzing the time series characteristics of the input IMU data, and outputs the corrected data; the IMU prediction module predicts the IMU data of the next frame based on the current input IMU data and the data output by the IMU bias correction module, and outputs the predicted data;

[0017] The neural network model completes model learning by minimizing the following total input-output loss:

[0018] L=L y +λ1L p +λ2L yc (1)

[0019] Among them, L represents the total input and output loss, L y represents the input and output root mean square error loss of the mask IMU module, L p represents the input and output root mean square error loss of the IMU bias correction module, L ycrepresents the input and output root mean square error loss of the IMU prediction module, λ1 is the weight coefficient of the IMU bias correction module, which is set to 10; λ2 is the weight coefficient of the IMU prediction module, which is set to 1;

[0020] The IMU correction data output by the IMU bias correction module of the learned neural network model is the final corrected IMU data.

[0021] Specifically, the specific process of calculating the reprojection error of the feature point is as follows: for any feature point in the image, a projection calculation is performed on it using the camera imaging principle to obtain the projection pixel coordinates of the feature point in other image frames, and the projection pixel coordinates are subtracted from the pixel coordinates of the point in the image frame that matches the feature point to obtain the reprojection error [δu, δv] of the feature point, where δu represents the reprojection error in the x direction and δv represents the reprojection error in the y direction; assuming that there are a total of n frames of images with points matching the feature point, the reprojection error vector of the feature point is err p =[δu1,δv1,δu2,δv2,…,δu n ,δv n ], where err p represents the reprojection error vector, δu i Represents the reprojection error of the feature point in the x direction in the i-th frame image, δv i Represents the reprojection error of the feature point in the y direction in the i-th frame image, i = 1, 2, …, n.

[0022] Specifically, the specific formula for updating the visual information weight using the obtained reprojection error described in step 7 is as follows:

[0023]

[0024] Among them, Σ p represents the updated visual information weight, w is calculated as follows:

[0025]

[0026] Among them, e thre The threshold of reprojection error is set to 2.5×10 -3 , w thre represents the maximum visual information weight, which is set to 1000, and k is the balance coefficient, which is set to 2. Represents the root mean square error of the feature point reprojection error vector.

[0027] Furthermore, step 4 is not adopted, and step 5 is changed to: integrating the inertial data obtained in step 1 to obtain the IMU estimated state quantity between adjacent image frames.

[0028] Furthermore, step 7 is not adopted, and the posture data obtained in step 6 is used as the final posture data.

[0029] The beneficial effects of the present invention are: by using a self-supervised learning neural network to correct the bias of the inertial data and then performing integration processing, the reliability and accuracy of the IMU data can be improved, thereby providing more accurate data for subsequent optimization; by integrating the visual data and the inertial data and dynamically adjusting the visual weight, more accurate posture information can be obtained. The present invention can improve the accuracy and robustness of visual inertial autonomous positioning and meet the needs of high-precision and high-reliability autonomous positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a visual inertial autonomous positioning method of the present invention;

[0031] Figure 2 It is a structural diagram of the IMU bias correction neural network of the self-supervised learning of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0033] The present invention provides a visual inertial autonomous positioning method, such as Figure 1 As shown, the specific implementation process is as follows:

[0034] Step 1: Get the environment image from the camera and the inertial data from the inertial sensor.

[0035] Step 2: Perform Shi-Tomasi feature extraction and Lucas-Kanade optical flow tracking on the acquired environment image to obtain the pixel coordinates of the feature points in the image and the corresponding relationship between the feature points between images.

[0036] Step 3: Triangulate the feature point information processed in step 2 to obtain the three-dimensional spatial coordinates of the feature points, and use the PnP algorithm to solve and calculate the feature point data of two adjacent frames of images to obtain the camera pose of the adjacent frames of images.

[0037] Step 4: Use the neural network model to perform bias correction on the acquired inertial data.

[0038] The present invention constructs a self-supervised learning IMU bias correction neural network model, which is composed of a mask IMU module, an IMU bias correction module and an IMU prediction module. The specific processing process is as follows: Figure 2 As shown:

[0039] (1) Input IMU data, the mask IMU module randomly masks part of its data and uses context information to predict the data of the masked part, and outputs the prediction result of the masked position. Figure 2 In the equation, u represents the raw IMU data input. represents the output result of the mask IMU module, d(,) represents the root mean square error loss, and f(,) represents the bias correction function, which is a learnable prediction function.

[0040] This process uses the contextual information of the mask data to help the correction method better understand the structure and characteristics of the IMU data. This method not only enhances the modeling ability of IMU data, but also improves the accuracy of the model's inference of unobserved data.

[0041] (2) The IMU bias correction module corrects the bias of the input IMU data by analyzing the time series characteristics of the data and outputs the corrected data. Figure 2 middle, represents the output result of the bias correction module, u represents the input raw IMU data, d(,) represents the root mean square error loss, and f(,) represents the bias correction function, which is a learnable prediction function.

[0042] (3) The IMU prediction module predicts the IMU data of the next frame based on the current input IMU data and the data output by the IMU bias correction module, and outputs the predicted data. Figure 2 In the above equation, h(,) is a function used to predict the next frame of IMU data. Indicates the next frame of IMU data obtained by predicting the output result of the bias correction module. Indicates the next frame of IMU data predicted from the original data, u F represents the next frame data of the original IMU, and d(,) represents the root mean square error loss.

[0043] The entire neural network model completes model learning by minimizing the following total input-output loss:

[0044] L=L y +λ1L p +λ2L yc (4)

[0045] Among them, L represents the total input and output loss, L y represents the input and output root mean square error loss of the mask IMU module, L p represents the input and output root mean square error loss of the IMU bias correction module, L yc It represents the input-output root mean square error loss of the IMU prediction module, λ1 is the weighting coefficient of the IMU bias correction module, which is set to 10; λ2 is the weighting coefficient of the IMU prediction module, which is set to 1.

[0046] The IMU correction data output by the IMU bias correction module of the learned neural network model is the final corrected IMU data.

[0047] Step 5: Integrate the inertial data processed in step 4 to obtain the IMU estimated state quantity between adjacent image frames. Alternatively, the inertial data obtained in step 1 may be directly integrated to obtain the IMU estimated state quantity between adjacent image frames.

[0048] Step 6: Subtract the IMU measured state from the IMU estimated state to obtain the IMU residual; calculate the feature point reprojection error and use it as the visual residual; use the IMU residual and visual residual to calculate the Jacobian matrix of the camera pose data, where a fixed value is used As the visual information weight, f represents the focal length of the camera; the camera pose data is optimized by least squares.

[0049] The specific process of calculating the reprojection error of the feature point is as follows: for any feature point in the image, use the camera imaging principle to perform projection calculation on it, obtain the projection pixel coordinates of the feature point in other image frames, subtract the projection pixel coordinates from the pixel coordinates of the point in the image frame that matches the feature point, and obtain the reprojection error [δu,δv] of the feature point, where δu represents the reprojection error in the x direction and δv represents the reprojection error in the y direction; assuming that there are a total of n frames of images with points matching the feature point, the reprojection error vector of the feature point is err p =[δu1,δv1,δu2,δv2,…,δu n ,δv n ], where err p represents the reprojection error vector, δu i Represents the reprojection error of the feature point in the x direction in the i-th frame image, δv i Represents the reprojection error of the feature point in the y direction in the i-th frame image, i = 1, 2, …, n.

[0050] Step 7: Calculate the feature point reprojection error based on the optimized camera pose, use the obtained reprojection error to update the visual information weight, repeat step 6, where the fixed value is replaced by the updated visual information weight, and the pose data after least squares optimization is output as the final camera pose data.

[0051] Among them, the specific formula for updating the visual information weight using the obtained reprojection error is as follows:

[0052]

[0053] Among them, Σ prepresents the updated visual information weight, w is calculated as follows:

[0054]

[0055] Among them, e thre The threshold of reprojection error is set to 2.5×10 -3 , w thre represents the maximum visual information weight, which is set to 1000, and k is the balance coefficient, which is set to 2. Represents the root mean square error of the feature point reprojection error vector.

[0056] It is also possible to not adopt step 7 and use the posture data obtained in step 6 as the final posture data.

Claims

1. A visual inertial autonomous positioning method, characterized in that Here are the steps: Step 1: Get the environment image from the camera and the inertial data from the inertial sensor; Step 2: Perform Shi-Tomasi feature extraction and Lucas-Kanade optical flow tracking on the acquired environment image to obtain the pixel coordinates of the feature points in the image and the corresponding relationship between the feature points between images; Step 3: Triangulate the feature point information processed in step 2 to obtain the three-dimensional spatial coordinates of the feature points, and use the PnP algorithm to solve and calculate the feature point data of two adjacent frames of images to obtain the camera pose of the adjacent frames of images; Step 4: Use the neural network model to perform bias correction processing on the acquired inertial data; Step 5: Integrate the inertial data processed in step 4 to obtain the IMU measurement state quantity between adjacent image frames; Step 6: Subtract the IMU measured state from the estimated state solved by PnP to obtain the IMU residual; Calculate the feature point reprojection error and use it as the visual residual; use the IMU residual and visual residual to find the Jacobian matrix of the camera pose data, where a fixed value is used As the visual information weight, f represents the focal length of the camera; the camera pose data is optimized by least squares; Step 7: Calculate the feature point reprojection error based on the optimized camera pose, use the obtained reprojection error to update the visual information weight, repeat step 6, where the fixed value is replaced by the updated visual information weight, and the pose data after least squares optimization is output as the final camera pose data.

2. A visual inertial autonomous positioning method as claimed in claim 1, characterized in that: The neural network model described in step 4 is composed of a masked IMU module, an IMU bias correction module and an IMU prediction module. The IMU data is input, and the masked IMU module randomly masks part of its data and uses context information to predict the masked part, and outputs the prediction result of the masked position; The IMU bias correction module corrects the bias of the input IMU data by analyzing the time series characteristics and outputs the corrected data; The IMU prediction module predicts the IMU data of the next frame based on the currently input IMU data and the data output by the IMU bias correction module, and outputs the predicted data; The neural network model completes model learning by minimizing the following total input-output loss: L=L y +λ1L p +λ2L yc (1) Among them, L represents the total input and output loss, L y represents the input and output root mean square error loss of the mask IMU module, L p represents the input and output root mean square error loss of the IMU bias correction module, L yc represents the input and output root mean square error loss of the IMU prediction module, λ1 is the weight coefficient of the IMU bias correction module, which is set to 10, and λ2 is the weight coefficient of the IMU prediction module, which is set to 1; The IMU correction data output by the IMU bias correction module of the learned neural network model is the final corrected IMU data.

3. A visual inertial autonomous positioning method as claimed in claim 1, characterized in that: The specific process of calculating the reprojection error of the feature point is as follows: for any feature point in the image, a projection calculation is performed on it using the camera imaging principle to obtain the projection pixel coordinates of the feature point in other image frames, and the projection pixel coordinates are subtracted from the pixel coordinates of the point in the image frame that matches the feature point to obtain the reprojection error [δu, δv] of the feature point, where δu represents the reprojection error in the x direction and δv represents the reprojection error in the y direction; assuming that there are a total of n frames of images with points matching the feature point, the reprojection error vector of the feature point is err p =[δu1, δv1, δu2, δv2,..., δu n ,δv n ], where err p represents the reprojection error vector, δu i Represents the reprojection error of the feature point in the x direction in the i-th frame image, δv i Represents the reprojection error of the feature point in the y direction in the i-th frame image, i = 1, 2, …, n.

4. A visual inertial autonomous positioning method as claimed in claim 1, characterized in that: The specific formula for updating the visual information weight using the obtained reprojection error described in step 7 is as follows: Among them, ∑ p represents the updated visual information weight, w is calculated as follows: Among them, e thre The threshold of reprojection error is set to 2.5×10 -3 , w thre represents the maximum visual information weight, which is set to 1000, and k is the balance coefficient, which is set to 2. Represents the root mean square error of the feature point reprojection error vector.

5. The visual inertial autonomous positioning method according to claim 1, characterized in that: Step 4 is not adopted, and step 5 is changed to: integrating the inertial data obtained in step 1 to obtain the IMU measurement state quantity between adjacent image frames.

6. A visual inertial autonomous positioning method as claimed in claim 1, characterized in that: Step 7 is not adopted, and the posture data obtained in step 6 is used as the final posture data.

Images