Vision-inertial- radar fusion self-localization method based on self-supervised neural network

Abstract

This invention relates to a vision-inertial navigation-radar fusion self-localization method based on a self-supervised neural network, comprising: acquiring a video frame sequence, inertial navigation data, and a lidar point cloud map; inputting the video frame sequence, inertial navigation data, and lidar point cloud map into a pose estimation network model to obtain a relative pose estimate of the input video frame; wherein, the pose estimation network model includes: a depth prediction network for obtaining a depth map based on the input video frame sequence; a feature extraction network for extracting feature information from the input video frame sequence, inertial navigation data, and lidar point cloud map respectively to obtain visual features, momentum features, and radar features; a feature fusion network for fusing visual features and radar features to obtain pre-fused features, and then fusing the pre-fused features with momentum features to obtain fused features; and a pose estimation network for predicting a pose transformation matrix based on the fused features. This invention improves the localization accuracy of the self-supervised depth localization algorithm.

Description

Technical Field

[0001] This invention relates to the field of self-localization technology, and in particular to a vision-inertial navigation-radar fusion self-localization method based on a self-supervised neural network. Background Technology

[0002] Self-localization technology is widely used in autonomous driving, SLAM, and other fields. In certain scenarios (buildings in cities, tunnels, dense forests in mountains, caves, or areas where satellite positioning is difficult), it's challenging to use radio wave technologies like satellite, BeiDou, and GPS for localization. In these situations, autonomous localization relies on cameras mounted on vehicles, drones, or robots. The basic process of traditional vision-based self-localization algorithms is: feature extraction - feature matching - pose calculation - backend optimization. While traditional self-localization algorithms achieve high positioning accuracy through complete global mapping and optimization, their computational process is complex, slow, and difficult to implement inference in real time. With the popularity and continuous development of deep learning, neural network-based visual self-localization algorithms—Deep Visual Odometry (VO)—can pre-train models offline using datasets and then directly use the trained model for inference. This eliminates the need for feature matching and backend optimization steps, making real-time inference easier. Deep VO improves localization performance by training and optimizing the target loss function through supervised or unsupervised learning methods. Unsupervised (or self-supervised) learning can train models without obtaining real pose labels, thus making use of a large amount of unlabeled data, and its training cost is lower than that of supervised methods.

[0003] Self-supervised VoIP was first proposed by Tinghui Zhou in 2017. Self-supervised VoIP first feeds consecutive input frames into a neural network for pose estimation and a neural network for depth estimation to calculate the depth map and pose transformation between the input images. Then, using the depth map and pose, it calculates the reprojection error between consecutive frames through a projection relationship established by epipolar geometry. Finally, it calculates the gradient of this error relative to the parameters of the pose and depth estimation neural networks and backpropagates to update the parameters, thereby achieving optimized estimation.

[0004] However, relying solely on single-modal data for positioning may face problems such as data gaps and insufficient information. In today's big data era, more and more sensors are being invented, and the types of data that can be acquired are increasing. Multimodal sensor fusion is the main trend in the future development of positioning technology. Compared with single-modal sensor positioning technology, multimodal sensor fusion positioning technology can combine the advantages of various modal data for information fusion and information compensation, thereby achieving higher accuracy positioning. Video frame signals can provide the most intuitive RGB time-series information, which is crucial for neural network learning feature detection. However, video frames may contain dynamic objects, leading to incorrect judgment of motion speed, and monocular video frame signals lack depth information. Inertial navigation signals can provide direct information on individual acceleration and velocity, but inertial navigation itself has various parameter biases and will generate error accumulation, which is not conducive to long-term operation. LiDAR signals can provide depth map information, but lack color visual features. Therefore, LiDAR signals can serve as a good supplement to monocular video frame signals that lack depth information. While binocular video signals can provide depth information, they require left-right eye matching algorithms, resulting in high computational complexity. Furthermore, binocular cameras are highly sensitive to changes in lighting and texture details; inconsistent lighting between the left and right eyes, as well as overly monotonous texture scenes, can lead to matching failures. Currently, various neural network-based dual-modal sensor fusion localization algorithms exist, such as depth vision-momentum odometry (VIO), depth radar vision odometry (VLO), and depth momentum radar odometry (LIO). However, these dual-modal sensor fusion-based depth odometry systems all lack certain important input information to varying degrees, and research on localization technologies based on three or more modal sensors remains at the traditional algorithm stage. Summary of the Invention

[0005] This invention provides a vision-inertial navigation-radar fusion self-localization method based on a self-supervised neural network, which solves the problem of insufficient self-localization information relying on a single mode or dual mode.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide a vision-inertial navigation-radar fusion self-localization method based on a self-supervised neural network, comprising the following steps:

[0007] Acquire video frame sequences, inertial navigation data, and lidar point cloud maps;

[0008] The video frame sequence, inertial navigation data, and lidar point cloud map are input into the pose estimation network model to obtain the relative pose estimate of the input video frame;

[0009] The pose estimation network model includes:

[0010] A depth prediction network is used to obtain a depth map based on the input video frame sequence;

[0011] The feature extraction network is used to extract feature information from the input video frame sequence, inertial navigation data and lidar point cloud map respectively, to obtain visual features, momentum features and radar features;

[0012] A feature fusion network is used to fuse the visual features and radar features to obtain pre-fused corrected visual features and corrected radar features, and then fuse the pre-fused corrected visual features and corrected radar features with the momentum features to obtain fused features.

[0013] A pose estimation network is used to predict the pose transformation matrix based on the fused features;

[0014] The parameter optimization module is used to calculate the loss function based on the depth map, pose transformation matrix, and video frames, and adjust the parameters of the pose estimation network model according to the loss function.

[0015] The feature extraction network includes:

[0016] In the first feature extraction part, a first convolutional network is used to extract visual features from the video frame sequence;

[0017] In the second feature extraction part, an LSTM network is used to extract momentum features from the inertial navigation data;

[0018] In the third feature extraction part, the lidar point cloud map is projected onto a 2D plane, and then the lidar point cloud map projected onto the 2D plane is encoded using a three-channel encoding method. Finally, the second convolutional network is used to extract the radar features.

[0019] The first and second convolutional networks have the same structure and share the weights of all network layers except the BN layer.

[0020] The first and second convolutional networks have the same structure and share the weights of all network layers except the BN layer.

[0021] The feature fusion network includes:

[0022] The first fusion part is used to fuse the visual features and radar features using a channel switching strategy to obtain pre-fused corrected visual features and corrected radar features.

[0023] In the second fusion section, the modified visual features and modified radar features used for pre-fusion are stitched together with the momentum features on a channel basis to obtain fused features.

[0024] The channel switching strategy is as follows: Among them, V′ k,c V represents the visual feature of the c-th channel output by the k-th convolutional layer after the swapping strategy.k,c L represents the visual feature of the c-th channel output by the k-th convolutional layer. k,c Let a represent the radar feature of the c-th channel output by the k-th convolutional layer. v,k,c b v,k,c σ v,k,c and μ v,k,c These represent the slope, bias, mean, and variance of the BN layer in the first convolutional network, respectively; a l,k,c b l,k,c σ l,k,c and μ l,k,c δ represents the slope, bias, mean, and variance of the BN layer in the second convolutional network, respectively; δ is the threshold.

[0025] The loss function includes reconstruction error, depth smoothing loss, and geometric consistency loss, and its expression is: Among them, L all Let L be the loss function. pe L represents the reconstruction error. smooth L represents the deep smoothing loss. geo Let l represent the geometric consistency loss, l represent the scale number, and ω1, ω2, and ω3 represent the weights of the reconstruction error, depth smoothing loss, and geometric consistency loss, respectively.

[0026] The expression for the reconstruction error is: Among them, I s For the source image, p s For source image I s The point on, I t For the target image, p′ s For source image I s Point p on s Corresponding to target image I t The points are defined by SSIM(), which is the structural similarity function, and λ1 and λ2 are the weight coefficients.

[0027] The expression for the reconstruction error is: L pe =λ1|I′ s -I s |+λ2SSIM(I′ s ,I s ), where I s For the source image, I′ s For target image I t The reconstructed source image, SSIM() is the structural similarity function, and λ1 and λ2 are weight coefficients.

[0028] The expression for the depth smoothing loss is: Among them, D t I is the depth map corresponding to the image at time t in the video frame sequence S.t For the target image, and These represent the partial derivatives of the coordinates of the two-dimensional image with respect to the x and y directions, respectively.

[0029] The expression for the geometric consistency loss is: Among them, D t Let D′ be the depth map corresponding to the image at time t in the video frame sequence S. t This represents the depth map at time t, reconstructed using the depth maps from subsequent time steps through pose transformation.

[0030] Beneficial effects

[0031] By adopting the above-mentioned technical solutions, this invention has the following advantages and positive effects compared with the prior art: This invention designs feature extraction networks for each modality and multimodal data fusion networks, and utilizes similar modality priority fusion strategy and channel exchange strategy to achieve full interaction and improvement between multimodal information, thereby improving the positioning accuracy of the self-supervised deep localization algorithm. Attached Figure Description

[0032] Figure 1 This is a flowchart of the vision-inertial navigation-radar fusion self-localization method based on a self-supervised neural network according to an embodiment of the present invention;

[0033] Figure 2 This is a framework diagram of the pose estimation network model in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating the principle of feature fusion in an embodiment of the present invention;

[0035] Figure 4 This is a comparison chart of the path trajectory output and the actual path in the KITTI datasets seq09 and seq10, based on an embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0037] The embodiments of the present invention relate to a vision-inertial navigation-radar fusion self-localization method based on a self-supervised neural network, such as... Figure 1 As shown, it includes the following steps:

[0038] Step 1: Acquire video frame sequences, inertial navigation data, and lidar point cloud maps;

[0039] Step 2: Input the video frame sequence, inertial navigation data and lidar point cloud map into the pose estimation network model to obtain the relative pose estimate of the input video frame.

[0040] Among them, the pose estimation network model is as follows Figure 2 As shown, this pose estimation network model takes a video frame sequence, inertial navigation data, and LiDAR point cloud map as input, and outputs the relative pose estimate for each moment of the input sequence. For the video frame sequence in the input, there are three parallel processing branches: one is the depth map D obtained through the DepthNet depth prediction network; the other is the output of the video frame sequence obtained through the inertial navigation data and LiDAR point cloud map. Figure 1 The first step involves inputting the data into a multimodal feature extraction network, followed by a pose estimation network (PoseNet) with feature fusion capabilities to obtain the relative pose estimate (pose transformation matrix T) at the corresponding time step. The second step combines the depth map (D) and pose transformation (T) to calculate the loss function. This pose estimation network model utilizes video frames, depth maps, and poses to calculate a self-supervised loss function and optimizes the network parameters through gradient descent backpropagation. Multiple training iterations are performed until the model converges.

[0041] The pose estimation network model in this embodiment includes: a depth prediction network, a feature extraction network, a feature fusion network, a pose estimation network, and a parameter optimization module.

[0042] The feature extraction network extracts features from each modality.

[0043] Visual features are extracted from video frame sequences from a monocular camera using a seven-layer convolutional network (ConvNet). Each convolutional layer is followed by a batch normalization (BN) layer and a non-linear activation function (ReLU) layer. The F-values ​​of each video frame in the input video frame sequence are then processed. t And its next frame F t+1 After being concatenated by channel, the data is input into the network to extract spatial and temporal information, resulting in visual features V. t :

[0044] V t =ConvNet(F t ,F t+1 )

[0045] For the IMU signal X from the inertial navigation system t An LSTM network is used for temporal information extraction, and the value of the last hidden layer of the LSTM network is taken as the output momentum feature I. t :

[0046] I t =LSTM(X) t )

[0047] For point cloud signals from lidar data in the format (x,y,z,i), where x,y,z represent three-dimensional spatial coordinates and i represents intensity (related to the object's surface illumination and reflectivity), cylindrical coordinate projection is used to first project the sparse and disordered point cloud onto a 2D plane:

[0048]

[0049]

[0050] α and β represent the pitch and yaw angles in cylindrical coordinates, respectively, and are also the horizontal and vertical coordinates of the planar projection map. When multiple 3D points are mapped to the same coordinate system, only the 3D point closest to the origin (the center of the lidar sensor) is taken as the corresponding projection point. A three-channel encoding method is used to encode the projected 2D projection map into... (The three-channel pixel values ​​at each coordinate in the projection image). For coordinates without corresponding 3D points, the channel values ​​at these coordinates are set to the default value of 0. The same ConvNet network used for processing video signals is directly applied to the projected radar projection image M. t Extracting features L t In this layer, all network layers except the Batch Normalization (BN) layer share weights:

[0051] L t =ConvNet(M t M t+1 )

[0052] A common approach to multimodal feature fusion based on neural networks involves first extracting features from each modality using different feature encoders and aligning them along a certain dimension. Then, the features from each modality are concatenated along that dimension before being fed into a fusion network for processing. This approach is most common in deep odometry based on bimodal fusion. However, in trimodal fusion, directly concatenating the three modal data channels requires the model to learn not only how a particular modality should be fused with the other two modalities (i.e., the fusion method), but also when and which modality it should be fused with (or simultaneously with the other two modalities) (i.e., the fusion order). This makes model training more difficult and slower.

[0053] The feature fusion network in this embodiment follows a fixed fusion order: first, it performs fusion between relatively simple and intuitive similar modalities, and then it performs fusion between relatively complex and abstract dissimilar modalities. Since radar point cloud data can be transformed into 2D image representations through projection transformation, it can first be fused with the features of video frame sequence signals of similar modalities, and then the fusion result is further fused with the features of IMU signals with large data format differences.

[0054] When fusing video and radar features, the data from the two similar modalities are first input into a feature extraction network (ConvNet) that shares all weights except for the Batch Normalization (BN) layer, and a channel-swapping strategy is employed.

[0055]

[0056] The two rows in the above equation represent the visual features V of the c-th channel output by the k-th convolutional layer. k,c and radar signature L k,c The expression passed through the BN layer (a v,k,c b v,k,c σ v,k,c and μ v,k,c These represent the slope, bias, mean, and variance of the BN layer in the first convolutional network, respectively; a l,k,c b l,k,c σ l,k,c and μ l,k,c , representing the slope, bias, mean, and variance of the BN layer in the second convolutional network, respectively. The channel swapping strategy determines the importance and completeness of each channel information in the output feature map of the previous convolutional layer based on the magnitude of the slope 'a' in the weight values ​​of the BN layer. If the slope value is too small (the threshold is set to δ here), it means that the parameter update weight of the model at that channel is small when updating parameters during gradient backpropagation, that is, the information of that channel is incomplete or unimportant. Therefore, in the next iteration, the component of the feature map output by the convolutional layer at that channel is replaced with the channel component at the corresponding position of the feature map of another similar modality (the case corresponding to the second row in the above formula is that the visual feature channel is replaced with the radar feature channel) to achieve the effect of supplementing missing information. This is because the feature components at the corresponding channel of the feature map output by similar modalities after going through some of the same network layers (weight sharing) should contain similar semantic information (illumination, depth, etc.). Therefore, this information between modalities can complement and improve each other.

[0057] The pre-fused corrected visual features V′ were obtained through similar modality fusion. t and radar signature L′ t Then, the pre-fused visual and radar features are combined with the momentum features output by the IMU feature extraction network. t By splicing the channels together, the fusion features are obtained.

[0058] In this embodiment, the pose estimation network is PoseNet, composed of an SE module and an LSTM. It takes the aforementioned fused features as input to obtain the predicted pose transformation matrix T. t :

[0059] T t =PoseNet(V′) t ,L′t ,I t )

[0060] In this embodiment, the depth prediction network performs prediction based on a given video frame image I. t , and its next frame image I s The concatenated images are input into a depth prediction network (DepthNet), which calculates I by comparing the images of the preceding and following frames. t Depth map D t :

[0061] D t =DepthNet(I t ,I s )

[0062] In this embodiment, the parameter optimization module calculates a loss function based on the depth map, pose transformation matrix, and video frames, and adjusts the parameters of the pose estimation network model according to the loss function. The loss function includes reconstruction error, depth smoothing loss, and geometric consistency loss.

[0063] When calculating the reconstruction error, given two consecutive frame images I t (target image) and I s (Source image), now their depth maps D have been obtained through the DepthNet and PoseNet networks. t and D s and their pose transformation T t→s For I s Any point p in s ∈I s According to the pixel-camera coordinate transformation formula, its position in I can be obtained. t Position p′ s :

[0064]

[0065] Where ~ represents the positional correspondence, i.e., I t point p′ on s and I s Point p on s These are the same point in the real world, and K represents the camera intrinsic parameter matrix. Similarly, all I... s All points on are projected onto I t By ignoring points that extend beyond the boundary after projection, all I can be obtained. s The point on I t The position above, at this point, can be determined based on I. t Reconstruct I′ using the RGB values ​​of the corresponding pixels. s, and then the reconstructed I′ s and I s Compare and calculate the reconstruction error L pe Reconstruction error L pe That is:

[0066] or L pe =λ1|I′ s -I s |+λ2SSIM(I′ s ,I s )

[0067] Where SSIM() is the structural similarity function, λ1 and λ2 are weight coefficients, and I′ s satisfy:

[0068]

[0069] To ensure that the depth map output by the DepthNet depth prediction network has smoothness and scale consistency, a depth smoothing loss Lo is required. smooth and geometric consistency loss L geo Additional constraints are imposed on DepthNet:

[0070]

[0071]

[0072] Among them, D t This is the depth map corresponding to the image at time t in the video frame sequence S. and D′ represents the partial derivatives of the two-dimensional image coordinates in the x and y directions, respectively. t This represents the depth map at time t, reconstructed using the depth maps from subsequent time steps through pose transformation.

[0073] To reduce invalid point matching during the projection process, the final loss function L all The expression is:

[0074]

[0075] Where l represents the scale number, and ω1, ω2, and ω3 represent the weights of the reconstruction error, depth smoothing loss, and geometric consistency loss, respectively. By calculating L... all Backpropagation updates the network parameters relative to the gradients of all network parameters, iterating multiple times until the model converges.

[0076] To verify the effectiveness of this implementation method, the performance of this implementation method and other localization algorithms based on unsupervised deep learning in recent years on the KITTI dataset was compared, as shown in Tables 1 and 2.

[0077] Table 1 Comparison of average translation error and average rotation error of various unsupervised neural network localization algorithms

[0078]

[0079] Table 2 Comparison of absolute trajectory error and relative pose error of various unsupervised neural network localization algorithms

[0080]

[0081] Table 1 shows the test results on the KITTI datasets seq09 and seq10. All algorithms were trained using seq00 to seq08 of the KITTI dataset and tested using seq09 and seq10, where t rel Represents rotational offset error, t tel These represent translational and rotational errors, respectively indicating the average translational and rotational deviation per 100 meters of travel. (Unit: m / 100m).

[0082] Table 2 shows the test results of this implementation method and other algorithms on the KITTI datasets seq09 and seq10, where ATE represents the root mean square of the difference between the predicted camera pose and the actual camera pose, and RPE represents the relative pose error between frames.

[0083] Figure 4 This is a schematic diagram comparing the path trajectory output from the test on the KITTI datasets seq09 and seq10 with the actual path in this embodiment. Figure 4 It can be seen that the two curves basically match, indicating that the positioning of this implementation method is accurate.

Claims

1. A vision-inertial navigation-radar fusion self-localization method based on a self-supervised neural network, characterized in that, Includes the following steps: Acquire video frame sequences, inertial navigation data, and lidar point cloud maps; The video frame sequence, inertial navigation data, and lidar point cloud map are input into the pose estimation network model to obtain the relative pose estimate of the input video frame; The pose estimation network model includes: A depth prediction network is used to obtain a depth map based on the input video frame sequence; A feature extraction network is used to extract feature information from the input video frame sequence, inertial navigation data, and lidar point cloud map, respectively, to obtain visual features, momentum features, and radar features; the feature extraction network includes: In the first feature extraction part, a first convolutional network is used to extract visual features from the video frame sequence; In the second feature extraction part, an LSTM network is used to extract momentum features from the inertial navigation data; In the third feature extraction part, the LiDAR point cloud map is projected onto a 2D plane, and then the LiDAR point cloud map projected onto the 2D plane is encoded using a three-channel encoding method. The radar features are then extracted using a second convolutional network. The first and second convolutional networks have the same structure and share the weights of all network layers except the BN layer. A feature fusion network is used to fuse the visual features and radar features to obtain pre-fused corrected visual features and corrected radar features, and then fuse the pre-fused corrected visual features and corrected radar features with the momentum features to obtain fused features; the feature fusion network includes: The first fusion part is used to fuse the visual features and radar features using a channel switching strategy to obtain pre-fused corrected visual features and corrected radar features. In the second fusion section, the modified visual features and modified radar features used for pre-fusion are stitched together with the momentum features on a channel basis to obtain the fused features. A pose estimation network is used to predict the pose transformation matrix based on the fused features; The parameter optimization module is used to calculate the loss function based on the depth map, pose transformation matrix, and video frames, and adjust the parameters of the pose estimation network model according to the loss function.

2. The vision-inertial navigation-radar fusion self-localization method based on self-supervised neural network according to claim 1, characterized in that, The channel switching strategy is as follows: Among them, V′ k,c V represents the visual feature of the c-th channel output by the k-th convolutional layer after the swapping strategy. k,c L represents the visual feature of the c-th channel output by the k-th convolutional layer. k,c Let a represent the radar feature of the c-th channel output by the k-th convolutional layer. v,k,c b v,k,c σ v,k,c and μ v,k,c Let a represent the slope, bias, mean, and variance of the BN layer in the first convolutional network, respectively; l,k,c b l,k,c σ l,k,c and μ l,k,c These represent the slope, bias, mean, and variance of the BN layer in the second convolutional network, respectively. δ is the threshold.

3. The vision-inertial navigation-radar fusion self-localization method based on self-supervised neural network according to claim 1, characterized in that, The loss function includes reconstruction error, depth smoothing loss, and geometric consistency loss, and its expression is: Among them, L all Let L be the loss function. pe L represents the reconstruction error. smooth L represents the deep smoothing loss. geo Let l represent the geometric consistency loss, l represent the scale number, and ω1, ω2, and ω3 represent the weights of the reconstruction error, depth smoothing loss, and geometric consistency loss, respectively.

4. The vision-inertial navigation-radar fusion self-localization method based on self-supervised neural network according to claim 3, characterized in that, The expression for the reconstruction error is: L pe =∑ ps∈Is (λ1|I t (p′ s )-I s (p s )|+λ2SSIM(I t (p′ s ),I s (p s ))), where I s For the source image, p s For source image I s The point on, I t For the target image, p′ s For source image I s Point p on s Corresponding to target image I t The points are defined by SSIM(), which is the structural similarity function, and λ1 and λ2 are the weight coefficients.

5. The vision-inertial navigation-radar fusion self-localization method based on self-supervised neural network according to claim 3, characterized in that, The expression for the reconstruction error is: L pe =λ1|I′ s -I s |+λ2SSIM(I′ s ,I s ), where I s For the source image, I′ s For target image I t The reconstructed source image, SSIM() is the structural similarity function, and λ1 and λ2 are weight coefficients.

6. The vision-inertial navigation-radar fusion self-localization method based on self-supervised neural network according to claim 3, characterized in that, The expression for the depth smoothing loss is: Among them, D t I is the depth map corresponding to the image at time t in the video frame sequence S. t For the target image, and These represent the partial derivatives of the coordinates of the two-dimensional image with respect to the x and y directions, respectively.

7. The vision-inertial navigation-radar fusion self-localization method based on self-supervised neural network according to claim 3, characterized in that, The expression for the geometric consistency loss is: Among them, D t Let D′ be the depth map corresponding to the image at time t in the video frame sequence S. t This represents the depth map at time t, reconstructed using the depth maps from subsequent time steps through pose transformation.

Images