Multi-sensor fusion cross-modal matching positioning method and system based on depth estimation

Abstract

The invention belongs to the field of robot positioning, and particularly relates to a multi-sensor fusion cross-modal matching positioning method and system based on depth estimation. Pixel-level depth estimation is carried out based on a deep learning technology in a mode of collaborative fusion of a camera and a millimeter-wave radar to obtain a pixel point cloud; through matching the pixel point cloud with the high-precision map point cloud, cross-modal matching positioning is completed, accurate positioning of the robot in an operation environment is realized, and a positioning result with relatively high accuracy can be obtained while the hardware cost of the working robot is greatly reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of robot positioning, and particularly relates to a multi-sensor fusion cross-modal matching positioning method and system based on depth estimation. BACKGROUND

[0002] In the robot mapping and positioning application scenario, the robot responsible for mapping will model the robot operating environment according to the laser radar point cloud, and construct a high-precision point cloud map, referred to as a high-precision point cloud map; in actual operation, the working robot can also be equipped with a laser radar, and online matching can be performed according to the real-time detected laser radar point cloud and the established high-precision point cloud map to calculate the pose of the working robot, thereby achieving accurate positioning in the operating environment. Although this method is logically intuitive, and both the map establishment and the online matching positioning are based on laser radar point cloud, the same modal matching positioning effect is excellent, but due to the relatively high price of the laser radar, in the actual operation process, installing a laser radar for all working robots often faces a high hardware cost. Although the sensor of the working robot can be replaced with a camera or a millimeter wave radar, the cross-modal matching positioning problem is still faced, that is, the problem of cross-modal matching positioning between the RGB image collected by the camera and the millimeter wave radar point cloud and the laser point cloud map. SUMMARY

[0003] To solve the above problems, the application provides a multi-sensor fusion cross-modal matching positioning method and system based on depth estimation, which adopts a camera and a millimeter wave radar in a cooperative fusion manner, performs depth estimation at a pixel level based on a depth learning technology to obtain pixel point cloud, and completes cross-modal matching positioning by matching the pixel point cloud with high-precision map point cloud, thereby achieving accurate positioning of the robot in the operating environment, greatly reducing the hardware cost of the working robot, and obtaining a positioning result with high accuracy.

[0004] The technical scheme adopted by the application is as follows:

[0005] A multi-sensor fusion cross-modal matching positioning method based on depth estimation comprises the following steps:

[0006] Step 1: synchronously collecting an RGB image and millimeter wave radar point cloud data of a target area; and constructing a high-precision environment map by a robot through laser radar point cloud data;

[0007] Step 2: extracting image features from the RGB image, and generating multi-scale guided features in the extraction process;

[0008] Step 3: sequentially performing densification and sparse coding processing on the point cloud data, then fusing the densified point cloud data with the multi-scale guided features to obtain intermediate sparse features and final millimeter wave radar features.

[0009] Step 4: Fuse image features, millimeter wave radar features, guidance features, and sparse features, and obtain pixel-level depth estimation and pixel point cloud through deconvolution decoding; calculate the scale ratio of radar sparse depth and deconvolution decoding depth through a global scale optimization module, multiply the scale ratio with the deconvolution decoding depth to adjust, and obtain the final predicted depth value and pixel point cloud;

[0010] Step 5: Use a feature extraction network based on a multi-scale FCGF descriptor (Fully-Convolutional Geometric Features) to extract feature point sets of the pixel point cloud and the high-precision point cloud map respectively; and based on the extraction results, construct a pixel point undirected graph and a high-precision point cloud map undirected graph for the two types of feature point sets respectively;

[0011] Step 6: Coarsely match the feature points between the pixel point cloud undirected graph and the high-precision point cloud map undirected graph, and delete obviously incorrect matches according to a set threshold;

[0012] Step 7: Based on the coarse matching result, extract the maximum clusters from the pixel point cloud undirected graph and the high-precision point cloud map undirected graph respectively; calculate the pose transformation relationship through the corresponding feature points in the matching clusters to obtain the matched maximum cluster point pairs;

[0013] Step 8: Align the complete point clouds between the pixel point cloud undirected graph and the high-precision point cloud map undirected graph according to the maximum cluster point pair information; then, calculate the pose through the ICP algorithm based on the least squares method to complete the multi-sensor fusion cross-modal matching positioning.

[0014] Further, the step 2 uses an RGB image feature extraction module based on an encoder-decoder structure to extract image features; the RGB image feature extraction module based on the encoder-decoder structure is composed of an encoder and a decoder, wherein the encoder uses ResNet-34 as the backbone network, and the decoder uses an Unet-shaped deconvolution structure.

[0015] Further, the implementation method of the step 3 includes:

[0016] The point cloud data is first sequentially processed by densification and sparse coding, and then fused with multi-scale guidance features to obtain intermediate sparse features and final millimeter wave radar features; the detailed implementation process includes:

[0017] Each point in the millimeter wave radar point cloud is regarded as a node in an undirected graph, and the distance, depth, and velocity correlation measure between points are taken as the weight of the edge; statistical analysis is performed on the multi-target radar point cloud data to determine the correlation measure threshold: when the correlation measure between two nodes exceeds the threshold, it is determined that the two nodes belong to the same target; based on this, interpolation is performed between nodes greater than the determination threshold to complete the densification processing of the millimeter wave radar point cloud data;

[0018] The interpolated point cloud is processed using a sparse invariant convolution encoder to obtain the original millimeter radar features and form intermediate sparse features; the intermediate sparse features are fused based on the guide features to generate radar sparse features that integrate RGB image features, i.e., the final millimeter wave radar features after integrating multi-modal information.

[0019] Further, the step 4 uses an element-by-element addition method to fuse image features, millimeter wave radar features, guide features, and sparse features.

[0020] Further, the step 6 uses a K-nearest neighbor matching algorithm to perform coarse matching between feature points of the pixel point cloud undirected graph and the high-precision point cloud map undirected graph.

[0021] Further, the step 6 uses a K-nearest neighbor matching algorithm to perform coarse matching between feature points of the pixel point cloud undirected graph and the high-precision point cloud map undirected graph.

[0022] (1) Calculate the descriptors of the high-precision point cloud and the pixel point cloud, respectively;

[0023] (2) Take the cosine similarity between the descriptor vectors as the similarity measure index, and use the K-nearest neighbor matching algorithm to find the most similar feature points in the high-precision point cloud map for each feature point of the pixel point cloud to complete the coarse matching of the feature points on the undirected graph: find the K most similar points in the high-precision point cloud map feature point set to the pixel point cloud feature points; at the same time, find the K most similar points in the high-precision point cloud map feature point set to the pixel point cloud. Traverse all points in turn, and all matching pairs can be obtained.

[0024] Further, the step 7 uses a maximum clique search algorithm to extract the maximum cliques from the pixel point cloud undirected graph and the high-precision point cloud map undirected graph.

[0025] A multi-sensor fusion cross-modal matching positioning system based on depth estimation adopts a depth estimation network of image and millimeter wave radar fusion to realize positioning, comprising an image feature extraction module, an image interpolation algorithm module, a depth encoder module, a depth decoder module and a feature extraction network module based on a multi-scale descriptor, and the above-mentioned multi-sensor fusion cross-modal matching positioning method based on depth estimation is executed through the modules.

[0026] Compared with the prior art, the present application has the following advantages:

[0027] The present application realizes pixel-level depth estimation by using deep learning technology to obtain a pixel point cloud, and then matches the pixel point cloud with a high-precision map point cloud, thereby not only solving the cross-modal matching positioning problem, but also having high precision. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 It is a structure diagram of the multi-sensor fusion cross-modal matching positioning system based on depth estimation;

[0029] Figure 2 It is a network structure schematic diagram of the image feature extraction module based on the encoder-decoder structure of the present application;

[0030] Figure 3 It is a depth encoder network structure schematic diagram based on the image and millimeter wave radar feature fusion of the present application;

[0031] Figure 4 It is a depth decoder network structure schematic diagram based on the image and millimeter wave radar feature fusion of the present application;

[0032] Figure 5 It is a feature extraction network structure schematic diagram based on the multi-scale FCGF descriptor of the present application;

[0033] Figure 6 It is a schematic diagram of intermediate processing and depth estimation results (inverse projection image) based on the VoD dataset of the present application;

[0034] Figure 7 It is a point cloud matching result (inverse projection image) schematic diagram based on the multi-scale FCGF descriptor of the present application;

[0035] Figure 8 It is a schematic diagram of an undirected graph based on point cloud features of the present application. DETAILED DESCRIPTION

[0036] The technical scheme of the present application will be described in detail below with reference to the drawings.

[0037] A multi-sensor fusion cross-modal matching positioning system based on depth estimation provided by the present embodiment adopts a depth estimation network of image and millimeter wave radar fusion to realize positioning, as shown inFigure 1 As shown, the image feature extraction module, the image interpolation algorithm module, the depth encoder, the depth decoder, and the feature extraction network based on the multi-scale descriptor are included, and the following multi-sensor fusion cross-modal matching positioning method based on depth estimation is executed through the modules.

[0038] A multi-sensor fusion cross-modal matching positioning method based on depth estimation, comprising the following steps:

[0039] Step 1: synchronously collecting the RGB image and the millimeter wave radar point cloud data of the target area; the robot constructs a high-precision environment map through the laser radar point cloud data;

[0040] Step 2: extracting the image features of the RGB image, and generating the multi-scale guided features in the extraction process. The embodiment uses an RGB image feature extraction module based on an encoder-decoder structure to extract the image features, and the module is as shown in the figure. Figure 2 As shown, the module includes an encoder and a decoder, wherein the encoder uses ResNet-34 as the backbone network, and the decoder uses the deconvolution structure of Unet. The RGB image is input into the encoder, and the image is encoded through the downsampling convolution operation to obtain the image features with reduced resolution but increased channel number; in this process, the encoder will generate multiple sets of intermediate feature maps with different resolutions; the input of the decoder is the image features output by the encoder, and when the spatial resolution is restored through the deconvolution upsampling, the current upsampling features will be fused with the intermediate features of the corresponding scale in the encoder to obtain the guided feature map and take it as one of the inputs of the next stage or the final output. The final fusion output of the decoder of the embodiment is 4 scales of guided features, which provides full-scale image information covering details and semantics for subsequent depth estimation and enhances the model understanding ability.

[0041] Step 3: sequentially performing densification and sparse coding processing on the point cloud data, and then fusing it with the multi-scale guided features to obtain the intermediate sparse features and the final millimeter wave radar features. The implementation process is as follows:

[0042] Each point in the millimeter wave radar point cloud is regarded as a node in the undirected graph, and the distance, depth and velocity correlation measure between the points are regarded as the weight of the edge. The correlation measure threshold is determined by counting the radar point cloud of multiple targets, that is, the nodes greater than the threshold represent the same target, and therefore the interpolation operation is performed between the nodes greater than the threshold to complete the densification processing of the millimeter wave radar point cloud data.

[0043] The interpolated point cloud is processed by sparse invariant convolution using a sparse data encoder to obtain original millimeter radar features, forming intermediate sparse features; the intermediate sparse features are fused based on the guide features to generate radar sparse features fused with RGB image features, i.e., the final millimeter wave radar features after integrating multi-modal information. The intermediate sparse features and the final millimeter wave radar features obtained in this embodiment are implemented using a deep encoder network based on image and millimeter wave radar feature fusion, and the execution process is as shown in Figure 3 .

[0044] (1) Channel feature weight generation and compression: generate channel feature weights by channel convolution and generate spatial feature weights by spatial convolution for each guide feature; compress the width-height (WH) dimensions of the channel feature weights to the same dimension by pooling processing, and compress the channel dimensions of the spatial feature weights into one dimension by pooling processing;

[0045] (2) Dimension alignment: adjust the compressed channel feature dimensions using a multi-layer perception (C-MLP) that processes the channel dimensions; adjust the compressed spatial feature dimensions using a WH convolution that processes the width-height dimensions, so that they are the same as the channels and width-height of the radar features;

[0046] (3) Feature adjustment: multiply the channel feature weights and the spatial feature weights that have undergone dimension alignment, respectively, with the radar features to obtain sparse features at different scales.

[0047] Step 4: Depth estimation and pixel point cloud acquisition. This embodiment uses a feature fusion depth decoder network based on image and millimeter wave radar to achieve depth estimation and pixel point cloud acquisition. First, the image features, millimeter wave radar features, guide features, and sparse features are fused by element-wise addition, and the pixel-level depth estimation (i.e., the initial decoded depth) and the pixel point cloud are obtained by inverse convolution decoding. Then, the scale ratio between the radar sparse depth and the decoded depth is calculated explicitly by a global scale optimization module, and the ratio is multiplied by the initial decoded depth to adjust it, thereby obtaining the final predicted depth value and the pixel point cloud. The specific process is shown in Figure 4 .

[0048] Step 5: Use a feature extraction network based on a multi-scale FCGF descriptor to extract feature point sets for the pixel point cloud and the high-precision point cloud map, respectively; and construct a pixel point undirected graph for the pixel point cloud feature point set and a high-precision point cloud map undirected graph for the high-precision point cloud map feature point set based on the extraction results.

[0049] The feature extraction network based on a multi-scale FCGF descriptor used in this embodiment performs multi-scale feature extraction. As shown in Figure 5The feature extraction network based on the multi-scale FCGF descriptor includes a feature extraction module based on a 3D Unet architecture and a deconvolution layer; each feature extraction module based on the 3D Unet architecture is composed of a plurality of Unet structures, each Unet structure performs down-sampling on the high-precision map point cloud and the pixel point cloud data respectively through 3D convolution layers with a step of 2, and then extracts features by using a structure containing a residual connection and a skip connection. Three Unets with different scales are constructed by using 64, 128 and 256 output channels in sequence to realize multi-level feature extraction, so as to enhance the robustness and scale invariance of the features. Finally, full convolution features are generated by up-sampling through a 3D deconvolution layer, as the descriptor. On this basis, a pixel point cloud undirected graph and a high-precision point cloud map undirected graph are respectively established for the feature point set of the pixel point cloud and the high-precision point cloud map, wherein the feature points are used as vertices, and the connection lines between the feature points are used as edges, so as to convert the features extracted by the feature extraction network based on the multi-scale FCGF descriptor into a structured correlation expression, which is specifically shown in Figure 8 .

[0050] Step 6: coarse matching is performed on the feature points between the pixel point cloud undirected graph and the high-precision point cloud map undirected graph, and obviously incorrect matching is deleted according to a set threshold. Specifically:

[0051] (1) the descriptors of the high-precision point cloud map and the pixel point cloud are respectively calculated;

[0052] (2) the cosine similarity between the descriptor vectors is used as a similarity measurement index, the K-nearest neighbor matching algorithm is used to find the most similar feature points in the high-precision point cloud map for each feature point of the pixel point cloud, and coarse matching of the feature points on the undirected graph is completed: in the feature point set of the pixel point cloud, the K most similar points to the feature points of the high-precision point cloud map are found; at the same time, in the feature point set of the high-precision point cloud map, the K most similar points to the pixel point cloud are found. All points are traversed in turn, and all matching pairs can be obtained.

[0053] Step 7: based on the coarse matching result, the maximum clique search algorithm is used to find the maximum clique in the pixel point cloud undirected graph and the high-precision point cloud map undirected graph respectively. In this embodiment, the Bron-Kerbosch algorithm is used to find the complete subgraph with the most vertices based on the undirected graph established in step 5, and the complete subgraph is the maximum clique. As shown in Figure 7 Since the matching point pairs in the maximum clique have high consistency, the estimation result based on these point pairs will be more accurate and robust.

[0054] Step 8: after obtaining the matching point pair information of the maximum clique, the ICP algorithm based on the least square method is used to calculate the pose, and the multi-sensor fusion cross-modal matching positioning is completed.

[0055] The beneficial effects of the present application are verified and described as follows:

[0056] To test the performance of the algorithm, the following algorithm verification is based on the Delft view dataset (VoD). In terms of depth estimation results, Figure 6 The intermediate processing and depth estimation results (inverse perspective image) based on the VoD dataset are shown; combined Figure 6 and shown in Table 1, as shown, the depth estimation network trained based on the RGB image and the millimeter wave radar multi-scale fusion in this embodiment has an RMSE error of 536.73 mm in the VoD dataset, which is the best among the performances of several classic depth estimation algorithms listed, and the scale threshold accuracy and Compared with other networks, there is a great improvement.

[0057] Table 1 Depth estimation results based on VoD dataset

[0058]

[0059] In terms of positioning accuracy, the present application is compared and analyzed with several listed existing methods. First, the Monodepth depth estimation and ICP registration method after millimeter wave radar global scale optimization are compared to explore the influence of the performance of the depth estimation network on positioning. Second, by comparing with directly using neural network regression positioning results, the effect of the direct regression method based on the depth learning network is discussed. Finally, the grid classification method (Grid Cls.+PnP) and the frustum classification method (Frus Cls.+6-DoF) in DeepI2P are compared to explore the effect of the classification matching method based on the depth learning, as shown in Compared with several listed existing methods, the positioning error and translation error of the method of the present application are the best.

[0060] Table 2 Positioning error results based on VoD dataset

[0061]

[0062] Of course, the present application can also have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, but these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.

Claims

1. A multi-sensor fusion cross-modal matching localization method based on depth estimation, characterized in that, Includes the following steps: Step 1: Simultaneously acquire RGB images and millimeter-wave radar point cloud data of the target area; the robot constructs a high-precision environmental map using the lidar point cloud data; Step 2: Extract image features from the RGB image and generate multi-scale guided features during the extraction process; Step 3: First, perform dense and sparse coding processing on the point cloud data in sequence. Then, fuse the dense point cloud data with multi-scale guided features to obtain intermediate sparse features and the final millimeter-wave radar features. Step 4: Fuse image features, millimeter-wave radar features, guidance features, and sparse features, and obtain pixel-level depth estimation and pixel point cloud through deconvolution decoding; calculate the scale ratio of radar sparse depth to deconvolution decoding depth through the global scale optimization module, and adjust it by multiplying it with the deconvolution decoding depth to obtain the final predicted depth value and pixel point cloud. Step 5: Use a feature extraction network based on multi-scale FCGF descriptors to extract feature point sets for pixel point clouds and high-precision point cloud maps respectively; and construct undirected graphs of pixel points and high-precision point cloud maps for the two types of feature point sets based on the extraction results respectively. Step 6: Perform coarse matching of feature points between the undirected pixel point cloud graph and the undirected high-precision point cloud map graph, and delete obviously erroneous matches according to the set threshold; Step 7: Based on the coarse matching results, extract the maximum clique from the pixel point cloud undirected graph and the high-precision point cloud map undirected graph respectively; calculate the pose transformation relationship by matching the corresponding feature points within the clique to obtain the matching maximum clique point pairs; Step 8: Based on the information of the largest clumped points, align the complete point cloud between the undirected graph of pixel point cloud and the undirected graph of high-precision point cloud map; then, use the ICP algorithm based on the least squares method to calculate the pose and complete the multi-sensor fusion cross-modal matching localization.

2. The method according to claim 1, characterized in that, Step 2 uses an RGB image feature extraction module based on an encoder-decoder structure to extract image features; The RGB image feature extraction module based on the encoder-decoder structure consists of an encoder and a decoder. The encoder uses ResNet-34 as the backbone network, and the decoder uses a Unet-shaped deconvolution structure.

3. The method according to claim 1, characterized in that, Step 3, which involves sequentially performing densification and sparse coding on the point cloud data, and then fusing the densified point cloud data with multi-scale guided features to obtain intermediate sparse features and the final millimeter-wave radar features, includes the following steps: Each point in the millimeter-wave radar point cloud is treated as a node in an undirected graph, and the distance, depth, and velocity correlation between points are used as edge weights. Statistical analysis is performed on the multi-target radar point cloud data to determine the threshold for correlation measurement: when the correlation measurement between two nodes exceeds the threshold, the two nodes are determined to belong to the same target. Based on this, interpolation is performed between nodes with a correlation greater than the threshold to complete the densification processing of the millimeter-wave radar point cloud data. The interpolated point cloud is processed by a sparse data encoder to perform sparse invariant convolution to obtain the original millimeter radar features, forming intermediate sparse features. The intermediate sparse features are then fused based on the guided features to generate radar sparse features that integrate RGB image features, which are the final millimeter-wave radar features after integrating multimodal information.

4. The method according to claim 1, characterized in that, In step 4, image features, millimeter-wave radar features, guidance features, and sparse features are fused by adding elements one by one.

5. The method according to claim 1, characterized in that, Step 6 employs a K-nearest neighbor matching algorithm to perform coarse matching of feature points between the undirected pixel point cloud graph and the high-precision point cloud map undirected graph.

6. The method according to claim 5, characterized in that, Step 6, which employs a K-nearest neighbor matching algorithm to perform coarse matching of feature points between the undirected pixel point cloud graph and the high-precision point cloud map undirected graph, includes the following steps: (1) Calculate the descriptors for the high-precision point cloud map and the pixel point cloud respectively; (2) Using the cosine similarity between descriptor vectors as a similarity metric, the K-nearest neighbor matching algorithm is used to find the most similar feature points in the high-precision point cloud map for each feature point of the pixel point cloud, thus completing the coarse matching of feature points on the undirected graph: in the feature point set of the pixel point cloud, find the K points most similar to the feature points of the high-precision point cloud map; at the same time, in the feature point set of the high-precision point cloud map, find the K points most similar to the pixel point cloud, and traverse all points in turn to obtain all matching pairs.

7. The method according to claim 5, characterized in that, Step 7 uses a maximum clique search algorithm to extract the maximum clique from the pixel point cloud undirected graph and the high-precision point cloud map undirected graph, respectively.

8. A multi-sensor fusion cross-modal matching localization system based on depth estimation, which uses a depth estimation network that fuses image data with millimeter-wave radar to achieve localization, characterized in that... The system includes an image feature extraction module, a graph interpolation algorithm module, a depth encoder module, a depth decoder module, and a feature extraction network module based on multi-scale descriptors. These modules execute the aforementioned multi-sensor fusion cross-modal matching localization method based on depth estimation.

Images