Positioning method for generating pseudo LiDAR descriptor for cross-modal matching based on visual depth estimation

Abstract

The invention discloses a positioning method for generating a pseudo LiDAR descriptor for cross-modal matching based on visual depth estimation, and the method comprises the following stages: in an offline training stage, obtaining synchronous multi-modal data, inputting a camera image into a pre-trained monocular depth estimation network, generating a dense depth map, and carrying out the alignment; training a global descriptor generation network with weight sharing; a laser LiDAR global descriptor database is constructed; an on-line positioning stage: acquiring a surrounding environment image; inputting the image into a pre-trained monocular depth estimation network to generate dense depth map alignment; inputting a global descriptor generation network, and generating a visual global descriptor; and performing cross-modal descriptor retrieval and matching. Generating a geometrically aligned pseudo LiDAR depth map through monocular depth estimation, and realizing cross-modal matching with a real LiDAR map; the system hardware threshold is obviously reduced; introducing a triple supervision mechanism based on geometric overlapping degree and a yaw angle invariant descriptor network of weight sharing; and a modal gap between the vision and the LiDAR is bridged.

Description

Technical Field

[0001] This invention relates to a cross-modal sensor data localization method, and more particularly to a localization method based on visual depth estimation to generate pseudo-LiDAR descriptors for cross-modal matching. Background Technology

[0002] Navigation and positioning technology, as one of the fundamental yet crucial capabilities of mobile robot systems, has always been a research hotspot in both academia and industry. With the increasing demands for positioning accuracy, robustness, and cost-effectiveness from applications such as autonomous driving, intelligent logistics, and indoor / outdoor collaborative operations, the field has seen significant advancements. In recent years, deep learning-based LiDAR point cloud descriptor methods, such as PointNetVLAD and OverlapTransformer, have significantly improved loop closure detection and global localization capabilities in large-scale scenarios. Among these, OverlapTransformer, by converting LiDAR scans into range images and incorporating them into the Transformer architecture, achieves yaw angle invariance and millisecond-level inference speeds, providing an efficient solution for pure LiDAR positioning systems. However, these methods heavily rely on expensive LiDAR sensors, making them difficult to deploy on low-cost mobile platforms.

[0003] Meanwhile, while visual localization methods offer advantages such as low cost and rich information, their robustness significantly decreases under complex conditions such as changes in lighting, seasonal transitions, or drastic changes in viewing angle. Although existing research has attempted to achieve visual localization through cross-modal matching between images and LiDAR maps, current methods generally face a core problem: a significant modal gap exists when directly matching images with the original LiDAR map, leading to difficulties in feature alignment and low recall rates. Therefore, enabling visual systems to directly, efficiently, and robustly query high-precision maps constructed from pure LiDAR without relying on LiDAR sensors has become a key challenge for achieving low-cost, high-precision autonomous localization. Summary of the Invention

[0004] This invention proposes a localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors, which can effectively solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A cross-modal localization method for visual depth-LiDAR descriptors includes an offline training phase and an online localization phase. The specific steps are as follows:

[0007] (a) During the offline training phase:

[0008] (1) Acquire synchronized multimodal data, including panoramic camera images, LiDAR point clouds, calibration parameters between the camera and LiDAR, and pose ground truth.

[0009] (2) Input the camera image into the pre-trained monocular depth estimation network to generate a dense depth map. Using the pose ground truth and the real LiDAR point cloud, estimate the global scale factor s through scale invariant loss, and back-project the depth map into a point cloud, which is spatially aligned with the real LiDAR point cloud.

[0010] A geometrically aligned pseudo-LiDAR depth map is generated through monocular depth estimation, and cross-modal matching with the real LiDAR map is achieved at the descriptor level.

[0011] (21) Assume that the surround vision system carried by the unmanned system includes There are 12 cameras, each with a limited field of view, which together cover a 360° horizontal field of view. For the 12th camera... One camera ( The RGB image it acquires is denoted as Each image Input a pre-trained monocular depth estimation network to generate the corresponding dense depth map. Since such networks typically output relative depth, which needs to be aligned with real LiDAR point clouds measured in meters, scale calibration is required.

[0012] Using the ground truth pose values ​​and LiDAR point clouds provided in the offline dataset, the global scale factor is estimated by minimizing the reprojection error or scale invariance loss. The calibrated absolute depth map is obtained:

[0013]

[0014] For any pixel in the depth map (Taking the top left corner of the image as the origin) For column indexes, (for row indexes), if its calibration depth If a pixel is located in an unmasked area, it is considered a valid pixel and participates in subsequent backprojection.

[0015] Using the first Intrinsic parameter matrix of each camera :

[0016]

[0017] in Focal length (unit: pixels). Using the principal point coordinates (in pixels, usually close to the image center), backproject the effective pixels to Euclidean 3D points in the camera coordinate system:

[0018]

[0019] (22) Using the extrinsic parameters from the camera to the LiDAR coordinate system provided by the public dataset Transform this point to the LiDAR coordinate system. Because... For 3D Euclidean vectors, only the rotation and translation components are needed:

[0020]

[0021] Perform the above operation on all cameras i and their effective pixels to obtain Group of local point clouds They are then directly stitched together to form a complete visual point cloud:

[0022]

[0023] Thanks to the precise calibration parameters, strict time synchronization, and scale calibration mechanisms provided by the public dataset, Compared with the real LiDAR point cloud at the same moment Alignment is achieved in spatial location, physical scale, and coordinate system, and can be directly used for subsequent cross-modal descriptor training.

[0024] (3) The point cloud generated by vision is converted into a pseudo LiDAR depth map by spherical projection, and the real LiDAR point cloud at the corresponding time is projected into a real depth map;

[0025] Geometric overlap is calculated using real LiDAR point clouds and their pose ground values, serving as a supervisory label for cross-modal triples;

[0026] A weight-shared global descriptor generation network is trained by minimizing the distance between co-located pseudo-LiDAR and real LiDAR descriptors and maximizing the distance between disjoint pairs.

[0027] (31) Convert the 3D point cloud into a spherical depth map for subsequent input into the image-based descriptor network. Let any point in the point cloud be... This is represented in the LiDAR coordinate system. To ensure that the visually generated pseudo-LiDAR depth map is strictly aligned with the real LiDAR depth map in terms of spatial coverage and effective area, only those visual points that have corresponding observations in the original LiDAR scan are retained: firstly, the real LiDAR point cloud of the current frame is projected as a standard-sized range map. And construct a binary effective mask. ,in This indicates an indicator function that takes a value of 1 when the condition within the parentheses is true, and 0 otherwise. Subsequently, for each point in the visually reconstructed point cloud... First, project it onto the spherical image coordinates in the LiDAR coordinate system. ,like If the value is positive, then keep the point; otherwise, discard it.

[0028] (32) Perform spherical projection on the filtered valid point set to map it to two-dimensional image coordinates. And record its radial distance.

[0029]

[0030] in and These are the width and height of the depth map, respectively. and The vertical field of view of the sensor, both upward and downward (in radians).

[0031] (33) Fill the depth map using the "nearest point priority" principle, that is, retain the minimum radial distance for the same pixel position:

[0032]

[0033] in This is a subset of the visual point cloud after LiDAR masking. This represents the spherical projection function defined by formula (6). For pixels not covered by any valid points, let... The resulting pseudo LiDAR depth map Compared with real LiDAR depth map Having exactly the same size Furthermore, visual depth values ​​are filled only within the actual LiDAR observation area, fundamentally eliminating modal differences introduced by inconsistent fields of view or invalid depth estimation, and providing geometrically aligned input for subsequent descriptor networks with shared weights.

[0034] (34) The global descriptor generation network includes a depth map encoder, a Transformer module and a global descriptor generator connected in sequence. It takes a single-channel spherical depth map as input and outputs a global descriptor vector of fixed dimensions.

[0035] The depth encoder employs a multi-layer convolutional neural network, downsampling only in the vertical direction while maintaining the original resolution in the horizontal direction. Furthermore, no padding is used in any of the convolutional layers to ensure that the output feature map maintains the same yaw angle variability as the input depth map in the horizontal direction. Let the input depth map be... ,in For vertical resolution, For horizontal resolution, the encoder outputs a feature map. Satisfying the yaw and other variable relationships:

[0036]

[0037] in, Indicates cyclic shift along the horizontal direction Each pixel corresponds to a change in the vehicle's yaw angle.

[0038] The Transformer module includes a multi-head self-attention mechanism (MHSA), a feedforward network (FFN), and a layer normalization (LN) unit, used for global context modeling of the feature sequence output by the range graph encoder. This module receives signals of size [size missing]. The feature map is regarded as The Transformer module also maintains the same yaw angle variability, meaning that when the input feature map undergoes a horizontal cyclic shift, the output feature map shifts synchronously.

[0039] set up From The query, key, and value matrix obtained by linear transformation along the channel dimension is then the multi-head self-attention output as follows:

[0040]

[0041] And satisfy ,in This refers to the Transformer module.

[0042] The global descriptor generator uses the NetVLAD aggregation layer to process the output of the Transformer module. Local features (in They are aggregated into a fixed-dimensional global descriptor. :

[0043]

[0044] in The number of cluster centers. For the first A learnable clustering center We assign weights softly. Since this operation is permutation-invariant to the input order, and combined with the yaw-angle invariance of the preceding modules, the final generated global descriptor possesses yaw angle invariance. This means that descriptors generated for the same geographical location under different orientations have consistent heights, satisfying the following:

[0045]

[0046] (35) During the training phase, the real LiDAR depth map and the visually generated pseudo-LiDAR depth map share the same set of network parameters. The triplet loss function constrains the alignment of visual and LiDAR modalities in the descriptor space by minimizing the Euclidean distance between cross-modal descriptor pairs at the same location and maximizing the distance between descriptor pairs at different locations. The loss function is:

[0047]

[0048] in, Query descriptors generated for pseudo-LiDAR graphs. For the actual LiDAR descriptor at the same location, For negative sample descriptors at different locations, This is the preset interval threshold.

[0049] (4) Using the trained global descriptor generation network, perform forward inference on the pre-collected real LiDAR point cloud map, generate and store the corresponding LiDAR global descriptor, and construct a laser LiDAR global descriptor database.

[0050] (41) Convert the real LiDAR point cloud data of the dataset selected in step 1 into a depth map in the manner described in (32) and (33).

[0051] (42) Input the real LiDAR depth map output in (41) into the global descriptor generation network trained in (35), perform forward inference, and output the global descriptor of the corresponding frame. And set the global descriptor It is stored in association with its corresponding global location and timestamp for easy indexing later.

[0052] (43) Repeat steps (41) and (42) to traverse all frames of the entire LiDAR map and finally construct a global descriptor library for LiDAR, which will be used for online cross-modal retrieval in subsequent stages.

[0053] (ii) Online positioning

[0054] (1) At time k, the surrounding environment image is acquired by the 360° camera system carried by the unmanned system;

[0055] That is, at time The unmanned system uses a 360° surround-view camera system to acquire multi-view RGB images of the surrounding environment. ,in This refers to the number of cameras.

[0056] (2) Input the camera image into the pre-trained monocular depth estimation network to generate a dense depth map. Directly reuse the global scale factor s fixed in the offline stage to back-project the depth map into a point cloud and align it spatially with the real LiDAR point cloud.

[0057] (21) Each image Input a pre-trained monocular depth estimation network to generate the corresponding dense relative depth map. Since no real LiDAR point clouds were available for scale calibration during the online phase, the global scale factor learned during the offline phase was used. (See formula (1)) Scale normalize the depth map to obtain the absolute depth map:

[0058]

[0059] (22) Based on the calibrated camera internal parameters Extrinsic parameters from camera to LiDAR coordinate system The effective pixels are back-projected onto the LiDAR coordinate system to generate a visual point cloud. Its calculation method is completely consistent with steps (1) and (2) of the offline stage, that is:

[0060]

[0061] in For the first The effective depth region in a single camera image.

[0062] (23) Use the same sensor parameters as in the offline phase. Perform spherical projection (Equation (6)).

[0063] (3) The point cloud generated by vision is converted into a pseudo LiDAR depth map by spherical projection. Based on steps (1) and (3), the pseudo LiDAR depth map is input into the global descriptor generation network to generate a visual global descriptor.

[0064] (31) Since there are no real LiDAR point clouds in the online phase, the typical effective LiDAR region templates statistically obtained during offline training are reused. Only The position retains the visual depth value, while the rest are set to 0.

[0065] (32) Fill the depth map using the "nearest point priority" principle (Formula (7)) to generate a pseudo LiDAR map that is aligned with the geometry of the real LiDAR depth map. .

[0066] (4) Based on the global descriptor database of LiDAR constructed in steps (1) and (4), cross-modal descriptor retrieval and matching are performed. The optimal matching position of the unmanned system in the map is obtained through nearest neighbor search, thereby achieving real-time high-precision positioning without LiDAR sensors.

[0067] That is, the global descriptor database of laser LiDAR constructed in steps (i) and (4) For reference, perform cross-modal nearest neighbor retrieval, where For the first Frame LiDAR descriptor, Its global pose:

[0068]

[0069] Return the index of the best matching frame. and output its corresponding pose. As an unmanned system, it is always The global positioning results.

[0070] Advantages compared to existing technologies:

[0071] Compared with the prior art, the present invention has the following significant advantages:

[0072] First, instead of relying on onboard LiDAR sensors for online positioning, this method generates a geometrically aligned pseudo-LiDAR depth map through monocular depth estimation and achieves cross-modal matching with the real LiDAR map at the descriptor level. This approach avoids the deployment costs of expensive LiDAR hardware while reusing existing pure LiDAR high-precision maps, significantly reducing the system's hardware requirements.

[0073] Secondly, a triplet supervision mechanism based on geometric overlap is introduced during the offline training phase, and a yaw angle-invariant descriptor network with weight sharing is adopted to align the visually generated descriptors with the real LiDAR descriptors in a unified embedding space. This mechanism effectively bridges the modal gap between vision and LiDAR. Attached Figure Description

[0074] Figure 1 This is a flowchart of a localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors.

[0075] Figure 2 Loss function-round number graph during offline training;

[0076] Figure 3 This is a learning rate-epoch graph during offline training.

[0077] Figure 4 To validate the accuracy-epoch graph during offline training;

[0078] Figure 5 The precision-recall curve for the offline training phase;

[0079] Figure 6 This is the Top-N recall curve for the online location phase. Detailed Implementation

[0080] Example 1, refer to Appendix Figure 1-6 A localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors includes the following steps:

[0081] S1 offline training and mapping:

[0082] S1.1 adopts a publicly available multimodal autonomous driving dataset, directly using its provided synchronous LiDAR point cloud, surround-view camera images, sensor calibration parameters, and pose ground truth, without the need to construct additional camera-LiDAR synchronous data;

[0083] During the offline training phase, this invention first utilizes existing publicly available multimodal autonomous driving datasets. These datasets were collected by professional organizations in real-world road environments and already include high-precision time-synchronized 3D LiDAR point clouds, 360° camera images, extrinsic parameter calibration matrices between sensors (i.e., rigid body transformation relationships between the camera and LiDAR), and pose ground truth values ​​provided by a high-precision GNSS / INS integrated navigation system.

[0084] This invention directly utilizes the original synchronized data provided in the above dataset, without the need for additional hardware synchronization modifications, timestamp alignment, or extrinsic parameter calibration of the camera and LiDAR, thereby significantly reducing data preparation costs and ensuring the spatiotemporal consistency of multimodal data.

[0085] For each frame of data in the dataset, the system can simultaneously acquire: RGB images from multiple surrounding cameras, the corresponding LiDAR 3D point cloud, camera intrinsic parameters and camera-to-LiDAR extrinsic calibration parameters, and the precise pose of that frame in the global coordinate system. This information provides a reliable foundation for subsequent depth map generation, point cloud alignment, geometric overlap calculation, and cross-modal triplet construction.

[0086] S1.2 Input the camera image into the pre-trained monocular depth estimation network to generate a dense depth map, and back-project it into a point cloud according to the calibration parameters, aligning it spatially with the real LiDAR point cloud.

[0087] Includes the following steps:

[0088] S1.2.1 Assume the surround-view vision system carried by the unmanned system includes There are 12 cameras, each with a limited field of view, which together cover a 360° horizontal field of view. For the 12th camera... One camera ( The RGB image it acquires is denoted as Each image Input a pre-trained monocular depth estimation network to generate the corresponding dense depth map. .

[0089] S1.2.2 Utilizing the true pose values ​​and LiDAR point clouds provided in the offline phase dataset, the global scale factor is estimated by minimizing the reprojection error. The calibrated absolute depth map is obtained:

[0090]

[0091] For any pixel in the depth map (Taking the top left corner of the image as the origin) For column indexes, (for row indexes), if its calibration depth If a pixel is located in an unmasked area, it is considered a valid pixel and participates in subsequent backprojection.

[0092] Using the first Intrinsic parameter matrix of each camera :

[0093]

[0094] in Focal length (unit: pixels). Using the principal point coordinates (in pixels, usually close to the image center), backproject the effective pixels to Euclidean 3D points in the camera coordinate system:

[0095]

[0096] S1.2.3 Using the extrinsic parameters from the camera to the LiDAR coordinate system provided by the public dataset Transform this point to the LiDAR coordinate system. Because... For 3D Euclidean vectors, only the rotation and translation components are needed:

[0097]

[0098] Perform the above operation on all cameras i and their effective pixels to obtain Group of local point clouds They are then directly stitched together to form a complete visual point cloud:

[0099]

[0100] (Thanks to the precise calibration parameters, strict time synchronization, and scale calibration mechanisms provided by the publicly available dataset,) Compared with the real LiDAR point cloud at the same moment Alignment is achieved in spatial location, physical scale, and coordinate system, and can be directly used for subsequent cross-modal descriptor training.

[0101] S1.3 The visual point cloud (the point cloud generated by vision mentioned above) is converted into a pseudo LiDAR depth map by spherical projection, and the real LiDAR point cloud at the corresponding time is projected into a real depth map.

[0102] Geometric overlap is calculated using real LiDAR point clouds and their pose ground values, serving as a supervisory label for cross-modal triples;

[0103] By minimizing the Euclidean distance between cross-modal descriptor pairs at the same location and maximizing the distance between descriptor pairs at different locations, a global descriptor generation network with shared weights is trained.

[0104] Includes the following steps:

[0105] S1.3.1 converts the visual point cloud into a spherical depth map for subsequent input into the image-based descriptor network. Let any point in the point cloud be... It is represented in the LiDAR coordinate system.

[0106] To ensure that the visually generated pseudo-LiDAR depth map is strictly aligned with the real LiDAR depth map in terms of spatial coverage and effective area, only those visual points that have corresponding observations in the original LiDAR scan are retained:

[0107] First, project the real LiDAR point cloud of the current frame as a standard-sized range map. And construct a binary effective mask. ,in This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise.

[0108] Subsequently, for each point in the visual point cloud First, project it onto the spherical image coordinates in the LiDAR coordinate system. ,like If the value is positive, then keep the point; otherwise, discard it.

[0109] S1.3.2 Perform spherical projection on the filtered valid point set to map it to two-dimensional image coordinates. And record its radial distance.

[0110]

[0111] in and These are the width and height of the depth map, respectively. and The vertical field of view of the sensor, both upward and downward (in radians).

[0112] S1.3.3 fills the depth map using the "nearest point priority" principle, which means retaining the minimum radial distance for the same pixel position:

[0113]

[0114] in This is a subset of the visual point cloud after LiDAR masking. This represents the spherical projection function defined by formula (6). For pixels not covered by any valid points, let... The resulting pseudo LiDAR depth map Compared with real LiDAR depth map Having exactly the same size Furthermore, visual depth values ​​are only filled within the actual LiDAR observation area, eliminating modal differences introduced by inconsistent fields of view or invalid depth estimation, and providing geometrically aligned input for subsequent descriptor networks with shared weights.

[0115] The global descriptor generation network described in S1.3.4 includes a depth map encoder, a Transformer module, and a global descriptor generator connected in sequence. It takes a single-channel spherical depth map as input and outputs a global descriptor vector with fixed dimensions.

[0116] The depth encoder employs a multi-layer convolutional neural network that downsamples only in the vertical direction while maintaining the original resolution in the horizontal direction. Furthermore, no padding is used in any of the convolutional layers to ensure that the output feature map maintains the same yaw angle variability as the input depth map in the horizontal direction.

[0117] Let the input depth map be ,in For vertical resolution, For horizontal resolution, the encoder outputs a feature map. Satisfying the yaw and other variable relationships:

[0118]

[0119] in, Indicates cyclic shift along the horizontal direction Each pixel corresponds to a change in the vehicle's yaw angle.

[0120] The Transformer module includes a multi-head self-attention mechanism (MHSA), a feedforward network (FFN), and a layer normalization (LN) unit, used for global context modeling of the feature sequence output by the range graph encoder. This module receives signals of size [size missing]. The feature map is regarded as The Transformer module also maintains the same yaw angle variability, meaning that when the input feature map undergoes a horizontal cyclic shift, the output feature map shifts synchronously.

[0121] set up From The query, key, and value matrix obtained by linear transformation along the channel dimension is then the multi-head self-attention output as follows:

[0122]

[0123] And satisfy ,in This refers to the Transformer module.

[0124] The global descriptor generator uses the NetVLAD aggregation layer to process the output of the Transformer module. Local features (in They are aggregated into a fixed-dimensional global descriptor. :

[0125]

[0126] in The number of cluster centers. For the first A learnable clustering center We assign weights softly. Since this operation is permutation-invariant to the input order, and combined with the yaw-angle invariance of the preceding modules, the final generated global descriptor possesses yaw angle invariance. This means that descriptors generated for the same geographical location under different orientations have consistent heights, satisfying the following:

[0127]

[0128] S1.3.5 During the training phase, the real LiDAR depth map and the visually generated pseudo-LiDAR depth map share the same set of network parameters. The triplet loss function constrains the alignment of visual and LiDAR modalities in the descriptor space by minimizing the Euclidean distance between cross-modal descriptor pairs at the same location and maximizing the distance between descriptor pairs at different locations. The loss function is:

[0129]

[0130] in, Query descriptors generated for pseudo-LiDAR graphs. For the actual LiDAR descriptor at the same location, For negative sample descriptors at different locations, This is the preset interval threshold.

[0131] S1.4 Utilize the trained global descriptor generation network to perform forward inference on the pre-collected real LiDAR point cloud map, generate and store the corresponding LiDAR global descriptors, and construct a laser LiDAR global descriptor database.

[0132] Includes the following steps:

[0133] S1.4.1 Convert the real LiDAR point cloud data of the dataset selected in S1.1 into a depth map in accordance with the methods in S1.3.2 and S1.3.3.

[0134] S1.4.2 Input the real LiDAR depth map output from S1.4.1 into the global descriptor generation network trained in S1.3.5, perform forward inference, and output the global descriptor for the corresponding frame. And set the global descriptor It is stored in association with its corresponding global location and timestamp for easy indexing later.

[0135] S1.4.3 Repeat steps S1.4.1 and S1.4.2 to traverse all frames of the entire LiDAR map and finally construct a global descriptor library for LiDAR, which will be used for online cross-modal retrieval in subsequent stages.

[0136] S2 Online Location:

[0137] At time k, S2.1 acquires images of the surrounding environment using the 360° camera system mounted on the unmanned system;

[0138] That is, at time The unmanned system uses a 360° surround-view camera system to acquire multi-view RGB images of the surrounding environment. ,in This refers to the number of cameras.

[0139] S2.2 Input the camera images into the pre-trained monocular depth estimation network to generate a dense depth map, and backproject it into a point cloud according to the calibration parameters, aligning it spatially with the real LiDAR point cloud; including the following steps:

[0140] S2.2.1 Each image Input a pre-trained monocular depth estimation network to generate the corresponding dense relative depth map. Since no real LiDAR point clouds are available for scale calibration during the online phase, the global scale factor from the offline phase is used. (See formula (1)) Scale normalize the depth map to obtain the absolute depth map:

[0141]

[0142] S2.2.2 Based on the calibrated camera intrinsic parameters Extrinsic parameters from camera to LiDAR coordinate system The effective pixels are back-projected onto the LiDAR coordinate system to generate a visual point cloud. Its calculation method is completely consistent with step (2) of the offline stage, that is:

[0143]

[0144] in For the first The effective depth region in a single camera image.

[0145] S2.2.3 Uses the same sensor parameters as in the offline phase. Perform spherical projection (Equation (6)).

[0146] S2.3 The point cloud generated by vision (visual point cloud) is converted into a pseudo LiDAR depth map by spherical projection. Based on step S1.3, the pseudo LiDAR depth map is input into the global descriptor generation network to generate a visual global descriptor.

[0147] S2.3.1 The visual point cloud is converted into a pseudo LiDAR depth map by spherical projection, including the following steps:

[0148] S2.3.1.1 Since there are no real LiDAR point clouds during the online phase, the typical effective LiDAR region templates obtained during offline training are reused. Only The position retains the visual depth value, while the rest are set to 0.

[0149] S2.3.1.2 The depth map is filled using the "nearest point priority" principle (Formula (7)) to generate a pseudo LiDAR map that is aligned with the geometry of the real LiDAR depth map. .

[0150] S2.3.2 Input the pseudo LiDAR depth map into the global descriptor generation network to generate a visual global descriptor, including the following steps:

[0151] A global descriptor database for laser LiDAR constructed using S1.4 For reference, perform cross-modal nearest neighbor retrieval, where For the first Frame LiDAR descriptor, Its global pose:

[0152]

[0153] Return the index of the best matching frame. and output its corresponding pose. As an unmanned system, it is always The global positioning results.

[0154] S2.4 Based on the global descriptor database of LiDAR constructed in step S1.4, cross-modal descriptor retrieval and matching are performed. The optimal matching position of the unmanned system in the map is obtained through nearest neighbor search, thereby achieving real-time high-precision positioning without LiDAR sensors.

[0155] Example 2, based on Example 1, further verifies the effectiveness and robustness of the method of the present invention through multiple sets of offline training and online localization experiments. Specifically, the model is trained and evaluated using the publicly available multimodal autonomous driving dataset UrbanLoco, and the training process and localization performance are quantitatively analyzed. The results are as follows:

[0156] Figure 2 The curves showing the change of the Lazy Triplet Loss with the number of training epochs during offline training are presented. It can be seen that the loss value decreases rapidly in the first 50 epochs and then tends to stabilize, indicating that the designed cross-modal descriptor network can converge effectively, and that the pseudo-LiDAR and real LiDAR descriptors gradually align in the embedding space.

[0157] Figure 3 The curve shows the ReduceLROnPlateau learning rate scheduling strategy used during training. The learning rate gradually decreases from the initial value, which helps the model fine-tune parameters in the later stages, avoids oscillations, and improves generalization ability.

[0158] Figure 4 The figure shows the trend of localization accuracy (measured by Top-1 recall) on the validation set as a function of training epochs. As training progresses, the validation accuracy continues to rise and then saturates after about 150 epochs, eventually reaching 95.3%, indicating that the model has good generalization performance.

[0159] Figure 5 This is the precision-recall (PR) curve during the offline training phase, used to evaluate the model's discriminative ability in cross-modal matching tasks. The curve shows the trend of precision versus recall at different confidence thresholds. The highest F1 score marked in the figure is 0.8837, corresponding to a recall of 0.811 and a precision of 0.971, indicating that the model maintains both high recall and extremely high precision, effectively balancing false matches and missed matches, and validating the effectiveness of the pseudo-LiDAR descriptor alignment in the unified embedding space.

[0160] Figure 6 The Top-N recall curve for the online localization phase is used to evaluate the localization performance of this method in actual deployment. The curve shows the probability that the correctly matched location is included when only the top N search results are retrieved. As can be seen from the figure, the recall rate rises rapidly with increasing N, then levels off after the Top-10, eventually stabilizing above 95%. This indicates that the visual global descriptor generated by this method can effectively match the LiDAR map library, achieving highly robust real-time localization without the need for vehicle-mounted LiDAR sensors.

Claims

1. A localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors, characterized in that: Includes the following steps: S1 offline training and graph construction; S1.1 Acquire synchronized multimodal data, including panoramic camera images, LiDAR point clouds, calibration parameters between the camera and LiDAR, and pose ground truth. S1.2 Input the camera image into the pre-trained monocular depth estimation network to generate a dense depth map. Using the pose ground truth and the real LiDAR point cloud, estimate the global scale factor s through scale invariant loss, and backproject the depth map into a point cloud, which is spatially aligned with the real LiDAR point cloud. S1.3 The point cloud generated by vision is converted into a pseudo LiDAR depth map by spherical projection, and the real LiDAR point cloud at the corresponding time is projected into a real depth map. Geometric overlap is calculated using real LiDAR point clouds and their pose ground values, serving as a supervisory label for cross-modal triples; A weight-shared global descriptor generation network is trained by minimizing the distance between co-located pseudo-LiDAR and real LiDAR descriptors and maximizing the distance between disjoint pairs. S1.4 Utilize the trained global descriptor generation network to perform forward inference on the pre-collected real LiDAR point cloud map, generate and store the corresponding LiDAR global descriptors, and construct a laser LiDAR global descriptor database. S2 Online Location: At time k, S2.1 acquires images of the surrounding environment using the 360° camera system mounted on the unmanned system; S2.2 Input the camera image into the pre-trained monocular depth estimation network to generate a dense depth map. The global scale factor s fixed in the offline stage is directly reused to backproject the depth map into a point cloud, which is spatially aligned with the real LiDAR point cloud. S2.3 The point cloud generated by vision is converted into a pseudo LiDAR depth map by spherical projection. Based on step S1.3, the pseudo LiDAR depth map is input into the global descriptor generation network to generate a visual global descriptor. S2.4 Based on the global descriptor database of LiDAR constructed in step S1.4, cross-modal descriptor retrieval and matching are performed. The optimal matching position of the unmanned system in the map is obtained through nearest neighbor search, thereby achieving real-time high-precision positioning without LiDAR sensors.

2. The localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors according to claim 1, characterized in that: S1.2 includes the following steps: S1.2.1 Assume the surround-view vision system carried by the unmanned system includes There are several cameras, each with a limited field of view, which together cover a 360° horizontal field of view; for the first... One camera ( The RGB image it acquires is denoted as ; each image Input a pre-trained monocular depth estimation network to generate the corresponding dense depth map. ; S1.2.2 Utilizing the true pose values ​​and LiDAR point clouds provided in the offline phase dataset, the global scale factor is estimated by minimizing the reprojection error. The calibrated absolute depth map is obtained: For any pixel in the depth map Taking the top left corner of the image as the origin, For column indexes, For row indexes, if their calibration depth If a pixel is located in an unmasked area, it is considered a valid pixel and participates in subsequent backprojection. Using the first Intrinsic parameter matrix of each camera : in Focal length (unit: pixels). Using the principal point coordinates (in pixels, usually close to the image center), backproject the effective pixels to Euclidean 3D points in the camera coordinate system: S1.2.3 Using the extrinsic parameters from the camera to the LiDAR coordinate system provided by the public dataset Transform this point to the LiDAR coordinate system; because For 3D Euclidean vectors, only the rotation and translation components are needed: Perform the above operation on all cameras i and their effective pixels to obtain Group of local visual point clouds They can be directly stitched together to form a complete visual point cloud: Compared with the real LiDAR point cloud at the same moment Alignment is achieved in spatial location, physical scale, and coordinate system, and can be directly used for subsequent cross-modal descriptor training.

3. The localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors according to claim 1, characterized in that: S1.3 includes the following steps: S1.3.1 Convert the visual point cloud into a spherical depth map for subsequent input into the image-based descriptor network; let any point in the visual point cloud be... , which is represented in the LiDAR coordinate system; To ensure that the visually generated pseudo-LiDAR depth map is strictly aligned with the real LiDAR depth map in terms of spatial coverage and effective area, only those visual points that have corresponding observations in the original LiDAR scan are retained: First, project the real LiDAR point cloud of the current frame as a standard-sized range map. And construct a binary effective mask. ,in This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Subsequently, for each point in the visual point cloud First, project it onto the spherical image coordinates in the LiDAR coordinate system. ,like If the value is correct, keep the point; otherwise, discard it. S1.3.2 Perform spherical projection on the filtered valid point set to map it to two-dimensional image coordinates. And record its radial distance. in and These are the width and height of the depth map, respectively. and The vertical field of view of the sensor, pointing upwards and downwards, in radians; S1.3.3 fills the depth map using the "nearest point priority" principle, which means retaining the minimum radial distance for the same pixel position: in This is a subset of the visual point cloud after LiDAR masking. Let represent the spherical projection function defined by formula (6); for pixels not covered by any valid points, let The resulting pseudo LiDAR depth map Compared with real LiDAR depth map Having exactly the same size Furthermore, visual depth values ​​are only filled within the actual LiDAR observation area, eliminating modal differences introduced by inconsistent field of view or invalid depth estimation; The global descriptor generation network described in S1.3.4 includes a depth map encoder, a Transformer module, and a global descriptor generator connected in sequence. It takes a single-channel spherical depth map as input and outputs a global descriptor vector with fixed dimensions. The depth encoder employs a multi-layer convolutional neural network that downsamples only in the vertical direction while maintaining the original resolution in the horizontal direction. Furthermore, no padding is used in any of the convolutional layers to ensure that the output feature map maintains the same yaw angle variability as the input depth map in the horizontal direction. Let the input depth map be ,in For vertical resolution, For horizontal resolution, the encoder outputs a feature map. Satisfying the yaw and other variable relationships: in, Indicates cyclic shift along the horizontal direction Each pixel corresponds to the change in the vehicle's yaw angle; The Transformer module includes a multi-head self-attention mechanism (MHSA), a feedforward network (FFN), and a layer normalization (LN) unit, used for global context modeling of the feature sequence output by the range graph encoder. This module receives signals of size [size missing]. The feature map is regarded as The Transformer module also maintains the same yaw angle variability, that is, when the input feature map undergoes horizontal cyclic shift, the output feature map shifts synchronously. set up From The query, key, and value matrix obtained by linear transformation along the channel dimension is then the multi-head self-attention output as follows: And satisfy ,in This refers to the Transformer module; The global descriptor generator uses the NetVLAD aggregation layer to process the output of the Transformer module. Local features (in They are aggregated into a fixed-dimensional global descriptor. : in The number of cluster centers. For the first A learnable clustering center We assign weights softly; since this operation is permutation-invariant to the input order, combined with the yaw-invariant property of the preceding modules, the final generated global descriptor is yaw-angle invariant, meaning that descriptors generated at the same geographical location under different orientations have consistent heights, satisfying: S1.3.5 During the training phase, the real LiDAR depth map and the visually generated pseudo-LiDAR depth map share the same set of network parameters. The triplet loss function constrains the alignment of visual and LiDAR modalities in the descriptor space by minimizing the Euclidean distance between cross-modal descriptor pairs at the same location and maximizing the distance between descriptor pairs at different locations. The loss function is: in, Query descriptors generated for pseudo-LiDAR graphs. For the actual LiDAR descriptor at the same location, For negative sample descriptors at different locations, This is the preset interval threshold.

4. The localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors according to claim 1, characterized in that: S1.4 includes the following steps: S1.4.1 Convert the real LiDAR point cloud data of the dataset selected in S1.1 into a depth map according to the methods in S1.3.2 and S1.3.3; S1.4.2 Input the real LiDAR depth map output from S1.4.1 into the global descriptor generation network trained in S1.3.5, perform forward inference, and output the global descriptor for the corresponding frame. And put the global descriptor It is stored in association with its corresponding global location and timestamp for easy subsequent indexing; S1.4.3 Repeat steps S1.4.1 and S1.4.2 to traverse all frames of the entire LiDAR map and finally construct a global descriptor library for LiDAR, which will be used for online cross-modal retrieval in subsequent stages.

5. The localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors according to claim 1, characterized in that: S2.2 includes the following steps: S2.2.1 Each image Input a pre-trained monocular depth estimation network to generate the corresponding dense relative depth map. Since no real LiDAR point cloud was available for scale calibration during the online phase, the global scale factor from the offline phase was used. Scale normalization of the depth map yields the absolute depth map: S2.2.2 Based on the calibrated camera intrinsic parameters Extrinsic parameters from camera to LiDAR coordinate system The effective pixels are back-projected onto the LiDAR coordinate system to generate a visual point cloud. Its calculation method is completely consistent with step (2) of the offline stage, that is: in For the first Effective depth region in each camera image; S2.2.3 Uses the same sensor parameters as in the offline phase. Perform spherical projection (Equation (6)).

6. The localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors according to claim 1, characterized in that: The point cloud generated visually in S2.3 is converted into a pseudo LiDAR depth map via spherical projection, including the following steps: S2.3.1.1 Since there are no real LiDAR point clouds during the online phase, the typical effective LiDAR region templates obtained during offline training are reused. Only The position retains the visual depth value, while the rest are set to 0; S2.3.1.2 The depth map is filled using the "nearest point priority" principle (Formula (7)) to generate a pseudo LiDAR map that is geometrically aligned with the real LiDAR depth map. .

7. The localization method for cross-modal matching based on visual depth estimation to generate pseudo-LiDAR descriptors according to claim 1, characterized in that: In step S2.3, the pseudo LiDAR depth map is input into the global descriptor generation network to generate a visual global descriptor, including the following steps: A global descriptor database for laser LiDAR constructed using S1.4 For reference, perform cross-modal nearest neighbor retrieval, where For the first Frame LiDAR descriptor, Its global pose: Return the index of the best matching frame. and output its corresponding pose. As an unmanned system, it is always The global positioning results.

Images