Cross-modal method for visual identification in large-scale point cloud map

Abstract

The invention relates to a cross-modal method for visual identification in a large-scale point cloud map, and the method comprises the steps: building a data set based on RGB image data and point cloud data collected in a global map; performing cross-modal conversion on the data set; training a preset network model by using the converted data set to obtain a cross-modal positioning model; wherein the preset network model comprises a multi-scale feature encoder, a cascade cross attention module and a projection converter; and inputting the RGB image data which are not included in the data set into the cross-modal positioning model, and obtaining the position of the sensor in the global map. According to the invention, the precision and stability of cross-modal position recognition from images to point clouds in unknown indoor and outdoor environments can be remarkably improved, and the deployment feasibility of the cross-modal position recognition device in practical application is improved due to the lightweight design.

Description

Technical Field

[0001] The present invention relates to the field of indoor positioning and navigation technology, and in particular to a cross-modal method for performing visual recognition in large-scale point cloud maps. Background Art

[0002] Visual place recognition plays a crucial role in mobile robotics, particularly in applications such as autonomous driving and simultaneous localization and mapping (SLAM). It relies on sensor data, such as images and point clouds, to achieve global localization and loop closure detection within pre-built maps and databases. Visual place recognition is highly competitive due to the small size and low cost of cameras. However, cameras are sensitive to lighting variations and field of view. LiDAR, on the other hand, is unaffected by illumination and can accurately detect distant objects. Furthermore, LiDAR prices are steadily decreasing due to declining costs. Leveraging the complementary strengths of different sensors is crucial in practical mobile robotics applications. Given the differences between the data modalities queried by mobile robots and those stored in databases, cross-modal place recognition addresses these challenges, significantly improving the accuracy and robustness of global place recognition. For example, a self-driving car can achieve accurate visual place recognition. This is achieved by using query images captured by inexpensive cameras and performing recognition on a pre-built, large-scale 3D point cloud map.

[0003] Given the complementary nature of image and point cloud data, various multimodal fusion methods have been developed to combine two-dimensional and three-dimensional data, significantly improving global localization accuracy. However, fusion remains a challenging task due to the inherent differences between the two modalities. Furthermore, existing multimodal methods have not yet fully addressed the problem of global location recognition from sensors of one modality to maps of another modality, the so-called cross-modal place recognition challenge. To overcome the data representation differences between image and point cloud information, some methods convert sensor data into intermediate representations such as bird's-eye views or depth images. Other methods aim to extract structural information from visual data for alignment with point clouds or directly use deep neural networks (DNNs) for data matching. However, these methods are often resource-intensive and involve a large number of model parameters. Specifically, processing three-dimensional point clouds often requires processing massive and complex data, resulting in high hardware costs. Furthermore, even with sufficient training data, models may struggle to maintain good performance when faced with unseen data.

[0004] Although significant progress has been made in single-modality place recognition, cross-modal location recognition from images to point clouds still suffers from low precision and accuracy. Therefore, there is still great potential for exploration in achieving location recognition from images to large-scale point cloud maps.

[0005] The training framework of the existing ModaLink framework. The point cloud is converted to a depth image through projection. The query image and depth image are then cropped to have the same field of view overlap. Based on the depth information augmentation module, the sparse depth image is upsampled to a dense depth image. A shared weight encoder then generates a global position descriptor. Finally, the existing ModaLink framework uses a triplet loss for supervised learning. However, this existing technology has the following shortcomings:

[0006] 1. Existing technologies use triplet loss for training, which results in a small number of negative samples. This makes it difficult for the model to learn discriminative features, resulting in insufficient discrimination of the embedded vectors in the feature space.

[0007] 2. The existing technology does not fully utilize the information between the interactive modalities after generating the descriptors of the two modalities and before supervised learning.

[0008] 3. The existing technology has low recall accuracy under the 1m high-precision distance limitation in unknown environments.

[0009] 4. The existing technology does not perform well in model transfer in out-of-domain scenarios with zero-sample training. Summary of the Invention

[0010] The purpose of the present invention is to provide a cross-modal method for visual recognition in large-scale point cloud maps to solve the problems existing in the above-mentioned prior art and increase the recognition accuracy and positioning precision of mobile robots equipped with cameras in point cloud map databases.

[0011] To achieve the above object, the present invention provides the following solutions:

[0012] A cross-modal approach for visual recognition in large-scale point cloud maps, including:

[0013] Construct a dataset based on the RGB image data and point cloud data collected from the global map;

[0014] Performing cross-modal transformation on the dataset;

[0015] Using the converted dataset, a preset network model is trained to obtain a cross-modal localization model; wherein the preset network model includes: a multi-scale feature encoder, a cascaded cross-attention module, and a projection converter;

[0016] The RGB image data not included in the dataset is input into the cross-modal localization model to obtain the position of the sensor in the global map.

[0017] Optionally, based on RGB image data and point cloud data collected from the global map, constructing a dataset includes:

[0018] Based on the RGB image data and point cloud data collected in the global map, the calibration data is calculated, a database of paired image data and point cloud data of consecutive frames in the global map is constructed, and the position coordinates of the paired image data and point cloud data in the global map are recorded in real time.

[0019] Optionally, performing cross-modal conversion on the dataset includes:

[0020] The RGB image data and point cloud data are converted into a depth map and a range map with distance information, respectively.

[0021] Optionally, using the converted data set to train a preset network model includes:

[0022] Inputting the converted depth map and range map into a preset network model to obtain distinguishable modal representation information;

[0023] Based on the mini-batch contrast loss function, the parameters of the entire network architecture are trained.

[0024] Optionally, inputting the converted depth map and range map into a preset network model includes:

[0025] Inputting the converted depth map and range map into a multi-scale feature encoder for feature extraction to obtain representation information of the image and point cloud in a unified embedding space;

[0026] The representation information of the image and point cloud in the unified embedding space is input into the cascade cross attention module for dynamic interactive fusion and pairing to obtain the vector features of the aligned image point cloud pair;

[0027] The vector features of the aligned image point cloud pairs are input into the projection transformer for projection transformation to obtain discriminative modal representation information.

[0028] Optionally, the cascaded crisscross attention module includes two identical crisscross attention sub-modules; each crisscross attention sub-module includes: a multi-head attention layer and a residual connection;

[0029] The projective converter includes two projection heads for dimensionality increase and dimensionality reduction, and each projection head includes a fully connected layer and a nonlinear activation layer.

[0030] Optionally, based on the mini-batch contrastive loss function, the parameters for training the entire network architecture include:

[0031] Set the batch size and temperature parameters, and update the parameters of the network module through backpropagation based on minimizing the loss function.

[0032] Optionally, inputting RGB image data not included in the data set into the cross-modal localization model to obtain the position of the sensor in the global map includes:

[0033] The RGB image data not included in the dataset is input into the cross-modal localization model, the most similar point cloud is retrieved in the dataset based on cosine similarity, and the position of the sensor in the global map is determined.

[0034] The beneficial effects of the present invention are:

[0035] The present invention provides a flow chart of a cross-modal method for visual recognition in large-scale point cloud maps. A dual-stream network architecture is designed to synchronously encode image-point cloud pairs and interactively fuse feature information from different modalities. After embedding spatial projection, network training is performed based on a small-batch contrast loss function. The trained model can achieve high-precision position recognition from query images to large-scale point cloud maps in unknown environments, contributing an important technical tool to the fields of mobile robotics and autonomous driving. Through this cross-modal method for visual recognition in large-scale point cloud maps, compared with related technologies, the present invention can significantly improve the accuracy and stability of image-to-point cloud cross-modal position recognition in unknown indoor and outdoor environments. At the same time, its lightweight design increases the feasibility of deploying the cross-modal position recognition device in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 It is the training framework of ModaLink framework;

[0038] Figure 2 1 is a flow chart of a cross-modal method for performing visual recognition in a large-scale point cloud map according to an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of hardware connection of an embodiment of the present invention;

[0040] Figure 4 A flowchart for database construction according to an embodiment of the present invention;

[0041] Figure 5 Schematic diagram of a cascaded cross attention module according to an embodiment of the present invention;

[0042] Figure 6Schematic diagram of a projection converter according to an embodiment of the present invention;

[0043] Figure 7 This is a flow chart of cross-modal positioning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 2 As shown, this embodiment proposes a cross-modal method for visual recognition in a large-scale point cloud map, including:

[0047] Construct a dataset based on the RGB image data and point cloud data collected from the global map;

[0048] Performing cross-modal transformation on the dataset;

[0049] Using the converted dataset, a preset network model is trained to obtain a cross-modal localization model; wherein the preset network model includes: a multi-scale feature encoder, a cascaded cross-attention module, and a projection converter;

[0050] The RGB image data not included in the dataset is input into the cross-modal localization model to obtain the position of the sensor in the global map.

[0051] The network architecture of this embodiment is as follows: RGB images are converted into depth images through monocular depth estimation (MDE). Point clouds are converted into range images through spherical projection (SP). Subsequently, they are processed through a multi-scale feature encoder based on the CrossViT backbone, followed by a cascaded cross attention module (CCAM) and a projection transformer (PT). Finally, the network is trained using a mini-batch contrast loss. Hardware connections are as follows Figure 3 shown.

[0052] The purpose of this embodiment is to provide a cross-modal method for visual recognition in large-scale point cloud maps, thereby increasing the recognition accuracy and positioning precision of mobile robots equipped with cameras in point cloud map databases. The specific technical solution is as follows:

[0053] Utilize mobile robots or vehicles equipped with cameras and lidar to capture closed-loop sensor data indoors or outdoors, calculate calibration data, build a database of paired images and point clouds for consecutive frames in a large-scale map, and record their position coordinates on the global map in real time. Obtain depth maps and range maps with distance information, converted from the collected RGB images and point clouds, respectively.

[0054] Depth maps and range maps are batch-inputted into a multi-scale encoder with the same architecture for feature extraction, obtaining their representations in a unified embedding space. A cascaded cross-attention module is designed to dynamically and interactively fuse the representations of paired images and point clouds in the unified embedding space, obtaining vector features of aligned image-point cloud pairs. A projective transformer is then designed to obtain more discriminative modal representations.

[0055] Based on mini-batch contrast learning, the representation information of paired image point clouds in a unified embedding space is brought closer and the unpaired representation information is excluded, thereby training and updating the parameters of the entire model.

[0056] Image data not included in the training set is input into the trained model, the most similar point cloud is retrieved from a large-scale point cloud database based on cosine similarity, and the position of the sensor in the global map is determined.

[0057] Furthermore, based on the RGB image data and point cloud data collected from the global map, the dataset is constructed including:

[0058] Based on the RGB image data and point cloud data collected in the global map, the calibration data is calculated, a database of paired image data and point cloud data of consecutive frames in the global map is constructed, and the position coordinates of the paired image data and point cloud data in the global map are recorded in real time.

[0059] Furthermore, performing cross-modal conversion on the dataset includes:

[0060] The RGB image data and point cloud data are converted into a depth map and a range map with distance information, respectively.

[0061] Specifically, in this embodiment, Figure 4 As shown, a database of indoor or outdoor calibrated continuous frame images and point clouds is constructed, and their position coordinates on the global map are recorded in real time. The depth map and range map with distance information converted from the collected RGB images and point clouds are applied, including:

[0062] Utilize mobile robots or vehicles equipped with cameras and lidar to capture closed-loop sensor data indoors or outdoors, calculate calibration data, build a database of paired images and point clouds for consecutive frames in a large-scale map, and record their position coordinates on the global map in real time. Obtain depth maps and range maps with distance information, converted from the collected RGB images and point clouds, respectively.

[0063] Furthermore, using the converted data set, training the preset network model includes:

[0064] Inputting the converted depth map and range map into a preset network model to obtain distinguishable modal representation information;

[0065] Based on the mini-batch contrast loss function, the parameters of the entire network architecture are trained.

[0066] Furthermore, inputting the converted depth map and range map into a preset network model includes:

[0067] Inputting the converted depth map and range map into a multi-scale feature encoder for feature extraction to obtain representation information of the image and point cloud in a unified embedding space;

[0068] The representation information of the image and point cloud in the unified embedding space is input into the cascade cross attention module for dynamic interactive fusion and pairing to obtain the vector features of the aligned image point cloud pair;

[0069] The vector features of the aligned image point cloud pairs are input into the projection transformer for projection transformation to obtain discriminative modal representation information.

[0070] Specifically, in this embodiment, a multi-scale feature encoder with the same architecture is used to obtain representation information of image point cloud pairs. Based on the representation information of a single modality, a cascaded cross-attention module is introduced to interactively fuse information from different modalities. A projective transformer is introduced to obtain more discriminative modal representation information, including:

[0071] The pre-trained crossvit_small_240 was used as the encoder backbone, with the last classifier layer removed. The designed cascaded cross-attention module consists of two identical cross-attention modules, primarily comprising a multi-head attention layer and residual connections. The projection transformer consists of two projection heads, one for dimensionality increase and one for dimensionality reduction, primarily comprising fully connected layers and nonlinear activation layers.

[0072] The detailed architecture of the cascaded cross attention module is as follows Figure 5 The formula is as follows:

[0073]

[0074] where QI ,K P ,V P They represent the image branch, the point cloud branch, the embedded vector of the point cloud branch output by the encoder, MHA (Multi-Head Attention) is the multi-head attention layer, MHA (Q I ,K P ,V P ) output indicates that the image embedding representation carries the paired point cloud information, Q I +MHA(Q I ,K P ,V P ) The residual form makes the image representation not lose the original information, and the output of the first cross attention module K as the next cross attention module I ,V I , Q P The embedding vector output by the encoder for the point cloud branch. The rest of the operations are the same as the first module. LN (Layer Normalization) is a normalization layer that can prevent gradient vanishing or exploding problems, making the training process smoother. FFN (Feedforward Neural Network) is a feedforward neural network that performs a series of nonlinear transformations to produce richer and more useful feature representations.

[0075] The detailed architecture of the projection converter module is as follows Figure 6 . Provisions f cross is the embedding vector output by the cascaded cross-attention module. The formula is as follows:

[0076] f fc =RELU(W fc1 f cross )

[0077] f pt =LN(W fc3 (GELU(W fc2 f fc ))+W fc2 f fc )

[0078] Where W fc1 is the weight matrix of the first layer linear transformation, which is used to transform the input f cross Mapped to another feature space. RELU is used to introduce nonlinear factors and enhance the model's expressiveness. fc2 and W fc3 Represent the weight matrices of the second and third layer linear transformations respectively. fc2 Used to W fc2 ffc Further mapping, and W fc3 It is used to re-map the result after GELU activation.

[0079] RELU (Rectified Linear Unit) is a rectified linear unit, and GELU (Gaussian Error Linear Unit) is a Gaussian error linear unit.

[0080] Furthermore, based on the mini-batch contrast loss function, the parameters for training the entire network architecture include:

[0081] Set the batch size and temperature parameters, and update the parameters of the network module through backpropagation based on minimizing the loss function.

[0082] Specifically, in this embodiment, the parameters of the entire network architecture are trained based on the mini-batch contrast loss function using the interactively fused two modal representation information, including:

[0083] Set the batch size and temperature parameters similar to the CLIP model, minimize the loss function, and update the parameters of the network module based on backpropagation.

[0084] Among them, the two modal representation information refers to the information representation results of the image branch and the point cloud branch after the two branches pass through the cascaded cross-attention module and the projection converter.

[0085] Specifically, given a batch of depth images and the range images that match them consistently N represents the batch size, so this batch contains N positive matching pairs and N 2 -N negative matching pairs. For each matching pair, the contrast loss can be calculated as follows:

[0086]

[0087] The temperature parameter τ is mainly used to adjust the distribution of similarity scores in the loss function. The target loss function of a batch class is defined as follows:

[0088]

[0089] Furthermore, inputting RGB image data not included in the data set into the cross-modal localization model to obtain the position of the sensor in the global map includes:

[0090] The RGB image data not included in the dataset is input into the cross-modal localization model, the most similar point cloud is retrieved in the dataset based on cosine similarity, and the position of the sensor in the global map is determined.

[0091] Specifically, in this embodiment, using a dataset that has never been seen by the model, determining the most similar point cloud in the large-scale point cloud map for the image data collected by the camera and the global position coordinates of the sensor includes:

[0092] like Figure 7 As shown, the input image is generated by the network model and the image embedding vector is calculated with the embedding vector set of the pre-built point cloud database for cosine similarity calculation. The position of the camera sensor in the global map is determined by the calculated best point cloud candidate.

[0093] This example uses mini-batch contrastive learning to train the network, significantly increasing the number of negative samples. After generating descriptors and before supervised learning, a cascaded cross-attention model is introduced to fully leverage information from the interaction between modalities. This enables high-precision and high-accuracy cross-modal localization in both unknown environments and out-of-domain scenarios with zero-shot training.

[0094] This embodiment provides a flow chart of a cross-modal method for visual recognition in large-scale point cloud maps. It designs a dual-stream network architecture, synchronously encodes image-point cloud pairs, and interactively fuses feature information from different modalities. After embedding spatial projection, the network is trained based on a small batch contrast loss function. The trained model can achieve high-precision position recognition from query images to large-scale point cloud maps in unknown environments, contributing an important technical tool to the fields of mobile robots and autonomous driving. Through this cross-modal method for visual recognition in large-scale point cloud maps, compared with related technologies, this embodiment can significantly improve the accuracy and stability of image-to-point cloud cross-modal position recognition in unknown indoor and outdoor environments. At the same time, due to its lightweight design, it increases the feasibility of deploying this cross-modal position recognition device in practical applications.

[0095] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A cross-modal method for visual recognition in large-scale point cloud maps, characterized by: include: Construct a dataset based on the RGB image data and point cloud data collected from the global map; Performing cross-modal transformation on the dataset; Using the converted dataset, a preset network model is trained to obtain a cross-modal localization model; wherein the preset network model includes: a multi-scale feature encoder, a cascaded cross-attention module, and a projection converter; The RGB image data not included in the dataset is input into the cross-modal localization model to obtain the position of the sensor in the global map.

2. The cross-modal method for visual recognition in large-scale point cloud maps according to claim 1, characterized in that: Based on the RGB image data and point cloud data collected from the global map, the dataset is constructed including: Based on the RGB image data and point cloud data collected in the global map, the calibration data is calculated, a database of paired image data and point cloud data of consecutive frames in the global map is constructed, and the position coordinates of the paired image data and point cloud data in the global map are recorded in real time.

3. The cross-modal method for visual recognition in large-scale point cloud maps according to claim 1, characterized in that: Performing cross-modal conversion on the dataset includes: The RGB image data and point cloud data are converted into a depth map and a range map with distance information, respectively.

4. The cross-modal method for visual recognition in large-scale point cloud maps according to claim 3, characterized in that: Using the converted dataset, training the preset network model includes: Inputting the converted depth map and range map into a preset network model to obtain distinguishable modal representation information; Based on the mini-batch contrast loss function, the parameters of the entire network architecture are trained.

5. The cross-modal method for visual recognition in large-scale point cloud maps according to claim 4, characterized in that: Inputting the converted depth map and range map into a preset network model includes: Inputting the converted depth map and range map into a multi-scale feature encoder for feature extraction to obtain representation information of the image and point cloud in a unified embedding space; The representation information of the image and point cloud in the unified embedding space is input into the cascade cross attention module for dynamic interactive fusion and pairing to obtain the vector features of the aligned image point cloud pair; The vector features of the aligned image point cloud pairs are input into the projection transformer for projection transformation to obtain discriminative modal representation information.

6. The cross-modal method for visual recognition in large-scale point cloud maps according to claim 5, characterized in that: The cascaded crisscross attention module includes two identical crisscross attention sub-modules; Each cross-attention submodule includes: multi-head attention layer and residual connection; The projective converter includes two projection heads for dimensionality increase and dimensionality reduction, and each projection head includes a fully connected layer and a nonlinear activation layer.

7. The cross-modal method for visual recognition in large-scale point cloud maps according to claim 4, characterized in that: Based on the mini-batch contrast loss function, the parameters for training the entire network architecture include: Set the batch size and temperature parameters, and update the parameters of the network module through backpropagation based on minimizing the loss function.

8. The cross-modal method for visual recognition in large-scale point cloud maps according to claim 1, characterized in that: Inputting the RGB image data not included in the dataset into the cross-modal localization model to obtain the position of the sensor in the global map includes: The RGB image data not included in the dataset is input into the cross-modal localization model, the most similar point cloud is retrieved in the dataset based on cosine similarity, and the position of the sensor in the global map is determined.

Images