A cross-modal method for visual recognition in large-scale point cloud maps

Abstract

The present application relates to a kind of cross-modal methods for visual identification in large-scale point cloud map, comprising: based on the RGB image data and point cloud data collected in global map, construct data set;The data set is converted cross-modal;Using converted data set, the preset network model is trained, and cross-modal positioning model is obtained;Wherein, the preset network model includes: multi-scale feature encoder, cascaded cross attention module and projection converter;RGB image data not included in the data set is input into the cross-modal positioning model, and the position of sensor in global map is obtained.The present application can significantly improve the image-to-point cloud cross-modal location recognition accuracy and stability in unknown indoor and outdoor environment, and simultaneously due to its lightweight design, increase the feasibility of the cross-modal location recognition device in practical application deployment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of indoor positioning and navigation, and particularly relates to a cross-modal method for visual recognition in a large-scale point cloud map. BACKGROUND

[0002] Visual place recognition plays a crucial role in mobile robotics, especially in applications such as autonomous driving and simultaneous localization and mapping (SLAM). It relies on sensor data such as images and point clouds to achieve global localization and loop closure detection in pre-built maps and databases. Visual place recognition is highly competitive due to the small size and low cost of cameras. However, cameras are sensitive to changes in lighting and field of view. On the other hand, lidar is not affected by light and can accurately detect objects at a distance. In addition, due to the reduction in cost, the price of lidar is gradually declining. In actual mobile robot applications, it is crucial to take advantage of the complementary advantages of different sensors. Given the differences between the data modalities queried by mobile robots and the data modalities stored in the database, cross-modal place recognition addresses the challenges posed by these modality differences, thereby significantly improving the accuracy and robustness of global place recognition. For example, an autonomous vehicle can achieve accurate visual place recognition. Using query images taken by a low-cost camera, recognition is achieved on a pre-built large-scale 3D point cloud map.

[0003] Given the complementarity of image data and point cloud data, various multi-modal fusion methods have been developed to combine two-dimensional and three-dimensional data, which significantly improves the accuracy of global localization. However, due to the inherent differences between the two modalities, fusion remains a challenging task. In addition, existing multi-modal methods have not fully addressed the problem of global place recognition from one modality of sensor to another modality of map, i.e. the so-called cross-modal place recognition challenge. In order to overcome the differences in data representation between images and point cloud information, some methods convert sensor data into intermediate representations such as bird's eye view or depth images. Other methods aim to extract structural information from visual data in order to align with point clouds, or directly use deep neural networks (DNNs) for data matching. However, these methods are usually resource-intensive and involve a large number of model parameters. Specifically, processing three-dimensional point clouds often requires processing vast and complex data, which results in high hardware costs. In addition, even with sufficient training data, the model may struggle to maintain good performance when faced with unseen data.

[0004] Currently, although significant progress has been made in single-modal place recognition, cross-modal place recognition from images to point clouds still has low accuracy and low accuracy. Therefore, there is still great potential for exploration in achieving image-based location recognition on large-scale point cloud maps.

[0005] The training framework of the prior art ModaLink framework. The point cloud is converted into a depth image by projection. Then, the query image and the depth image are cropped to have the same field of view overlap. Based on the depth information augmentation module, the sparse depth image is upsampled into a dense depth image. Next, the global location descriptor is generated by a shared weight encoder. Finally, the prior art ModaLink adopts a triplet loss for supervised learning. However, the prior art has the following disadvantages:

[0006] 1. The prior art adopts a triplet loss for training, and the number of negative samples is small, so that the model is difficult to learn discriminative features, resulting in insufficient discrimination of the embedding vectors in the feature space.

[0007] 2. The prior art does not sufficiently interact the information between the two modalities after generating the descriptors of the two modalities before supervised learning.

[0008] 3. The prior art has a low recall accuracy under the condition of a 1m high-precision distance limit in an unknown environment.

[0009] 4. The prior art has a poor model migration effect in a zero-shot training out-of-domain scene. SUMMARY

[0010] The purpose of the present application is to provide a cross-modal method for visual recognition in a large-scale point cloud map to solve the above-mentioned problems of the prior art and increase the recognition accuracy and positioning precision of a mobile robot equipped with a camera in a point cloud map database.

[0011] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0012] A cross-modal method for visual recognition in a large-scale point cloud map, comprising:

[0013] constructing a data set based on RGB image data and point cloud data collected in a global map;

[0014] cross-modal conversion of the data set;

[0015] training a preset network model using the converted data set to obtain a cross-modal positioning model; wherein the preset network model comprises a multi-scale feature encoder, a cascaded cross-attention module, and a projection converter;

[0016] inputting RGB image data not included in the data set into the cross-modal positioning model to obtain the position of a sensor in the global map.

[0017] Optionally, constructing a data set based on RGB image data and point cloud data collected in a global map comprises:

[0018] Based on the RGB image data and the point cloud data collected in the global map, calibration data is calculated, a database of paired image data and point cloud data of consecutive frames in the global map is constructed, and the location coordinates of the paired image data and point cloud data in the global map are recorded in real time.

[0019] Optionally, the cross-modal conversion of the data set comprises:

[0020] The RGB image data and the point cloud data are respectively converted into a depth map with distance information and a range map.

[0021] Optionally, training the preset network model using the converted data set comprises:

[0022] The converted depth map and range map are input into the preset network model to obtain distinctive modal representation information.

[0023] Parameters of the entire network architecture are trained based on a small batch contrast loss function.

[0024] Optionally, inputting the converted depth map and range map into the preset network model comprises:

[0025] The converted depth map and range map are input into a multi-scale feature encoder for feature extraction to obtain representation information of images and point clouds in a unified embedding space.

[0026] The representation information of images and point clouds in the unified embedding space is input into a cascaded cross-attention module for dynamic interactive fusion pairing to obtain vector features of an aligned image-point cloud pair.

[0027] The vector features of the aligned image-point cloud pair are input into a projection converter for projection conversion to obtain distinctive modal representation information.

[0028] Optionally, the cascaded cross-attention module comprises two identical cross-attention submodules; each cross-attention submodule comprises a multi-head attention layer and a residual connection.

[0029] The projection converter comprises two projection heads for dimensionality increase and dimensionality reduction, and each projection head comprises a fully connected layer and a nonlinear activation layer.

[0030] Optionally, training parameters of the entire network architecture based on the small batch contrast loss function comprises:

[0031] A batch size and a temperature parameter are set, and parameters of the network module are updated through back propagation based on a minimization loss function.

[0032] Optionally, the RGB image data not included in the data set is input into the cross-modal positioning model to obtain the position of the sensor in the global map, comprising:

[0033] The RGB image data not included in the data set is input into the cross-modal positioning model, the most similar point cloud is retrieved in the data set based on cosine similarity, and the position of the sensor in the global map is determined.

[0034] The present application has the following advantages:

[0035] The present application provides a cross-modal method flowchart for visual recognition in a large-scale point cloud map, a double-flow network architecture is designed, image point cloud pairs are synchronously encoded and interactively fused with feature information of different modalities, network training is performed based on a small batch contrast loss function after projection in an embedding space, and the trained model can realize high-precision position recognition of a query image to a large-scale point cloud map in an unknown environment, thereby contributing an important technical tool to the fields of mobile robots and autonomous driving. Compared with related technologies, the cross-modal method for visual recognition in a large-scale point cloud map can significantly improve the image-to-point cloud cross-modal position recognition accuracy and stability in unknown indoor and outdoor environments, and simultaneously, due to the lightweight design, the feasibility of deploying the cross-modal position recognition device in actual applications is increased. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described in the following only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0037] Figure 1 It is a training framework of ModaLink framework;

[0038] Figure 2 It is a cross-modal method flowchart for visual recognition in a large-scale point cloud map according to an embodiment of the present application;

[0039] Figure 3 It is a hardware connection schematic diagram according to an embodiment of the present application;

[0040] Figure 4 It is a database construction flowchart according to an embodiment of the present application;

[0041] Figure 5 It is a cascaded cross-attention module schematic diagram according to an embodiment of the present application;

[0042] Figure 6This is a schematic diagram of a projection converter according to an embodiment of the present invention;

[0043] Figure 7 This is a flowchart illustrating the cross-modal positioning process according to an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 2 As shown, this embodiment proposes a cross-modal method for visual recognition in large-scale point cloud maps, including:

[0047] A dataset is constructed based on RGB image data and point cloud data collected from the global map;

[0048] Perform cross-modal transformation on the dataset;

[0049] Using the transformed dataset, a pre-defined network model is trained to obtain a cross-modal localization model; wherein, the pre-defined network model includes: a multi-scale feature encoder, a cascaded cross-attention module, and a projection converter;

[0050] RGB image data not included in the dataset is input into the cross-modal localization model to obtain the sensor's location on the global map.

[0051] In this embodiment, the network architecture is as follows: RGB images are converted into depth images using monocular depth estimation (MDE). Point clouds are converted into distance images using spherical projection (SP). Subsequently, they are processed by a multi-scale feature encoder based on a CrossViT backbone, followed by a cascaded cross-attention module (CCAM) and a projection transformer (PT). Finally, the network is trained using mini-batch contrastive loss. The hardware connections are as follows: Figure 3 As shown.

[0052] The purpose of this embodiment is to provide a cross-modal method for visual recognition in large-scale point cloud maps, thereby increasing the recognition accuracy and positioning precision of camera-equipped mobile robots in point cloud map databases. The specific technical solution is as follows:

[0053] A mobile robot or a driving vehicle equipped with a camera and a laser radar is used to capture sensor data in a closed loop indoors or outdoors, calculate calibration data, construct a database of paired images and point clouds of consecutive frames in a large-scale map, and record their position coordinates in the global map in real time. Depth maps and range maps with distance information converted from the captured RGB images and point clouds, respectively, are obtained.

[0054] Batch input of depth maps and range maps into a multi-scale encoder of the same architecture for feature extraction obtains their representation information in a unified embedding space. A cascaded cross-attention module is designed to dynamically interact and fuse the representation information of paired images and point clouds in the unified embedding space, obtaining vector features of aligned image-point cloud pairs. A projection converter is designed to obtain more discriminative modal representation information.

[0055] Based on small-batch contrastive metric learning, the representation information of paired images and point clouds in the unified embedding space is pulled closer, and the representation information of unpaired images and point clouds is repelled, thereby training and updating the parameters of the entire model.

[0056] Inputting image data not included in the training set into the trained model, based on cosine similarity, the most similar point cloud is retrieved from the large-scale point cloud database, and the position of the sensor in the global map is determined.

[0057] Further, based on the RGB image data and point cloud data collected in the global map, a data set is constructed, including:

[0058] Based on the RGB image data and point cloud data collected in the global map, calibration data is calculated, a database of paired image data and point cloud data of consecutive frames in the global map is constructed, and the position coordinates of the paired image data and point cloud data in the global map are recorded in real time.

[0059] Further, the cross-modal conversion of the data set includes:

[0060] The RGB image data and point cloud data are respectively converted into depth maps and range maps with distance information.

[0061] Specifically, in the present embodiment, as shown in Figure 4 a database of paired images and point clouds of consecutive frames calibrated indoors or outdoors is constructed, and their position coordinates in the global map are recorded in real time. Depth maps and range maps with distance information converted from the captured RGB images and point clouds, respectively, are applied, including:

[0062] The mobile robot or driving vehicle equipped with a camera and a laser radar is used to capture closed-loop sensor data indoors or outdoors, calculate calibration data, construct a database of paired images and point clouds of consecutive frames in a large-scale map, and record their position coordinates in a global map in real time. Depth maps and range maps with distance information are obtained by converting the collected RGB images and point clouds, respectively.

[0063] Further, training the preset network model using the converted data set comprises:

[0064] inputting the converted depth map and range map into the preset network model to obtain distinctive modal representation information;

[0065] training the parameters of the entire network architecture based on a small batch contrast loss function.

[0066] Further, inputting the converted depth map and range map into the preset network model comprises:

[0067] inputting the converted depth map and range map into a multi-scale feature encoder for feature extraction to obtain representation information of the image and the point cloud in a unified embedding space;

[0068] inputting the representation information of the image and the point cloud in the unified embedding space into a cascaded cross-attention module for dynamic interactive fusion pairing to obtain vector features of the aligned image point cloud pair;

[0069] inputting the vector features of the aligned image point cloud pair into a projection converter for projection conversion to obtain distinctive modal representation information.

[0070] Specifically, in the present embodiment, the multi-scale feature encoder of the same architecture is used to obtain representation information of the image point cloud pair, based on the obtained single modal representation information, a cascaded cross-attention module is introduced to interactively fuse information of different modalities, a projection converter is introduced to obtain more distinctive modal representation information, which comprises:

[0071] The pre-trained crossvit_small_240 is used as the encoder backbone, and the last layer classifier is removed. The designed cascaded cross-attention module is composed of two identical cross-attention modules, mainly including a multi-head attention layer and a residual connection. The projection converter is composed of two projection heads of dimension increasing and dimension decreasing, mainly including a fully connected layer and a nonlinear activation layer.

[0072] The detailed architecture of the cascaded cross-attention module is as Figure 5 The formula is as follows:

[0073]

[0074] where QI ,K P ,V P respectively represent the image branch, the point cloud branch, the embedding vector output by the encoder of the point cloud branch, MHA (Multi-Head Attention) is a multi-head attention layer, MHA (Q I ,K P ,V P ) output represents that the image embedding representation carries paired point cloud information, Q I + MHA (Q I ,K P ,V P ) residual form makes the image representation not lose the original information, and the output of the first cross-attention module As the K I ,V I , Q P of the next cross-attention module, the embedding vector output by the encoder of the point cloud branch, the remaining operations are the same as the first module. LN (Layer Normalization) is a normalization layer, which can prevent the problem of gradient vanishing or explosion, and make the training process more stable. FFN (Feedforward Neural Network) is a feedforward neural network, which performs a series of nonlinear transformations to produce more rich and useful feature representations.

[0075] The detailed architecture of the projection converter module is as Figure 6 . f cross is the embedding vector output by the cascaded cross-attention module. The formula is as follows:

[0076] f fc = RELU (W fc1 f cross )

[0077] f pt = LN (W fc3 (GELU (W fc2 f fc ))+ W fc2 f fc )

[0078] Where W fc1 is the weight matrix of the first layer linear transformation, which is used to map the input f cross to another feature space. RELU is used to introduce a nonlinear factor to enhance the model expression ability. W fc2 and W fc3 represent the weight matrices of the second and third layer linear transformations respectively. Among them, W fc2 is used to map W fc2 ffc Further mapping, while W fc3 is used to map the results of GELU activation again.

[0079] RELU (Rectified Linear Unit) is a rectified linear unit, and GELU (Gaussian Error Linear Unit) is a Gaussian error linear unit.

[0080] Further, based on the small batch contrast loss function, the parameters of the entire network architecture are trained, including:

[0081] Set the batch size and temperature parameter, based on the minimization of the loss function, the parameters of the network module are updated by back propagation.

[0082] Specifically, in this embodiment, the two modal representation information after interaction fusion is used, and the parameters of the entire network architecture are trained based on the small batch contrast loss function, including:

[0083] Set the batch size and temperature parameter similar to the CLIP model, minimize the loss function, and update the parameters of the network module based on back propagation.

[0084] Among them, the two modal representation information refers to the information representation results of the image branch and the point cloud branch after the cascade cross attention module and the projection converter.

[0085] Specifically, given a batch of depth images and the range images consistent with them N represents the batch size, then this batch contains N positive matching pairs and N 2 -N negative matching pairs. For each matching pair, the contrast loss size can be calculated as follows:

[0086]

[0087] Where the temperature parameter τ is mainly used to adjust the distribution of the similarity score in the loss function. The target loss function of a batch class is defined as follows:

[0088]

[0089] Further, the RGB image data not included in the data set is input into the cross-modal positioning model to obtain the position of the sensor in the global map, including:

[0090] The RGB image data not included in the data set is input into the cross-modal positioning model, the most similar point cloud is retrieved in the data set based on the cosine similarity, and the position of the sensor in the global map is determined.

[0091] Specifically, in this embodiment, using a dataset not seen by the model, the most similar point cloud and the global position coordinates of the sensor in the large-scale point cloud map are determined from the image data acquired by the camera, including:

[0092] like Figure 7 As shown, the input image is processed by a network model to generate an image embedding vector, and the cosine similarity is calculated between the input image and the embedding vector set of the pre-built point cloud database. The position of the camera sensor in the global map is determined by the calculated best point cloud candidate.

[0093] This embodiment employs mini-batch contrastive learning to train the network, significantly increasing the number of negative samples. After descriptor generation and before supervised learning, a cascaded cross-attention model is introduced to fully interact with information between modalities. This achieves high-precision and high-accuracy cross-modal localization in unknown environments and out-of-domain scenarios with zero-shot training.

[0094] This embodiment provides a schematic flowchart of a cross-modal method for visual recognition in large-scale point cloud maps. It designs a dual-stream network architecture, synchronously encodes image-point cloud pairs, and interactively fuses feature information from different modalities. After embedding spatial projection, the network is trained based on a mini-batch contrastive loss function. The trained model can achieve high-precision location recognition from query images to large-scale point cloud maps in unknown environments, contributing an important technical tool to the fields of mobile robotics and autonomous driving. Compared with related technologies, this embodiment significantly improves the accuracy and stability of image-to-point cloud cross-modal location recognition in unknown indoor and outdoor environments. Furthermore, its lightweight design increases the feasibility of deploying this cross-modal location recognition device in practical applications.

[0095] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A cross-modal method for visual recognition in large-scale point cloud maps, characterized in that, The method comprises the following steps: constructing a data set based on RGB image data and point cloud data collected in a global map; cross-modal conversion is performed on the data set, which comprises the following steps: the RGB image data and the point cloud data are respectively converted into a depth map with distance information and a range map; a preset network model is trained using the converted data set to obtain a cross-modal positioning model, wherein the preset network model comprises a multi-scale feature encoder, a cascaded cross-attention module, and a projection converter; training the preset network model using the converted data set comprises the following steps: the converted depth map and range map are input into the preset network model to obtain distinctive modal representation information; parameters of the entire network architecture are trained based on a small-batch contrast loss function; inputting the converted depth map and range map into the preset network model comprises the following steps: the converted depth map and range map are input into the multi-scale feature encoder for feature extraction to obtain representation information of images and point clouds in a unified embedding space; the representation information of images and point clouds in the unified embedding space is input into the cascaded cross-attention module for dynamic interactive fusion pairing to obtain vector features of an aligned image-point cloud pair; the vector features of the aligned image-point cloud pair are input into the projection converter for projection conversion to obtain distinctive modal representation information; RGB image data not included in the data set is input into the cross-modal positioning model to obtain the position of a sensor in the global map.

2. The cross-modal method of visual recognition in large-scale point cloud maps according to claim 1, characterized in that, constructing a data set based on RGB image data and point cloud data collected in a global map comprises the following steps: based on the RGB image data and the point cloud data collected in the global map, calibration data is calculated, a database of paired image data and point cloud data of consecutive frames in the global map is constructed, and the position coordinates of the paired image data and point cloud data in the global map are recorded in real time.

3. The cross-modal method for visual recognition in a large-scale point cloud map according to claim 1, characterized in that: the cascaded cross-attention module comprises two identical cross-attention submodules; each cross-attention submodule comprises a multi-head attention layer and a residual connection; the projection converter comprises two projection heads for dimensionality increase and dimensionality reduction, and each projection head comprises a fully connected layer and a nonlinear activation layer.

4. The cross-modal method of visual recognition in large-scale point cloud maps of claim 1, wherein, training the parameters of the entire network architecture based on the small-batch contrast loss function comprises the following steps: the batch size and the temperature parameter are set, and the parameters of the network modules are updated through back propagation based on minimizing the loss function.

5. The cross-modal method of visual recognition in large-scale point cloud maps of claim 1, wherein, inputting the RGB image data not included in the data set into the cross-modal positioning model to obtain the position of the sensor in the global map comprises the following steps: the RGB image data not included in the data set is input into the cross-modal positioning model, the most similar point cloud is retrieved in the data set based on the cosine similarity, and the position of the sensor in the global map is determined.

Images