Method and system for constructing place recognition model based on text-point cloud matching

Abstract

The present invention proposes a method and system for constructing a place recognition model based on text-point cloud matching, which relates to the field of robotics and computer vision technology. The problem addressed is that traditional methods rely on a single modality for place matching, are easily affected by environmental factors, and have poor accuracy and robustness. The text description is generated by the advanced open source GPT-4 tool, and combined with the KITTI 360 point cloud subgraph, semantic alignment and cross-modal feature fusion of text to point cloud are achieved. Specifically, text features and point cloud features are extracted respectively by the text encoding module and the point cloud encoding module, and contrastive learning is used to map the text description and the point cloud subgraph to a unified semantic embedding space. The cascaded cross-modal attention selective state space module is used to further optimize the feature matching between text and point cloud, thereby improving the generalization performance of the model in complex environments and the accuracy of cross-modal matching, and solving the problem of poor robustness and accuracy of single-modality place recognition technology in complex scenes.

Description

Technical Field

[0001] The present invention belongs to the field of robotics and computer vision technology, and in particular, relates to a method and system for constructing a place recognition model based on text-point cloud matching, and specifically to a method and system for constructing a cross-modal place recognition model from text to point cloud using a state space model that incorporates an attention mechanism. Background Art

[0002] With the development of smart devices and autonomous driving technology, place recognition, as one of the core technologies, has been widely used in robot navigation, autonomous driving, virtual reality and other fields. Traditional place recognition methods based on vision or lidar usually rely only on single-modal data, relying on 2D images or 3D point clouds for matching, and extracting features through cameras or radars. However, due to the influence of external factors such as changes in ambient lighting and changes in viewing angles, their accuracy and robustness are limited. However, with the popularization of multimodal data, cross-modal place recognition technology that combines text descriptions with point cloud maps has gradually become a research hotspot. Text information and point cloud information have different expressions. How to effectively integrate the feature information of the two to improve the accuracy and robustness of place recognition is an important challenge facing current technology.

[0003] Through the above analysis, it can be seen that the existing location recognition technology has certain shortcomings:

[0004] (1) Traditional positioning methods rely only on single-modality image or point cloud features, which can easily lead to positioning failures in scenes with changing environmental conditions (such as lighting, seasons) and perspectives (such as occlusion, large-angle rotation), and it is difficult to ensure robustness and accuracy.

[0005] (2) Existing cross-modal methods fail to fully utilize the complementary information of text descriptions and point cloud maps. Especially in point cloud feature extraction, they usually adopt a simple attention mechanism or directly aggregate the features of point clouds, which leads to the semantic space misalignment of point clouds and text features, limiting the cross-modal fusion effect. Summary of the invention

[0006] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a method and system for constructing a place recognition model based on text-point cloud matching, which effectively realizes cross-modal place matching from text to point cloud by introducing a state space model based on the attention mechanism in text encoding, point cloud encoding and cross-modal fusion, so as to overcome the above-mentioned technical problems existing in the existing related technologies.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] In a first aspect, a method for constructing a location recognition model based on text-point cloud matching is disclosed, comprising:

[0009] Obtain text data, where the text description is generated by the GPT-4 large model tool according to the KITTI360 point cloud submap;

[0010] The text description and point cloud submap are input into the feature encoding module, and the attention features and spatial features of each modality are extracted through the attention feature extraction mechanism of the text encoding module and the point cloud encoding module;

[0011] As a further technique, the key features of spatial relations in natural language descriptions are extracted through the text encoding module, and the long-range geometric features of the point cloud submap are captured through the point cloud encoding module to obtain the point cloud attention features;

[0012] Cross-modally fuse text attention features with point cloud attention features to obtain unified semantic embedding features;

[0013] As a further technique, text features and point cloud features are input into a contrastive learning network, and a cross-modal semantic space is generated through contrastive learning training to achieve cross-modal feature semantic alignment;

[0014] The aligned text features and point cloud features are input into the cross-modal semantic space, and the Cascade Attention Selective State-space Model (CASSM) is used to fuse the cross-modal features of text and point cloud to further enhance the spatial and semantic matching relationship of the target location.

[0015] As a further technique, the candidate submaps are accurately regressed, and the candidate submap features and text features in the rough positioning stage are input into CASSM. Through CASSM, the cross-modal features of text and point cloud are fused to further enhance the spatial and semantic matching relationship of the target location;

[0016] According to the fused matching embedding, the specific location offset ΔL of the target is calculated to accurately locate the described target location. The calculation formula is as follows:

[0017] ΔL=f(M TP )

[0018] Among them, ΔL represents the specific location offset of the target, M TP represents the generation of the final matching embedding, and f(·) is the function used for place regression.

[0019] In a second aspect, a place recognition model construction system based on text-point cloud matching is disclosed, comprising:

[0020] The text generation module is used to generate text description data corresponding to the KITTI360 point cloud sub-map dataset using the GPT-4 large model tool;

[0021] The text encoding module is used to encode text descriptions into embedded features to facilitate cross-modal feature matching between text and point cloud data;

[0022] Point cloud encoding module, which is used to extract features and encode 3D point cloud data so as to match them with the embedding of text descriptions;

[0023] The text-point cloud cross-modal matching module is used to align text and point cloud features across modalities to achieve cross-modal scene matching and positioning;

[0024] The cascaded attention selective state space model module is used to further refine the matching relationship between text and point cloud, and finally obtain high-precision target location coordinates.

[0025] The third aspect of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0026] A fourth aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which executes the steps of the above method when executed by a processor.

[0027] A fifth aspect of the present invention is to provide a computer program product, comprising a computer program, which performs the above-described method when executed by a processor.

[0028] One or more of the above technical solutions have the following beneficial effects:

[0029] In this example, the multi-scale expression capabilities of text semantic features and point cloud spatial features are enhanced through the text encoding module and the point cloud encoding module. The semantic alignment of text and point cloud features is optimized by introducing contrastive learning. The deep fusion of cross-modal features is achieved through CASSM, which improves the accuracy and robustness of target positioning. Compared with single-modal methods, this example overcomes the problem that traditional single-modal recognition methods that rely only on vision or point clouds are prone to failure in complex scenes. At the same time, it also solves the shortcomings of ambiguity and lack of spatial correlation in positioning methods based solely on text descriptions, significantly improving the generalization performance of the model and the accuracy of cross-modal matching, and achieving more accurate and efficient target positioning.

[0030] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 Schematic diagram of the place recognition model framework based on text-point cloud matching in the first embodiment.

[0033] Figure 2 This is a flow chart of the method for constructing a place recognition model based on text-point cloud matching in the first embodiment.

[0034] Figure 3 4 is a structural diagram of the cascaded attention selectivity state space model in the first embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0036] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0037] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0038] The overall idea proposed by the present invention is as follows: the present invention first generates a text description corresponding to a point cloud submap through an open source GPT-4 model tool, and then uses a text encoding module and a point cloud encoding module to respectively extract the semantic features of the text description and the spatial features of the point cloud submap, and fuses them to generate a global semantic embedding feature; the global semantic embedding feature is input into a contrastive learning network, and the semantic alignment of text and point cloud features is optimized through contrastive learning to generate a cross-modal matching feature; the cross-modal matching feature is input into CASSM to further enhance the fusion characteristics of text and point cloud and generate a cross-modal fusion feature; according to the fused matching embedding, the specific location offset of the target is calculated to accurately locate the described target location.

[0039] Embodiment 1

[0040] This embodiment discloses a method for constructing a location recognition model based on text-point cloud matching. In order to more clearly explain this embodiment, the implementation process of constructing a location recognition model based on text-point cloud matching includes a coarse positioning stage and a fine positioning stage:

[0041] The coarse positioning stage includes a text generation module, a text encoding module, a point cloud encoding module, and a text-point cloud cross-modal matching module, which are used to quickly narrow the search scope and screen out areas where the target may exist from a large environment, providing candidate locations for the next step of fine positioning.

[0042] The precise positioning stage includes the CASSM module, which is used to further refine the matching relationship between text and point cloud, and finally obtain high-precision coordinates of the target location.

[0043] A method for constructing a place recognition model based on text-point cloud matching, specifically comprising:

[0044] S1. Text data generation: Based on the spatial features of the point cloud submap (such as geometric shape, relative location, etc.), the open source GPT-4 model is used to generate descriptive text with spatial relationships, such as "the target is located on the south side of the white building next to the green tree";

[0045] S2: Extract text semantic features, input the text description into the pre-trained T5 model, generate basic text embedding vectors; extract key semantic features in the text, including spatial relationships and target location keywords, through the Selective State-space Model (SSM);

[0046] S3, extract point cloud spatial features, input the point cloud submap into the PointNet++ model, pre-process the 3D point cloud data P to extract key features, and apply SSM to capture long-range spatial relationship information in the point cloud;

[0047] S4, map text features and point cloud features to a unified semantic embedding space, input text feature descriptors and point cloud feature descriptors into a contrastive learning network, and generate a cross-modal semantic space through contrastive learning optimization. In this space, the feature distances of matched text and point cloud submaps are closer, and the feature distances of non-matching pairs are farther, thereby achieving coarse positioning and obtaining candidate submap regions;

[0048] S5. Perform precise location regression on the candidate point cloud sub-image, and input the candidate point cloud sub-image features and text features in the rough positioning stage into CASSM. Through CASSM, the cross-modal features of text and point cloud are fused to further enhance the spatial and semantic matching relationship of the target location;

[0049] S6. Predict the target location. Based on the fused matching embedding, calculate the specific location offset of the target to accurately locate the described target location.

[0050] like Figure 2 As shown, in step S1, the GPT-4 large model tool is used to generate text description data corresponding to the point cloud sub-image of the KITTI360 dataset;

[0051] As a further technical solution, natural language description data is generated based on spatial feature information. The point cloud feature vector is set as P, and the geometric features and spatial relationship information extracted by preprocessing are recorded as F P , the specific processing includes;

[0052] The point cloud features are represented as prompt input for the GPT model to generate descriptions (prompts):

[0053] Prompt=format(F P ) (1)

[0054] Among them, format(·) represents the prompt function that formats the point cloud features into natural language descriptions.

[0055] Input the prompt into the GPT-4 model and get the description text T:

[0056] T=GPT-4(Prompt) (2)

[0057] Among them, GPT-4(·) represents an open source large language generation model.

[0058] like Figure 1 , Figure 2 As shown, in step S2, the text description is encoded into an embedded feature to facilitate text-point cloud cross-modal feature matching;

[0059] As a further technical solution, the text description is encoded into E through the pre-trained T5 model. T , and use SSM to extract the spatial location features of the text. The specific processing includes:

[0060] The generated text T is input into the text encoding model T5 to extract the preliminary embedding vector E T :

[0061] E T =TextEncoder(T) (3)

[0062] Among them, TextEncoder(·) represents the open source T5 text encoder.

[0063] The initial embedding vector E T Input spatial position feature extraction module, and strengthen the spatial position information through attention model and SSM;

[0064] Use the multi-head attention mechanism to calculate the spatial attention weight matrix A:

[0065]

[0066] Among them, WQ and W K is the weight matrix of “query” and “key” in multi-head attention, d K is the dimension of the attention vector and softmax(·) represents the classification function.

[0067] For E T Apply SSM to filter out the key spatial location features S T :

[0068] S T =SSM(A·E T ) (5)

[0069] Here, SSM(·) represents the selective state space function.

[0070] like Figure 2 As shown, in step S3, feature extraction and encoding of the point cloud submap are performed to match the embedding of the text description;

[0071] As a further technical solution, Figure 1 As shown in the figure, the open source PointNet++ model is used to preprocess the point cloud feature P to extract key features; SSM is applied to capture the long-distance spatial relationship information in the point cloud to ensure that the geometric information can fully reflect the environment structure. The specific processing includes:

[0072] Extract the basic feature representation of point cloud through point cloud encoding model (PointNet++) P :

[0073] E P =PointNet++(P) (6)

[0074] Among them, E P It is the basic feature description of point cloud. PointNet++(·) represents an open source point cloud feature extraction model.

[0075] Use the SSM module to strengthen the spatial relationship of the point cloud and obtain the filtered point cloud spatial features S P :

[0076] S P =SSM(E P ) (7)

[0077] The point cloud spatial features are normalized to obtain the point cloud features after the point cloud features are aligned with the text features:

[0078]

[0079] Among them, S P is the point cloud spatial feature, S′ Pis the aligned point cloud feature, ‖·‖ is the norm.

[0080] like Figure 2 As shown, in step S4, the text features and the point cloud features are matched to achieve cross-modal matching and positioning;

[0081] As a further technical solution, Figure 1 As shown, the text feature S T With point cloud feature S′ P Perform cross-modal matching, including:

[0082] Contrastive learning is used to project text features and point cloud features into a unified semantic space to ensure matching accuracy. Contrastive learning loss is defined as L contraset , so that the distance between the matching text feature and the point cloud feature pair is the smallest, and the distance between the non-matching pairs is the largest:

[0083]

[0084] Among them, sim(·,·) is the similarity measurement function, τ is the temperature parameter, Represents negative sample features.

[0085] like Figure 2 As shown, in step S5, the CASSM module is used to further refine the matching relationship between the text and the point cloud, and finally obtain the high-precision coordinates of the target location;

[0086] As a further technical solution, Figure 1 As shown in the figure, the CASSM module is used to realize the cross-modal fusion of text features and point cloud features. The specific processing includes:

[0087] The text S is converted into T With point cloud feature S′ P Perform multi-scale fusion to generate the final matching embedding M TP :

[0088] M TP =CASSM(S T ,S′ P ) (10)

[0089] Among them, CASSM(·) represents the processing function related to the cascaded attention state space model, such as formulas (11)-(14); its structure is as follows Figure 3 As shown, the relevant processing process is steps S51-S54.

[0090] S51, such as Figure 3As shown in the figure, in the CASSM module, the point cloud features and text features are fused through a multi-head attention mechanism, where the point cloud is used as the query Q and the text features are used as the key K and value V:

[0091] Z=AddNorm(Q,MHA(Q,K,V)) (11)

[0092] Among them, AddNorm(·) represents the residual normalization function, MHA(·) represents the multi-head attention function, and Z is the fusion feature of the multi-head attention output.

[0093] S52, the multi-head attention fusion feature Z is processed by the feedforward neural network layer and outputs Z′:

[0094] Z′=AddNorm(Z,FFN(Z)) (12)

[0095] Where FFN(·) represents the feed-forward neural network function.

[0096] S53, SSM module further processes the output features of the feedforward neural network layer. Here, SSM is used to generate new query Q', key K' and value V':

[0097] Q′,K′,V′=SSM(Z′) (13)

[0098] S54: Input the new query, key and value generated by the SSM module into the cross attention module, and obtain the matching embedding M after further processing. TP :

[0099] M TP =CrossAtten(Q′,K′,V′) (14)

[0100] Among them, CrossAtten(·) represents the cross attention processing function.

[0101] like Figure 1 , Figure 2 As shown, in step S6, according to the fused matching embedding M TP , calculate the target's specific location offset ΔL to accurately locate the target location described:

[0102] ΔL=f(M TP )

[0103] Where f(·) is the location regression function, which is used to calculate the final positioning offset.

[0104] The place recognition model based on text-point cloud matching is applied to the text-point cloud cross-modal place dataset for performance evaluation. The results are shown in Table 1.

[0105] As shown in Table 1, the recognition method in this embodiment is compared with the most advanced benchmarks. It can be seen that when the robot positioning recall rate at 5 meters / 10 meters / 15 meters is compared on the same validation set and test set, the recall rate of the method in this embodiment when the candidate locations are set to 1 / 5 / 10 respectively is significantly higher than that of the other three methods, indicating that the recognition method in this embodiment is superior to all other advanced methods. This is mainly because the method in this embodiment uses contrastive learning to map text descriptions and point cloud subgraphs to a unified semantic embedding space, and uses a cascaded cross-modal attention selective state space module to further optimize the feature matching of text and point clouds, thereby improving the generalization performance of the model in complex environments and the accuracy of cross-modal matching. Among them, "Top (1, 5, 10)" means the recall rate of one correct match when the candidate locations are 1, 5, and 10 respectively.

[0106] Table 1 Performance comparison between this method and other advanced methods

[0107]

[0108] Embodiment 2

[0109] The purpose of this embodiment is to provide a place recognition model construction system based on text-point cloud matching, including:

[0110] The text generation module is used to generate text description data corresponding to the KITTI360 point cloud dataset using the GPT-4 large model tool;

[0111] The text encoding module is used to encode text descriptions into embedded features to facilitate cross-modal feature matching between text and point cloud data;

[0112] Point cloud encoding module, which is used to extract features and encode 3D point cloud data so as to match them with the embedding of text descriptions;

[0113] The text-point cloud cross-modal matching module is used to match text and point cloud features to achieve cross-modal scene matching and positioning;

[0114] The CASSM module is used to further refine the matching relationship between text and point cloud, and finally obtain high-precision target location coordinates.

[0115] Based on providing a place recognition model construction system based on text-point cloud matching, the method steps in embodiment 1 are implemented.

[0116] Embodiment 3

[0117] This example provides a computer device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the program, each step of the text-to-point cloud cross-modal positioning method is implemented.

[0118] Embodiment 4

[0119] The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the text-to-point cloud cross-modal positioning method described in Example 3.

[0120] Embodiment 5

[0121] The purpose of this embodiment is to provide a computer program product containing instructions, which, when executed on a computer, enables the computer to execute the text-to-point cloud cross-modal positioning method and function involved in Embodiment 3 or Embodiment 4.

[0122] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0123] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0124] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for constructing a location recognition model based on text-point cloud matching, characterized in that: The specific steps are as follows: Step S1: Obtain text data, where the text description is generated by the GPT-4 large model tool according to the KITTI 360 point cloud submap; Generate text descriptions for each point cloud sub-image in the KITTI 360 dataset using the open source GPT-4 model; Step S2: Input the text description and the point cloud submap into the feature encoding module, and extract the attention features and spatial features of each modality through the attention feature extraction mechanism of the text encoding module and the point cloud encoding module; Step S3: Cross-modally fuse the text attention features with the point cloud attention features to obtain a unified semantic embedding feature; Step S4: Input the aligned text features and point cloud features into the cross-modal semantic space, and use CASSM to fuse the cross-modal features of text and point cloud to obtain matching embedding, further enhancing the spatial and semantic matching relationship of the target location; Step S5: Based on the fused matching embedding, the specific location offset ΔL of the target is calculated to accurately locate the described target location.

2. The method for constructing a location recognition model based on text-point cloud matching according to claim 1, characterized in that: In step S1, the extraction process of each text description feature is as follows: Step S11: Input the text description into the pre-trained T5 model to generate a basic text embedding vector; Step S12: extract key semantic features in the text through SSM, including spatial relations and target location keywords.

3. The method for constructing a location recognition model based on text-point cloud matching according to claim 1, characterized in that: In step S3, the process of extracting features of each point cloud sub-map is as follows: Step S21: input the point cloud submap into the PointNet++ model to pre-process the three-dimensional point cloud data P to extract key features; Step S22: Apply SSM to capture long-range spatial relationship information in the point cloud.

4. The method for constructing a place recognition model based on text-point cloud matching according to claims 2 and 3, characterized in that: In step S4, the extracted text features and point cloud features are mapped to a unified semantic embedding space, and the specific steps are as follows: Step S23: inputting the text feature descriptor and the point cloud feature descriptor into a contrastive learning network, and generating a cross-modal semantic space through contrastive learning optimization; Step S24: In this space, the matched text is closer to the feature distance of the point cloud sub-map, and the feature distance of the non-matching pair is farther, thereby achieving rough positioning and obtaining the candidate sub-map area.

5. The method for constructing a location recognition model based on text-point cloud matching according to claim 4, characterized in that: The process of performing precise location regression on the candidate point cloud sub-image in step S5 is as follows: Input the candidate point cloud sub-image features and text features from the coarse positioning stage into the cascaded attention selectivity state space model; By fusing the cross-modal features of text and point cloud through CASSM, the spatial and semantic matching relationship of the target location is further enhanced. The process of fusing the cross-modal features of text and point cloud using CASSM is as follows: The point cloud features and text features are fused through a multi-head attention mechanism, where the point cloud is used as the query Q, the text features are used as the key K and value V, and the output is Z; The fusion feature Z is processed by the feedforward neural network layer and then outputs Z ′ ; The SSM module is used to further process the output feature Z of the feedforward neural network layer. Here, SSM is used to generate a new query Q ′ , key K ′ Sum value V ′ ; The new query, key and value (Q ′ ,K ′ ,V ′ ) is input to the cross attention module and further processed to obtain the matching embedding M TP .

6. The method for constructing a location recognition model based on text-point cloud matching according to claim 5, characterized in that: The step S6 predicts the target location, specifically: embedding M according to the fused matching TP , calculate the specific location offset ΔL of the target to accurately locate the described target location.

7. A place recognition model construction system based on text-point cloud matching, characterized in that: include: The text generation module is used to generate text description data corresponding to the KITTI 360 point cloud dataset using the GPT-4 large model tool; The text encoding module is used to encode text descriptions into embedded features to facilitate cross-modal feature matching between text and point cloud data; Point cloud encoding module, which is used to extract features and encode 3D point cloud data so as to match them with the embedding of text descriptions; The text-point cloud cross-modal matching module is used to match text and point cloud features to achieve cross-modal scene matching and positioning; The CASSM module is used to further refine the matching relationship between text and point cloud, and finally obtain high-precision target location coordinates.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are performed.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images