Position identification model construction method and system based on multi-view cross-modal matching

Abstract

The invention relates to a position recognition model construction method and system based on multi-view cross-modal matching, relates to the technical field of computer vision and natural language processing, and aims to solve the problems that a traditional vision position recognition method is difficult to keep high precision in a complex environment and a multi-view scene and cannot effectively process natural language description. In order to solve the problem, a multi-view image and natural language text description are combined, and text coding and visual coding are adopted to carry out feature extraction on a text and an image respectively; then, clustering the image features by using a clustering algorithm, and splicing the multi-view image features of each position into global image features; and finally, performing position matching by calculating the similarity between the text features and the image features. By combining vision and text information, the problems of poor robustness and accuracy of a traditional method under complex scenes and view angle changes are solved, and the method can be widely applied to the field of unmanned system navigation.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and natural language processing, and in particular to a cross-modal matching method and system for realizing visual position recognition through text description, which is applicable to the fields of mobile robot positioning, autonomous driving, scene reconstruction, etc. Background Art

[0002] With the development of artificial intelligence and robotics, location recognition and positioning technologies have been widely used in autonomous driving, logistics distribution, intelligent navigation and other fields. Traditional visual place recognition technology (Visual Place Recognition, VPR) mainly relies on a single visual modality information to determine the target location by matching with images in the database. However, these technologies face significant limitations when dealing with complex scenes or natural language description tasks. Existing visual place recognition methods usually use pre-trained deep learning models to extract image features and match them through global descriptors. Such methods perform well in simple environments, but when they need to process rich semantic information or multi-view scenes, there are problems with image detail loss and feature matching difficulties.

[0003] In recent years, the rise of cross-modal technology has brought new development directions for location recognition. By combining natural language descriptions with environmental information, cross-modal location recognition is expected to achieve more natural user interaction. At present, some studies have tried to use point clouds and text labels for positioning, such as Text2Pos and MambaPlace. Although these methods are effective, their text descriptions are too simplified, usually based on fixed object labels, and contain only a small amount of information, which cannot meet the positioning requirements in complex scenarios. In addition, these methods rely on point cloud data, which has a high acquisition cost and limits the actual application scenarios.

[0004] Through the above analysis, it can be seen that the existing location recognition technology has certain shortcomings:

[0005] (1) Traditional visual position recognition methods mainly rely on visual information and achieve positioning through image matching. However, they suffer from low accuracy, loss of feature details, and high computational complexity when processing natural language descriptions in complex environments. In addition, some methods rely on point cloud data, which has high acquisition costs and limits practical applications.

[0006] (2) Cross-modal location recognition attempts to combine natural language with visual information, but the language descriptions of existing methods are too simplified to meet the needs of complex scenarios. Moreover, most of them rely on point clouds or fixed labels, lacking flexibility and accuracy. Summary of the invention

[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a method and system for constructing a location recognition model based on multi-perspective cross-modal matching. Through a multi-perspective cross-modal matching method, visual location recognition based on text description is realized to overcome the above-mentioned technical problems existing in the existing related technology.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] In a first aspect, a method for constructing a location recognition model based on multi-view cross-modal matching is disclosed, comprising:

[0010] Segment the 360° panoramic image acquired by the panoramic camera into multiple perspectives and generate a natural language description for each perspective;

[0011] As a further technique, the 360° images acquired by the panoramic camera are segmented into four perspectives: east, west, south, and north. Then, the GPT-4 open source tool is used to generate a text description of each perspective.

[0012] The generated language description is input into the text encoding module, and the frozen pre-trained T5 model is used to extract the embedding features of the text description, combining the attention mechanism with the Transformer layer to capture the global features of the text;

[0013] Encode multi-view scene images and use Vision Transformer (ViT) as the image encoder skeleton network to extract multi-level image features;

[0014] As a further technique, ViT is used to divide the image into blocks and extract local visual features. The Si nkhorn algorithm is used to cluster the local features extracted by Vi T to generate a compact global image vector, and a learnable temperature coefficient is introduced to improve the feature matching effect.

[0015] Cross-modally fuse text features with image features to obtain unified semantic embedding features;

[0016] As a further technique, text features and image features are input into the contrastive learning network. In the training phase, a one-to-one image-text contrastive learning strategy is adopted instead of overall matching training of multiple sets of texts and images. In addition, a dynamic temperature parameter τ is introduced into the contrastive learning loss function to optimize the feature distance between positive and negative samples, making the positive samples more closely matched while maximizing the feature interval of negative samples.

[0017] Align and match the semantic embeddings of multiple text features and multiple image features at the same location, output the matching embedding, and improve the positioning accuracy through multi-view information fusion;

[0018] As a further technique, multiple view images and corresponding text descriptions of the same scene are combined to generate high-dimensional feature representations through vector concatenation. Then, the multi-view features are subjected to dimensionality reduction processing to reduce feature redundancy while maintaining the distinguishability between features. Low-dimensional matching embedding features are output to improve inference speed.

[0019] According to the fused low-dimensional matching embedding, the specific position offset ΔL of the target is calculated to accurately locate the described target position. The calculation formula is as follows:

[0020] ΔL=f(M TV )

[0021] Among them, ΔL represents the specific position offset of the target, M TV represents the generation of the final matching embedding, and f(·) is the function used for position regression.

[0022] In a second aspect, a location recognition model construction system based on multi-view cross-modal matching is disclosed, comprising:

[0023] The data acquisition and preprocessing module is used to obtain a 360° panoramic image of the scene, segment it into multi-view images (east, south, west, and north), and generate a detailed text description corresponding to each view;

[0024] The text feature encoding module is used to extract text features, generate fixed-length text embeddings, and capture the global semantic features of the text;

[0025] Image feature encoding module, used to encode and cluster features of multi-view images to generate compact global image descriptors;

[0026] The multi-view fusion module is used to integrate multi-view text descriptions and image features to generate a unified scene representation vector, and achieve precise positioning based on similarity calculation.

[0027] The cross-modal feature matching module is used to optimize the similarity between text and image features, increase the distance between non-matching samples, and fuse text and image features from multiple perspectives to achieve high-dimensional feature matching through vector splicing;

[0028] The third aspect of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0029] A fourth aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which executes the steps of the above method when executed by a processor.

[0030] A fifth aspect of the present invention is to provide a computer program product, comprising a computer program, which performs the above-described method when executed by a processor.

[0031] One or more of the above technical solutions have the following beneficial effects:

[0032] In this example, by combining text and multi-view image information, high precision and high adaptability of cross-modal location recognition are achieved, especially in complex scenes; the fusion of multi-view features effectively improves the robustness of the system and reduces the impact of a single view or missing description on positioning performance; the introduction of the Si nkhorn algorithm and attention mechanism makes the matching of image and text features more accurate, while reducing feature redundancy and improving computational efficiency; it supports matching tasks in the case of partial text description or image view missing, significantly enhancing the practicality of the system; through contrastive learning strategies and multi-view feature splicing, the matching accuracy of text to image is greatly improved, achieving a leading positioning accuracy within a range of 5 meters, which can be widely used in fields such as autonomous driving, robot navigation and logistics distribution, and meet the needs of efficient and accurate positioning in different scenarios.

[0033] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the location recognition model framework based on multi-view cross-modal matching in the first embodiment.

[0035] Figure 2 A flowchart for constructing a location recognition model based on multi-view cross-modal matching in the first embodiment.

[0036] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention. DETAILED DESCRIPTION

[0037] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0038] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0039] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0040] The overall idea proposed by the present invention is as follows: the present invention divides a 360° panoramic image into multiple perspectives through multi-perspective data collection, and generates a text description for each perspective; uses a frozen pre-trained T5 model and Vi T to extract the global features and local features of text and image respectively; clusters image features through the Si nkhorn algorithm, and optimizes text feature extraction in combination with the attention mechanism; adopts a contrastive learning method to align the embedding space of text and image features in the cross-modal matching stage, and generates a unified scene representation vector through multi-perspective feature fusion in the cross-modal fusion stage, so as to realize accurate matching and position recognition from text to image.

[0041] Embodiment 1

[0042] This embodiment discloses a method for constructing a location recognition model based on multi-view cross-modal matching.

[0043] In order to more clearly illustrate this embodiment, the implementation process of the location recognition model construction method based on multi-view cross-modal matching can be specifically described as follows:

[0044] A method for constructing a location recognition model based on multi-view cross-modal matching, comprising:

[0045] S1, obtaining a multi-view scene image, where the multi-view image includes four side views of east, south, west, and north;

[0046] S2. Input the multi-view images into the open source language generation model GPT-4 tool to generate text descriptions of the corresponding view images;

[0047] S3, input the text description into the pre-trained T5 model, and extract global semantic text features by combining the attention mechanism and the Transformer layer;

[0048] S4, using ViT to divide the image into blocks, extracting local visual features, and then using the Sinkhorn algorithm to cluster the local features extracted by ViT to generate compact global image features, and introducing a learnable temperature coefficient to improve the feature matching effect;

[0049] S5. Splice and fuse the multi-view text and image embeddings at the same position to generate a high-dimensional scene representation vector. Through similarity calculation, select candidate positions to achieve accurate matching.

[0050] S6. Use contrastive learning methods to embed and optimize text features and image features, reduce the feature distance between matching samples, increase the difference between non-matching samples, and use a one-to-one image-text contrastive learning strategy to enhance feature capture capabilities. At the same time, use the Adam optimizer to further optimize the contrastive learning loss function to improve position matching accuracy.

[0051] like Figure 2 As shown, in step S1, a 360° multi-view scene image is acquired by a panoramic camera, and the multi-view image includes four side views of east, west, south, and north;

[0052] In step S2, the multi-view images are input into the open source language generation model GPT-4 tool to generate text descriptions of the corresponding view images. i,j , which represents the image of the i-th scene at the j-th perspective;

[0053] The text description of each position is d i,j Input into the frozen pre-trained T5 model for encoding to generate semantic embedding t i,j , the formula is:

[0054] t i,j =f(d i,j θ t ) (1)

[0055] Among them, θ t are frozen pre-trained parameters to ensure the stability of text representation, and f(·;·) represents the encoding function of the T5 model.

[0056] Then, if Figure 1 In the model training, the maximum pooling layer and multi-layer perception layer are used to further extract high-dimensional semantic features a i,j , capturing the key information in the text, the formula is:

[0057] a i,j =MLP(MaxPooling(t i,j )) (2)

[0058] Among them, MLP(·) and MaxPooling(·) represent the multi-layer perception function and the maximum pooling function respectively.

[0059] Then, by Figure 1 In the model training, Transformer captures long-distance dependencies and integrates the semantic connections between multiple text sentences into the final global feature m. i,j In the formula:

[0060] m i,j = Transformer(a i,j ) (3)

[0061] Among them, Transformer(·) represents an open source encoding model.

[0062] like Figure 1 As shown, in step S4, the ViT encoder is first used to convert the multi-view image x i,jBlock processing generates a set of global and local feature representations (tokens) T = {T global ; T1, T2, …, T n}, where T global is the extracted global feature, T n is the nth local feature.

[0063] And, if Figure 1 The Sinkhorn algorithm is used for feature clustering in the model training. By solving the optimal transmission problem, local features are assigned to the cluster center P, and then a compact global descriptor F is generated. j , the formula is:

[0064]

[0065] F j =∑P i,j ·f i (5)

[0066] Among them, S is the cost matrix of local features and cluster centers, u and v are auxiliary variables, τ is a learnable temperature parameter used to adjust the "softness" of feature allocation, and f i and F j They are the local and global feature representations of each cluster center generated after clustering.

[0067] like Figure 1 As shown in the model reasoning stage in , the multi-view text features m at the same position are i,j and image embedding feature F j The stitching and fusion are performed separately to generate multi-view text and image representation vectors of high-dimensional scenes, and the candidate positions are screened out through similarity calculation.

[0068] S51, the features F of all cluster centers j Spliced ​​into a unified vector to represent the global feature q of the view image i,k , the formula is:

[0069]

[0070] S52, the global features q of all view images at each position i,k Combined into a high-dimensional feature representation Q′ i , the formula is:

[0071]

[0072] S53: Process the text description corresponding to the view image at each position in the same manner as S51 to obtain a multi-view text description M′ i , the formula is:

[0073]

[0074] in, It is a characteristic operation, which means concatenating two or more vectors in sequence to generate a new vector.

[0075] S54, using cosine similarity to calculate text feature M' i With each position image feature Q′ i The similarity is:

[0076]

[0077] Among them, sim(·) represents the cosine similarity, which is used to measure the matching degree between text and image.

[0078] The generated text feature M′ i and image features Q′ i Input to the contrastive learning network, and optimize the matching of text and image features through contrastive learning method. The loss function formula is:

[0079]

[0080] Among them, q i,k Represents the global descriptor of the k-th view image of the ith location.

[0081] S61. Use the Adam optimizer to further optimize the loss function and update the model parameters according to the gradient of the feature. The formula is:

[0082]

[0083] Among them, α is the learning rate of network training, which represents the loss function L contrast The gradient of the parameter θ.

[0084] S62, such as Figure 1 As shown in the figure, all locations are sorted by similarity, and the candidate locations with the highest similarity are selected. According to the set Top-k strategy, the most likely location is selected as the matching result. In the location output, the matching location identifier i and related information are returned, including the specific perspective image, description and location label of the location, to achieve accurate location recognition.

[0085] The location recognition model based on multi-view cross-modal matching is applied to the Google Street View dataset with generated text descriptions for performance evaluation. The results are shown in Table 1. The recognition method in this embodiment is compared with the most advanced benchmark. When there is only one candidate location, by comparing the positioning recall rates on the validation set and the test set at 5 meters / 10 meters / 15 meters, it can be seen that the recall rate of the location recognition method based on text-image cross-modality in this embodiment is significantly higher than that of several other location recognition methods based on text-point cloud. This is mainly because the method in this embodiment uses multi-view stitching technology and cross-modal alignment technology, while other methods do not use multi-view stitching technology.

[0086] Table 1 Performance comparison of this method and other advanced methods on Google Street View

[0087]

[0088] Embodiment 2

[0089] The purpose of this embodiment is to provide a location recognition model construction system based on multi-view cross-modal matching, including:

[0090] The data acquisition and preprocessing module is used to obtain a 360° panoramic image of the scene, segment it into multi-view images (side views in four directions: east, south, west, and north), and generate a detailed text description corresponding to each view;

[0091] The text feature encoding module is used to extract text features, generate fixed-length text embeddings, and capture the global semantic features of the text;

[0092] Image feature encoding module, used to encode and cluster features of multi-view images to generate compact global image descriptors;

[0093] The multi-view fusion module is used to integrate multi-view text descriptions and image features to generate a unified scene representation vector, and achieve precise positioning based on similarity calculation.

[0094] The cross-modal feature matching module is used to optimize the similarity between text and image features, increase the distance between non-matching samples, and fuse text and image features from multiple perspectives to achieve high-dimensional feature matching through vector splicing;

[0095] Based on providing a location recognition model construction system based on multi-view cross-modal matching, the method steps in embodiment 1 are implemented.

[0096] Embodiment 3

[0097] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0098] Embodiment 4

[0099] The purpose of this embodiment is to provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the above method are performed.

[0100] Embodiment 5

[0101] The purpose of this embodiment is to provide a computer program product containing instructions, which, when running on a computer, enables the computer to execute the methods and functions involved in any of the above embodiments.

[0102] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0103] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0104] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for constructing a location recognition model based on multi-view cross-modal matching, characterized in that: The specific steps include: Step S1: Acquire a multi-view scene image, wherein the multi-view image includes four side views of east, south, west, and north; Step S2: Input the multi-view images into the open source language generation model GPT-4 tool to generate text descriptions of the corresponding view images; Step S3: Input the text description into the pre-trained T5 model, and extract global semantic text features by combining the attention mechanism and the Transformer layer; Step S4: Use ViT to divide the image into blocks and extract local visual features. Then use the Sinkhorn algorithm to cluster the local features extracted by ViT to generate compact global image features, and introduce a learnable temperature coefficient to improve the feature matching effect. Step S5: splice and fuse the multi-view text and image embeddings at the same position to generate a high-dimensional scene representation vector, and screen out candidate positions through similarity calculation to achieve accurate matching; Step S6: Use contrastive learning method to embed and optimize text features and image features, reduce the feature distance between matching samples, increase the difference between non-matching samples, and adopt one-to-one image-text contrastive learning strategy to enhance feature capture capability; at the same time, use Adam optimizer to further optimize the contrastive learning loss function to improve position matching accuracy.

2. The method for constructing a location recognition model based on multi-view cross-modal matching according to claim 1, characterized in that: The image features are extracted through ViT, and the text features are extracted through a pre-trained T5 model.

3. The method for constructing a location recognition model based on multi-view cross-modal matching according to claim 1, characterized in that: The clustering of the image features uses the Sinkhorn algorithm, and the generation of cluster centers includes weighted summation of local image features.

4. The method for constructing a location recognition model based on multi-view cross-modal matching according to claim 1, characterized in that: The concatenation operation of the text description and the image features is a step-by-step concatenation operation of feature vectors to generate a unified high-dimensional feature representation.

5. The method for constructing a location recognition model based on multi-view cross-modal matching according to claim 1, characterized in that: The similarity calculation adopts the cosine similarity formula:

6. The method for constructing a location recognition model based on multi-view text-visual cross-modal matching according to claim 1, characterized in that: The contrastive learning method is used to embed and optimize text features and image features. The contrastive learning loss formula is:

7. A location recognition model construction system based on multi-view cross-modal matching, characterized in that: include: The data acquisition and preprocessing module is used to obtain a 360° panoramic image of the scene, segment it into multi-view images (side views in four directions: east, south, west, and north), and generate a detailed text description corresponding to each view; The text feature encoding module is used to extract text features, generate fixed-length text embeddings, and capture the global semantic features of the text; Image feature encoding module, used to encode and cluster features of multi-view images to generate compact global image descriptors; The multi-view fusion module is used to integrate multi-view text descriptions and image features to generate a unified scene representation vector, and achieve precise positioning based on similarity calculation. The cross-modal feature matching module is used to optimize the similarity between text and image features, increase the distance between non-matching samples, fuse text and image features from multiple perspectives, and achieve high-dimensional feature matching through vector splicing.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are performed.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images