A method and system for visual localization in large-scale urban scenes using text

Abstract

The application provides a method and system for visual positioning in a large-scale urban scene using text, the method comprising: collecting multi-vision images and position coordinate information of an agent, obtaining a place text description corresponding to the multi-vision images through GPT, and constructing a database of text and multi-view image pairing; constructing a visual positioning model and training the visual positioning model based on the database of text and multi-view image pairing; and inputting a place text description of an agent to be positioned into the trained visual positioning model to realize visual positioning of the agent to be positioned. The application can significantly improve the precision and stability of cross-modal recognition of text to multi-view image places in unknown indoor and outdoor environments, and due to the design of hard negative sample perception, the cross-modal position recognition device has a faster convergence speed than the prior art in actual training, saving the cost of computing power.

Description

Technical Field

[0001] This invention belongs to the field of visual positioning technology, specifically relating to a method and system for visual positioning in large-scale urban scenes using text. Background Technology

[0002] Traditional self-localization systems typically support single-modal queries (text, image, or point cloud). Advanced systems should support cross-modal queries, enabling users to more effectively guide robots to localize using conceptual frameworks. Future intelligent robots should collaborate with humans, utilizing large-scale scenes and natural language descriptions for self-localization, suitable for tasks such as goods delivery and vehicle pick-up. Cross-modal methods can improve robots' navigation capabilities in complex urban environments. For example, delivery robots with natural language processing capabilities can use verbal descriptions from humans to identify scenes and find their destination. However, challenges include the high computational complexity of handling large-scale point cloud data in text-to-point cloud cross-modal solutions, which can slow down operations. Furthermore, in text-to-image cross-modal solutions, the lack of multi-view scene descriptions limits contextual cues during self-localization.

[0003] like Figure 1 As shown, the existing technical framework uses a frozen T5 model to extract local tokens from multiple natural language descriptions, capturing fine-grained semantic elements. These tokens are then encoded as image features by a Visual Transformer (ViT) and projected into a shared embedding space via a Multilayer Perceptron (MLP). A Cluster Set module dynamically aggregates similar features to form cluster tokens, mitigating structural ambiguity. The Transformer encoder fuses local and global contexts and generates query embeddings through max pooling. Finally, the similarity matrix between the text query and the cluster token sum is calculated using a bidirectional InfoNCE loss objective function to bring positive samples closer and separate negative samples, achieving global horizontal alignment between text and multi-view images. This design improves the localization accuracy in urban scenes to some extent.

[0004] While current self-localization methods perform well in coarse-grained retrieval, their fine-grainedness remains insufficient, mainly due to two core limitations: (1) Insufficient modeling of fine-grained local alignment between modalities. Existing models mostly use global pooling to align text and image embeddings, ignoring local semantic correspondences (such as "the red sign on the left" in the text corresponding to a specific region in the image). This results in a high false positive rate in structurally similar scenarios. (2) Suboptimal utilization of hard negative samples during training. Traditional contrastive learning only uses random negative samples, while true hard negatives in urban environments (such as adjacent blocks) are more likely to expose ambiguity. However, the mining and injection mechanisms are often inefficient, leading to slow model convergence. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method and system for visual localization using text in large-scale urban scenes, aiming to improve the accuracy and stability of cross-modal recognition of text-to-multi-view image locations in unknown indoor and outdoor environments.

[0006] To achieve the above objectives, the present invention provides the following solution: A method for visual localization in large-scale urban scenes using text, the method comprising: Collect multi-view images and location coordinate information of the intelligent agent, obtain the location text descriptions corresponding to the multi-view images through GPT, and build a database of text and multi-view image pairings; A visual localization model was constructed and trained based on a database of text and multi-view image pairings. The location text description of the agent to be located is input into the trained visual localization model to achieve visual localization of the agent.

[0007] Preferably, the visual positioning model includes: text branch, image branch, and local soft alignment module; The text branch uses T5-small as the encoder backbone to obtain textual representation information of location text descriptions; Image branching utilizes the ViT encoder to obtain image representation information from multiple visual images; The local soft alignment module is used to optimize text representation information and image representation information, and to achieve a many-to-many correspondence between text representation information and image representation information.

[0008] Preferably, the method for optimizing text representation information and image representation information using the local soft alignment module to achieve a many-to-many correspondence between text representation information and image representation information includes: Calculate the transmission cost matrices for text-to-image and image-to-text, solve for the entropy-regularized optimal transmission plan using the Sinkhorn algorithm, construct a soft alignment loss, and achieve self-supervised many-to-many correspondences: ; ; ; in, and All are predicted probability distributions. l and k These are the index variables for text tokens and image clustering, respectively. and It is an entropy-regularized optimal transmission plan based on text and image perspectives. The soft alignment loss function is... and These are the cross-entropy loss functions based on text and image perspectives, respectively.

[0009] Preferably, the visual localization model further includes: introducing a global contrastive loss that incorporates hard negative sample perception, and using cross-entropy loss to reveal the parameters of the entire network architecture to accelerate the training of difficult samples. Set the batch size, add a hard negative sample suppression term based on the standard symmetric InfoNCE loss, use the same temperature parameter as the standard symmetric InfoNCE loss, minimize the loss function, and update the parameters of the network modules based on backpropagation; Calculate the cosine similarity matrix: ; in, The cosine similarity matrix is... , These represent image features and text features, respectively. and These represent the image feature unit vector and the text feature unit vector after L2 normalization, respectively. Let temperature parameter The partitioning functions for the row and column directions are: ; Define the probability mass of assigning each sample to the hardest negative sample in both modalities as: ; ; in, , They represent the first i The maximum similarity of a sample to all unpaired samples in both modal directions; Variations of hard-negative regularization: Logarithmic loss function: ; Exponential loss function: ; Mean absolute error loss function: ; Tangent loss function: ; The final global training objective is defined as: ; in, Represents any variant of the hard-negative regularization term.

[0010] The present invention also provides a system for visual localization in large-scale urban scenes using text, the system being used to implement the aforementioned method, the system comprising: a data acquisition module, a model building module, and a visual localization module; The data acquisition module is used to collect multi-view images and location coordinate information of the intelligent agent, obtain location text descriptions corresponding to the multi-view images through GPT, and build a database of text and multi-view image pairings; The model building module is used to build a visual localization model and train the visual localization model based on a database of text and multi-view image pairings. The visual localization module is used to input the location text description of the agent to be localized into the trained visual localization model to achieve visual localization of the agent.

[0011] Preferably, the visual positioning model includes: text branch, image branch, and local soft alignment module; The text branch uses T5-small as the encoder backbone to obtain textual representation information of location text descriptions; Image branching utilizes the ViT encoder to obtain image representation information from multiple visual images; The local soft alignment module is used to optimize text representation information and image representation information, and to achieve a many-to-many correspondence between text representation information and image representation information.

[0012] Preferably, the method for optimizing text representation information and image representation information using the local soft alignment module to achieve a many-to-many correspondence between text representation information and image representation information includes: Calculate the transmission cost matrices for text-to-image and image-to-text, solve for the entropy-regularized optimal transmission plan using the Sinkhorn algorithm, construct a soft alignment loss, and achieve self-supervised many-to-many correspondences: ; ; ; in, and All are predicted probability distributions. l and k These are the index variables for text tokens and image clustering, respectively. and It is an entropy-regularized optimal transmission plan based on text and image perspectives. The soft alignment loss function is... and These are cross-entropy loss functions based on text and image perspectives, respectively.

[0013] Preferably, the visual localization model further includes: introducing a global contrastive loss that incorporates hard negative sample perception, and using cross-entropy loss to reveal the parameters of the entire network architecture to accelerate the training of difficult samples. Set the batch size, add a hard negative sample suppression term based on the standard symmetric InfoNCE loss, use the same temperature parameter as the standard symmetric InfoNCE loss, minimize the loss function, and update the parameters of the network modules based on backpropagation: Calculate the cosine similarity matrix: ; in, The cosine similarity matrix is... , These represent image features and text features, respectively. and These represent the image feature unit vector and the text feature unit vector after L2 normalization, respectively. Let temperature parameter The partitioning functions for the row and column directions are: ; Define the probability mass of assigning each sample to the hardest negative sample in both modalities as: ; ; in, , They represent the first i The maximum similarity of a sample to all unpaired samples in both modal directions; Variations of hard-negative regularization: Logarithmic loss function: ; Exponential loss function: ; Mean absolute error loss function: ; Tangent loss function: ; The final global training objective is defined as: ; in, Represents any variant of the hard-negative regularization term.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a method and system for visual localization in large-scale urban scenes using text. A dual-branch network architecture is designed to synchronously encode text-image pairs. Through a local soft alignment module, self-supervised training is performed at the text token and image cluster levels to achieve many-to-many correspondences. Furthermore, a global contrastive loss with hard negative sample awareness is introduced, and cross-entropy loss is used to explicitly mine difficult samples, accelerating the training of the entire network architecture parameters and avoiding getting stuck in the model's optimal solution. The trained model can achieve high-precision location recognition from text descriptions to large-scale multi-view image maps in unknown environments, contributing an important technical tool for intelligent agents such as mobile robots and autonomous driving. Compared with related technologies, this method and system for visual localization in large-scale urban scenes using text significantly improves the accuracy and stability of cross-modal location recognition from text to multi-view images in unknown indoor and outdoor environments. Simultaneously, due to the hard negative sample awareness design, this cross-modal location recognition device has a faster convergence speed in actual training than previous technologies, saving computational costs. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of an existing self-localization model framework in the background art of this invention; Figure 2 This is a schematic diagram of the visual positioning model framework constructed for an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Example 1 This invention provides a method for visual localization in large-scale urban scenes using text, including: Collect multi-view images and location coordinate information of the intelligent agent, obtain the location text descriptions corresponding to the multi-view images through GPT, and build a database of text and multi-view image pairings; A visual localization model was constructed and trained based on a database of text and multi-view image pairings. The location text description of the agent to be located is input into the trained visual localization model to achieve visual localization of the agent.

[0020] The specific implementation process of this invention is as follows: Using intelligent agents (mobile robots equipped with cameras, driving vehicles, etc.), global closed loop (global closed loop refers to the process of introducing long-distance loop closure constraints and performing global optimization on the entire pose graph / map when the current pose is detected to have revisited a region in the previously constructed map, thereby eliminating cumulative drift and ensuring the global consistency of trajectory and map) sensor data capture indoors or outdoors. At each location, images from four directions are captured simultaneously to construct a continuous multi-view image database in the global map (multi-view image data of each location is encoded into tensors as candidates for cosine similarity calculation) and their position coordinates on the global map are recorded in real time (each location has multiple view images (taken by the camera), and recording their global position is the position coordinate of the camera sensor on the global map, which is used as ground truth coordinates to judge the quality of the trained model during training and evaluation). Subsequently, GPT was used to perform simple tagging on the collected multi-view images based on human location description habits (for example, when humans describe a location, they usually describe the surrounding buildings, landscapes, and include directional words, such as: 1. A brick building with large windows, 2. AFedEx Kinkos storefront, 3. Traffic lights and street signs indicating rules, and 4. A pedestrian walking across the street). This yielded location text descriptions corresponding to the multi-view images, and a database of text and multi-view image pairings was constructed.

[0021] The visual positioning model constructed in this invention includes: text branch, image branch, and local soft alignment module; The text branch uses T5-small as the encoder backbone to obtain textual representation information of location text descriptions; Image branching utilizes the ViT encoder to obtain image representation information from multiple visual images; The local soft alignment module is used to optimize text representation information and image representation information, and to achieve a many-to-many correspondence between text representation information and image representation information.

[0022] like Figure 2 As shown, the visual localization model constructed in this invention adopts a dual-branch architecture to achieve fine-grained alignment and global contrastive learning across text and images. The text branch extracts token sequences using a frozen pre-trained T5 encoder, first capturing contextual dependencies through a trainable Transformer block; then, it aggregates these sequences using mean pooling to generate global sentence embeddings, projects them onto the target dimension via a multilayer perceptron (MLP), and finally compresses them using a second Transformer block to improve semantic abstraction. The image branch uses a ViT-based encoder to extract patch-level embeddings, fine-tuning only the last two layers; standard optimal transport (OT) aggregates patches into clusters, flattens them along the last dimension, and projects them onto a unified feature space via an MLP. Through self-supervised learning of the local soft alignment loss in the local soft alignment module, a many-to-many correspondence between text tokens and image tokens is achieved. Furthermore, a global hard negative perceptual contrastive loss term is added to the standard symmetric InfoNCE loss to explicitly suppress hard negative samples and improve discriminability.

[0023] Specifically: The location text descriptions and multi-view images from the database that are paired with text and multi-view images are input into the corresponding text branch and image branch, respectively, to obtain text representation information (text token) and image representation information (image token) in a unified embedding space.

[0024] Based on a local soft alignment loss module, optimal bidirectional transmission of text tokens and image tokens is achieved, capturing many-to-many consistency and obtaining alignment representation information between different modalities, including: Calculate the transmission cost matrix for text-to-image and image-to-text, use the Sinkhorn algorithm to solve for the entropy-regularized optimal transmission plan, construct the soft alignment loss, and realize self-supervised many-to-many correspondence.

[0025] The algorithm of the local soft alignment module of this invention specifically includes: Given a batch of samples, represent the text token as follows: , ,in, L The total number of text tokens. D For the embedding dimension; image clustering (the process of clustering several image tokens into an image cluster using the optimal transmission algorithm is called image clustering) is represented as , ,in, K The total number of image clusters; the similarity matrix is ​​defined as: , , l =1,2,..., L , k =1,2,..., K ,in, Indicates the first The text token and the first The similarity score between image clusters (calculated from the dot product of feature vectors) Indicates the first A text token, Indicates the first Image clustering, l and k These are the index variables for text tokens and image clusters, respectively.

[0026] Predicted distribution (Token → Cluster): ; Predicted distribution (Cluster→Token): ; in, This represents the predicted probability distribution from text tokens to image clusters (Token→Cluster), specifically referring to the probability distribution given the first... In the case of the first text token, it is related to the second... Image clustering achieves aligned normalized probabilities; This represents the predicted probability distribution from image clustering to text token (Cluster→Token), specifically referring to the probability distribution given the first... In the case of image clustering, it is similar to the first image clustering. The normalized probability of alignment is achieved by each text token; Indicates the first The text token and the first Similarity scores between image clusters; Indicates the first The text token and the first Similarity scores between image clusters; , All are used as summation index variables in the denominator, used to traverse all tokens and image clusters in the text sequence, and to calculate the normalization constant of the Softmax function by accumulating the exponent values ​​of each term. The temperature parameter is used to adjust the smoothness of the probability distribution after Softmax normalization. The smaller the value, the sharper the distribution, and the more the model tends to focus only on the most similar matches; The larger the value, the smoother the distribution and the "softer" the alignment. In contrastive learning or alignment tasks, it is often used to control the model's discriminatory power towards negative samples and its focus on positive samples.

[0027] Transmission cost matrix: ; ; OT Plan: ; ; Loss function: ; ; ; in, and Let represent the transmission cost matrices from text to image and from image to text, respectively. They are constructed from the similarity matrix and its transpose, with the aim of transforming the problem of maximizing similarity into the problem of minimizing transmission cost. and It is an entropy-regularized optimal transmission plan obtained based on the Sinkhorn algorithm, which serves as a "soft label" for guiding alignment; and This represents a preset edge distribution (usually set to uniform distribution). The entropy regularization coefficient is used to control the smoothness of the distribution. This represents the number of iterations of the algorithm. and These are the cross-entropy loss functions based on text and image perspectives, respectively. and It is an entropy-regularized optimal transmission plan based on text and image perspectives. The soft alignment loss function minimizes the model's predicted distribution. With optimal transmission plan The differences between them enable local soft alignment optimization between modalities.

[0028] Furthermore, this invention introduces a global contrastive loss with hard negative sample awareness, and uses cross-entropy loss to explicitly mine hard samples, accelerating the training of the entire network architecture parameters. By explicitly suppressing hard negative samples, it avoids the model training from getting stuck in local optima to a certain extent, maintaining the representational information of paired text to multi-view images in a unified embedding space, and no longer treating every negative sample equally, thereby accelerating the convergence speed of the entire model. Specifically: Set the batch size, add a hard negative sample suppression term based on the standard symmetric InfoNCE loss, use the same temperature parameter as the standard symmetric InfoNCE loss, minimize the loss function, and update the parameters of the network modules based on backpropagation.

[0029] The hard negative sample-aware global contrastive loss algorithm of this invention specifically includes: Given a set of paired embedding vectors , , These represent image features and text features, respectively. This represents the batch size, i.e., the number of paired image-text samples in the current batch of input data; first, L2 normalization is performed on them to obtain unit vectors, and then the cosine similarity matrix is ​​calculated: , i =1,2,..., B , j =1,2,..., B ; in, The cosine similarity matrix is... and These represent the image feature unit vector and text feature unit vector after L2 normalization, respectively. Specifically, they are obtained by normalizing the original feature vectors... and In addition to their respective The norm is obtained, and this step ensures that the magnitude of the vector is 1, so that the subsequent dot product operation of the vectors is directly equivalent to the cosine similarity calculation.

[0030] Let temperature parameter The partitioning functions for the row and column directions are: , ; The symmetric InfoNCE loss is defined as: ; To explicitly model hard negatives, the probability mass of each sample assigned to the hardest negative sample in both modalities (row direction and column direction) is defined as follows: ; ; in, , They represent the first i The maximum similarity of a sample to all unpaired samples in both modal directions; Variations of hard-negative regularization: Logarithmic loss function: ; Exponential loss function: ; Mean Absolute Error Loss Function: ; Tangent loss function: ; The final global training objective is defined as: ; in, This represents any one of the above hard-negative regular expressions.

[0031] Furthermore, the location text description of the agent to be located is input into the trained visual localization model. Based on cosine similarity, the most similar visual location is retrieved from a pre-set large-scale multi-view image database, enabling the agent to determine its own position in the global map and achieving visual localization of the agent to be located.

[0032] During the retrieval process, the model initializes pre-trained weights. The location text description first enters a pre-trained and frozen T5 encoder to obtain an embedding, then passes through n trainable transformer blocks and mean pooling, and finally enters the MLP to adjust the feature dimension of the text embedding to 2048 dimensions. For all multi-view images in the map database, a location has images in four directions. Each image in one direction is processed by a pre-trained DINO V2 encoder to obtain an image patch embedding (resolution size is 14). 14) The n image patches are clustered into m image clusters through optimal transmission (n>>m). The embeddings of these n image clusters are then flattened and stitched together into a large embedding. The feature dimension is then transformed into 2048 dimensions through an MLP. In this way, all the image embeddings in the database can be multiplied by the input text embedding. The one with the largest value is considered to have the highest similarity, which is the most similar visual location found.

[0033] In summary, this invention provides a method for visual localization in large-scale urban scenes using text. It designs a dual-branch network architecture, synchronously encodes text-image pairs, and performs self-supervised training at the text token and image cluster levels through a local soft alignment module, achieving many-to-many correspondences. Furthermore, it introduces a global contrastive loss with hard negative sample awareness, and uses cross-entropy loss to explicitly mine difficult samples, accelerating the training of the entire network architecture parameters and avoiding getting stuck in the model's optimal solution. The trained model can achieve high-precision location recognition from text descriptions to large-scale multi-view image maps in unknown environments, contributing an important technical tool for intelligent agents such as mobile robots and autonomous driving. Compared with related technologies, this invention significantly improves the accuracy and stability of cross-modal location recognition from text to multi-view images in unknown indoor and outdoor environments. Simultaneously, due to the hard negative sample awareness design, this cross-modal location recognition device exhibits faster convergence speed during actual training compared to previous technologies, saving computational costs.

[0034] Example 2 Based on the same inventive concept, the present invention also provides a system for visual positioning in large-scale urban scenes using text, for implementing the method described in the foregoing embodiments. The system includes: a data acquisition module, a model building module, and a visual positioning module. The data acquisition module is used to collect multi-view images and location coordinate information of the intelligent agent, obtain location text descriptions corresponding to the multi-view images through GPT, and build a database of text and multi-view image pairings; The model building module is used to build a visual localization model and train the visual localization model based on a database of text and multi-view image pairings. The visual localization module is used to input the location text description of the agent to be localized into the trained visual localization model to achieve visual localization of the agent.

[0035] Furthermore, the visual localization model includes: text branch, image branch, and local soft alignment module; The text branch uses T5-small as the encoder backbone to obtain textual representation information of location text descriptions; Image branching utilizes the ViT encoder to obtain image representation information from multiple visual images; The local soft alignment module is used to optimize text representation information and image representation information, and to achieve a many-to-many correspondence between text representation information and image representation information.

[0036] Furthermore, the local soft alignment module optimizes text and image representation information, and the methods for achieving many-to-many correspondence between text and image representation information include: Calculate the transmission cost matrices for text-to-image and image-to-text, solve for the entropy-regularized optimal transmission plan using the Sinkhorn algorithm, construct a soft alignment loss, and achieve self-supervised many-to-many correspondences: ; ; ; in, and All are predicted probability distributions. l and k These are the index variables for text tokens and image clustering, respectively. and It is an entropy-regularized optimal transmission plan based on text and image perspectives. The soft alignment loss function is... and These are cross-entropy loss functions based on text and image perspectives, respectively.

[0037] Furthermore, the visual localization model also includes: introducing a global contrastive loss that incorporates hard negative sample perception, and using cross-entropy loss to reveal the parameters of the entire network architecture to accelerate the training of difficult samples. Set the batch size, add a hard negative sample suppression term based on the standard symmetric InfoNCE loss, use the same temperature parameter as the standard symmetric InfoNCE loss, minimize the loss function, and update the parameters of the network modules based on backpropagation; Calculate the cosine similarity matrix: ; in, The cosine similarity matrix is... , Representing image features and text features respectively. and These represent the image feature unit vector and the text feature unit vector after L2 normalization, respectively. Let temperature parameter The partitioning functions for the row and column directions are: ; Define the probability mass of assigning each sample to the hardest negative sample in both modalities as: ; ; in, , They represent the first i The maximum similarity of a sample to all unpaired samples in both modal directions; Variations of hard-negative regularization: Logarithmic loss function: ; Exponential loss function: ; Mean absolute error loss function: ; Tangent loss function: ; The final global training objective is defined as: ; in, Represents any variant of the hard-negative regularization term.

[0038] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for visual localization in large-scale urban scenes using text, characterized in that, The method includes: Collect multi-view images and location coordinate information of the intelligent agent, obtain the location text descriptions corresponding to the multi-view images through GPT, and build a database of text and multi-view image pairings; A visual localization model was constructed and trained based on a database of text and multi-view image pairings. Input the location text description of the agent to be located into the trained visual localization model to achieve visual localization of the agent. The visual localization model includes: text branch, image branch, and local soft alignment module; The text branch uses T5-small as the encoder backbone to obtain textual representation information of location text descriptions; Image branching utilizes the ViT encoder to obtain image representation information from multiple visual images; The local soft alignment module is used to optimize text representation information and image representation information, and realize many-to-many correspondence between text representation information and image representation information; The local soft alignment module optimizes text and image representation information, and methods for achieving many-to-many correspondence between text and image representation information include: Calculate the transmission cost matrix between text and image and between image and text, use the Sinkhorn algorithm to solve the entropy regularized optimal transmission plan, construct the soft alignment loss, and realize self-supervised many-to-many correspondence; The visual localization model also includes: introducing a global contrastive loss with hard negative sample perception, and using cross-entropy loss to reveal and accelerate the training of parameters for the entire network architecture by mining hard samples.

2. The method according to claim 1, characterized in that, The local soft alignment module optimizes text and image representation information, and methods for achieving many-to-many correspondence between text and image representation information include: ； ； ； in, and All are predicted probability distributions. l and k These are the index variables for text tokens and image clustering, respectively. and It is an entropy-regularized optimal transmission plan based on text and image perspectives. The soft alignment loss function is... and These are cross-entropy loss functions based on text and image perspectives, respectively.

3. The method according to claim 2, characterized in that, The visual positioning model also includes: Set the batch size, add a hard negative sample suppression term based on the standard symmetric InfoNCE loss, use the same temperature parameter as the standard symmetric InfoNCE loss, minimize the loss function, and update the parameters of the network modules based on backpropagation; Calculate the cosine similarity matrix: ； in, The cosine similarity matrix is... , These represent image features and text features, respectively. and These represent the image feature unit vector and the text feature unit vector after L2 normalization, respectively. Let temperature parameter The partitioning functions for the row and column directions are: ； Where B is the batch size, i.e., the number of paired image-text samples in the current batch of input data; the probability mass of each sample being assigned to the hardest negative sample in both modalities is defined as: ； ； in, , They represent the first The maximum similarity of a sample to all unpaired samples in both modal directions; Variations of hard-negative regularization: Logarithmic loss function: ; Exponential loss function: ; Mean absolute error loss function: ; Tangent loss function: ; The final global training objective is defined as: ； in, L represents any variant of the hard-negative regularization term. NCE The loss is a standard symmetric InfoNCE loss.

4. A system for visual localization in large-scale urban scenes using text, the system being used to implement the method according to any one of claims 1-3, characterized in that, The system includes: a data acquisition module, a model building module, and a visual positioning module; The data acquisition module is used to collect multi-view images and location coordinate information of the intelligent agent, obtain location text descriptions corresponding to the multi-view images through GPT, and build a database of text and multi-view image pairings; The model building module is used to build a visual localization model and train the visual localization model based on a database of text and multi-view image pairings. The visual localization module is used to input the location text description of the agent to be localized into the trained visual localization model to achieve visual localization of the agent.

5. The system according to claim 4, characterized in that, The visual localization model includes: text branch, image branch, and local soft alignment module; The text branch uses T5-small as the encoder backbone to obtain textual representation information of location text descriptions; Image branching utilizes the ViT encoder to obtain image representation information from multiple visual images; The local soft alignment module is used to optimize text representation information and image representation information, and to achieve a many-to-many correspondence between text representation information and image representation information.

6. The system according to claim 5, characterized in that, The local soft alignment module optimizes text and image representation information, and methods for achieving many-to-many correspondence between text and image representation information include: Calculate the transmission cost matrices for text-to-image and image-to-text, solve for the entropy-regularized optimal transmission plan using the Sinkhorn algorithm, construct a soft alignment loss, and achieve self-supervised many-to-many correspondences: ； ； ； in, and All are predicted probability distributions. l and k These are the index variables for text tokens and image clustering, respectively. and It is an entropy-regularized optimal transmission plan based on text and image perspectives. The soft alignment loss function is... and These are cross-entropy loss functions based on text and image perspectives, respectively.

7. The system according to claim 6, characterized in that, The visual localization model also includes: introducing a global contrastive loss that incorporates hard negative sample perception, and using cross-entropy loss to reveal the parameters of the entire network architecture to accelerate the training of difficult samples. Set the batch size, add a hard negative sample suppression term based on the standard symmetric InfoNCE loss, use the same temperature parameter as the standard symmetric InfoNCE loss, minimize the loss function, and update the parameters of the network modules based on backpropagation; Calculate the cosine similarity matrix: ； in, The cosine similarity matrix is... , These represent image features and text features, respectively. and These represent the image feature unit vector and the text feature unit vector after L2 normalization, respectively. Let temperature parameter The partitioning functions for the row and column directions are: ； Where B is the batch size, i.e., the number of paired image-text samples in the current batch of input data; the probability mass of each sample being assigned to the hardest negative sample in both modalities is defined as: ； ； in, , They represent the first The maximum similarity of a sample to all unpaired samples in both modal directions; Variations of hard-negative regularization: Logarithmic loss function: ; Exponential loss function: ; Mean absolute error loss function: ; Tangent loss function: ; The final global training objective is defined as: ； in, L represents any variant of the hard-negative regularization term. NCE The loss is the standard symmetric InfoNCE loss.

Images