Serialization-oriented visual location identification method and device, equipment and storage medium

Abstract

The invention relates to a serialization-oriented visual location identification method and device, equipment and a storage medium. The method comprises the following steps: inputting a batch of location image training data to a visual location identification model, extracting and splicing multiple layers of feature maps through a backbone network, inputting a first feature mixing unit of a shared sequence branch network, and adjusting parameters of the first feature mixing unit by im2im loss. A batch of sequence image training data is input, a backbone network extracts feature maps and splices the feature maps, then the feature maps are respectively input into sharing and special sequence branch networks to obtain a plurality of descriptors, the sharing and special sequence descriptors are spliced to calculate seq2seq loss, and attention separation loss is calculated according to the sharing and special image descriptors, so that the parameters of the two branch networks are adjusted. Using the image and the sequence location to identify each batch of data training model of the data set to obtain a trained model; a visual site sequence descriptor is generated by using the model, and a site identification result is retrieved, so that the distinction degree and the positioning precision of the sequence descriptor can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for visual location recognition oriented towards serialization. Background Technology

[0002] Visual Place Recognition (VPR) is a key capability for autonomous mobile robots, enabling them to identify previously seen locations even when the Global Positioning System (GPS) is unavailable. This technology achieves this by matching current visual observations (whether images or video streams) with stored geotagged locations. Research in this field plays a crucial role in simultaneous localization and mapping, augmented reality, and autonomous driving.

[0003] Based on matching characteristics, Visual Location Recognition (VPR) is mainly divided into two categories: image-to-image VPR (im2im) and sequence-to-sequence VPR (seq2seq). im2im is considered a retrieval problem. Both the query image and individual images in the database are converted into global descriptors, and then the database image most similar to the query image is retrieved through nearest neighbor search. To utilize the inherent temporal information in robot motion, seq2seq was proposed, where both the query and database image sequences are converted into matchable sequence descriptors. However, labeled sequence data is far scarcer than labeled individual images. A recent Joint Image and Sequence Training (JIST) method addresses this problem by using a unified network architecture optimized for joint training on seq2seq and im2im tasks. However, the generated sequence descriptors are highly dependent on image descriptors trained on the im2im task, reducing their discriminative power. This leads to decreased matching accuracy, reduced retrieval robustness, and the inability to leverage the advantages of sequences. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, device, and storage medium for serialized visual location recognition that can improve the discriminative power of generated sequence descriptors and thus enhance the positioning accuracy of sequence location recognition, in order to address the aforementioned technical problems.

[0005] A method for visual location recognition oriented towards serialization, the method comprising: A visual location recognition model is constructed. The visual location recognition network includes a parameter-frozen backbone network and two independent sequence branch networks, which respectively include a dedicated sequence branch network and a shared sequence branch network. Obtain an image location recognition dataset and a sequence location recognition dataset, wherein the image location recognition dataset includes multiple batches of location image training data, and the sequence location recognition dataset includes multiple batches of sequence image training data; A batch of location image training data is input into the visual location recognition model. First, the backbone network extracts multi-layer feature maps of the location images. After the multi-layer feature maps are concatenated, they are input into the first feature mixing unit of the shared sequence branch network. Then, the im2im loss is calculated through the im2im branch, and the adjustable parameters in the first feature mixing unit are adjusted using the im2im loss. In the im2im branch, the image descriptor is obtained by dimensionality reduction through the GeM pooling layer and the fully connected layer. Then, the im2im loss is obtained by using the difference between the image descriptor at the same location and different locations. A batch of sequential image training data is input into the visual location recognition model. The multi-layer feature maps extracted by the backbone network are concatenated and then input into the shared sequence branch network and the dedicated sequence branch network respectively to obtain shared image descriptors, shared sequence descriptors, dedicated image descriptors, and dedicated sequence descriptors. After concatenating the shared sequence descriptors and dedicated sequence descriptors, the seq2seq loss is calculated. Attention separation loss is also calculated based on the shared image descriptors and dedicated image descriptors. The adjustable parameters in the dedicated sequence branch network and the shared sequence branch network are adjusted using the seq2seq loss and attention separation loss. The visual location recognition model is trained sequentially using the image location recognition dataset and the batches of recognition datasets in the sequence location recognition dataset to obtain a trained visual location recognition model. A sequence of visual locations to be identified is obtained, and a sequence descriptor of the visual location sequence is obtained using the trained visual location recognition model. The location recognition result is obtained by searching based on the sequence descriptor.

[0006] In one embodiment, the location image training data in each batch includes multiple images of the same location taken from different angles, as well as multiple images of different locations; Each batch of training data contains multiple image sequences for the same location and multiple image sequences for different locations.

[0007] In one embodiment, the dedicated sequence branch network and the shared sequence branch network have the same structure, each including a feature mixing unit, an L2 normalization layer, a GeM pooling layer, a SeqGeM aggregation layer, and a fully connected layer. In the dedicated sequence branch network and the shared sequence branch network: The input data is processed by the feature mixing unit to obtain a feature map, and then passed through the L2 normalization layer. The feature map is passed through a GeM pooling layer and an L2 normalization layer in sequence to obtain an image descriptor; The image descriptor is passed sequentially through a SeqGeM aggregation layer, a fully connected layer, and an L2 normalization layer to obtain a sequence descriptor; The feature mixing unit in the shared sequence branch network is the first feature mixing unit.

[0008] In one embodiment, the feature mixing unit includes a convolutional layer, a ReLU function layer, and a core mixing logic layer connected in sequence; The multi-layer feature map first undergoes basic transformations via the convolutional layer and the ReLU function layer to obtain enhanced features; The core hybrid logic layer performs dimensional rearrangement of the enhanced features by transposing them, and after layer normalization, it uses a multilayer perceptron to perform cross-dimensional association to obtain the cross-dimensional hybrid output features.

[0009] In one embodiment, the im2im loss employs the CosFace loss function.

[0010] In one embodiment, the seq2seq loss employs the Triplet loss function.

[0011] In one embodiment, when calculating the attention separation loss: After performing matrix multiplication on the shared feature map and the shared image descriptor, the image descriptor is processed by the L2 normalization layer, and the first attention vector is obtained by measuring the similarity between the image descriptor and each position of the image feature using cosine similarity. After performing matrix multiplication on the dedicated feature map and dedicated image descriptor, the image descriptor is processed by an L2 normalization layer, and the second attention vector is obtained by measuring the similarity between the image descriptor and each position of the image feature through cosine similarity. The attention separation loss is calculated using the cosine distance between the first attention vector and the second attention vector.

[0012] This application also provides a visual location recognition device for serialization, the device comprising: A visual location recognition model construction module is used to construct a visual location recognition model. The visual location recognition network includes a parameter-frozen backbone network and two independent sequence branch networks, which respectively include a dedicated sequence branch network and a shared sequence branch network. The training dataset acquisition module is used to acquire image location recognition dataset and sequence location recognition dataset. The image location recognition dataset includes multiple batches of location image training data, and the sequence location recognition dataset includes multiple batches of sequence image training data. The first feature mixing unit training module is used to input a batch of location image training data into the visual location recognition model. First, the backbone network extracts multi-layer feature maps of the location images. After the multi-layer feature maps are concatenated, they are input into the first feature mixing unit of the shared sequence branch network. Then, the im2im loss is calculated through the im2im branch, and the adjustable parameters in the first feature mixing unit are adjusted using the im2im loss. In the im2im branch, the image descriptor is obtained by dimensionality reduction through the GeM pooling layer and the fully connected layer. Then, the im2im loss is obtained by using the difference between the image descriptor at the same location and different locations. The dual-sequence branch network training module is used to input a batch of sequential image training data into the visual location recognition model. After concatenating the multi-layer feature maps extracted by the backbone network, the modules are input into the shared sequence branch network and the dedicated sequence branch network respectively to obtain shared image descriptors, shared sequence descriptors, dedicated image descriptors, and dedicated sequence descriptors. After concatenating the shared sequence descriptors and dedicated sequence descriptors, the seq2seq loss is calculated. Furthermore, the attention separation loss is calculated based on the shared image descriptors and dedicated image descriptors. The adjustable parameters in the dedicated sequence branch network and the shared sequence branch network are adjusted using the seq2seq loss and the attention separation loss. The model iterative training module is used to train the visual location recognition model sequentially using the image location recognition dataset and the batches of recognition datasets in the sequence location recognition dataset to obtain the trained visual location recognition model. The visual location recognition module is used to acquire a sequence of visual locations to be recognized, obtain a sequence descriptor of the visual location sequence using the trained visual location recognition model, and retrieve the location recognition result based on the sequence descriptor.

[0013] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the specific steps in the above-described serialization-oriented visual location recognition method.

[0014] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the specific steps of the above-described serialization-oriented visual location recognition method.

[0015] The aforementioned method, apparatus, device, and storage medium for serialized visual location recognition, by inputting a batch of location image training data into a visual location recognition model, firstly, the backbone network extracts multi-layer feature maps of the location images, then concatenates these multi-layer feature maps and inputs them into the first feature mixing unit of the shared sequence branch network, then calculates the im2im loss through the im2im branch, and adjusts the adjustable parameters in the first feature mixing unit using the im2im loss. Then, a batch of sequential image training data is input into the visual location recognition model again, and the multi-layer feature maps extracted by the backbone network are concatenated and input into the shared sequence branch network and the dedicated sequence branch network respectively, to obtain shared image descriptors and shared sequence descriptors. This method employs dedicated image descriptors and dedicated sequence descriptors, concatenating the shared sequence descriptors and dedicated sequence descriptors, and calculating a seq2seq loss. It also calculates an attention separation loss based on the shared and dedicated image descriptors. The adjustable parameters in the dedicated sequence branch network and the shared sequence branch network are adjusted using the seq2seq loss and attention separation loss. The visual location recognition model is trained sequentially using batches of recognition datasets from both the image location recognition dataset and the sequence location recognition dataset, resulting in a trained visual location recognition model. Finally, the sequence descriptors of the visual location sequences are obtained using the trained visual location recognition model, and the location recognition result is obtained by searching based on the sequence descriptors. This method can improve the discriminative power of the generated sequence descriptors, thereby improving the localization accuracy of sequence location recognition. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a visual location recognition method for serialization in one embodiment; Figure 2 This is a schematic diagram of the structure of a visual location recognition model in one embodiment; Figure 3 This is a flowchart illustrating the model training steps in one embodiment of the method. Figure 4 This is a schematic diagram of the experimental results in an experiment; Figure 5 This is a structural block diagram of a visual location recognition device for serialization in one embodiment; Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0017] In addressing the issue that during the optimization of neural networks jointly trained on seq2seq and im2im tasks, the generated sequence descriptors are highly dependent on the image descriptors trained on the im2im task, leading to reduced discriminative power, this application addresses this problem. Figure 1 As shown, a method for visual location recognition oriented towards serialization is provided, which specifically includes the following steps: Step S100: Construct a visual location recognition model. The visual location recognition network includes a parameter-frozen backbone network and two independent sequence branch networks, namely a dedicated sequence branch network and a shared sequence branch network.

[0018] Step S110: Obtain the image location recognition dataset and the sequence location recognition dataset. The image location recognition dataset includes multiple batches of location image training data, and the sequence location recognition dataset includes multiple batches of sequence image training data.

[0019] Step S120: Input a batch of location image training data into the visual location recognition model. First, the backbone network extracts multi-layer feature maps of the location images. After the multi-layer feature maps are concatenated, they are input into the first feature mixing unit of the shared sequence branch network. Then, the im2im loss is calculated through the im2im branch, and the adjustable parameters in the first feature mixing unit are adjusted using the im2im loss. In the im2im branch, the image descriptor is obtained by dimensionality reduction through the GeM pooling layer and the fully connected layer. Then, the im2im loss is obtained by using the difference between the image descriptor at the same location and different locations.

[0020] Step S130: Input a batch of sequence image training data into the visual location recognition model. After concatenating the multi-layer feature maps extracted by the backbone network, input them into the shared sequence branch network and the dedicated sequence branch network respectively to obtain shared image descriptors, shared sequence descriptors, dedicated image descriptors, and dedicated sequence descriptors. After concatenating the shared sequence descriptors and dedicated sequence descriptors, calculate the seq2seq loss. Also, calculate the attention separation loss based on the shared image descriptors and dedicated image descriptors. Use the seq2seq loss and attention separation loss to adjust the adjustable parameters in the dedicated sequence branch network and the shared sequence branch network.

[0021] Step S140: The visual location recognition model is trained sequentially using the recognition datasets of each batch in the image location recognition dataset and the sequence location recognition dataset to obtain the trained visual location recognition model.

[0022] Step S150: Obtain the visual location sequence to be identified, use the trained visual location recognition model to obtain the sequence descriptor of the visual location sequence, and perform retrieval based on the sequence descriptor to obtain the location recognition result.

[0023] This application provides an image-sequence decoupled joint training method for sequential visual location recognition, employing a frozen backbone network and two independent sequence branches. One branch is supervised by both im2im and seq2seq losses, while the other branch is supervised only by seq2seq loss. Furthermore, an attention separation loss is introduced between the two sequence branches, forcing different branches to focus on different parts of the image, thereby generating more informative sequence descriptors to overcome the problem of low discriminative power.

[0024] In step S100, the constructed visual location recognition model includes a frozen backbone network and two independent sequence branch networks. The backbone network uses the large visual model DINOv2. During model training, the parameters are kept frozen to ensure the sequence branch networks remain independent. The model structure is as follows: Figure 2 As shown, in Figure 2 The red line represents the data flow when the input is an image location recognition dataset, the blue line represents the data flow when the input is a sequence location recognition dataset, the green line points to an optimizable model structure that can pass through both data flows, and the black line represents a non-optimizable (i.e., frozen) model structure. The inputs of the two sequence branch networks are concatenated with the final M-layer block features of the backbone network output, meaning that the multi-layer feature maps output by the backbone network are the input data for the two sequence branch networks.

[0025] In step S110, when training the visual location recognition model, two training datasets are used: an image location recognition dataset and a sequence location recognition dataset. Both training datasets include multiple batches of training data. Each batch of image training data contains multiple images of the same location taken from different angles, as well as multiple images of different locations. Similarly, each batch of sequence image training data contains multiple sequences of images of the same location, as well as multiple sequences of images of different locations.

[0026] In this embodiment, before inputting the image location recognition dataset into the visual location recognition model, it is preprocessed by dividing the dataset, which contains UTM coordinates and heading angle labels, into different categories based on location and orientation. Each cell is divided into 12 categories along the direction / heading, i.e., the geographic region is divided into small 10×10 meter square cells, with each category being 30° wide. This ensures that all images in the same category observe the same scene (because they have similar locations and orientations). This method of dividing the continuous label space into a finite number of categories enables the use of highly scalable loss functions in large-scale image retrieval.

[0027] Furthermore, considering that directly using all classes would be problematic when training a shared sequence branch network using, for example, CosFace loss as an im2im loss, because images in adjacent classes may visually overlap significantly, leading to gradient instability during optimization, training only considers images from a subset. This subset does not simultaneously use adjacent classes. Specifically, during preprocessing, each cell is divided into 12 classes, and the subset uses 6, 4, or 3 classes, excluding adjacent classes. This subset is not fixed but iteratively changes during training to traverse all images in the dataset.

[0028] Specifically, the SF-XL dataset can be used for image location recognition.

[0029] In step S120, the training data of location images from each batch in the image location recognition dataset are input into the visual location recognition model. After multi-layer feature extraction by the backbone network, the first feature mixing unit of the shared sequence branch network performs multi-layer feature mixing enhancement and then enters the im2im branch to perform im2im loss calculation. The calculation results are then used to train the first feature mixing unit.

[0030] In this embodiment, the im2im loss uses the CosFace loss function. In other embodiments, loss functions such as the Triplet loss function and the Contrastive loss function may also be used.

[0031] Specifically, in the im2im branch, after the enhanced features output by the first feature mixing unit are processed by the L2 normalization layer, a feature map is obtained. This feature map is then passed through the GeM pooling layer, the fully connected layer, and the L2 normalization layer to obtain an image descriptor. The CosFace loss function is then calculated using the image descriptor and the ground truth label.

[0032] In step S130, using the sequence location recognition dataset, supervised training is performed on the network parameters of the dedicated sequence branch network and the shared sequence branch network using seq2seq loss and attention separation loss. The dedicated sequence branch network and the shared sequence branch network have the same structure, including feature mixing units, L2 normalization layers, GeM pooling layers, SeqGeM aggregation layers, and fully connected layers, respectively.

[0033] Specifically, in the dedicated sequence branch network and the shared sequence branch network: the input data is feature-mixed by the feature mixing unit, and then passes through the L2 normalization layer to obtain a feature map. The feature map then passes through the GeM pooling layer and the L2 normalization layer to obtain an image descriptor. The image descriptor then passes through the SeqGeM aggregation layer, the fully connected layer, and the L2 normalization layer to obtain a sequence descriptor. The feature mixing unit in the shared sequence branch network is the first feature mixing unit mentioned above, then refer to... Figure 2 The feature mixing unit in the dedicated sequence branch network is the second feature mixing unit. These two feature mixing units have the same model structure and independent network weights.

[0034] Furthermore, the feature mixing unit includes a convolutional layer, a ReLU function layer, and a core mixing logic layer connected in sequence. The multi-layer feature map first undergoes basic transformations through the convolutional layer and the ReLU function layer to obtain enhanced features. The core mixing logic layer then performs dimensional rearrangement of the enhanced features by transposing them and normalizing them through layers. Finally, a multilayer perceptron is used to perform cross-dimensional association to obtain the cross-dimensional mixed output features.

[0035] Specifically, the feature mixing unit uses channel and spatial feature fusion and includes a 1×1 convolution, a ReLU activation function, and two block mixing layers, namely the core mixing logic layer.

[0036] In this embodiment, the Triplet loss function is used as the seq2seq loss. In other embodiments, loss functions such as the Multi-Similarity loss function and the Contrastive loss function can also be used. The Triplet loss function is calculated after concatenating the sequence descriptors output by the dedicated sequence branch network and the shared sequence branch network.

[0037] Specifically, the Triplet loss function comprises three elements: a query sample, a positive sample (i.e., a sequence from the same position as the query sample), and a negative sample. To achieve optimal training results, "hard negative samples" are selected, rather than randomly chosen negative samples. This is because randomly selected negative samples tend to form trivial triples—those triples with a loss value of zero. Specifically, negative samples are selected based on their proximity to the query sample in the feature space. After determining the triples consisting of the query sample, positive sample, and negative sample, the Triplet loss function is defined as follows:

[0038] In the above formula, , , These represent the features of the query sample, positive sample, and negative sample, respectively. This represents the marginal value of the triplet loss. This represents the Euclidean distance between two features.

[0039] To encourage the two sequence branches to focus on different key regions in the image, this method proposes an attention separation loss, which is calculated only when the input is a sequence of images.

[0040] In this embodiment, a batch of sequential image training data is input into the visual location recognition model. At that time, among them, For sequence length, , , These represent the image's length, width, and number of channels, respectively. First, the frozen backbone network... Then, they are respectively passed through the first feature mixing unit in the shared sequence branch network and the dedicated sequence branch network. Mixed with the second feature unit Enhanced features were obtained respectively. and The corresponding image descriptors are obtained by performing GeM global pooling operations on both. and Then, based on the image descriptor and and enhanced features and Calculate the attention separation loss.

[0041] Specifically, when calculating the attention separation loss: after performing matrix multiplication on the shared feature map and the shared image descriptor, the data is processed by an L2 normalization layer, and the first attention vector is obtained by measuring the similarity between the image descriptor and each position of the image feature using cosine similarity. Simultaneously, after performing matrix multiplication on the dedicated feature maps and dedicated image descriptors, and processing them through an L2 normalization layer, a second attention vector is obtained by measuring the similarity between the image descriptors and image features at each position using cosine similarity. Through the first attention vector and the second attention vector The cosine distance between them is used to calculate the attention separation loss. Furthermore, the specific process of calculating the attention separation loss after inputting the sequence image training data into the visual location recognition model can be represented as follows:

[0042] In the above formula, Indicates a branch index. Corresponding sequence position, Corresponding spatial location, Corresponding channel dimension.

[0043] After L2 normalization, the attention separation loss is achieved by maximizing the cosine distance between the attention vectors output by the two branches:

[0044] In this embodiment, the total loss function for training the visual location recognition model is expressed as:

[0045] In the above formula, , , This is a hyperparameter.

[0046] In step 140, during iterative training of the visual location recognition model, in each iteration, the first feature mixing unit and the two sequence branch networks are trained sequentially using different batches of training datasets from the image location recognition dataset and the sequence location recognition dataset, until the termination criteria are met. The training process is as follows: Figure 3 As shown.

[0047] In step S150, after the visual location recognition model is trained, the im2im branch is removed, leaving only the backbone network and two independent sequence branch networks. The location sequence data to be recognized is input into the visual location recognition model, and the two sequence branch networks output two sequence descriptors respectively. These two sequence descriptors are concatenated and then searched in the map sequence database to achieve the recognition of the current location.

[0048] This paper also demonstrates the effectiveness of the proposed method through experiments. In the experiments, the MSLS training set and the SF-XL training set were used to train the seq2seq task and the im2im task, respectively, and the MSLS validation set was used to test the model. The evaluation criterion adopted is Recall@N, commonly used in sequence visual location recognition tasks, defined as the percentage of query images in which at least one of the first N predictions originates from the same location as the query image. If the distance between at least one frame of a prediction and a frame of a query is less than 25 meters, the prediction is considered correct. In this embodiment, the JIST training method is compared with the DJIST training method proposed in this invention. Both use DINOv2 as the backbone network. Figure 4 As shown, the DJIST method has a significantly higher Recall@N value than the JIST method for different N values.

[0049] In the aforementioned methods for sequence-based visual location recognition, a decoupled joint training method for image and sequence is proposed to improve the discriminative power of the generated sequence descriptors, thereby enhancing the localization accuracy of sequence location recognition. This method constructs a visual location recognition model containing a frozen backbone network and two independent sequence branches. Input image data is processed through the feature mixing module in the frozen backbone network and the shared sequence branch, then through a GeM pooling layer and a fully connected layer to generate image descriptors. The difference between descriptors for the same location and those for different locations is calculated to obtain an im2im loss, which is used to optimize the weights in the shared sequence branch. Input sequence data is processed separately through the frozen backbone network and the two sequence branches to obtain two parts of sequence descriptors. These parts are combined to generate the final sequence descriptor. The difference between the descriptors for the same location and those for different locations is calculated to obtain a seq2seq loss. Simultaneously, an attention separation loss is constructed to encourage the two sequence branches to focus on different regions of the same data. The seq2seq loss and attention separation loss are backpropagated to optimize the weights of the two sequence branches. The input of image and sequence data is performed alternately until a specified number of iterations is reached, at which point training stops. Compared with existing methods, this method can significantly improve the localization accuracy of sequence location identification. It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0050] In one embodiment, such as Figure 5 As shown, a visual location recognition device for serialization is provided, comprising: a visual location recognition model building module 200, a training dataset acquisition module 210, a first feature mixing unit training module 220, a dual-sequence branch network training module 230, a model iterative training module 240, and a visual location recognition module 250, wherein: The visual location recognition model construction module 200 is used to construct a visual location recognition model. The visual location recognition network includes a parameter-frozen backbone network and two independent sequence branch networks, which respectively include a dedicated sequence branch network and a shared sequence branch network. The training dataset acquisition module 210 is used to acquire an image location recognition dataset and a sequence location recognition dataset. The image location recognition dataset includes multiple batches of location image training data, and the sequence location recognition dataset includes multiple batches of sequence image training data. The first feature mixing unit training module 220 is used to input a batch of location image training data into the visual location recognition model. First, the backbone network extracts multi-layer feature maps of the location images. After the multi-layer feature maps are concatenated, they are input into the first feature mixing unit of the shared sequence branch network. Then, the im2im loss is calculated through the im2im branch, and the adjustable parameters in the first feature mixing unit are adjusted using the im2im loss. In the im2im branch, the image descriptor is obtained by dimensionality reduction through the GeM pooling layer and the fully connected layer. Then, the im2im loss is obtained by using the difference between the image descriptor at the same location and different locations. The dual-sequence branch network training module 230 is used to input a batch of sequence image training data into the visual location recognition model. After concatenating the multi-layer feature maps extracted by the backbone network, it inputs them into the shared sequence branch network and the dedicated sequence branch network respectively to obtain shared image descriptors, shared sequence descriptors, dedicated image descriptors, and dedicated sequence descriptors. After concatenating the shared sequence descriptors and dedicated sequence descriptors, it calculates the seq2seq loss. It also calculates the attention separation loss based on the shared image descriptors and dedicated image descriptors. The adjustable parameters in the dedicated sequence branch network and the shared sequence branch network are adjusted using the seq2seq loss and the attention separation loss. The model iterative training module 240 is used to train the visual location recognition model sequentially using the recognition datasets of each batch in the image location recognition dataset and the sequence location recognition dataset to obtain the trained visual location recognition model. The visual location recognition module 250 is used to acquire a sequence of visual locations to be recognized, obtain a sequence descriptor of the visual location sequence using the trained visual location recognition model, and retrieve the location recognition result based on the sequence descriptor.

[0051] Specific limitations regarding the serialization-oriented visual location recognition device can be found in the limitations of the serialization-oriented visual location recognition method described above, and will not be repeated here. Each module in the aforementioned serialization-oriented visual location recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0052] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a serialization-oriented visual location recognition method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0053] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0054] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: A visual location recognition model is constructed. The visual location recognition network includes a parameter-frozen backbone network and two independent sequence branch networks, which respectively include a dedicated sequence branch network and a shared sequence branch network. Obtain an image location recognition dataset and a sequence location recognition dataset, wherein the image location recognition dataset includes multiple batches of location image training data, and the sequence location recognition dataset includes multiple batches of sequence image training data; A batch of location image training data is input into the visual location recognition model. First, the backbone network extracts multi-layer feature maps of the location images. After the multi-layer feature maps are concatenated, they are input into the first feature mixing unit of the shared sequence branch network. Then, the im2im loss is calculated through the im2im branch, and the adjustable parameters in the first feature mixing unit are adjusted using the im2im loss. In the im2im branch, the image descriptor is obtained by dimensionality reduction through the GeM pooling layer and the fully connected layer. Then, the im2im loss is obtained by using the difference between the image descriptor at the same location and different locations. A batch of sequential image training data is input into the visual location recognition model. The multi-layer feature maps extracted by the backbone network are concatenated and then input into the shared sequence branch network and the dedicated sequence branch network respectively to obtain shared image descriptors, shared sequence descriptors, dedicated image descriptors, and dedicated sequence descriptors. After concatenating the shared sequence descriptors and dedicated sequence descriptors, the seq2seq loss is calculated. Attention separation loss is also calculated based on the shared image descriptors and dedicated image descriptors. The adjustable parameters in the dedicated sequence branch network and the shared sequence branch network are adjusted using the seq2seq loss and attention separation loss. The visual location recognition model is trained sequentially using the image location recognition dataset and the batches of recognition datasets in the sequence location recognition dataset to obtain a trained visual location recognition model. A sequence of visual locations to be identified is obtained, and a sequence descriptor of the visual location sequence is obtained using the trained visual location recognition model. The location recognition result is obtained by searching based on the sequence descriptor.

[0055] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: A visual location recognition model is constructed. The visual location recognition network includes a parameter-frozen backbone network and two independent sequence branch networks, which respectively include a dedicated sequence branch network and a shared sequence branch network. Obtain an image location recognition dataset and a sequence location recognition dataset, wherein the image location recognition dataset includes multiple batches of location image training data, and the sequence location recognition dataset includes multiple batches of sequence image training data; A batch of location image training data is input into the visual location recognition model. First, the backbone network extracts multi-layer feature maps of the location images. After the multi-layer feature maps are concatenated, they are input into the first feature mixing unit of the shared sequence branch network. Then, the im2im loss is calculated through the im2im branch, and the adjustable parameters in the first feature mixing unit are adjusted using the im2im loss. In the im2im branch, the image descriptor is obtained by dimensionality reduction through the GeM pooling layer and the fully connected layer. Then, the im2im loss is obtained by using the difference between the image descriptor at the same location and different locations. A batch of sequential image training data is input into the visual location recognition model. The multi-layer feature maps extracted by the backbone network are concatenated and then input into the shared sequence branch network and the dedicated sequence branch network respectively to obtain shared image descriptors, shared sequence descriptors, dedicated image descriptors, and dedicated sequence descriptors. After concatenating the shared sequence descriptors and dedicated sequence descriptors, the seq2seq loss is calculated. Attention separation loss is also calculated based on the shared image descriptors and dedicated image descriptors. The adjustable parameters in the dedicated sequence branch network and the shared sequence branch network are adjusted using the seq2seq loss and attention separation loss. The visual location recognition model is trained sequentially using the image location recognition dataset and the batches of recognition datasets in the sequence location recognition dataset to obtain a trained visual location recognition model. A sequence of visual locations to be identified is obtained, and a sequence descriptor of the visual location sequence is obtained using the trained visual location recognition model. The location recognition result is obtained by searching based on the sequence descriptor.

[0056] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0057] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0058] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for visual location recognition oriented towards serialization, characterized in that, The method includes: A visual location recognition model is constructed. The visual location recognition network includes a parameter-frozen backbone network and two independent sequence branch networks, which respectively include a dedicated sequence branch network and a shared sequence branch network. Obtain an image location recognition dataset and a sequence location recognition dataset, wherein the image location recognition dataset includes multiple batches of location image training data, and the sequence location recognition dataset includes multiple batches of sequence image training data; A batch of location image training data is input into the visual location recognition model. First, the backbone network extracts multi-layer feature maps of the location images. After the multi-layer feature maps are concatenated, they are input into the first feature mixing unit of the shared sequence branch network. Then, the im2im loss is calculated through the im2im branch, and the adjustable parameters in the first feature mixing unit are adjusted using the im2im loss. In the im2im branch, the image descriptor is obtained by dimensionality reduction through the GeM pooling layer and the fully connected layer. Then, the im2im loss is obtained by using the difference between the image descriptor at the same location and different locations. A batch of sequential image training data is input into the visual location recognition model. The multi-layer feature maps extracted by the backbone network are concatenated and then input into the shared sequence branch network and the dedicated sequence branch network respectively to obtain shared image descriptors, shared sequence descriptors, dedicated image descriptors, and dedicated sequence descriptors. After concatenating the shared sequence descriptors and dedicated sequence descriptors, the seq2seq loss is calculated. Attention separation loss is also calculated based on the shared image descriptors and dedicated image descriptors. The adjustable parameters in the dedicated sequence branch network and the shared sequence branch network are adjusted using the seq2seq loss and attention separation loss. The visual location recognition model is trained sequentially using the image location recognition dataset and the batches of recognition datasets in the sequence location recognition dataset to obtain a trained visual location recognition model. A sequence of visual locations to be identified is obtained, and a sequence descriptor of the visual location sequence is obtained using the trained visual location recognition model. The location recognition result is obtained by searching based on the sequence descriptor.

2. The visual location recognition method for serialization according to claim 1, characterized in that, Each batch of location image training data includes multiple images taken from different angles at the same location, as well as multiple images taken at different locations; Each batch of training data contains multiple image sequences for the same location and multiple image sequences for different locations.

3. The visual location recognition method for serialization according to claim 2, characterized in that, The dedicated sequence branch network and the shared sequence branch network have the same structure, including feature mixing units, L2 normalization layers, GeM pooling layers, SeqGeM aggregation layers, and fully connected layers, respectively. In the dedicated sequence branch network and the shared sequence branch network: The input data is processed by the feature mixing unit to obtain a feature map, and then passed through the L2 normalization layer. The feature map is passed through a GeM pooling layer and an L2 normalization layer in sequence to obtain an image descriptor; The image descriptor is passed sequentially through a SeqGeM aggregation layer, a fully connected layer, and an L2 normalization layer to obtain a sequence descriptor; The feature mixing unit in the shared sequence branch network is the first feature mixing unit.

4. The visual location recognition method for serialization according to claim 3, characterized in that, The feature mixing unit includes a convolutional layer, a ReLU function layer, and a core mixing logic layer connected in sequence. The multi-layer feature map first undergoes basic transformations via the convolutional layer and the ReLU function layer to obtain enhanced features; The core hybrid logic layer performs dimensional rearrangement of the enhanced features by transposing them, and after layer normalization, it uses a multilayer perceptron to perform cross-dimensional association to obtain the cross-dimensional hybrid output features.

5. The visual location recognition method for serialization according to claim 4, characterized in that, The im2im loss uses the CosFace loss function.

6. The visual location recognition method for serialization according to claim 5, characterized in that, The seq2seq loss uses the Triplet loss function.

7. The visual location recognition method for serialization according to claim 6, characterized in that, When calculating the attention separation loss: After performing matrix multiplication on the shared feature map and the shared image descriptor, the image descriptor is processed by the L2 normalization layer, and the first attention vector is obtained by measuring the similarity between the image descriptor and each position of the image feature using cosine similarity. After performing matrix multiplication on the dedicated feature map and dedicated image descriptor, the image descriptor is processed by an L2 normalization layer, and the second attention vector is obtained by measuring the similarity between the image descriptor and each position of the image feature through cosine similarity. The attention separation loss is calculated using the cosine distance between the first attention vector and the second attention vector.

8. A visual location recognition device for serialization, characterized in that, The device includes: A visual location recognition model construction module is used to construct a visual location recognition model. The visual location recognition network includes a parameter-frozen backbone network and two independent sequence branch networks, which respectively include a dedicated sequence branch network and a shared sequence branch network. The training dataset acquisition module is used to acquire image location recognition dataset and sequence location recognition dataset. The image location recognition dataset includes multiple batches of location image training data, and the sequence location recognition dataset includes multiple batches of sequence image training data. The first feature mixing unit training module is used to input a batch of location image training data into the visual location recognition model. First, the backbone network extracts multi-layer feature maps of the location images. After the multi-layer feature maps are concatenated, they are input into the first feature mixing unit of the shared sequence branch network. Then, the im2im loss is calculated through the im2im branch, and the adjustable parameters in the first feature mixing unit are adjusted using the im2im loss. In the im2im branch, the image descriptor is obtained by dimensionality reduction through the GeM pooling layer and the fully connected layer. Then, the im2im loss is obtained by using the difference between the image descriptor at the same location and different locations. The dual-sequence branch network training module is used to input a batch of sequential image training data into the visual location recognition model. After concatenating the multi-layer feature maps extracted by the backbone network, the modules are input into the shared sequence branch network and the dedicated sequence branch network respectively to obtain shared image descriptors, shared sequence descriptors, dedicated image descriptors, and dedicated sequence descriptors. After concatenating the shared sequence descriptors and dedicated sequence descriptors, the seq2seq loss is calculated. Furthermore, the attention separation loss is calculated based on the shared image descriptors and dedicated image descriptors. The adjustable parameters in the dedicated sequence branch network and the shared sequence branch network are adjusted using the seq2seq loss and the attention separation loss. The model iterative training module is used to train the visual location recognition model sequentially using the image location recognition dataset and the batches of recognition datasets in the sequence location recognition dataset to obtain the trained visual location recognition model. The visual location recognition module is used to acquire a sequence of visual locations to be recognized, obtain a sequence descriptor of the visual location sequence using the trained visual location recognition model, and retrieve the location recognition result based on the sequence descriptor.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images