Training of visual location recognition models, visual location recognition methods and devices

Abstract

This invention provides a training method and apparatus for a visual location recognition model, belonging to the field of robot visual localization technology. It includes: extracting a global aggregated descriptor from a query image and its positive and negative sample images; constructing difference features between the query image and the positive sample descriptors; encoding and generating reference condition features and reconstructed difference features; calibrating the global aggregated descriptor sequence using the reference condition features; calculating the comparison and reconstruction loss based on the calibration results and reconstructed difference features; and updating the visual location recognition model. This invention explicitly models the appearance differences of the same physical location under different conditions, using these differences as reference conditions to perform residual calibration on the descriptors. This enables the model to learn a more stable and consistent representation of the location, effectively overcoming feature fluctuations caused by environmental changes such as viewing angle and lighting. During the testing phase, no encoding and calibration modules are required, significantly improving the robustness of visual location recognition without increasing inference computation.

Description

Technical Field

[0001] This invention relates to the field of robot visual positioning technology, and in particular to a training method and apparatus for visual location recognition model, visual location recognition method, and apparatus. Background Technology

[0002] Robot visual location recognition is a crucial foundation for robot localization, relocalization, and loop closure detection. The overall technical requirement is to accurately match the currently acquired query image with a pre-established database of images labeled with location information, thereby determining the physical location corresponding to the currently queried image.

[0003] To meet the aforementioned location identification requirements, existing technologies typically first utilize convolutional neural networks to extract local features from images, and then generate image-level descriptors through global feature aggregation methods. During the model training phase, existing methods mainly rely on directly constraining the descriptor distance between the query image and positive and negative sample images in the training samples to optimize the network, and complete image retrieval based on descriptor similarity during the testing phase.

[0004] However, in real-world robot operating scenarios, environmental changes such as perspective, lighting, and weather often occur, causing significant alterations in the appearance of images corresponding to the same physical location. Existing model training constraints struggle to effectively adapt global descriptors to these environmental variations when faced with image feature fluctuations at the same location under different shooting conditions. This results in poor descriptor stability and a tendency to mismatch visual locations. Summary of the Invention

[0005] This invention provides a training method and apparatus for a visual location recognition model, addressing the shortcomings of existing visual location recognition methods that fail to explicitly model descriptor differences between images of the same location caused by environmental changes such as viewpoint and lighting. This results in insufficient adaptability of the generated descriptors to environmental changes and a tendency for location mismatches. The invention encodes the descriptor differences between the query image and positive sample images of the same location as reference conditional features during the model training phase. These reference conditional features are then used to calibrate the global aggregated descriptors, enabling the visual location recognition model to learn more stable location consistency representations. This improves the robustness of the global aggregated descriptors to environmental changes and effectively increases the accuracy of robot visual location recognition without increasing the inference computation during the testing phase.

[0006] This invention provides a training method for a visual location recognition model, comprising: A reference condition calibration training network is constructed, which includes a feature extraction module, a VLAD aggregation module, a reference condition encoding module, and a reference condition calibration module. Obtain the query image samples in the training set, as well as the positive and negative sample images of the same location corresponding to the query image samples; The query image sample, the positive sample image, and the negative sample image are sequentially input into the feature extraction module and the VLAD aggregation module to obtain the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor, respectively. Based on the descriptor differences between the global aggregated descriptor of the query and the global aggregated descriptor of the positive sample, construct sample pair difference features; The sample pair difference features are input into the reference condition encoding module to obtain reference condition features and reconstructed difference features used to characterize appearance differences under changing environmental conditions. The query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor are combined to form an initial descriptor sequence, which is then input together with the reference condition feature into the reference condition calibration module to obtain the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor. The contrast loss is calculated based on the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor; Based on the reconstructed difference features and the sample pair difference features, the variational autoencoder reconstruction loss is calculated; By combining the contrast loss and the variational autoencoder reconstruction loss, the network parameters of the reference condition calibration training network are jointly updated, so that the updated feature extraction module and the VLAD aggregation module can be used as the trained visual location recognition model.

[0007] According to a training method for a visual location recognition model provided by the present invention, the step of inputting the sample pair difference features into the reference conditional encoding module to obtain reference conditional features for characterizing appearance differences under changing environmental conditions and reconstructed difference features includes: The sample is subjected to a linear encoding operation, a batch normalization encoding operation, and an activation encoding operation in sequence to obtain the first hidden feature; The first hidden feature is subjected to mean mapping and variance mapping respectively to obtain the latent variable mean feature and the latent variable log-variance feature. Based on the log-variance feature of the latent variable, the standard deviation feature of the latent variable is obtained, and random noise features with the same size as the standard deviation feature of the latent variable are generated; Reparameterized sampling is performed based on the latent variable mean feature, the latent variable standard deviation feature, and the random noise feature to obtain the sampled latent features; Based on the latent variable mean feature and the latent variable standard deviation feature, a first reference condition feature is generated as the reference condition feature; The sampled latent features are sequentially subjected to a decoding linear operation and a decoding normalization operation to obtain a second reconstructed difference feature, which is then used as the reconstructed difference feature.

[0008] According to a training method for a visual location recognition model provided by the present invention, the step of reparameterizing sampling based on the latent variable mean feature, the latent variable standard deviation feature, and the random noise feature to obtain sampled latent features includes: The random noise features are modulated using the latent variable standard deviation feature to obtain the noise distribution features; The noise distribution features are fused with the latent variable mean features to obtain the sampling latent features; Accordingly, generating the first reference condition feature based on the latent variable mean feature and the latent variable standard deviation feature includes: The first reference condition feature is obtained by fusing the latent variable standard deviation feature with the latent variable mean feature.

[0009] According to a training method for a visual location recognition model provided by the present invention, the step of forming an initial descriptor sequence by combining the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor, and inputting it together with the reference condition features into the reference condition calibration module to obtain calibrated query descriptors, calibrated positive sample descriptors, and calibrated negative sample descriptors includes: The reference condition feature is then expanded and copied in dimensions to obtain the second reference condition feature; The initial descriptor sequence is concatenated with the second reference condition feature to obtain the descriptor-reference condition joint feature; The joint features of the descriptor reference conditions are pooled and mapped respectively to obtain the compensation features and calibration weights. The compensation features are modulated using the calibration weights to obtain weighted compensation features; The initial descriptor sequence and the weighted compensation features are residually fused and then normalized to obtain the second calibration descriptor; The second calibration descriptor is split according to the sample sequence dimension to obtain the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor.

[0010] According to a training method for a visual location recognition model provided by the present invention, the step of sequentially inputting the query image sample, the positive sample image, and the negative sample image into the feature extraction module and the VLAD aggregation module respectively to obtain a query global aggregation descriptor, a positive sample global aggregation descriptor, and a negative sample global aggregation descriptor includes: The feature extraction module is used to extract local features from the image to be processed, resulting in a first local feature map. In the VLAD aggregation module, the first local feature map is normalized to obtain the first normalized local feature map; The first normalized local feature map is subjected to soft-allocation feature extraction and normalization mapping to obtain a soft-allocation weight map; Calculate the first residual feature between the first normalized local feature map and the preset learnable cluster centers; The first residual feature is modulated using the soft-assigned weight map to obtain the first weighted residual feature; The first weighted residual features are accumulated and aggregated in the spatial dimension, and then normalized and flattened to obtain the target global aggregated descriptor; Wherein, when the images to be processed are respectively the query image sample, the positive sample image, and the negative sample image, the corresponding target global aggregation descriptors are respectively the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor.

[0011] According to a training method for a visual location recognition model provided by the present invention, the step of calculating the variational autoencoder reconstruction loss based on the reconstructed difference features and the sample pair difference features includes: Calculate the feature reconstruction error between the reconstructed differential features and the sample pair differential features; Based on the mean characteristics and log-variance characteristics of the latent variables, the distribution divergence loss of the latent variables is calculated; Based on preset weighting coefficients, the feature reconstruction error and the latent variable distribution divergence loss are weighted and fused to obtain the variational autoencoder reconstruction loss.

[0012] According to a training method for a visual location recognition model provided by the present invention, the step of calculating the contrast loss based on the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor includes: Calculate the positive sample descriptor similarity between the calibration query descriptor and the calibration positive sample descriptor; Calculate the negative sample descriptor similarity between the calibration query descriptor and each of the calibration negative sample descriptors; Based on a preset temperature coefficient, the similarity of the positive sample descriptors and the similarity of each negative sample descriptor are exponentially normalized to obtain the target comparison probability. The contrast loss is obtained by calculating the logarithmic loss based on the target contrast probability.

[0013] The present invention also provides a visual location recognition method, which, after training a visual location recognition model based on any of the aforementioned visual location recognition model training methods, further includes: Obtain a query image of the scene to be identified, input the query image into the visual location recognition model, and obtain a global aggregate descriptor for the query; The global aggregate descriptor of the query is subjected to dimensionality reduction processing to obtain a dimensionality-reduced global aggregate descriptor of the query; Obtain the dimensionality-reduced global aggregate descriptor of each database image in the environment map. The dimensionality-reduced global aggregate descriptor is obtained by the visual location recognition model after feature extraction and dimensionality reduction of the database images. Calculate the feature distance between the global aggregate descriptor of the dimensionality reduction query and each of the global aggregate descriptors of the dimensionality reduction database; The database image to which the dimensionality reduction database global aggregation descriptor corresponding to the minimum feature distance among all the feature distances belongs is taken as the visual location recognition result of the scene to be identified.

[0014] The present invention also provides a training apparatus for a visual location recognition model, comprising: A network construction module is used to construct a reference condition calibration training network, which includes a feature extraction module, a VLAD aggregation module, a reference condition encoding module, and a reference condition calibration module. The sample acquisition module is used to acquire query image samples in the training set, as well as positive and negative sample images at the same location corresponding to the query image samples; The initial feature extraction module is used to input the query image sample, the positive sample image, and the negative sample image into the feature extraction module and the VLAD aggregation module respectively, so as to obtain the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor. The difference construction module is used to construct sample pair difference features based on the descriptor differences between the query global aggregate descriptor and the positive sample global aggregate descriptor; The encoding and reconstruction module is used to input the difference features of the sample pair into the reference condition encoding module to obtain the reference condition features and reconstructed difference features used to characterize the appearance differences under changes in environmental conditions. The descriptor calibration module is used to form an initial descriptor sequence from the query global aggregate descriptor, the positive sample global aggregate descriptor, and the negative sample global aggregate descriptor, and input it together with the reference condition feature into the reference condition calibration module to obtain the calibrated query descriptor, the calibrated positive sample descriptor, and the calibrated negative sample descriptor; The loss calculation module is used to calculate the contrast loss based on the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor, and to calculate the variational autoencoder reconstruction loss based on the reconstruction difference features and the sample pair difference features. The model update module is used to jointly update the network parameters of the reference condition calibration training network by combining the contrast loss and the variational autoencoder reconstruction loss, so as to use the updated feature extraction module and the VLAD aggregation module as the trained visual location recognition model.

[0015] The present invention also provides a visual location recognition device, which is executed using a visual location recognition model trained by any of the aforementioned training methods, including: The query feature extraction module is used to acquire a query image of the scene to be identified, input the query image into the visual location recognition model, and obtain a global aggregated descriptor for the query. The query dimensionality reduction module is used to perform dimensionality reduction processing on the global aggregate descriptor of the query to obtain a dimensionality-reduced global aggregate descriptor of the query. The database feature acquisition module is used to acquire the dimensionality-reduced global aggregate descriptor of each database image in the environment map. The dimensionality-reduced global aggregate descriptor is obtained by the visual location recognition model after feature extraction and dimensionality reduction of the database image. The distance calculation module is used to calculate the feature distance between the global aggregate descriptor of the dimensionality reduction query and each of the global aggregate descriptors of the dimensionality reduction database; The result determination module is used to take the database image to which the dimensionality reduction database global aggregation descriptor corresponding to the minimum feature distance among all the feature distances belongs as the visual location recognition result of the scene to be identified.

[0016] The visual location recognition model training, method, and apparatus provided by this invention obtain local feature maps through a feature extraction module, and input these local feature maps into a VLAD aggregation module to obtain global aggregated descriptors. Simultaneously, during the training phase, a difference in global aggregated descriptors between a query image and positive sample images of the same location is constructed. This difference is used to obtain reference condition features through a reference condition encoding module, and a reference condition calibration module is used to perform residual calibration on the global aggregated descriptors, thereby improving the robustness of the global aggregated descriptors to changes in environmental conditions. Furthermore, during the testing phase, only the feature extraction module and VLAD aggregation module obtained from the above training are used for single-image descriptor extraction, thus improving the robustness of robot visual location recognition without increasing the computational load. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the training method for the visual location recognition model provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the encoding and decoding reconstruction process of the reference condition encoding module provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the reference condition encoding module provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the process by which the reference condition calibration module provided by the present invention calibrates the initial descriptor sequence.

[0022] Figure 5 This is a schematic diagram of the reference condition calibration module provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the process for obtaining a global aggregation descriptor provided by the present invention.

[0024] Figure 7 This is a schematic diagram of the feature extraction module provided by the present invention.

[0025] Figure 8 This is a schematic diagram of the VLAD aggregation module provided by the present invention.

[0026] Figure 9 This is a flowchart illustrating the visual location recognition method provided by the present invention.

[0027] Figure 10 This is a schematic diagram of the structure of the training device for the visual location recognition model provided by the present invention.

[0028] Figure 11 This is a schematic diagram of the structure of the visual location recognition device provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0031] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0032] Currently, robot vision location recognition methods typically first use convolutional neural networks to extract local features of images, then generate image-level descriptors through global feature aggregation methods, and finally complete the retrieval based on descriptor similarity.

[0033] However, in real-world robot operation scenarios, environmental changes such as perspective, lighting, and weather often lead to variations in the appearance of images corresponding to the same location. Existing methods typically focus on constraining the descriptor distance between the query image and positive / negative sample images in the training phase, rarely explicitly modeling the descriptor difference between the query image and positive sample images of the same location. Since this descriptor difference includes acceptable feature variations of the same location under different shooting conditions, failing to fully utilize this difference weakens the descriptor's adaptability to environmental changes, thus easily leading to location mismatches.

[0034] Based on this, the present invention considers using the differences between positive samples from the same location as a reference condition for the model during the training phase, and calibrating the descriptor based on the reference condition, so that the model learns a more stable location consistency expression. This can improve the robustness of visual location recognition by using the difference information of sample pairs during the training phase without increasing the network complexity during the testing phase.

[0035] The following is combined with Figures 1-11 This invention describes the training of the visual location recognition model, the visual location recognition method, and the apparatus provided by the present invention.

[0036] It should be noted in advance that the execution entity of the training method for a visual location recognition model provided by this invention can be an electronic device with data processing capabilities, such as a server, workstation, personal computer, or robot control system equipped with a processor, or it can be a software module or computer program running in the aforementioned electronic device. For ease of explanation, the model training processor is used as the execution entity in the following embodiments, but this does not constitute a limitation on the execution entity of this invention.

[0037] Figure 1 This is a flowchart illustrating the training method for the visual location recognition model provided by the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps: Step 11: Construct a reference condition calibration training network, which includes a feature extraction module, a Vector of Locally Aggregated Descriptors (VLAD) aggregation module, a reference condition encoding module, and a reference condition calibration module.

[0038] The reference condition calibration training network refers to the complete network structure used in the network training phase to optimize the parameters of the visual location recognition model to be trained. In an optional embodiment, a reference condition calibration training network and a visual location descriptor generation network can be pre-constructed, wherein the reference condition calibration training network is used in the network training phase, and the visual location descriptor generation network is used in the network testing phase. The visual location descriptor generation network includes a feature extraction module and a VLAD aggregation module obtained by joint optimization by the reference condition calibration training network.

[0039] Of the four modules described above—feature extraction, VLAD aggregation, reference condition encoding, and reference condition calibration—the feature extraction module extracts local features from the input image to obtain feature representations that reflect the visual information of local areas of the image. The VLAD aggregation module aggregates the local features output by the feature extraction module into an image-level global aggregated descriptor. The reference condition encoding module encodes the descriptor differences between the query image and positive sample images from the same location to obtain reference condition features that characterize the appearance differences of the same location under changing environmental conditions. The reference condition calibration module calibrates the global aggregated descriptor based on these reference condition features. It is understood that the reference condition encoding and reference condition calibration modules in this embodiment only function during the network training phase and do not participate in the inference process during the testing phase.

[0040] Step 12: Obtain the query image samples in the training set, as well as the positive and negative sample images at the same location corresponding to the query image samples.

[0041] The acquired training set refers to the image data set used to train the reference condition calibration training network, which typically contains a large number of query images and database images with location annotations. Query image samples are the images currently used for training, whose corresponding physical locations are to be determined. Positive sample images of the same location are database images that correspond to the same physical location as the query image samples, while negative sample images are database images that correspond to different physical locations than the query image samples.

[0042] Step 13: Input the query image sample, the positive sample image, and the negative sample image into the feature extraction module and the VLAD aggregation module respectively in sequence to obtain the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor.

[0043] Specifically, the query image sample, positive sample image, and negative sample image can be used as images to be processed, and then processed sequentially by the feature extraction module and the VLAD aggregation module. The image to be processed first undergoes feature extraction to obtain local features reflecting its local region information. These local features are then aggregated by the VLAD aggregation module to obtain an image-level global aggregated descriptor corresponding to the image to be processed.

[0044] When the images to be processed are respectively query image samples, positive sample images, and negative sample images, the corresponding global aggregated descriptors are the query global aggregated descriptor, the positive sample global aggregated descriptor, and the negative sample global aggregated descriptor, respectively. The query global aggregated descriptor, the positive sample global aggregated descriptor, and the negative sample global aggregated descriptor are all vector representations that can comprehensively characterize the visual features of the location corresponding to the image, and can be used for subsequent descriptor similarity comparisons.

[0045] Step 14: Construct sample pair difference features based on the descriptor differences between the query global aggregate descriptor and the positive sample global aggregate descriptor.

[0046] Since the query image samples and the positive sample images correspond to the same physical location, the differences in their appearance mainly stem from variations in environmental conditions such as viewing angle, lighting, and weather. Therefore, the descriptor differences between the query global aggregate descriptor and the positive sample global aggregate descriptor contain information on acceptable feature variations of the same location under different shooting conditions. Based on this, sample pair difference features can be constructed according to the descriptor differences between the query global aggregate descriptor and the positive sample global aggregate descriptor.

[0047] In an optional embodiment, the global aggregate descriptor of the query can be subtracted from the global aggregate descriptor of the positive sample, and the result of the subtraction can be used as the difference feature of the sample pair.

[0048] Step 15: Input the difference features of the sample pair into the reference condition encoding module to obtain the reference condition features and reconstructed difference features used to characterize the appearance differences under changing environmental conditions.

[0049] In this embodiment, after the sample pair difference features are input into the reference condition encoding module, the module processes them and outputs both reference condition features and reconstructed difference features. The reference condition features are those extracted from the sample pair difference features that characterize the appearance differences of the same physical location under changing environmental conditions; they can serve as a reference for subsequent calibration of the global aggregate descriptor.

[0050] Reconstructed difference features refer to the features obtained by the reference conditional coding module after reconstructing the difference features of the sample pairs, which are used to compare with the original difference features of the sample pairs to constrain the coding process. The appearance difference under changing environmental conditions refers to the difference in visual appearance of the same physical location on the image due to changes in external environmental factors such as viewing angle, lighting, and weather.

[0051] Step 16: The query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor are combined to form an initial descriptor sequence, which is then input together with the reference condition feature into the reference condition calibration module to obtain the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor.

[0052] The initial descriptor sequence is a sequence formed by arranging the query global aggregate descriptor, positive sample global aggregate descriptor, and negative sample global aggregate descriptor according to the sample dimension. After the initial descriptor sequence and reference condition features are input into the reference condition calibration module, the module uses the environmental condition change information carried by the reference condition features to calibrate each global aggregate descriptor in the initial descriptor sequence, thereby obtaining the calibrated query descriptor, calibrated positive sample descriptor, and calibrated negative sample descriptor. The calibrated descriptors, compared to the uncalibrated global aggregate descriptors, exhibit stronger adaptability to changes in environmental conditions.

[0053] Step 17: Calculate the contrast loss based on the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor.

[0054] Contrast loss measures the proximity between a calibration query descriptor and a calibration positive sample descriptor, as well as the distance between a calibration query descriptor and a calibration negative sample descriptor. By constraining the contrast loss, calibration query descriptors are encouraged to move closer to calibration positive sample descriptors corresponding to the same physical location in the feature space, while moving further away from calibration negative sample descriptors corresponding to different physical locations. This allows the visual location recognition model to learn discriminative location representations.

[0055] Step 18: Calculate the variational autoencoder reconstruction loss based on the reconstructed difference features and the sample pair difference features.

[0056] Variational autoencoder reconstruction loss refers to the loss used to measure the degree of difference between the reconstructed difference features output by the reference conditional encoding module and the original sample pair difference features. This embodiment, by constraining the variational autoencoder reconstruction loss, can encourage the reconstructed difference features output by the reference conditional encoding module to approximate the original sample pair difference features as closely as possible, thereby enabling the reference conditional encoding module to fully encode the environmental condition change information contained in the sample pair difference features.

[0057] Step 19: Combine the contrast loss and the variational autoencoder reconstruction loss to jointly update the network parameters of the reference condition calibration training network, so as to use the updated feature extraction module and the VLAD aggregation module as the trained visual location recognition model.

[0058] Specifically, the contrastive loss can be combined with the variational autoencoder reconstruction loss to obtain a joint loss for optimizing the reference condition calibration training network. Based on this joint loss, the network parameters of the feature extraction module, VLAD aggregation module, reference condition encoding module, and reference condition calibration module in the reference condition calibration training network can be jointly updated.

[0059] In an optional embodiment, a stochastic gradient descent optimizer can be used to iteratively update the network parameters until the joint loss converges or a preset number of training rounds are reached, thereby completing the training of the reference condition calibration training network.

[0060] After training is completed, the updated feature extraction module and VLAD aggregation module are combined to form a visual location recognition model, while the reference condition encoding module and reference condition calibration module, which are only used during the training phase, are not included in the visual location recognition model.

[0061] It should be noted that although the visual location recognition model after training only includes a feature extraction module and a VLAD aggregation module in form, its parameters are obtained through joint optimization of the feature extraction module, VLAD aggregation module, reference conditional encoding module, reference conditional calibration module, and joint loss during the training phase. Therefore, the visual location recognition model can inherit the feature robustness improvement brought by the reference conditional calibration mechanism during the training phase while maintaining a lightweight inference process for single images.

[0062] The training method for the visual location recognition model provided by this invention constructs sample pair difference features between the query global aggregate descriptor and the global aggregate descriptor of positive samples of the same location during the training phase. These difference features contain the acceptable feature changes of the same location under different shooting conditions. After being encoded into reference condition features by the reference condition encoding module, the global aggregate descriptor is calibrated by the reference condition calibration module. This explicitly introduces the appearance difference information of the same physical location under changing environmental conditions into the optimization process of the network, prompting the feature extraction module and the VLAD aggregation module to learn a more stable and consistent representation of the location under changing environmental conditions.

[0063] Meanwhile, since the reference condition encoding module and the reference condition calibration module only participate in the training phase and not in the inference phase of the testing phase, the trained visual location recognition model effectively improves the robustness of the robot's visual location recognition to environmental changes such as viewpoint, lighting, and weather without increasing the inference computation in the testing phase, thereby reducing the probability of location mismatch.

[0064] Based on the training method of the visual location recognition model described in the above embodiments, the present invention further provides a specific implementation method for how to extract and reconstruct descriptor differences caused by environmental changes through a reference conditional coding module.

[0065] Figure 2 This is a schematic diagram of the encoding and decoding reconstruction process of the reference condition encoding module provided by the present invention, as shown below. Figure 2As shown, the step of inputting the sample pair difference features into the reference condition encoding module for encoding and decoding reconstruction to generate reference condition features and reconstructed difference features characterizing appearance differences under changing environmental conditions can specifically include the following implementation steps: Step 21: Perform encoding linear operation, encoding batch normalization operation, and encoding activation operation on the differential features of the sample in sequence to obtain the first hidden feature.

[0066] When the sample pair difference features, such as those obtained by subtracting the global aggregated descriptor of positive samples from the same location, are input into the reference conditional encoding module, they first need to be mapped from the high-dimensional descriptor feature space to the low-dimensional latent variable space. Specifically, a linear encoding operation can be performed on the sample pair difference features to achieve linear dimensionality reduction. Subsequently, a batch encoding normalization operation is performed to normalize the dimensionality-reduced feature distribution, thereby accelerating network convergence and mitigating the internal covariate shift problem. Finally, an encoding activation operation is performed to introduce nonlinear expressive power. After the above series of encoding and extraction operations, the first hidden feature, which has been dimensionality reduced and contains nonlinear features, is obtained.

[0067] Step 22: Perform mean mapping and variance mapping on the first hidden feature to obtain the latent variable mean feature and the latent variable log-variance feature respectively.

[0068] After obtaining the first hidden feature, this embodiment employs the design concept of a variational autoencoder structure to map it into statistical parameters representing the distribution of environmental differences. Specifically, the first hidden feature is input into the first projection branch for mean mapping to extract the mean feature of the latent variable representing the average trend of environmental changes; simultaneously, the first hidden feature is input into the second projection branch in parallel for variance mapping to extract the log-variance feature of the latent variable representing the uncertainty or fluctuation range of environmental changes.

[0069] Step 23: Obtain the standard deviation feature of the latent variable based on the log-variance feature of the latent variable, and generate random noise features with the same size as the standard deviation feature of the latent variable.

[0070] To support subsequent generative sampling, this embodiment considers converting the extracted log-variance to standard deviation. Specifically, this can be achieved by scaling and exponential operations on the latent variable log-variance feature to obtain the latent variable standard deviation feature. Simultaneously, a random noise sampling operation is triggered, randomly generating a noise matrix from a standard normal distribution. This noise matrix serves as the random noise feature, and its data size is configured to strictly match the size of the latent variable standard deviation feature.

[0071] Step 24: Perform reparameterized sampling based on the latent variable mean feature, the latent variable standard deviation feature, and the random noise feature to obtain the sampled latent features.

[0072] Furthermore, this embodiment utilizes a reparameterized sampling mechanism to combine deterministic statistical features with random features. By performing mathematical operations (such as multiplication and addition) on the latent variable mean features, latent variable standard deviation features, and random noise features, not only is the randomness of exploring the continuously changing space of the environment introduced into the visual location recognition model to be trained, but the random sampling process is also successfully separated from the gradient calculation path of the network parameters, ensuring that error backpropagation can proceed smoothly in the subsequent training process. The fused features are the sampled latent features used for subsequent decoding and reconstruction.

[0073] Step 25: Generate a first reference condition feature as the reference condition feature based on the latent variable mean feature and the latent variable standard deviation feature.

[0074] In addition to generating latent features for reconstruction, the reference condition encoding module also outputs prior information to guide descriptor calibration. Therefore, this embodiment combines the mean feature of latent variables reflecting the center of environmental difference distribution with the standard deviation feature of latent variables reflecting difference tolerance, encoding both together to generate the first reference condition feature. This first reference condition feature encompasses the core visual differences and their reasonable variation boundaries under different shooting conditions at the same location. This embodiment outputs it as the final reference condition feature to the subsequent reference condition calibration module.

[0075] Step 26: Perform decoding linear operation and decoding normalization operation on the sampled latent features in sequence to obtain the second reconstructed difference feature as the reconstructed difference feature.

[0076] To ensure that the encoding process truly learns the effective information in the sample pair difference features, this embodiment performs a decoding linear operation on the sampled latent features, reprojecting and expanding them from the low-dimensional latent variable space back to the original high-dimensional descriptor space. Subsequently, a decoding normalization operation is performed, numerically normalizing the feature dimensions to obtain the second reconstructed difference feature. This second reconstructed difference feature can then be used as the reconstructed difference feature to compare the reconstruction error with the original input sample pair difference feature when calculating the variational autoencoder reconstruction loss.

[0077] Figure 3 This is a schematic diagram of the reference condition encoding module provided by the present invention. To further clearly illustrate the data flow and tensor size changes within the above-mentioned reference condition encoding module, as shown below... Figure 3 As shown, in a specific optional embodiment, the underlying operators and feature dimensions of the above operations are configured as follows: The reference condition encoding module includes the following operations: EncLinear_1 (encoding linear operation), EncBN_1 (encoding batch normalization operation), EncAct_1 (encoding activation operation), MuLinear_1 (mean mapping operation), LogVarLinear_1 (variance mapping operation), StdExp_1 (standard deviation calculation operation), NoiseSample_1 (random noise sampling operation), Reparam_1 (reparameterized sampling operation), CondAdd_1 (reference condition generation operation), DecLinear_1 (decoding linear operation), and DecNorm_1 (decoding normalization operation).

[0078] Optionally, the encoded linear operation EncLinear_1 can be implemented by a linear mapping function of the neural network (such as the nn.Linear function in the torch library), with an input dimension of 32768 and an output dimension of 512.

[0079] The batch normalization operation EncBN_1 can be implemented by a one-dimensional batch normalization function (such as nn.BatchNorm1d function) with a feature dimension of 512.

[0080] The encoded activation operation EncAct_1 can be implemented by a leaky modified linear unit (such as nn.LeakyReLU).

[0081] The mean mapping operation MuLinear_1 and the variance mapping operation LogVarLinear_1 can both be implemented by linear mapping functions with both input and output dimensions of 512, such as the nn.Linear(512,512) function.

[0082] Decoding the linear operation DecLinear_1 can be achieved by a linear mapping function with an input dimension of 512 and an output dimension of 32768, such as the nn.Linear(512,32768) function.

[0083] The decoding normalization operation DecNorm_1 can be implemented using L2 norm normalization functions (such as the normalize function in the torch library).

[0084] In addition, the reparameterized sampling operation Reparam_1 is implemented by element-wise multiplication and element-wise addition, while the reference condition generation operation CondAdd_1 can be implemented by element-wise addition.

[0085] During the training phase, for each input query image sample... I q Determine a positive sample image from the database images in the training set. and N negative sample images To form a training tuple ,in, q This indicates the sequence number of the image sample being queried. Indicates the first q The positive sample image corresponding to each query image sample. Indicates the first q The first query image sample n One negative sample image.

[0086] As an alternative implementation, weakly supervised learning can be used for data preparation during the training phase. Images in the training set typically come from publicly available street view datasets with Global Positioning System (GPS) coordinate labels, such as sets of images of the same location taken at different times and from different perspectives. Since such data often lacks fine-grained, pixel-level manual annotation, the inherent geographic location information (such as GPS coordinates) of the images can be used for weakly supervised sample partitioning.

[0087] As an optional embodiment, for any given query image sample, the specific method for determining the corresponding positive sample image at the same location is as follows: First, the GPS coordinate geographic distance between each image in the training set database and the query image is calculated. If the geographic distance between an image in the database and the query image is less than a preset first distance threshold (e.g., 10 meters), then the image is classified as a potential positive sample image.

[0088] In real-world scenarios, even two images geographically close can exhibit significant differences in content due to factors such as camera perspective deviations or temporary occlusions. Therefore, during training, this embodiment calculates the feature distance between the query image and all potential positive sample images in the feature mapping space. Based on the calculated feature distance, it automatically selects the image with the smallest feature distance (i.e., the most similar visual features) from all potential positive sample images as the best-matching positive sample image for the current training iteration step.

[0089] The above-mentioned secondary screening operation based on feature distance minimization can effectively solve the problem of visual matching ambiguity of positive samples in weakly supervised data.

[0090] As another alternative embodiment, negative sample images can also be determined in a similar manner: First, negative samples are defined based on geographical distance. If the GPS coordinate geographical distance between an image in the database and the query image is greater than a preset second distance threshold (e.g., 25 meters), then it is confirmed that the two images do not belong to the same location in physical space, and the image is directly defined as a candidate negative sample image.

[0091] In large-scale datasets, the vast majority of randomly selected negative sample images differ greatly in visual appearance from the query image, making them easily distinguishable by the model. These negative samples contribute very little to the gradient updates of the model's network parameters. To significantly improve training efficiency and the model's feature discrimination capability, this embodiment prioritizes selecting the most valuable negative sample images. Specifically, from the random negative sample pool (e.g., containing 1000 candidate negative sample images) constructed from the aforementioned candidate negative sample images, their feature distances or loss function values ​​with the query image are calculated. A few candidate negative sample images (e.g., 10) that are relatively close to the query image in the feature space, are most easily confused with positive samples, and thus generate the largest loss value are specifically selected as the negative sample images in the current training tuple.

[0092] By combining the above-mentioned positive and negative sample selection strategy with geographic coordinate distance threshold determination and candidate negative sample image mining, this embodiment can continuously provide high-quality and challenging training tuples for the reference condition calibration training network with extremely low annotation cost. This not only effectively avoids the waste of computing resources on simple samples, but also forces the visual location recognition model to focus on learning how to distinguish the visually most easily confused location features, thereby greatly improving the model's feature expression ability and retrieval accuracy in complex scenarios.

[0093] Based on the above embodiments, query image samples can be... I q Positive sample images and negative sample images The data is processed sequentially through the feature extraction module and the VLAD aggregation module to obtain the global aggregated descriptor for the query. GD q Positive Sample Global Aggregation Descriptor Global aggregation descriptor for negative samples All are 32768 in size.

[0094] Furthermore, this can be achieved by using the query global aggregation descriptor. GD q Global aggregation descriptor with positive samples Subtracting them allows us to obtain the difference features between the sample pairs. E 1. Its dimensions are 32768.

[0095] During the data flow process, the differential features of sample pairs with a size of 32768 E 1. First, the feature is processed sequentially by EncLinear_1, EncBN_1, and EncAct_1 to obtain the first hidden feature with a size of 512. H 1. Its dimensions are 512.

[0096] Furthermore, the first hidden featureH After processing with MuLinear_1 and LogVarLinear_1 respectively, the latent variable mean features of size 512 are obtained. MU 1 and the log-variance characteristics of latent variables LV 1. All sizes are 512.

[0097] Subsequently, the log-variance characteristics of the latent variables LV 1. After processing with StdExp_1 (e.g., scaling the input vector by 0.5 and then performing an exponentiation), the latent variable standard deviation feature of size 512 is obtained. STD 1.

[0098] Meanwhile, the characteristics of the standard deviation of latent variables STD 1 was fed into NoiseSample_1 to generate latent variable standard deviation features. STD 1. Random noise features of the same size EPS 1. For example, it can be achieved using a function that generates random numbers that follow a standard normal distribution.

[0099] Next, Reparam_1 will be based on the latent variable mean characteristics. MU 1. Characteristics of latent variable standard deviation STD 1 and random noise characteristics EPS 1. Generate sampled latent features of size 512. Z 1. In one optional embodiment, the specific fusion logic may be set as follows: Z 1= MU 1+ EPS 1 STD 1, the corresponding size is 512.

[0100] CondAdd_1 will include the latent variable mean feature. MU 1 and latent variable standard deviation characteristics STD Adding 1 element by element (i.e., performing element-wise addition) yields the first reference condition feature with a size of 512. CF 1. Use the obtained first reference condition feature as the subsequent reference condition feature.

[0101] Sampling latent features Z After processing with DecLinear_1, the first reconstructed difference feature with a size of 32768 is obtained. RD 1. Its dimensions are 32768.

[0102] Finally, the first reconstruction of differential features RD 1. The feature dimension is L2 normalized using DecNorm_1, resulting in a second reconstructed differential feature of size 32768. RD 2. The dimensions are 32768.

[0103] This invention constructs a reference condition encoding module based on variational inference for encoding and decoding reconstruction. It decomposes the hard descriptor differences between the query image and the positive sample image caused by factors such as viewpoint and illumination into mean features representing the average offset and variance / standard deviation features representing the tolerance interval. Random noise is introduced for reparameterization, enabling the visual location recognition model to be trained to explicitly and smoothly model the feature fluctuation distribution caused by environmental changes, rather than simply memorizing and fixing differences. This endows the output reference condition features with tolerance to complex environmental interference, significantly enhancing the robustness and generalization ability of the robot in performing location recognition under complex conditions such as variable lighting and seasonal changes in the real world.

[0104] Based on the above embodiments, as an optional embodiment, a specific implementation method is provided for how to perform reparameterized sampling and how to generate reference condition features.

[0105] In the process of encoding the differential features of samples, in order to enable the visual location recognition model to be trained to learn a continuous and smooth latent variable space, thereby better representing the uncertainty brought about by changes in environmental conditions, this embodiment provides a specific design for how to obtain the sampled latent features. Specifically, the step of reparameterizing the latent variable mean feature, the latent variable standard deviation feature, and the random noise feature to obtain the sampled latent features may include: The random noise features are modulated using the latent variable standard deviation feature to obtain the noise distribution features.

[0106] In practical applications, directly sampling from a Gaussian distribution parameterized by specific mean and variance lacks a deterministic computational path that facilitates backpropagation of conventional errors, hindering end-to-end optimization of the model during training. Therefore, this embodiment first randomly samples random noise features (e.g., a 512-dimensional feature vector) from a standard normal distribution (mean 0, variance 1) without carrying specific environmental parameters. Subsequently, the pre-extracted latent variable standard deviation feature is used to modulate these random noise features.

[0107] As an optional implementation, the feature modulation operation can be achieved through element-wise multiplication, that is, by using the latent variable standard deviation feature to scale the amplitude of the random noise feature. The feature obtained after scaling and modulation is the noise distribution feature. This step enables the originally standard random noise to have distribution characteristics that conform to the fluctuation amplitude of the current real environment.

[0108] Furthermore, the noise distribution features and the latent variable mean features are fused to obtain the sampling latent features.

[0109] After modulating the standard deviation, this embodiment further introduces an average offset benchmark for environmental differences. Specifically, the noise distribution features obtained above are fused with the latent variable mean features extracted in the previous step.

[0110] As an alternative implementation, this feature fusion operation can be achieved through element-wise addition, that is, superimposing the mean feature representing the average trend of change onto the noise distribution feature with a reasonable fluctuation range.

[0111] Let's illustrate this with a specific example: Suppose the latent variable means feature MU 1. Characteristics of latent variable standard deviation STD 1 and random noise characteristics EPS The dimensions of all 1 are 512-dimensional. In this case, the reparameterization sampling operation Reparam_1 generates sampled latent features. Z The calculation method for 1 can be configured as follows: Z 1= MU 1+ EPS 1 STD 1.

[0112] As another optional embodiment, besides constructing latent features for decoding reconstruction through reparameterized sampling, another core task of the reference conditional encoding module is to output an explicit conditional prior to the reference conditional calibration module to guide descriptor calibration. Accordingly, generating the first reference conditional feature based on the latent variable mean feature and the latent variable standard deviation feature may include: fusing the latent variable standard deviation feature with the latent variable mean feature to obtain the first reference conditional feature.

[0113] When guiding the global aggregation descriptor to perform calibration, the model needs a deterministic and robust reference. If only the mean feature is passed to the reference condition calibration module, the reasonable fluctuation range information that may be caused by environmental changes will be lost. On the other hand, if the sampled latent features containing random noise are passed directly, unnecessary random disturbances may be introduced into the calibration process, resulting in unstable calibration results.

[0114] Therefore, this embodiment chooses to fuse the latent variable standard deviation feature with the latent variable mean feature. As an optional implementation, this feature fusion operation can also be achieved through element-wise addition.

[0115] Optionally, the reference condition generation operation CondAdd_1 will generate latent variable mean features, all of which are 512-dimensional. MU 1 and latent variable standard deviation characteristics STD 1. Perform element-wise addition to obtain the first reference condition feature with a size of 512 dimensions. CF 1. Characteristics of the first reference condition obtained CF 1 will then be used as the reference condition feature in the residual calibration process of the initial descriptor sequence.

[0116] Figure 4 This is a schematic diagram of the process by which the reference condition calibration module provided by the present invention calibrates the initial descriptor sequence. Figure 5 This is a schematic diagram of the reference condition calibration module provided by the present invention, combined with... Figure 4 and Figure 5 As shown, the query global aggregation descriptor GD q The positive sample global aggregation descriptor and the negative sample global aggregation descriptor An initial descriptor sequence is formed and input together with the reference condition features into the reference condition calibration module to obtain calibration query descriptors, calibration positive sample descriptors, and calibration negative sample descriptors, which mainly include but are not limited to the following implementation steps.

[0117] To facilitate the precise flow of the underlying feature dimensions, in an optional embodiment, suppose that the training tuple constructed for a query image in the training batch contains one positive sample image and... N One negative sample image.

[0118] Before performing calibration, the input global aggregate descriptors for queries, positive samples, and negative samples need to be processed. Assume that the feature size of each of the above global aggregate descriptors is 32768.

[0119] The reference condition calibration module used in this embodiment mainly includes, but is not limited to, the following: tuple concatenation operation TupleConcat_1, conditional copy operation CondRepeat_1, joint concatenation operation JointConcat_1, compensated pooling operation CompPool_1, weight pooling operation WeightPool_1, weight linear mapping operation WeightLinear_1, residual weighting operation ResidualWeight_1, residual fusion operation ResidualAdd_1, and calibration normalization operation CalNorm_1.

[0120] The tuple concatenation operation TupleConcat_1 and the joint concatenation operation JointConcat_1 can be implemented using the cat function in the torch library; the conditional copy operation CondRepeat_1 can be implemented using the repeat function in the torch library; and the compensated pooling operation CompPool_1 is implemented using the nn.AdaptiveAvgPool1d function in the torch library, with the output dimension configured to 32768.

[0121] Additionally, the weight pooling operation WeightPool_1 can be implemented using the nn.AdaptiveAvgPool1d function in the torch library, with the output dimension configured to 64; the weight linear mapping operation WeightLinear_1 is implemented using the nn.Linear(64,1) function; the residual weighting operation ResidualWeight_1 can be implemented using element-wise multiplication; the residual fusion operation ResidualAdd_1 is implemented using element-wise addition; and the calibration normalization operation CalNorm_1 is implemented using the normalize function in the torch library.

[0122] Step 41: Perform dimensional expansion and copying of the reference condition feature to obtain the second reference condition feature.

[0123] In one alternative implementation, the tuple concatenation operation TupleConcat_1 is first used to concatenate the global aggregate descriptor of the query. GD q Positive Sample Global Aggregation Descriptor Global aggregation descriptor for negative samples A new sample dimension is added to the first dimension, and features are concatenated along this sample dimension to obtain the first descriptor sequence. DS 1. The first descriptor sequence DS 1 is the initial descriptor sequence, whose size is ( N +2) 32768.

[0124] Subsequently, in order to make the reference condition features compatible with the aforementioned first descriptor sequence DS 1. Align in dimension with the first reference condition feature of size 512 output from the preceding step. CF 1. Perform dimensional expansion copying. Specifically, this can be done through the conditional copy operation CondRepeat_1, on the first reference conditional feature. CF Add a new dimension to the first dimension of 1 and copy it. N +2 times, to obtain the second reference condition feature. CF 2, its dimensions are ( N +2) 512.

[0125] Step 42: The initial descriptor sequence is concatenated with the second reference condition feature to obtain the descriptor-reference condition joint feature.

[0126] Specifically, after dimensional alignment is completed, a joint concatenation operation, JointConcat_1, is performed to concatenate the first descriptor sequence. DS 1 and the second reference condition features CF 2. Feature concatenation is performed along the feature dimension. This feature concatenation operation constructs a descriptor-reference conditional joint feature that simultaneously perceives the current image structure and environmental differences. JF 1, its size is expanded to ( N +2) 33280.

[0127] Step 43: Perform pooling and mapping processing on the joint features of the descriptor reference conditions to obtain the corresponding compensation features and calibration weights.

[0128] On the one hand, this embodiment will consider the joint features of the descriptor reference conditions. JF 1. The first branch is processed by the Compensated Pooling operation CompPool_1 to filter out redundant noise in the joint features and extract compensating features specifically used to make up for feature loss. R 1, its size is pooled and compressed to ( N +2) 32768.

[0129] On the other hand, this embodiment will combine descriptor reference conditions with features. JF The parallel input is fed into the second branch, where it is processed sequentially by the weight pooling operation WeightPool_1 and the weight linear mapping operation WeightLinear_1. This compresses the joint features along the feature dimension and maps each feature position to a scalar weight, resulting in calibration weights used to control the compensation intensity. A 1. Its dimensions are ( N +2) 1.

[0130] Step 44: Use the calibration weights to perform feature modulation on the compensation features to obtain weighted compensation features.

[0131] Simultaneously obtain compensation features R 1 and calibration weights A After step 1, this embodiment will perform a residual weighting operation, ResidualWeight_1. Specifically, the calibration weights will be... A 1 and the compensation feature R1. Perform element-wise multiplication and use the weights to scale the feature strength of the compensation features, thereby obtaining the weighted compensation features. WR 1. Its dimensions are ( N +2) 32768.

[0132] Step 45: Perform residual fusion of the initial descriptor sequence and the weighted compensation features, and obtain the second calibration descriptor after normalization.

[0133] Optionally, this embodiment employs a residual learning structure, performing a residual fusion operation ResidualAdd_1, which uses the backbone features, i.e., the first descriptor sequence, as the skip connection. DS 1. Weighted compensation characteristics as bypass compensation signal WR 1. By adding element by element, we obtain a size of ( N +2) 32768's first calibration descriptor CD 1.

[0134] Subsequently, the calibration normalization operation CalNorm_1 is performed on the first calibration descriptor. CD 1. L2 normalization is performed on the feature dimension to eliminate scale fluctuations caused by extreme values, resulting in a final stable and uniform second calibration descriptor. CD 2, its dimensions remain as ( N +2) 32768.

[0135] Step 46: Split the second calibration descriptor according to the sample sequence dimension to obtain the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor.

[0136] Due to the second calibration descriptor CD Formally, it is still a sequence tensor that integrates multiple samples, and it needs to be segmented and split according to the sample sequence dimensions. Specifically, in this embodiment, the second calibration descriptor will generally be... CD The first descriptor in step 2 is extracted as the calibration query descriptor. CGD q The second descriptor is extracted as the calibration positive sample descriptor. and the rest N Each descriptor was extracted and used as a calibration negative sample descriptor. After splitting, the size of all types of calibration descriptors was restored to 32768, which is convenient for subsequent calculation of contrast loss.

[0137] Based on the training method of the visual location recognition model described in the above embodiments, the present invention further provides a detailed implementation method for how to obtain local features through the feature extraction module and generate a global aggregated descriptor through the VLAD aggregation module.

[0138] Figure 6 This is a schematic diagram of the process for obtaining a global aggregation descriptor provided by the present invention, such as... Figure 6 As shown, the step of sequentially inputting the query image sample, the positive sample image, and the negative sample image into the feature extraction module and the VLAD aggregation module to obtain the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor can specifically include the following steps: Step 61: Use the feature extraction module to extract local features from the image to be processed to obtain a first local feature map.

[0139] Figure 7 This is a schematic diagram of the feature extraction module provided by the present invention, combined with... Figure 7 As shown, when the image to be processed (i.e., any one of the query image sample, positive sample image, or negative sample image) is input into the feature extraction module, the BaseNet convolutional feature extraction unit in the feature extraction module will perform hierarchical feature extraction on it.

[0140] As an optional implementation, the BaseNet convolutional feature extraction unit can be obtained by preserving the network layers in the feature extraction sequence of the pre-trained VGG network, excluding the terminal ReLU activation layer and the terminal max-pooling layer. For example... Figure 7 As shown, this basic convolutional feature extraction unit is specifically composed of five consecutive convolutional feature blocks connected in series: Block 1 contains two 3×3 convolutional layers with 64 output channels, equipped with the ReLU activation function, and a 2×2 max pooling layer. Block 2 contains two 3×3 convolutional layers with 128 output channels, equipped with the ReLU activation function, and a 2×2 max pooling layer. Block 3 contains three 3×3 convolutional layers with 256 output channels, equipped with the ReLU activation function, and one 2×2 max pooling layer. Block 4 contains three 3×3 convolutional layers with 512 output channels, equipped with the ReLU activation function, and one 2×2 max pooling layer. Block 5 contains three 3×3 convolutional layers with a total of 512 output channels.

[0141] In the flow of underlying feature dimensions, for a size of 3 H W (Where 3 represents the number of RGB channels,) H and W The original color image to be processed (representing the height and width of the original color image, respectively). I After processing by each block of the BaseNet convolutional feature extraction unit, the first local feature map is obtained. F 1. The first local feature map F The feature size of 1 is 512. H / 16 W The image has a spatial resolution of 1 / 16, which is reduced to 1 / 16 compared to the original image, and the number of channels is increased to 512, thereby extracting richer visual features while preserving the local spatial structure.

[0142] After obtaining the first local feature map F After step 1, this embodiment will input it into the subsequent VLAD aggregation module for processing.

[0143] Figure 8 This is a schematic diagram of the VLAD aggregation module provided by the present invention, combined with... Figure 8 As shown, the VLAD aggregation module is used to aggregate the first local feature map. F 1. Aggregation is a global aggregation descriptor. The VLAD aggregation module mainly includes, but is not limited to: first L2 normalization operation L2Norm_1, soft-assigned convolution operation SoftConv_1, second-dimensional reshaping operation Reshape_2, first Softmax normalization operation Softmax_1, third-dimensional reshaping operation Reshape_3, feature expansion operation FeatExpand_1, and learnable cluster centers. Cluster The cluster center expansion operation CenExpand_1, weight expansion operation WeightUnsqueeze_1, first element-wise multiplication operation Multi_1, spatial dimension accumulation operation Sum_1, second L2 normalization operation L2Norm_2, and fourth dimension rearrangement operation Reshape_4, etc.

[0144] The first L2 normalization operation L2Norm_1 and the second L2 normalization operation L2Norm_2 can be configured to normalize the input vector along the channel or feature dimension using the L2-norm.

[0145] The soft-assignment convolution operation SoftConv_1 can be implemented by the nn.Conv2d function in the torch library, with a kernel size of 1. 1. Step size is (1,1), fill size is (0,0), input channel number is set to 512, output channel number is set to 64.

[0146] The second-dimensional rearrangement operation Reshape_2, the third-dimensional rearrangement operation Reshape_3, and the fourth-dimensional rearrangement operation Reshape_4 can all be implemented using the reshape function in the torch library to deform and flatten the tensor dimensions.

[0147] The feature expansion operation FeatExpand_1 and the cluster center expansion operation CenExpand_1 can be broadcast along a specific dimension using the expand function in the torch library.

[0148] The learnable clustering centers Cluster It can be wrapped in a 64-bit array using the nn.Parameter function in the torch library. The 512 trainable parameter tensor can be initialized by referring to the mechanism of the currently open-source NetVLAD model, and assigned values ​​through K-Means pre-cluster centers.

[0149] The weight expansion operation WeightUnsqueeze_1 can be implemented by the unsqueeze function in the torch library to insert a new dimension of dimension 1 at a specific position.

[0150] The spatial dimension accumulation operation Sum_1 can be implemented by the sum function in the torch library.

[0151] Based on the above structure, the parallel branching and aggregation operations within this VLAD aggregation module are specifically implemented through the following steps 62 to 66: Step 62: In the VLAD aggregation module, the first local feature map is normalized to obtain a first normalized local feature map.

[0152] Specifically, due to the first local feature map F 1. Local descriptors at different spatial locations may have different response amplitudes, and some high-response locations may occupy an excessive proportion in subsequent feature processing. To reduce the amplitude scale differences between local descriptors and improve the stability of feature representation, this embodiment will process the first local feature map. F 1. Perform the first L2 normalization operation L2Norm_1. This first L2 normalization operation L2Norm_1 performs normalization along the channel dimension for a size of 512. H / 16 W / 16 First local feature map F 1. Perform L2 normalization to obtain a first normalized local feature map with a smoother feature response. NF 1. Its size remains 512. H / 16 W / 16.

[0153] Step 63: Perform soft-assignment feature extraction and normalization mapping on the first normalized local feature map to obtain a soft-assignment weight map.

[0154] like Figure 8 As shown, in the soft-assigned weight branch, the first normalized local feature map NF First, the local features are processed by the soft-assignment convolution operation SoftConv_1, which maps the local features to the cluster center space, resulting in the first soft-assignment score map. SA 1, its size is 64 H / 16 W / 16, where 64 is the preset number of cluster centers.

[0155] Subsequently, the first soft-allocation score graph SA 1. After the second-dimensional rearrangement operation Reshape_2 flattens the spatial dimensions, the second soft-assigned fractional map is obtained. SA 2. The rearranged dimensions are 64. ( H / 16 W / 16).

[0156] Next, the second soft-allocation score diagram SA 2. After the first Softmax normalization operation Softmax_1, normalization is performed along the cluster center dimension, so that the sum of the weights of each local feature to each cluster center is 1, thus obtaining the soft assignment weight map representing the soft assignment probability. SW 1, its size is 64 ( H / 16 W / 16).

[0157] Step 64: Calculate the first residual feature between the first normalized local feature map and the preset learnable cluster centers.

[0158] like Figure 8 As shown, in the feature residual branch, the first normalized local feature map NF1. After the third-dimensional rearrangement operation Reshape_3, the spatial dimensions are flattened, resulting in the local descriptor matrix. FD 1, its dimensions are 512 ( H / 16 W / 16).

[0159] The above local descriptor matrix FD 1. Subsequently, the feature expansion operation FeatExpand_1 is performed to replicate and expand along the cluster center dimension, resulting in the first expanded feature. EF 1. Size is 64 512 ( H / 16 W / 16).

[0160] Meanwhile, this embodiment further extracts the internal dimensions of the module to be 64. 512 Learnable Cluster Centers Cluster This learnable clustering center Cluster The cluster center expansion operation CenExpand_1 expands the cluster centers spatially, resulting in the first expanded cluster centers. EC 1. Size is 64 512 ( H / 16 W / 16).

[0161] Subsequently, an element-wise subtraction operation is performed to transform the first extended feature. EF 1 and the first extended cluster center EC 1. Perform element-wise subtraction to obtain the first residual feature that characterizes the degree to which local features deviate from the cluster center. RF 1. Size is 64 512 ( H / 16 W / 16).

[0162] Step 65: Use the soft-assigned weight map to perform feature modulation on the first residual feature to obtain the first weighted residual feature.

[0163] To ensure that the first residual feature corresponds to its soft assignment probability, this embodiment will use the soft assignment weight map obtained in the previous steps. SW 1. After processing with the weight expansion operation WeightUnsqueeze_1, the weights are expanded into the first expanded weights along the feature channel dimension. EW 1, its size becomes 64 1 ( H / 16 W / 16).

[0164] Subsequently, the first element-wise multiplication operation Multi_1 is performed, and the first residual feature is... RF 1 and the first extended weight EW 1. Perform element-wise multiplication to modulate the residuals according to the assigned weights, thus obtaining the first weighted residual feature. WRF 1. Size is 64 512 ( H / 16 W / 16).

[0165] Step 66: Accumulate and aggregate the first weighted residual features in the spatial dimension, and then normalize and flatten them to obtain the target global aggregated descriptor.

[0166] Finally, in order to aggregate the discrete local residuals into an image-level global representation, a spatial dimension accumulation operation Sum_1 is performed to accumulate the first weighted residual features along the spatial location dimension. WRF 1. Summation is performed to eliminate the spatial location dimension, resulting in the first VLAD aggregated feature. VF 1. Size is 64 512.

[0167] The first VLAD aggregation feature VF 1. The descriptor undergoes a second L2 normalization operation (L2Norm_2) and a fourth-dimensional rearrangement operation (Reshape_4) sequentially. The second L2 normalization operation normalizes the residual aggregation vectors corresponding to each cluster center along the feature dimension to reduce the magnitude differences between feature vectors corresponding to different cluster centers and balance the contribution of each cluster center to the final descriptor. Subsequently, the fourth-dimensional rearrangement operation flattens the normalized aggregation features into a one-dimensional descriptor vector, resulting in the target global aggregation descriptor. GD Its dimension is 32768 (i.e., the product of 64 and 512).

[0168] Since the above process is a feature extraction mechanism, when the input images to be processed are the query image, the positive sample image, and the negative sample image, the target global aggregate descriptors output by the entire forward propagation pipeline are the query global aggregate descriptor, the positive sample global aggregate descriptor, and the negative sample global aggregate descriptor, respectively.

[0169] The training method for the visual location recognition model provided by this invention achieves high translation invariance and robustness to illumination and viewpoint at the image feature representation level through the aforementioned local feature extraction and VLAD aggregation steps. First, by truncating the pooling layer at the end of the VGG network to obtain the first local feature map, the loss of key spatial details caused by downsampling in traditional CNN models at this stage is avoided. Second, the VLAD aggregation module employs a method based on 1... The soft allocation mechanism of convolution and Softmax transforms the non-differentiable hard allocation of traditional clustering algorithms into a smooth, end-to-end differentiable operation, significantly improving the backpropagation efficiency of network gradients. Crucially, L2 normalization is applied to local features before and after aggregation to reduce the magnitude differences between different local descriptors and cluster residual vectors, balance the contribution of different cluster centers to the final descriptor, and mitigate the excessive influence of repetitive texture features such as dense leaves and repeated windows on the global descriptor. Through these processes, a variable number of local features can be aggregated into a fixed-dimensional global aggregated descriptor, which helps improve the robot's feature discrimination and anti-interference capabilities during location identification in large-scale, long-cycle environmental inspections.

[0170] Based on the training method of the visual location recognition model provided in the above embodiments, after obtaining the reconstructed difference features used to characterize the appearance differences under changing environmental conditions, and various calibrated descriptor sequences, the embodiments of the present invention further provide a specific implementation method for how to construct a multi-task joint loss function to drive the iterative update of the entire network parameters.

[0171] Before describing the specific loss calculation steps, the training environment and hyperparameter configuration of this embodiment will be explained first. As an optional embodiment, the entire reference condition calibration training network can use a publicly available location identification dataset, such as the training set in the Pitts250k dataset, as the training data. During the model parameter optimization phase, a Stochastic Gradient Descent (SGD) optimizer can be used, with the total number of training epochs preferably set to 8. To ensure rapid exploration in the early stages of training and smooth convergence in the later stages, the initial learning rate of the SGD optimizer can be set to 0.001, the momentum parameter to 0.9, and the weight decay parameter to 0.001. Simultaneously, a step-wise learning rate decay strategy can be adopted during training; for example, the learning rate automatically decays to 0.5 times its original value every two generations.

[0172] Within this training framework, the calculation of the variational autoencoder reconstruction loss specifically includes, but is not limited to, the following steps: Step 1: Calculate the feature reconstruction error between the reconstructed difference features and the sample pair difference features.

[0173] To ensure that the reference conditional encoding module retains environmental change information as completely as possible during compression and decoding, this embodiment extracts the second reconstructed difference feature output by the decoder in the preceding steps and compares it with the feature dimension of the original input sample difference feature.

[0174] In a specific computational embodiment, the mean squared error algorithm can be used to calculate the squared difference between the second reconstructed difference feature and the sample pair difference feature in each feature dimension, and average all the squared differences to obtain the feature reconstruction error. This feature reconstruction error intuitively reflects the severity of the loss of original visual difference information after the model has undergone feature compression, reparameterized sampling and decoding reconstruction.

[0175] Step 2: Calculate the distribution divergence loss of the latent variables based on the mean characteristics and log-variance characteristics of the latent variables.

[0176] To prevent overfitting of the latent variable space and to force it to conform to the statistical prior of a standard normal distribution, this embodiment extracts the latent variable mean and log-variance features generated during the encoding stage. For example, by using the KL divergence algorithm, the distance between the current distribution formed by the mean and log-variance and the standard normal distribution is quantified. This distance calculation result is the latent variable distribution divergence loss, which is essentially a regularization penalty term used to constrain the smoothness of the feature distribution.

[0177] Step 3: Based on preset weight coefficients, the feature reconstruction error and the latent variable distribution divergence loss are weighted and fused to obtain the variational autoencoder reconstruction loss.

[0178] In this embodiment, the calculated feature reconstruction error and the latent variable distribution divergence loss are linearly summed using preset weighting coefficients. Let the weighting coefficient for the feature reconstruction error be... The weighting coefficients for the divergence loss of latent variable distributions are: .

[0179] In a hyperparameter configuration example It can be set to 1. β It can be set to 0.01, then the final variational autoencoder reconstruction loss will be... Lr The mathematical expression can be: ; in, Indicates the second reconstruction difference feature in the first... j The numerical value of the dimension; Indicates the difference features of the sample pair in the th case. j The numerical value of the dimension; and These represent the latent variable mean characteristics and the latent variable logarithmic variance characteristics, respectively, on the 1st... i The numerical values ​​of the dimensions; 32768 and 512 are the total dimensions of the corresponding features.

[0180] Accordingly, the calculation of the contrast loss based on the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor specifically includes the following steps: Step 1: Calculate the positive sample descriptor similarity between the calibration query descriptor and the calibration positive sample descriptor.

[0181] After residual calibration using reference condition features, calibration query descriptors and calibration positive sample descriptors for the same physical location are obtained. In this embodiment, the spatial distance between the two is evaluated by the inner product operation (or cosine similarity operation) of feature vectors, and the inner product value calculated using a preset descriptor similarity calculation function is the positive sample descriptor similarity, representing the ability of the visual location recognition model to bring the features of images of the same location closer in the current feature space.

[0182] Step 2: Calculate the negative sample descriptor similarity between the calibration query descriptor and each of the calibration negative sample descriptors.

[0183] For training batches, the images correspond to different physical locations as query images. N For each calibration negative sample descriptor, repeat the similarity calculation function described above. Calculate the inner product between the calibration query descriptor and each calibration negative sample descriptor to obtain... N The similarity of negative sample descriptors is used to measure the degree of incorrect confusion the model makes about non-homogeneous locations.

[0184] Step 3: Based on a preset temperature coefficient, perform exponential normalization on the similarity of the positive sample descriptors and the similarity of each negative sample descriptor to obtain the target comparison probability.

[0185] To make the loss function more sensitive to difficult negative samples (i.e., negative samples with abnormally high similarity), a preset temperature coefficient is introduced. Divide the similarity of positive sample descriptors and the similarity of all negative sample descriptors by the temperature coefficient. Then, an exponential operation is performed. Subsequently, the exponential result of the positive sample is divided by the sum of the exponential results of the positive and negative samples, completing a normalization operation similar to Softmax, thereby obtaining a target contrast probability between 0 and 1.

[0186] In an alternative embodiment, the temperature coefficient The preferred setting is 0.07.

[0187] Step 4: Calculate the logarithmic loss based on the target contrast probability to obtain the contrast loss.

[0188] The natural logarithm of the obtained target contrast probabilities is taken, and the result is then negative. Finally, the average of all query images and their negative samples in the training batch is calculated to obtain the final contrast loss. : ; in, Q Indicates the number of query images in the training batch; N This indicates the number of negative sample images corresponding to each query image; This represents the function for calculating descriptor similarity. , and These represent the calibration query descriptor, the calibration positive sample descriptor, and the first... n A calibration negative sample descriptor.

[0189] After calculating the variational autoencoder reconstruction loss respectively Compared with loss Then, the two are combined to construct the overall objective loss function: ; Finally, using the SGD optimizer, based on this total loss... The parameter gradient is calculated, and the network parameters of the feature extraction module, VLAD aggregation module, reference condition encoding module, and reference condition calibration module in the reference condition calibration training network are jointly updated in reverse.

[0190] This invention constructs a dual constraint mechanism at the parameter optimization level by combining variational autoencoder reconstruction loss and the contrastive loss of the InfoNCE architecture. On one hand, the reconstruction error and KL divergence in the variational autoencoder reconstruction loss work together to constrain the reference condition encoding module to encode environmental differences into continuous, regular, and near-standard normal prior distribution low-dimensional latent variable representations. This helps reduce the risk of the model overfitting specific lighting or viewpoint changes in the training set and improves the generalization ability of the extracted reference condition features. On the other hand, the contrastive loss with a temperature coefficient is precisely applied to the calibrated descriptor space. By increasing the similarity between query image features and positive sample features and decreasing the similarity between negative sample features and query image features, the network can autonomously learn smaller intra-class differences and larger inter-class differences after fully perceiving environmental interference. The joint backpropagation of these two loss functions enables the finally trained visual location recognition model to possess robustness against complex external environmental fluctuations and high-precision location recall during the testing phase.

[0191] Based on the training method of the visual location recognition model described in the above embodiments, after the joint optimization of network parameters is completed, only the updated feature extraction module and VLAD aggregation module are retained as the final visual location recognition model. This invention further provides a specific implementation method for performing robot visual location recognition in a real-world scenario using this trained model.

[0192] Figure 9 This is a flowchart illustrating the visual location recognition method provided by the present invention. Figure 9 As shown, after training the visual location recognition model, the visual location recognition method may specifically include the following steps: Step 91: Obtain the query image of the scene to be identified, input the query image into the visual location recognition model, and obtain the global aggregate descriptor of the query.

[0193] In practical robot inspection or autonomous driving tasks, robots can use their onboard visual sensors, such as RGB color cameras, to capture images of their surrounding environment in real time. These captured images are then used as query images for the scene to be identified and directly input into the previously trained visual location recognition model.

[0194] After local feature extraction by the feature extraction module and spatial accumulation and normalization by the VLAD aggregation module in the visual location recognition model, the model outputs a high-dimensional one-dimensional feature vector during forward propagation, which is the query global aggregation descriptor. It should be noted that this inference stage completely discards the reference conditional encoding module and calibration module used during training, making the extraction process extremely lightweight and efficient.

[0195] Step 92: Perform dimensionality reduction processing on the global aggregate descriptor of the query to obtain a dimensionality-reduced global aggregate descriptor of the query.

[0196] Since high-dimensional global aggregation descriptors consume significant memory bandwidth and substantially increase matching time during large-scale map retrieval, this embodiment introduces a dimensionality reduction mapping mechanism after feature output. Specifically, dimensionality reduction processing is performed on the query global aggregation descriptor.

[0197] In one optional embodiment, principal component analysis (PCA) can be used for linear dimensionality reduction compression. In other optional embodiments, nonlinear dimensionality reduction techniques such as locality-preserving projections (LPP) or pre-trained autoencoders can be used to map high-dimensional descriptors to low-dimensional subspaces, thereby obtaining compact dimensionality-reduced global aggregate descriptors that retain the most discriminative core spatial visual information while significantly eliminating redundant feature dimensions.

[0198] Step 93: Obtain the dimensionality-reduced global aggregate descriptor of each database image in the environment map.

[0199] To enable location comparison, this embodiment pre-builds an environmental map, which typically consists of a series of database images collected by the robot during the mapping phase, each with precise physical coordinates or topological node annotations.

[0200] To ensure real-time online recognition, feature extraction from these database images is typically performed offline. Specifically, the dimensionality-reduced global aggregated descriptor is pre-generated by performing forward inference extraction on each database image using the identical visual location recognition model and processing it with the same dimensionality reduction algorithm (e.g., a shared PCA projection matrix). During the robot's online operation, this set of reference features, composed of the various dimensionality-reduced global aggregated descriptors, can be directly retrieved from memory or the database.

[0201] Step 94: Calculate the feature distance between the global aggregate descriptor of the dimensionality reduction query and each of the global aggregate descriptors of the dimensionality reduction database.

[0202] After obtaining the compact feature representation of the current scene and the reference feature base library of the environment map, this embodiment begins to perform similarity retrieval. Specifically, in the feature space, the feature distance between the global aggregate descriptor of the dimensionality reduction query and each global aggregate descriptor of the dimensionality reduction database is calculated one by one. As an optional implementation, this feature distance can be achieved by calculating the Euclidean distance between two vectors; the smaller the distance value, the more similar the two are visually and semantically. In other optional embodiments, cosine distance or Manhattan distance can also be used as a metric to measure the deviation in the feature space.

[0203] Step 95: The database image to which the dimensionality reduction database global aggregation descriptor corresponding to the minimum feature distance among all the feature distances belongs is taken as the visual location recognition result of the scene to be identified.

[0204] After completing the full database comparison, all calculated feature distances are sorted in ascending order. This includes extracting the feature distance with the smallest value and using a reverse index to find the dimensionality-reduced global aggregate descriptor of the database that generated the smallest feature distance. Further, the specific database image to which this descriptor belongs is traced. Since the database images are pre-bound with explicit physical location attributes, the location information of the best-matching database image can be directly output as the visual location recognition result for the robot's current scene, thus completing the localization loop from image pixels to physical location.

[0205] The visual location recognition method provided by this invention significantly compresses high-dimensional features by introducing dimensionality reduction processing such as PCA before retrieval, effectively eliminating redundant noise in the feature space. The combination of highly robust aggregated descriptors and low-dimensional distance calculation enables robots to achieve high-frequency, real-time closed-loop detection and location relocalization on resource-constrained mobile hardware without increasing any online inference computing power burden, fundamentally reducing the location mismatch rate caused by environmental changes.

[0206] Figure 10 This is a schematic diagram of the structure of the training device for the visual location recognition model provided by the present invention, as shown below. Figure 10 As shown, it mainly includes, but is not limited to: The network construction module 101 is used to construct a reference condition calibration training network, which includes a feature extraction module, a VLAD aggregation module, a reference condition encoding module, and a reference condition calibration module. The sample acquisition module 102 is used to acquire query image samples in the training set, as well as positive and negative sample images at the same location corresponding to the query image samples. The initial feature extraction module 103 is used to input the query image sample, the positive sample image, and the negative sample image into the feature extraction module and the VLAD aggregation module respectively in sequence, so as to obtain the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor. The difference construction module 104 is used to construct sample pair difference features based on the descriptor difference between the query global aggregate descriptor and the positive sample global aggregate descriptor; The encoding and reconstruction module 105 is used to input the difference features of the sample pair into the reference condition encoding module to obtain reference condition features and reconstructed difference features used to characterize the appearance differences under changes in environmental conditions. The descriptor calibration module 106 is used to form an initial descriptor sequence from the query global aggregate descriptor, the positive sample global aggregate descriptor, and the negative sample global aggregate descriptor, and input it together with the reference condition feature into the reference condition calibration module to obtain the calibrated query descriptor, the calibrated positive sample descriptor, and the calibrated negative sample descriptor; The loss calculation module 107 is used to calculate the contrast loss based on the calibration query descriptor, the calibration positive sample descriptor and the calibration negative sample descriptor, and to calculate the variational autoencoder reconstruction loss based on the reconstruction difference features and the sample pair difference features. The model update module 108 is used to jointly update the network parameters of the reference condition calibration training network by combining the contrast loss and the variational autoencoder reconstruction loss, so as to use the updated feature extraction module and the VLAD aggregation module as the trained visual location recognition model.

[0207] It should be noted that the training device for the visual location recognition model provided by the present invention can execute the training method for the visual location recognition model described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0208] The training device for the visual location recognition model provided by this invention explicitly models the appearance differences of the same physical location under different conditions, using these differences as reference conditions to perform residual calibration on the descriptors. This enables the model to learn a more stable and consistent representation of the location, effectively overcoming feature fluctuations caused by environmental changes such as viewing angle and lighting. No calibration module is needed during the testing phase, significantly improving the robustness of visual location recognition without increasing inference computation.

[0209] Figure 11 This is a schematic diagram of the structure of the visual location recognition device provided by the present invention, as shown below. Figure 11 As shown, it mainly includes, but is not limited to: The query feature extraction module 111 is used to acquire a query image of the scene to be identified, input the query image into the visual location recognition model, and obtain a global aggregated descriptor for the query. The query dimensionality reduction module 112 is used to perform dimensionality reduction processing on the global aggregate descriptor of the query to obtain a dimensionality-reduced global aggregate descriptor of the query. The database feature acquisition module 113 is used to acquire the dimensionality-reduced global aggregate descriptor of each database image in the environment map. The dimensionality-reduced global aggregate descriptor of the database is obtained by the visual location recognition model after feature extraction and dimensionality reduction of the database image. Distance calculation module 114 is used to calculate the feature distance between the global aggregate descriptor of the dimensionality reduction query and each of the global aggregate descriptors of the dimensionality reduction database; The result determination module 115 is used to take the database image to which the dimensionality reduction database global aggregation descriptor corresponding to the minimum feature distance among all the feature distances belongs as the visual location recognition result of the scene to be identified.

[0210] It should be noted that, in actual operation, the visual location recognition device provided by this invention will call the visual location recognition model trained by the training method of the visual location recognition model provided in any of the above embodiments to perform subsequent actions. The training method of the visual location recognition model described in any of the above embodiments will not be described in detail in this embodiment.

[0211] The visual location recognition device provided by this invention obtains local feature maps through a feature extraction module, and inputs these local feature maps into a VLAD aggregation module to obtain a global aggregated descriptor. Simultaneously, during the training phase, a difference in the global aggregated descriptor between a query image and a positive sample image of the same location is constructed. This difference is used to obtain reference condition features through a reference condition encoding module, and a reference condition calibration module is used to perform residual calibration on the global aggregated descriptor, thereby improving the robustness of the global aggregated descriptor to changes in environmental conditions. During the testing phase, only the feature extraction module and VLAD aggregation module obtained from the above training are used for single-image descriptor extraction, thus improving the robustness of the robot's visual location recognition without increasing the inference computation load.

[0212] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0213] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a visual location recognition model, characterized in that, include: A reference condition calibration training network is constructed, which includes a feature extraction module, a VLAD aggregation module, a reference condition encoding module, and a reference condition calibration module. Obtain the query image samples in the training set, as well as the positive and negative sample images of the same location corresponding to the query image samples; The query image sample, the positive sample image, and the negative sample image are sequentially input into the feature extraction module and the VLAD aggregation module to obtain the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor, respectively. Based on the descriptor differences between the global aggregated descriptor of the query and the global aggregated descriptor of the positive sample, construct sample pair difference features; The sample pair difference features are input into the reference condition encoding module to obtain reference condition features and reconstructed difference features used to characterize appearance differences under changing environmental conditions. The query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor are combined to form an initial descriptor sequence, which is then input together with the reference condition feature into the reference condition calibration module to obtain the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor. The contrast loss is calculated based on the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor; Based on the reconstructed difference features and the sample pair difference features, the variational autoencoder reconstruction loss is calculated; By combining the contrast loss and the variational autoencoder reconstruction loss, the network parameters of the reference condition calibration training network are jointly updated, so that the updated feature extraction module and the VLAD aggregation module can be used as the trained visual location recognition model.

2. The training method for the visual location recognition model according to claim 1, characterized in that, The step of inputting the sample pair difference features into the reference condition encoding module to obtain reference condition features and reconstructed difference features for characterizing appearance differences under changing environmental conditions includes: The sample is subjected to a linear encoding operation, a batch normalization encoding operation, and an activation encoding operation in sequence to obtain the first hidden feature; The first hidden feature is subjected to mean mapping and variance mapping respectively to obtain the latent variable mean feature and the latent variable log-variance feature. Based on the log-variance feature of the latent variable, the standard deviation feature of the latent variable is obtained, and random noise features with the same size as the standard deviation feature of the latent variable are generated; Reparameterized sampling is performed based on the latent variable mean feature, the latent variable standard deviation feature, and the random noise feature to obtain the sampled latent features; Based on the latent variable mean feature and the latent variable standard deviation feature, a first reference condition feature is generated as the reference condition feature; The sampled latent features are sequentially subjected to a decoding linear operation and a decoding normalization operation to obtain a second reconstructed difference feature, which is then used as the reconstructed difference feature.

3. The training method for the visual location recognition model according to claim 2, characterized in that, The step of reparameterizing sampling based on the latent variable mean feature, the latent variable standard deviation feature, and the random noise feature to obtain sampled latent features includes: The random noise features are modulated using the latent variable standard deviation feature to obtain the noise distribution features; The noise distribution features are fused with the latent variable mean features to obtain the sampling latent features; Accordingly, generating the first reference condition feature based on the latent variable mean feature and the latent variable standard deviation feature includes: The first reference condition feature is obtained by fusing the latent variable standard deviation feature with the latent variable mean feature.

4. The training method for the visual location recognition model according to claim 1, characterized in that, The step of assembling the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor into an initial descriptor sequence, and inputting it together with the reference condition features into the reference condition calibration module to obtain calibrated query descriptors, calibrated positive sample descriptors, and calibrated negative sample descriptors includes: The reference condition feature is then expanded and copied in dimensions to obtain the second reference condition feature; The initial descriptor sequence is concatenated with the second reference condition feature to obtain the descriptor-reference condition joint feature; The joint features of the descriptor reference conditions are pooled and mapped respectively to obtain the compensation features and calibration weights. The compensation features are modulated using the calibration weights to obtain weighted compensation features; The initial descriptor sequence and the weighted compensation features are residually fused and then normalized to obtain the second calibration descriptor; The second calibration descriptor is split according to the sample sequence dimension to obtain the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor.

5. The training method for the visual location recognition model according to claim 1, characterized in that, The step of sequentially inputting the query image sample, the positive sample image, and the negative sample image into the feature extraction module and the VLAD aggregation module to obtain the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor includes: The feature extraction module is used to extract local features from the image to be processed, resulting in a first local feature map. In the VLAD aggregation module, the first local feature map is normalized to obtain the first normalized local feature map; The first normalized local feature map is subjected to soft-allocation feature extraction and normalization mapping to obtain a soft-allocation weight map; Calculate the first residual feature between the first normalized local feature map and the preset learnable cluster centers; The first residual feature is modulated using the soft-assigned weight map to obtain the first weighted residual feature; The first weighted residual features are accumulated and aggregated in the spatial dimension, and then normalized and flattened to obtain the target global aggregated descriptor; Wherein, when the images to be processed are respectively the query image sample, the positive sample image, and the negative sample image, the corresponding target global aggregation descriptors are respectively the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor.

6. The training method for the visual location recognition model according to claim 2, characterized in that, The step of calculating the variational autoencoder reconstruction loss based on the reconstructed difference features and the sample pair difference features includes: Calculate the feature reconstruction error between the reconstructed differential features and the sample pair differential features; Based on the mean characteristics and log-variance characteristics of the latent variables, the distribution divergence loss of the latent variables is calculated; Based on preset weighting coefficients, the feature reconstruction error and the latent variable distribution divergence loss are weighted and fused to obtain the variational autoencoder reconstruction loss.

7. The training method for the visual location recognition model according to claim 1, characterized in that, The calculation of contrast loss based on the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor includes: Calculate the positive sample descriptor similarity between the calibration query descriptor and the calibration positive sample descriptor; Calculate the negative sample descriptor similarity between the calibration query descriptor and each of the calibration negative sample descriptors; Based on a preset temperature coefficient, the similarity of the positive sample descriptors and the similarity of each negative sample descriptor are exponentially normalized to obtain the target comparison probability. The contrast loss is obtained by calculating the logarithmic loss based on the target contrast probability.

8. A visual location recognition method, characterized in that, After training the visual location recognition model based on the training method of any one of claims 1 to 7, the method further includes: Obtain a query image of the scene to be identified, input the query image into the visual location recognition model, and obtain a global aggregate descriptor for the query; The global aggregate descriptor of the query is subjected to dimensionality reduction processing to obtain a dimensionality-reduced global aggregate descriptor of the query; Obtain the dimensionality-reduced global aggregate descriptor of each database image in the environment map. The dimensionality-reduced global aggregate descriptor is obtained by the visual location recognition model after feature extraction and dimensionality reduction of the database images. Calculate the feature distance between the global aggregate descriptor of the dimensionality reduction query and each of the global aggregate descriptors of the dimensionality reduction database; The database image to which the dimensionality reduction database global aggregation descriptor corresponding to the minimum feature distance among all the feature distances belongs is taken as the visual location recognition result of the scene to be identified.

9. A training device for a visual location recognition model, characterized in that, include: A network construction module is used to construct a reference condition calibration training network, which includes a feature extraction module, a VLAD aggregation module, a reference condition encoding module, and a reference condition calibration module. The sample acquisition module is used to acquire query image samples in the training set, as well as positive and negative sample images at the same location corresponding to the query image samples; The initial feature extraction module is used to input the query image sample, the positive sample image, and the negative sample image into the feature extraction module and the VLAD aggregation module respectively, so as to obtain the query global aggregation descriptor, the positive sample global aggregation descriptor, and the negative sample global aggregation descriptor. The difference construction module is used to construct sample pair difference features based on the descriptor differences between the query global aggregate descriptor and the positive sample global aggregate descriptor; The encoding and reconstruction module is used to input the difference features of the sample pair into the reference condition encoding module to obtain the reference condition features and reconstructed difference features used to characterize the appearance differences under changes in environmental conditions. The descriptor calibration module is used to form an initial descriptor sequence from the query global aggregate descriptor, the positive sample global aggregate descriptor, and the negative sample global aggregate descriptor, and input it together with the reference condition feature into the reference condition calibration module to obtain the calibrated query descriptor, the calibrated positive sample descriptor, and the calibrated negative sample descriptor; The loss calculation module is used to calculate the contrast loss based on the calibration query descriptor, the calibration positive sample descriptor, and the calibration negative sample descriptor, and to calculate the variational autoencoder reconstruction loss based on the reconstruction difference features and the sample pair difference features. The model update module is used to jointly update the network parameters of the reference condition calibration training network by combining the contrast loss and the variational autoencoder reconstruction loss, so as to use the updated feature extraction module and the VLAD aggregation module as the trained visual location recognition model.

10. A visual location recognition device, characterized in that, Execution of a visual location recognition model trained using the training method of any one of claims 1 to 7, comprising: The query feature extraction module is used to acquire a query image of the scene to be identified, input the query image into the visual location recognition model, and obtain a global aggregated descriptor for the query. The query dimensionality reduction module is used to perform dimensionality reduction processing on the global aggregate descriptor of the query to obtain a dimensionality-reduced global aggregate descriptor of the query. The database feature acquisition module is used to acquire the dimensionality-reduced global aggregate descriptor of each database image in the environment map. The dimensionality-reduced global aggregate descriptor is obtained by the visual location recognition model after feature extraction and dimensionality reduction of the database image. The distance calculation module is used to calculate the feature distance between the global aggregate descriptor of the dimensionality reduction query and each of the global aggregate descriptors of the dimensionality reduction database; The result determination module is used to take the database image to which the dimensionality reduction database global aggregation descriptor corresponding to the minimum feature distance among all the feature distances belongs as the visual location recognition result of the scene to be identified.

Images