A long-term visual localization method based on a relational guidance network

Abstract

The present invention relates to the field of computer vision technology, and discloses a long-term visual positioning method based on an association guidance network. The model training stage includes perceptual network training and concept network training; after the concept network training is completed, the concept network is used as a pre-training model to train the perceptual network, and the concept network is optimized at the same time; the association guidance mechanism performs information interaction; in the image retrieval stage, the trained perceptual network is used to perform image retrieval. The invention uses domain adaptive learning and an association guidance mechanism to use the features in the concept network to guide the feature learning in the perceptual network, so that the features finally obtained by the model training are robust under environmental changes. In addition, the present invention uses a concept database that does not require additional data to perform self-inspirational learning on the concept network, so as to better guide the perceptual network with domain features, thereby improving the final image retrieval performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a long-term visual positioning method based on an association guidance network. BACKGROUND

[0002] Visual positioning plays a crucial role in autonomous driving and mobile robots, especially in the SLAM system in long-term simultaneous localization and mapping. In long-term visual positioning, changes in factors such as light and season will have a significant impact on the positioning results, so it is of great significance to effectively solve the problem of long-term visual positioning. At present, the image retrieval method is a convenient and effective image-based positioning technology. This method finds the most similar location to the query image from the database through image retrieval to complete positioning and obtain a rough result, which can be used as the position recognition of the loop in SLAM (Simultaneous localization and Mapping, SLAM) or regression of the high-precision six-degree-of-freedom camera pose.

[0003] The document "IEEE / CAA Journal of Automatica Sinica, 313-328, 2021" proposes a new gradient-weighted similarity activation mapping loss. The network enhances the contrast learning of the embedding in a self-supervised manner to improve the accuracy of image matching. Although this method designs a gradient-weighted similarity activation mapping loss and improves the performance of the model, it still cannot achieve ideal results in some complex environmental changes. The document "Engineering Applications of Artificial Intelligence, 104793, 2022" proposes a method that fuses depth and semantic information in the scene. A scene depth and semantic information fusion module is used to extract the scene representation that remains unchanged when the environment changes and to improve the accuracy of long-term visual positioning through image retrieval. Although this method introduces additional effective information such as semantics and depth to improve the overall performance of the network, it is also difficult to obtain these additional information in actual situations and increases the computational complexity of the model. SUMMARY

[0004] The purpose of the present application is to provide a long-term visual positioning method based on an association guidance network to improve the accuracy and robustness of visual positioning.

[0005] The technical solution of the present application is as follows: a long-term visual positioning method based on an association guidance network, comprising a model training stage and an image retrieval stage.

[0006] The model training stage includes perception network training and concept network training; after the concept network training is completed, the concept network is used as a pre-training model to train the perception network, and the concept network is optimized at the same time; the encoding features of the perception network and the encoding features of the concept network are extracted respectively, information interaction is performed through the association guidance mechanism, which is used as an influencing factor of the concept network loss function and the perception network loss function, and the perception network is updated in reverse;

[0007] In the image retrieval stage, the trained perception network is used for image retrieval; the perception network is used for feature extraction of the image to be queried, the obtained features are compared with the features of the pictures in the retrieval database, and the picture with the most similar features is found and the pose of the picture is obtained, and the pose is assigned to the query picture to obtain the pose of the query picture.

[0008] The concept network is a multi-domain network structure, each domain is composed of a pair of encoder, decoder and discriminator, and the framework formula of the concept network is as follows:

[0009]

[0010] Among them, x is a picture i The encoding feature in the concept network domain i, x is a picture The decoding result in the concept network domain j, x is a picture The encoding feature in the concept network domain j, x is a picture i The reconstruction picture result in the concept network domain i; The encoder of the concept network domain i, The decoder of the concept network domain j;

[0011] In the training process of the concept network, the pictures randomly extracted in each domain are stored and the association guidance database of each domain is established, and when the accumulated data reaches a certain amount, the association guidance mechanism is activated; the association guidance mechanism extracts features similar to the input image by using the encoder in the concept network through the data in the association guidance database, and assists the concept network to generate features with discriminative characteristics.

[0012] The training of the concept network includes four loss functions, which are ComboGAN loss function Feature consistency loss function SAM loss function And adaptive triple feature loss function

[0013] The overall loss function of the concept network is represented as follows:

[0014]

[0015] The ComboGAN loss function includes an adversarial loss and a cycle consistency loss;

[0016] is defined as follows:

[0017]

[0018] The converted image in the encoding-decoding process is converted through the encoder E i C and the decoder D i C Find the cycle consistency loss, defined as follows:

[0019]

[0020] In addition, the translated image is found through the discriminator G j Adversarial loss, defined as follows:

[0021]

[0022] The overall representation of the feature consistency loss function is as follows:

[0023]

[0024] Where λ FCL and θ CFL are hyperparameters;

[0025] In order to obtain the features across different domains, an additional loss on the latent space is used, for the image x i in the i domain, its expression is defined as:

[0026]

[0027] In order to further improve the extraction of latent features by the concept network, the association guidance mechanism is used for the self- heuristic learning of the concept network, which is expressed as follows:

[0028]

[0029] Where p j is the picture that the concept network associates from the association guidance database;

[0030] The feature loss between domains is defined as follows:

[0031]

[0032] The SAM loss function is represented as a whole:

[0033]

[0034] Where θ SAM and θ AAM are hyperparameters;

[0035] The Grad-SAM loss function is represented as follows:

[0036]

[0037] The SAM loss function is introduced into the association guidance mechanism, and is defined as follows:

[0038]

[0039] Where, is the similarity response graph of the picture associated with the concept network from the association guidance database; L i，m is the similarity response graph of the input image;

[0040] The adaptive triple feature loss function is defined as a whole:

[0041]

[0042] Where m f is the margin, α f is a weight parameter; is the feature obtained by decoding in the i domain and then encoding in the j domain, is the feature obtained by encoding in the j domain.

[0043] The association guidance database stores a limited number of data. When the data stored in the current domain reaches the upper limit, one of the previously stored data in the current domain is randomly deleted, and the current data is stored, so as to update the association guidance database of each domain.

[0044] The structure of the perception network is composed of multiple domains, each domain containing a pair of encoder and decoder, and the framework of the perception network is formulated as follows:

[0045]

[0046] The perception network completes the final retrieval, is the picture x i is the encoded feature in the i domain in the perception network, is the is the result of decoding in the j domain in the perception network, is the is the encoded feature in the j domain in the perception network, is the picture xi reconstructed picture result of domain i in the perception network; for predicting an output prediction, a predicted picture and a predicted pose for the output.

[0047] The loss function of the perception network mainly consists of three parts, namely the feature consistency loss function the SAM loss function and the adaptive triple feature loss function

[0048] The overall loss function of the perception network is represented as:

[0049]

[0050] the SAM loss function in the perception network the SAM loss function in the concept network are the same;

[0051] Therefore, the feature consistency loss function is defined as:

[0052]

[0053] The feature guidance loss function of the concept network corresponding to the domain of the perception network is represented as follows:

[0054]

[0055] The overall definition of the adaptive triple feature loss function is as follows:

[0056]

[0057]

[0058] wherein, is the adaptive triple feature loss obtained in the perception network, is the adaptive triple feature loss obtained in the concept network.

[0059] The association guidance mechanism provides an information interaction mechanism; the information interaction mechanism guides the concept network training by storing the trained images in the process of concept network training and activating after reaching a certain number, extracting features by searching for similar images to the trained images in the stored association guidance database, and guiding the concept network training; the information interaction mechanism guides the perception network training by comparing the loss functions in the perception network training and the concept network.

[0060] Beneficial effects of the present invention: In response to the problems of non-robust feature extraction in previous long-term visual positioning algorithms, the present invention proposes a long-term visual positioning method based on an association guidance network. This invention uses domain-adaptive learning and an association guidance mechanism to use features in the concept network to guide feature learning in the perception network, making the features finally trained by the model robust under environmental changes. Furthermore, the present invention uses a concept database that does not require additional data for self-inspired learning of the concept network, thereby better guiding the perception network with domain features and improving the final image retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Flowchart of a long-term visual localization method based on association guidance network.

[0062] Figure 2 A conceptual network diagram.

[0063] Figure 3 Schematic diagram of the perception network. DETAILED DESCRIPTION

[0064] Figure 1 This is the main flow chart of the technical solution of the present invention. Figure 1 As shown in FIG, the long-term visual positioning algorithm based on the association guidance network proposed in the present invention includes the following steps:

[0065] (1) Concept Network Training: In the algorithm network framework proposed in this paper, the concept network is trained first. The concept network is a multi-domain network structure, each domain consists of a pair of encoders, decoders, and discriminators. The proposed network model is mainly aimed at the image retrieval problem of changing scenes. The domain here generally refers to an environment, such as a snowy scene, a rainy scene, etc. The framework of the concept network is formulated as follows:

[0066]

[0067] in, For picture x i The encoded features in domain i, for The result of decoding in domain j is, for The encoded features in domain j, For picture x i Reconstructed image results in domain i.

[0068] In the process of training the concept network, the present invention is inspired by the association mechanism of the human brain and introduces an association guidance mechanism to help the network better solve image retrieval problems under changes in lighting, seasons, etc., thereby providing more stable feature guidance to the perception network.

[0069] In the concept network, the association guidance mechanism mainly plays a self-priming role. In the training process of the concept network, the pictures randomly extracted in each domain are stored and the association guidance database of each domain is established, and when the accumulated data reaches a certain amount, the association guidance mechanism is activated to provide corresponding guidance to the concept network. Due to the limited number of database storage, if the data stored in the current domain reaches the upper limit, one data previously stored in the current domain will be randomly deleted, and the current data will be stored, so as to update the database of each domain.

[0070] The concept network mainly includes four loss functions, which are ComboGAN loss function feature consistency loss function SAM loss function and adaptive triple feature loss function

[0071] The translated image is found through the encoder and the decoder The cycle consistency loss is found, which is defined as follows:

[0072]

[0073] In addition, the translated image is found through the discriminator G j to find the adversarial loss, which is defined as follows:

[0074]

[0075] For the pure multi-domain image translation task, namely ComboGAN, the total loss only includes the adversarial loss and the cycle consistency loss, which is defined as follows:

[0076]

[0077] In order to obtain the features across different domains, an additional loss on the latent space is used, which is called feature consistency loss. Under the above assumption, for the image x i in the i domain, its expression is defined as:

[0078]

[0079] In order to further improve the extraction of latent features by the network, the association guidance mechanism is used for self-priming learning of the concept network, which is expressed as follows:

[0080]

[0081] Where, p jThe picture of the concept network is associated from the multi-domain concept database.

[0082] In addition, the application also uses the feature loss between domains to enable the model to better obtain the domain adaptation model, which is defined as follows:

[0083]

[0084] For this, the overall representation of the feature consistency loss function is as follows:

[0085]

[0086] In order to better obtain effective features, Grad-SAM loss function is used, and the formula is as follows:

[0087]

[0088] In addition, the application introduces the SAM loss function into the association guidance mechanism, which is defined as follows:

[0089]

[0090] Wherein, The picture of the concept network is associated from the multi-domain concept database.

[0091] The overall representation of the SAM loss function is as follows:

[0092]

[0093] The triple loss function is widely used in the field of image recognition, and the adaptive triple feature loss function used in this paper is defined as:

[0094]

[0095]

[0096] Wherein, m f Margin, alpha f Weight parameter.

[0097] The overall definition of the adaptive triple feature loss function is as follows:

[0098]

[0099] The overall loss function of the concept network is represented as follows:

[0100]

[0101] After the concept network is well trained, it becomes a pre-trained network and further provides stable feature guidance for the domain adaptation of the perception network. The concept network is also optimized during the training of the perception network.

[0102] (2) Perception network training: After the training of the concept network is completed, the concept network will train the perception network as a pre-trained model. The concept network will continue to be optimized and will not be excluded from the framework. The perception network has a similar structure to the concept network, but it needs the associated guidance of the concept network during the training process, such as in Figure 1 The framework of the perception network is formulated as follows:

[0103]

[0104] Formula (16) is similar to formula (1), except that the superscript C in formula (1) represents the concept network, and the superscript P in formula (16) represents the perception network. In addition, the perception network also needs to complete the final retrieval, for the output prediction of , the predicted picture and pose for the output.

[0105] During the training process of the entire framework network, the selected pictures are simultaneously input into the concept network and the perception network. Through the guidance of the concept network to the perception network, the perception network can obtain better domain adaptation features. The triplet loss function structure of the concept network and the perception network is similar, so the concept network has a better guiding effect when it guides the corresponding domain of the perception network.

[0106] Compared with the training of the concept network, the loss function of the perception network is relatively simple and mainly consists of three parts, namely the feature consistency loss function SAM loss function and adaptive triplet feature loss function

[0107] SAM loss function in the perception network is the same as the SAM loss function in the concept network except for the superscript difference.

[0108] In the perception network, the feature consistency loss function is defined as:

[0109]

[0110] The feature guidance loss function of the concept network to the corresponding domain of the perception network is represented as follows:

[0111]

[0112] ​The feature loss function between domains is consistent with the concept network structure, and therefore, the total feature consistency loss function is defined as:

[0113]

[0114] The adaptive triple feature loss function in the perception network is defined as:

[0115]

[0116] The adaptive triple feature loss function is defined as follows:

[0117]

[0118] Therefore, the overall loss function of the perception network is represented as:

[0119]

[0120] (3) The association guidance mechanism is mainly an interaction mechanism in the two networks, and each network obtains features according to its own encoder and optimizes them. The role of the association guidance mechanism in the concept network is mainly self-heuristic, which mainly stores the trained images in the process of training the concept network and activates them after reaching a certain number, and extracts features from similar images in the stored association guidance database to perform so-called guidance. In the perception network, the interaction in the loss function is mainly reflected, which mainly performs so-called guidance through the comparison of the loss functions in the two networks.

[0121] (4) Image retrieval: In the inference process, only the perception network is used for the final image retrieval, and the concept network is excluded from the framework. Through the feature extraction of the image to be queried by the perception network, the obtained features are compared with the pictures in the retrieval database, and the picture with the most similar features is found and the pose of the picture is obtained, and the pose is assigned to the query picture to obtain the pose of the query picture.

[0122] To verify the effectiveness of the algorithm in long-term visual positioning accuracy, the present invention was tested on 17 sequences of the CMU-Seasons dataset. According to the experimental results, the algorithm proposed in this invention achieved high-precision and medium-precision positioning results of 24.1% and 48.7% in urban scenarios, 11.6% and 27.6% in suburban scenarios, and 13.3% and 33.3% in park scenarios. Compared with the DISMA method, the proposed method improved the high-precision and medium-precision positioning results by 1.5% and 1.4% in urban scenarios, 0.5% and 0.1% in suburban scenarios, and 0.7% and 2% in park scenarios.

Claims

1. A long-term visual localization method based on association guidance network, characterized in that: Includes model training phase and image retrieval phase; The model training phase includes perceptual network training and concept network training. After the concept network training is completed, the concept network is used as a pre-training model to train the perceptual network, and the concept network is optimized at the same time. Extracting the encoding features of the perception network and the encoding features of the concept network respectively, performing information interaction through an association guidance mechanism, taking them as influencing factors of the concept network loss function and the perception network loss function, and updating the perception network in reverse; In the image retrieval stage, the trained perceptual network is used to perform image retrieval. The perceptual network is used to extract features of the image to be queried, and the obtained features are compared with the images in the retrieval database. The image with the most similar features is found and the pose of the image is obtained. The pose is assigned to the query image to obtain the pose of the query image. The concept network is a multi-domain network structure, each domain consists of a pair of encoders, decoders and discriminators. The framework of the concept network is formulated as follows: in, For picture x i The encoded features in concept network domain i, for The result of decoding in the concept network domain j, for The encoded features in concept network domain j, For picture x i The reconstructed image results in concept network domain i; is the encoder of the concept network domain i, is the decoder of concept network domain j; During the training process of the concept network, randomly extracted images from each domain are stored and an association guidance database for each domain is established. When the accumulated data reaches a certain amount, the association guidance mechanism is activated. The association guidance mechanism extracts similar features from the data in the association guidance database using the encoder in the concept network, assisting the concept network in generating discriminative features. The training of the concept network includes four loss functions, namely ComboGAN loss function Feature consistency loss function SAM loss function and adaptive triple feature loss function The overall loss function of the concept network is expressed as follows: The ComboGAN loss function includes adversarial loss and cycle consistency loss; The definition is as follows: The converted image in the encoding and decoding process is passed through the encoder and decoder Find the cycle consistency loss, defined as follows: Furthermore, the translated image is passed through the discriminator G j To find the adversarial loss, it is defined as follows: The overall representation of the feature consistency loss function is as follows: Among them, λ FCL and λ CFL is a hyperparameter; In order to obtain features across different domains, an additional loss on the latent space is used. For the image x in domain i i , whose expression is defined as: In order to further improve the extraction of potential features by the concept network, the association guidance mechanism is used for the self-inspiration learning of the concept network, which is expressed as follows: Among them, p j The images associated with the concept network from the association guidance database; The feature loss between domains is defined as follows: The SAM loss function is expressed as follows: Among them, λ SAM and λ AAM is a hyperparameter; Grad-SAM loss function, the formula is as follows: The SAM loss function is introduced into the association guidance mechanism and is defined as follows: in, is the similarity response graph of the pictures associated by the concept network from the association guidance database; L i,m is the similarity response map of the input image; The adaptive triple feature loss function is defined as follows: Among them, m f is margin,α f is the weight parameter; is the feature obtained by encoding in the j domain after decoding in the i domain, is the feature obtained by encoding in the j domain.

2. The long-term visual positioning method based on association guidance network according to claim 1 is characterized in that The number of data stored in the association guidance database is limited. When the data stored in the current domain reaches the upper limit, a previously stored data is randomly deleted from the current domain and the current data is stored in it, so as to update the association guidance database of each domain.

3. The long-term visual positioning method based on association guidance network according to claim 1 is characterized in that The structure of the perception network consists of multiple domains, each of which contains a pair of encoders and decoders. The framework of the perception network is formulated as follows: The perception network completes the final retrieval, For picture x i The encoded features in domain i in the perception network, for The result of domain j decoding in the perceptual network, for The encoded features in domain j in the perception network, For picture x i The reconstructed image result of domain i in the perception network; Used for The output prediction of The output predicted image and predicted pose.

4. The long-term visual positioning method based on association guidance network according to claim 3 is characterized in that The loss function of the perception network consists of three parts: feature consistency loss function SAM loss function and adaptive triple feature loss function The overall loss function of the perception network is expressed as: SAM loss function in perception network SAM loss function in the same concept network same; Therefore, the feature consistency loss function is defined as; The feature-guided loss function of the concept network for the corresponding domain of the perception network is expressed as follows: The adaptive triple feature loss function is defined as follows: in, is the adaptive triple feature loss obtained in the perception network, Adaptive triplet feature loss obtained for concept networks.

5. The long-term visual positioning method based on association guidance network according to claim 1 is characterized in that The association guidance mechanism provides an information interaction mechanism; during the concept network training, the information interaction mechanism stores trained images during the concept network training process and activates after a certain number of trained images are reached. The information interaction mechanism then searches for image features similar to the training images in the stored association guidance database to provide concept network training guidance. The information interaction mechanism guides the perceptual network training by comparing the loss functions in the perceptual network training and the concept network.

Images