Long-term visual positioning method based on association guidance network

Abstract

The invention belongs to the technical field of computer vision, and discloses a long-term vision positioning method based on an association guidance network. The model training stage comprises perception network training and concept network training; after the concept network training is completed, the concept network serves as a pre-training model to train the sensing network, and the concept network is optimized at the same time; performing information interaction by the association guidance mechanism; and an image retrieval stage: performing image retrieval by adopting the trained sensing network. According to the method, the features in the concept network are used for guiding feature learning in the perception network through domain adaptive learning and an association guidance mechanism, so that the features obtained by final training of the model have robustness under environment change. Besides, a concept database which does not need to be formed by additional data is used for carrying out self-heuristic learning on the concept network, so that domain feature guidance is better carried out on the sensing network, and the final image retrieval performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a long-term visual positioning method based on an association guidance network. Background Art

[0002] Visual positioning plays a vital role in autonomous driving and mobile robots, especially in SLAM systems for long-term simultaneous positioning and mapping. In long-term visual positioning, changes in factors such as lighting and seasons will have a significant impact on the positioning results. Therefore, it is of great significance to effectively solve the problem of long-term visual positioning. At present, the image retrieval method is a convenient and effective image-based positioning technology. This method retrieves the query image from the database to find the most similar position to complete the positioning and obtain a rough result, which can be used as a closed-loop position recognition in SLAM (Simultaneous localization and Mapping, SLAM) or regress the high-precision six-degree-of-freedom camera pose.

[0003] The document "IEEE / CAA Journal of Automatica Sinica, 313-328, 2021" proposes a new gradient-weighted similarity activation mapping loss. The network enhances the contrastive learning of embedding in a self-supervised manner, thereby improving the accuracy of image matching. Although this method designs a gradient-weighted similarity activation mapping loss and improves the performance of the model, it still cannot achieve the desired effect in some environments with complex environmental changes. The document "Engineering Applications of Artificial Intelligence, 104793, 2022" proposes a method for fusing depth and semantic information in the scene. It uses a depth and semantic information fusion module in the scene to extract scene representations that remain unchanged when the environment changes and improve the accuracy of long-term visual positioning through image retrieval. Although the method introduces additional effective information such as semantics and depth to improve the performance of the overall network, this additional information is also difficult to obtain in actual situations and will increase the computational complexity of the model. Summary of the invention

[0004] The purpose of the present invention is to propose a long-term visual positioning method based on an association guidance network to improve the accuracy and robustness of visual positioning.

[0005] The technical solution of the present invention is as follows: a long-term visual localization method based on an association guidance network, comprising a model training phase and an image retrieval phase;

[0006] The model training stage includes perceptual network training and concept network training; after the concept network training is completed, the concept network is used as a pre-training model to train the perceptual network, and the concept network is optimized at the same time; the encoding features of the perceptual network and the encoding features of the concept network are extracted respectively, and information interaction is performed through the association guidance mechanism, which is used as the influencing factor of the concept network loss function and the perceptual network loss function, and the perceptual network is updated in reverse;

[0007] In the image retrieval stage, the trained perceptual network is used for image retrieval. The perceptual network is used to extract features of the image to be queried, and the obtained features are compared with the images in the retrieval database. A picture with the most similar features is found and the pose of the picture is obtained. The pose is assigned to the query picture to obtain the pose of the query picture.

[0008] The concept network is a multi-domain network structure, each domain consists of a pair of encoders, decoders and discriminators. The framework of the concept network is formulated as follows:

[0009]

[0010] in, For picture x i The encoded features in concept network domain i, for The result of decoding in the concept network domain j, for The encoded features in the concept network domain j, For picture x i The reconstructed image results in concept network domain i; is the encoder of the concept network domain i, is the decoder of concept network domain j;

[0011] In the training process of the concept network, randomly extracted pictures in each domain are stored and an association guidance database of each domain is established. When the accumulated data reaches a certain amount, the association guidance mechanism is activated; the association guidance mechanism extracts features similar to the input image by associating the data in the guidance database and using the encoder in the concept network to assist the concept network in generating discriminative features.

[0012] The training of the concept network includes four loss functions, namely ComboGAN loss function Feature consistency loss function SAM loss function and adaptive triple feature loss function

[0013] The overall loss function of the concept network is expressed as follows:

[0014]

[0015] The ComboGAN loss function includes adversarial loss and cycle consistency loss;

[0016] The definition is as follows:

[0017]

[0018] The converted image in the encoding and decoding process is passed through the encoder E i C and decoder D i C Find the cycle consistency loss, defined as follows:

[0019]

[0020] Furthermore, the translated image is passed through the discriminator G j To find the adversarial loss, it is defined as follows:

[0021]

[0022] The overall representation of the feature consistency loss function is as follows:

[0023]

[0024] Among them, λ FCL and θ CFL is a hyperparameter;

[0025] In order to obtain features across different domains, an additional loss on the latent space is used. For an image x in domain i i , whose expression is defined as:

[0026]

[0027] In order to further improve the extraction of potential features by the concept network, the association guidance mechanism is used for the self-inspiration learning of the concept network, which is expressed as follows:

[0028]

[0029] Among them, p j The images associated with the concept network from the association guidance database;

[0030] The feature loss between domains is defined as follows:

[0031]

[0032] The SAM loss function is generally expressed as:

[0033]

[0034] Among them, θ SAM and θ AAM is a hyperparameter;

[0035] Grad-SAM loss function, the formula is as follows:

[0036]

[0037] The SAM loss function is introduced into the association guidance mechanism and is defined as follows:

[0038]

[0039] in, is the similarity response graph of the pictures associated by the concept network from the association guidance database; L i，m is the similarity response map of the input image;

[0040] The overall definition of the adaptive triple feature loss function is as follows:

[0041]

[0042] Among them, m f is margin,α f is the weight parameter; is the feature obtained by encoding in the j domain after decoding in the i domain, is the feature obtained by encoding in the j domain.

[0043] The number of data stored in the association guidance database is limited. When the data stored in the current domain reaches the upper limit, a previously stored data is randomly deleted from the current domain, and the current data is stored in it, so as to update the association guidance database of each domain.

[0044] The structure of the perception network consists of multiple domains, each of which contains a pair of encoders and decoders. The framework of the perception network is formulated as follows:

[0045]

[0046] The perception network completes the final retrieval. For picture x i The encoded features in domain i in the perceptual network, for The result of domain j decoding in the perceptual network, for The encoded features in domain j in the perceptual network, For picture xi The reconstructed image result of domain i in the perception network; Used for The output prediction of The output predicted image and predicted pose.

[0047] The loss function of the perception network is mainly composed of three parts: feature consistency loss function SAM loss function and adaptive triple feature loss function

[0048] The overall loss function of the perception network is expressed as:

[0049]

[0050] SAM loss function in perception network SAM loss function in the same concept network same;

[0051] Therefore, the feature consistency loss function is defined as;

[0052]

[0053] The feature-guided loss function of the concept network to the corresponding domain of the perception network is expressed as follows:

[0054]

[0055] The overall definition of the adaptive triple feature loss function is as follows:

[0056]

[0057]

[0058] in, is the adaptive triple feature loss obtained in the perception network, Adaptive triplet feature loss obtained for concept networks.

[0059] The association guidance mechanism provides an information interaction mechanism; in the concept network training, the information interaction mechanism stores trained images during the concept network training process and activates them after reaching a certain number, and guides the concept network training by searching for image extraction features similar to the training images in the stored association guidance database; in the perception network training, the information interaction mechanism guides the perception network training by comparing the loss functions in the perception network training and the concept network.

[0060] Beneficial effects of the present invention: In view of the problems that the extracted features in the previous long-term visual positioning algorithms are not robust, the present invention proposes a long-term visual positioning method based on an association guidance network. The present invention uses the features in the concept network to guide the feature learning in the perception network through domain adaptive learning and association guidance mechanisms, so that the features finally trained by the model are robust under environmental changes. In addition, the present invention uses a concept database that does not require additional data to perform self-inspired learning on the concept network, so as to better guide the domain features of the perception network, thereby improving the final image retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Flowchart of a long-term visual localization method based on association guidance network.

[0062] Figure 2 A conceptual network diagram.

[0063] Figure 3 Schematic diagram of the sensing network. DETAILED DESCRIPTION

[0064] Figure 1 It is the main flow chart of the technical solution of the present invention. Figure 1 As shown, the long-term visual positioning algorithm based on the association guidance network proposed in the present invention includes the following steps:

[0065] (1) Concept network training: In the algorithm network framework proposed in the present invention, the concept network is trained first. The concept network is a multi-domain network structure, each domain consists of a pair of encoders, decoders and discriminators. The proposed network model is mainly aimed at the image retrieval problem of changing scenes. The domain here generally refers to an environment, such as snowy scenes, rainy scenes, etc. The framework of the concept network is formulated as follows:

[0066]

[0067] in, For picture x i The encoded features in domain i, for The result of decoding in domain j is, for The encoded features in domain j, For picture x i Reconstructed image results in domain i.

[0068] In the process of training the concept network, the present invention is inspired by the association mechanism of the human brain and introduces an association guidance mechanism to help the network better solve image retrieval problems under changes in lighting, seasons, etc., thereby providing more stable feature guidance for the perception network.

[0069] In the concept network, the association guidance mechanism mainly plays a role of self-inspiration. During the training process of the concept network, the association guidance database of each domain is established by storing randomly extracted pictures in each domain. When the accumulated data reaches a certain amount, the association guidance mechanism is activated to provide corresponding guidance to the concept network. Since the amount of database storage is limited, if the data stored in the current domain reaches the upper limit, a previously stored data will be randomly deleted from the current domain and the current data will be stored in it, so as to update the database of each domain.

[0070] The concept network mainly includes four loss functions, namely ComboGAN loss function Feature consistency loss function SAM loss function and adaptive triple feature loss function

[0071] The translated image is passed through the encoder and decoder Find the cycle consistency loss, defined as follows:

[0072]

[0073] Furthermore, the translated image is passed through the discriminator G j To find the adversarial loss, it is defined as follows:

[0074]

[0075] For the pure multi-domain image translation task, namely ComboGAN, the total loss only includes the adversarial loss and the cycle consistency loss, which are defined as follows:

[0076]

[0077] In order to obtain features across different domains, an additional loss on the latent space is used, called feature consistency loss. Under the above assumptions, for an image x in domain i i , whose expression is defined as:

[0078]

[0079] In order to further improve the network's extraction of potential features, the present invention proposes an association guidance mechanism for self-inspired learning of concept networks, which is expressed as follows:

[0080]

[0081] Among them, p jConcept network association images from a multi-domain concept database.

[0082] In addition, the present invention also uses feature loss between domains to enable the model to better obtain a domain adaptation model, which is defined as follows:

[0083]

[0084] To this end, the overall representation of the feature consistency loss function is as follows:

[0085]

[0086] In order to better obtain effective features, the Grad-SAM loss function is used, and the formula is as follows:

[0087]

[0088] In addition, the present invention introduces the SAM loss function into the association guidance mechanism, which is defined as follows:

[0089]

[0090] in, Similarity response graph for images associated with the concept network from the database.

[0091] The overall SAM loss function is expressed as:

[0092]

[0093] The triple loss function is widely used in the field of image recognition. The adaptive triple feature loss function used in this paper is defined as:

[0094]

[0095]

[0096] Among them, m f is margin,α f is the weight parameter.

[0097] The overall definition of the adaptive triple feature loss function is as follows:

[0098]

[0099] The overall loss function of the concept network is expressed as follows:

[0100]

[0101] After the concept network is well trained, it becomes a pre-trained network and further provides stable feature guidance for the domain adaptation of the perception network. When the perception network is trained, the concept network is also optimized.

[0102] (2) Perceptual network training: After the concept network training is completed, the concept network will be used as a pre-training model to train the perceptual network. The concept network will continue to be optimized and will not be removed from the framework. The perceptual network has a similar structure to the concept network, but it needs the association guidance of the concept network during the training process, such as Figure 1 As described in , the framework of the perception network is formulated as follows:

[0103]

[0104] Formula (16) is similar to Formula (1), except that the superscript C in Formula (1) represents the concept network, and the superscript P in Formula (16) represents the perception network. In addition, the perception network also needs to complete the final retrieval. Used for The output prediction of The predicted image and pose are output.

[0105] During the training process of the entire framework network, the selected images are simultaneously input into the concept network and the perception network. Through the guidance of the concept network to the perception network, the perception network can obtain better domain adaptation features. The ternary loss function structure of the concept network and the perception network is similar, so the concept network will have a better guiding effect when guiding the perception network in the corresponding domain.

[0106] Compared with the training of the concept network, the loss function of the perception network is relatively simple, mainly composed of three parts: feature consistency loss function SAM loss function and adaptive triple feature loss function

[0107] SAM loss function in perception network SAM loss function in the same concept network The structures are the same except for the superscript differences.

[0108] In the perception network, the feature consistency loss function is defined as;

[0109]

[0110] The feature-guided loss function of the concept network to the corresponding domain of the perception network is expressed as follows:

[0111]

[0112] The feature loss function between domains is consistent with the concept network structure, so the overall feature consistency loss function is defined as;

[0113]

[0114] The adaptive triple feature loss function in the perception network is defined as:

[0115]

[0116] The overall definition of the adaptive triple feature loss function is as follows:

[0117]

[0118] Therefore, the overall loss function of the perception network is expressed as:

[0119]

[0120] (3) The association guidance mechanism is mainly an interaction mechanism between the two networks. Each network obtains features based on its own encoder and optimizes them. The role of the association guidance mechanism in the concept network is mainly self-inspiration. It is mainly achieved by storing trained images during the training of the concept network and activating them after reaching a certain number. It provides so-called guidance by searching for images similar to the training images in the stored association guidance database to extract features. In the perception network, it is mainly reflected in the interaction of loss functions. It provides so-called guidance mainly by comparing the loss functions of the two networks.

[0121] (4) Image retrieval: During the reasoning process, only the perception network is needed for the final image retrieval, and the concept network is removed from the framework. The perception network is used to extract features from the image to be queried, and the obtained features are compared with the images in the retrieval database. The image with the most similar features is found and the pose of the image is obtained. This pose is then assigned to the query image to obtain the pose of the query image.

[0122] In order to verify the effectiveness of the algorithm in long-term visual positioning accuracy, the present invention was tested on 17 sequences of the CMU-Seasons dataset. According to the experimental results, the algorithm proposed in the present invention has high-precision and medium-precision positioning results of 24.1% and 48.7% in urban scenes, 11.6% and 27.6% in suburban scenes, and 13.3% and 33.3% in park scenes. Compared with the DISMA method, the proposed method has improved the high-precision and medium-precision positioning results by 1.5% and 1.4% in urban scenes, 0.5% and 0.1% in suburban scenes, and 0.7% and 2% in park scenes.

Claims

1. A long-term visual localization method based on an association guidance network, characterized in that: Includes model training phase and image retrieval phase; The model training stage includes perceptual network training and concept network training; after the concept network training is completed, the concept network is used as a pre-training model to train the perceptual network, and the concept network is optimized at the same time; the encoding features of the perceptual network and the encoding features of the concept network are extracted respectively, and information interaction is performed through the association guidance mechanism, which is used as the influencing factor of the concept network loss function and the perceptual network loss function, and the perceptual network is updated in reverse; In the image retrieval stage, the trained perceptual network is used for image retrieval. The perceptual network is used to extract features of the image to be queried, and the obtained features are compared with the images in the retrieval database. A picture with the most similar features is found and the pose of the picture is obtained. The pose is assigned to the query picture to obtain the pose of the query picture.

2. The long-term visual positioning method based on the association guidance network according to claim 1 is characterized in that: The concept network is a multi-domain network structure, each domain consists of a pair of encoders, decoders and discriminators. The framework of the concept network is formulated as follows: in, For picture x i The encoded features in concept network domain i, for The result of decoding in the concept network domain j, for The encoded features in the concept network domain j, For picture x i The reconstructed image results in concept network domain i; is the encoder of the concept network domain i, is the decoder of concept network domain j; In the training process of the concept network, randomly extracted pictures in each domain are stored and an association guidance database of each domain is established. When the accumulated data reaches a certain amount, the association guidance mechanism is activated; the association guidance mechanism extracts similar features by associating the data in the guidance database and using the encoder in the concept network to assist the concept network in generating discriminative features.

3. The long-term visual positioning method based on the association guidance network according to claim 2 is characterized in that: The training of the concept network includes four loss functions, namely ComboGAN loss function Feature consistency loss function SAM loss function and adaptive triple feature loss function The overall loss function of the concept network is expressed as follows: The ComboGAN loss function includes adversarial loss and cycle consistency loss; The definition is as follows: The converted image in the encoding and decoding process is passed through the encoder and decoder Find the cycle consistency loss, defined as follows: Furthermore, the translated image is passed through the discriminator G j To find the adversarial loss, it is defined as follows: The overall representation of the feature consistency loss function is as follows: Among them, θ FCL and θ CFL is a hyperparameter; In order to obtain features across different domains, an additional loss on the latent space is used. For an image x in domain i i , whose expression is defined as: In order to further improve the extraction of potential features by the concept network, the association guidance mechanism is used for the self-inspiration learning of the concept network, which is expressed as follows: Among them, p j The images associated with the concept network from the association guidance database; The feature loss between domains is defined as follows: The SAM loss function is generally expressed as: Among them, λ SAM and λ AAM is a hyperparameter; Grad-SAM loss function, the formula is as follows: The SAM loss function is introduced into the association guidance mechanism and is defined as follows: in, is the similarity response graph of the pictures associated by the concept network from the association guidance database; L i,m is the similarity response map of the input image; The overall definition of the adaptive triple feature loss function is as follows: Among them, m f is margin,α f is the weight parameter; is the feature obtained by encoding in the j domain after decoding in the i domain, is the feature obtained by encoding in the j domain.

4. The long-term visual positioning method based on the association guidance network according to claim 2 or 3, characterized in that: The number of data stored in the association guidance database is limited. When the data stored in the current domain reaches the upper limit, a previously stored data is randomly deleted from the current domain, and the current data is stored in it, so as to update the association guidance database of each domain.

5. The long-term visual positioning method based on association guidance network according to claim 1 is characterized in that: The structure of the perception network consists of multiple domains, each of which contains a pair of encoders and decoders. The framework of the perception network is formulated as follows: The perception network completes the final retrieval. For picture x i The encoded features in domain i in the perceptual network, for The result of domain j decoding in the perceptual network, for The encoded features in domain j in the perceptual network, For picture x i The reconstructed image result of domain i in the perception network; Used for The output prediction of The output predicted image and predicted pose.

6. The long-term visual positioning method based on association guidance network according to claim 5 is characterized in that: The loss function of the perception network is mainly composed of three parts: feature consistency loss function SAM loss function and adaptive triple feature loss function The overall loss function of the perception network is expressed as: SAM loss function in perception network SAM loss function in the same concept network same; Therefore, the feature consistency loss function is defined as; The feature-guided loss function of the concept network to the corresponding domain of the perception network is expressed as follows: The overall definition of the adaptive triple feature loss function is as follows: in, is the adaptive triple feature loss obtained in the perception network, Adaptive triplet feature loss obtained for concept networks.

7. The long-term visual positioning method based on association guidance network according to claim 1 is characterized in that: The association guidance mechanism provides an information interaction mechanism; the information interaction mechanism stores trained images during the concept network training process and activates after reaching a certain number, and performs concept network training guidance by searching for image extraction features similar to the training images in the stored association guidance database; The information interaction mechanism guides the perceptual network training by comparing the loss functions in the perceptual network training and the concept network during the perceptual network training.

Images