Data set enhancement method for visual position identification

Abstract

The invention relates to a data set enhancement method for visual position identification, which comprises the following steps of: firstly, processing a Nordland data set containing complete seasonal image distribution of all places by utilizing a Conditional GAN, and realizing image seasonal conversion based on a Pix2pix model; then, the obtained 12 Pix2pix models are used for supplementing season information missing in some places for an existing GSV-Cimes data set; images in the obtained GSV-Ciences data set with more complete season information are input into a feature extractor for training; and carrying out performance analysis on the obtained trained image feature extraction model, and comparing with the current several VPR algorithms. Results show that the method provided by the invention has more excellent performance when seasons change and illumination and weather change.

Description

Technical Field

[0001] The present invention relates to the fields of visual position recognition and image processing, and in particular to a data set enhancement method for visual position recognition. Background Art

[0002] Visual place recognition (VPR) uses camera sensors on mobile devices, such as cars and robots, to obtain raw images and perform matching calculations to determine whether certain images represent the same location. Therefore, VPR is essentially an image retrieval problem: searching for images in a database constructed from a global environment. However, in real-world environments, variations in lighting, weather, seasons, viewpoints, and perceptual aliasing can pose numerous challenges in practical applications.

[0003] As deep neural networks have become mainstream in image recognition and detection, achieving state-of-the-art results, researchers are using unsupervised or weakly supervised methods to extract feature descriptors from images for location recognition. Virtual reality prediction (VPR) uses feature extraction methods to transform images into a low-dimensional space, saving storage costs and ensuring geometric and even cross-domain invariance. Therefore, feature extraction methods are crucial for VPR when illumination and viewpoint changes occur. However, some existing methods lack robustness in specific environments, such as those facing changing environmental conditions, and are susceptible to perceptual aliasing. To mitigate the discrepancies between different images of the same location caused by environmental changes, several techniques can be used to process images for better matching. These techniques include illumination-invariant images, shadow removal, appearance change prediction, linear regression, and deep learning-based generative adversarial networks (GANs). Recent research trends have focused on training deep architectures on various large-scale datasets to achieve robustness to both local and global features.

[0004] Most current research on VPR focuses on improving model architecture to extract more robust image descriptors, but ignores the limitations of datasets. Data augmentation can artificially introduce certain types of variation into training datasets, making the training data more diverse and thus improving the model's generalization performance and avoiding overfitting. Seasonal change is a common change in environmental conditions. Many existing datasets cover a limited number of seasons, and images taken at the same location in different seasons are not comprehensive, making it difficult to fully cover seasonal information. Furthermore, current network architectures for extracting image features do not efficiently and fully exploit the correlations between images from different seasons. Summary of the Invention

[0005] The purpose of the present invention is to address the problem of visual location recognition and propose a dataset enhancement method for visual location recognition, which mainly uses Conditional GAN ​​to achieve seasonal conversion and enhance the seasonal coverage of the dataset. The present invention obtains the conversion ability of images in the four seasons of spring, summer, autumn and winter by training Conditional GAN ​​on a dataset with complete seasonal distribution of all locations, and performs seasonal padding on the training dataset used for feature extraction to cover sufficient seasonal information. A general neural network skeleton is used to obtain image descriptors, and the backbone network preliminarily extracts and reaggregates local features to obtain the final global image features. The correlation of seasonal image information is implicitly incorporated into the design of the loss function, and the seasonal information is fully utilized during the back propagation of the image representation framework to enhance the correlation between images of different seasons at the same location and maintain the independence of images at different locations. Ultimately, the robustness of VPR in changing seasons is improved, and the perceptual aliasing caused by seasonal changes in the environment is overcome.

[0006] The above purpose is achieved through the following technical solutions:

[0007] The present invention proposes a dataset enhancement method for visual location recognition, comprising the following steps:

[0008] S1. We used Conditional GAN ​​to process the Nordland dataset, which contains the complete seasonal distribution of images from all locations. We then implemented seasonal conversion of images based on the Pix2pix model, gaining the ability to convert images between the four seasons of spring, summer, autumn, and winter. We obtained a total of 12 different Pix2pix models for the four seasons.

[0009] S2. Using the 12 Pix2pix models obtained in step S1, we supplement the missing seasonal information of some locations in an existing GSV-Cities dataset, thereby enhancing the dataset's complete seasonal coverage and obtaining a GSV-Cities dataset with more complete seasonal information.

[0010] S3: Input the images from the GSV-Cities dataset (with more complete seasonal information, obtained in step S2) into a feature extractor for training, resulting in a trained image feature extraction model. The feature extraction process involves inputting the images into a ResNet-50 network to obtain local features, and then aggregating these local features using GeM to obtain the final global features. The entire feature extraction process is performed in a fully supervised manner, using a multi-similarity loss and an online hard mining strategy.

[0011] S4. Perform a performance analysis on the trained image feature extraction model obtained in step S3 and compare it with several currently advanced VPR algorithms, including AVG, NetVLAD, CosPlace, and GeM. Comparative experiments are conducted on three existing datasets: Pitts250k-test, MSLS-val, and SPED. These datasets have high sample similarity and include seasonal variations, allowing for evaluation of the VPR algorithm's performance under extreme environmental conditions. Recall@K (R@K) is used as the evaluation metric.

[0012] Furthermore, the seasonal conversion of images based on the Pix2pix model is a translation of images, which can realize the conversion of images from one domain to another, that is, domain migration. The present invention realizes seasonal conversion of images through the Pix2pix model.

[0013] Furthermore, the Conditional GAN ​​was used to process the Nordland dataset, which contains a complete seasonal distribution of images from all locations. The Conditional GAN ​​model consists of a generator and a discriminator. Throughout the training process, the generator and discriminator compete with each other until reaching an equilibrium point, at which point the optimal solution is obtained. The generator of the Conditional GAN ​​model uses a modified "U-Net" architecture based on the encoder-decoder with skip connections, and the discriminator uses a convolutional PatchGAN classifier to determine whether each small patch of the image is real or fake.

[0014] Furthermore, the backbone of the feature extractor in step S3 uses the efficient and lightweight ResNet-50 to extract preliminary features, and the aggregation method uses GeM to aggregate local features. The image is output as a tensor through the backbone network Where H is the height, W is the width, H×W is the resolution, and C is the number of filters in the last convolutional layer of the backbone network. The specific form of GeM is where c∈{1,…,C}, is a set of feature maps. The parameter k can be set manually or learned.

[0015] Furthermore, the training process uses Multi-Similarity Loss and online hard mining strategy for training. MS loss combines advanced pair weighting scheme, and the MS loss function is: Where M is the number of images in a batch, for each query image I q , P q Refers to the set of samples that are in direct correspondence with the query (the images represent the same location), Nq Refers to the set of samples that form negative pairs with the query (the images represent different locations). qp and S qn Represent the cosine distance between the query and the positive and negative samples respectively. α, β and m are hyperparameters.

[0016] Beneficial effects:

[0017] 1. This method implicitly incorporates seasonal image correlation into the design of the loss function. This information is fully utilized during backpropagation within the image representation framework, enhancing the correlation between images from the same location across different seasons while maintaining the independence of images from different locations. This ultimately improves the robustness of VPR across seasonal variations and overcomes perceptual aliasing caused by seasonal environmental changes.

[0018] 2. This paper trains the pix2pix model using the Nordland dataset, which exhibits seasonal variations. The images in this dataset exhibit distinct seasonal characteristics, as well as variations in lighting and weather. This dataset also provides ground truth information, which is commonly used to test the performance of VPR algorithms under significant environmental changes. The GSV-Cities dataset, used for image feature extraction, is a training set with rich geographic diversity, spanning a wide range of time and geographic areas.

[0019] 3. The comparative experiments described in this paper include several currently advanced VPR algorithms: AVG, NetVLAD, CosPlace, and GeM. CosPlace utilizes data augmentation methods, including random cropping and color dithering. Recall@K (R@K) is used as the experimental evaluation metric. The experimental datasets used are Pitts250k-test, MSLS-val, and SPED. These datasets have high sample similarity and include seasonal variations, allowing for evaluation of the VPR algorithm's performance under extreme environmental conditions.

[0020] 4. This paper proposes a dataset enhancement method for visual location recognition, improving the dataset's scene coverage and combating perceptual aliasing caused by environmental changes. The global features extracted by this method achieve better performance in the face of seasonal changes, even changes in lighting and weather.

[0021] 5. The method proposed in this invention can also be further extended to other challenging extreme conditions, such as extreme lighting changes (day-night matching), indoor scenes without texture or repeated structure, and dynamic changes over long time spans. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a data set enhancement method for visual location recognition according to the present invention;

[0023] Figure 2 The conditional adversarial network model of the present invention;

[0024] Figure 3 These are experimental results of an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is a further detailed description of a data set enhancement method for visual position recognition proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purposes, features and advantages of the present invention more obvious and easy to understand, please refer to the drawings. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the implementation conditions of the present invention, so they have no technical significance. Any modification of the structure, change in the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the efficacy and purpose that can be achieved by the present invention.

[0026] like Figure 1 As shown, the present invention proposes a data set enhancement method for visual location recognition, which includes the following steps:

[0027] S1. We used Conditional GAN ​​to process the Nordland dataset, which contains the complete seasonal distribution of images of all locations. We implemented seasonal conversion of images based on the Pix2pix model and obtained the ability to convert images between the four seasons of spring, summer, autumn and winter. A total of 12 different Pix2pix models were obtained for the four seasons.

[0028] S2. Use the 12 Pix2pix models obtained in step S1 to supplement the missing seasonal information for certain locations in an existing GSV-Cities dataset, enhancing the dataset's complete seasonal coverage. The GSV-Cities dataset covers 40 cities across all continents, covering a wide range of time spans and geographic areas.

[0029] S3. The GSV-Cities dataset, which contains more complete seasonal information, is obtained through step S2. Images from this dataset are fed into a feature extractor for training, resulting in a trained feature extraction model. The feature extraction process involves feeding the image into a ResNet-50 network to obtain local features, which are then aggregated using GeM to obtain the final global features. The entire feature extraction process is fully supervised, employing a multi-similarity loss and an online hard mining strategy.

[0030] S4. Perform a performance analysis on the trained image feature extraction model obtained in step S3 and compare it with several currently advanced VPR algorithms, including AVG, NetVLAD, CosPlace, and GeM. The comparative experiments are conducted on three existing Pitts250k-test, MSLS-val, and SPED datasets. The samples in these datasets have high similarity and contain seasonal variations, which can evaluate the performance of the VPR algorithm under extreme environmental changes. Recall@K (R@K) is used as the evaluation metric of the experiment. It represents the ratio of at least one successful match among the top K similarity-ranked database images retrieved. The criterion for a successful match is that the physical distance between the two is within a pre-set threshold. Here, we set the threshold to 25 meters.

[0031] The results show that the method proposed in the present invention has better performance in seasonal changes and changes in light and weather. Furthermore, the method proposed in the present invention can also be migrated to other challenging situations, such as extreme lighting changes (day and night matching), scenes without texture or structural repetition (indoor scenes), dynamic changes over long time scales, etc. S1. The present invention mainly considers the performance of image features in the face of seasonal changes, uses Conditional GAN ​​for image processing, and realizes image season conversion based on the Pix2pix model. A dataset containing complete seasonal distribution images of all locations is trained to obtain the ability to convert images between the four seasons of spring, summer, autumn and winter. A total of 12 different Pix2pix models combining two of the four seasons are obtained.

[0032] The Pix2pix model described in step S1 is primarily designed to enable seasonal image conversion. The Pix2pix model primarily addresses image translation, enabling image conversion from one domain to another, also known as domain transfer. Examples include converting stick figures into photographs, restoring images with missing pixels, and converting from day to night. The present invention utilizes the Pix2pix model to achieve seasonal image conversion.

[0033] In Conditional GANs, for an image x in source domain X and target domain Y i ,i=1,…,N,x∈X and y j ,j=1,…,N,y∈Y, whose data distribution is x~p data (x), y~p data (y). In addition, the random noise variable z is z~p z (z). The model includes a generator and a discriminator, where the generator G:{x,z}→y converts the image in the X domain to the Y domain, that is, G(x,z)=y′. Therefore, the generator is also a mapping function, and the so-called condition in the conditional adversarial network is the real image x in the source domain. The corresponding discriminator D compares and discriminates the real image x and the generated image y′, as well as the images x and y in the source domain and the target domain. D can be used to judge the quality of the image generated by the generator G and the degree of compliance with the conditions. If the discriminator is good enough, D(x,y)=1 for the real image and D(x,y′)=0 for the generated image. But for the generator G, it hopes that the discriminator will make a wrong judgment and regard the generated image as real. During the entire training process, the generator and the discriminator compete with each other until they reach an equilibrium point, at which point the optimal solution can be obtained.

[0034] Use conditional adversarial loss for the mapping function:

[0035]

[0036] The goal is To ensure the correlation between the input and output of the entire network, that is, the similarity between the generated image y′ and the original input image x, an L1 loss is added. At the same time, the L1 distance can reduce blur. The L1 loss function is expressed as:

[0037]

[0038] The final loss function is

[0039] Loss=L(G,D,X,Y)

[0040] =L cGAN +λL L1

[0041] The generator uses a modified "U-Net" architecture based on the encoder-decoder with skip connections. The discriminator uses a convolutional PatchGAN classifier to determine whether each small patch of the image is real or fake. The network is trained using a dataset with distinct seasonal characteristics, enabling it to transform images between the four seasons of spring, summer, autumn, and winter.

[0042] The dataset of complete seasonal distribution images of all locations described in step S1 uses the Nordland dataset. This dataset is shot by fixing a camera on the locomotive. The shooting route is a 728km railway track in northern Norway. It was shot four times in four seasons. The images have obvious seasonal characteristics and changes in lighting and weather. The images in the dataset are mainly natural scenes, and also include a small number of cities and stations passed along the way. At the same time, they provide ground truth information, which is often used to detect the performance of VPR algorithms under obvious environmental changes. The present invention uses this dataset containing seasonal changes to train the pix2pix model.

[0043] The training set used for image feature extraction in step S2 uses the GSV-Cities dataset, which covers 40 cities across all continents and exhibits rich geographical diversity. It contains over 67,000 locations, each captured on 4 to 20 different dates. This dataset covers a wide range of time spans and geographic areas. After being padded using the seasonal conversion module, the GSV-Cities dataset includes more comprehensive seasonal information.

[0044] The backbone network for extracting global features of the image described in step S3 uses the efficient and lightweight ResNet-50 to extract preliminary features, and the aggregation method uses GeM to aggregate local features. The image is output as a tensor through the backbone network. Where H is the height, W is the width, H×W is the resolution, and C is the number of filters in the last convolutional layer of the backbone network. After obtaining the feature map F, it needs to be aggregated into a more discriminative image feature representation through the aggregation layer. The aggregation layer can usually be completed by a pooling operation. The present invention uses generalized average pooling (GeM) as the aggregation layer, which can better preserve spatial position information compared with traditional pooling methods. The specific form of GeM is

[0045]

[0046] Among them, c∈{1,…,C}, is a set of feature maps. The parameter k can be set manually or learned.

[0047] The model is trained in a fully supervised manner as described in step S4, using Multi-Similarity Loss and online hard mining strategy. MS loss combines an advanced pair weighting scheme, and the MS loss function is:

[0048]

[0049] Where M is the number of images in a batch, for each query image I q , P qRefers to the set of samples that are in direct correspondence with the query (the images represent the same location), N q Refers to the set of samples that form negative pairs with the query (the images represent different locations). qp and S qn Represent the cosine distance between the query and the positive and negative samples respectively. α, β and m are hyperparameters.

[0050] The experimental portion described in step S5 trained 12 Pix2pix models using TensorFlow on an NVIDIA RTX3090. A visual feature extraction framework was fine-tuned using a pretrained ResNet50 model as the backbone and GeM as the aggregation layer. The Pix2pix models were trained using the Adam optimizer with a learning rate of 2e-4 and an exponential decay of 0.5 for the first-order moment estimate. When fine-tuning the visual feature extraction framework, batches contained 100 different locations, each containing images from four seasons. Stochastic gradient descent was used as the optimizer with a momentum of 0.9 and a decay of 0.001. The initial learning rate was 0.03, which was multiplied by 0.3 every five epochs. Training was performed for 30 epochs, with all images resized to 320×320.

[0051] The comparative experiment includes several currently advanced VPR algorithms, namely AVG, NetVLAD, CosPlace and GeM. Among them, CosPlace uses data enhancement methods, including random adjustment of cropping size and color jittering. Recall@K (R@K) is used as the evaluation indicator of the experiment. It represents the ratio of at least one successful match among the top K similarity-ranked database images retrieved. The criterion for a successful match is that the physical distance between the two is within a preset threshold. Here, the present invention sets the threshold to 25 meters. The experimental data sets used are Pitts250k-test, MSLS-val, and SPED. The samples in these data sets have high similarity and contain seasonal changes, which can be used to evaluate the performance of the VPR algorithm under extreme environmental changes. Figure 2 The results of the proposed method on experimental datasets are shown for four comparative methods, each demonstrating the performance in terms of Recall@K (R@K). Specifically, it surpasses other methods on the MSLS-val and SPED datasets and also improves on GeM on the Pitts250k-test dataset. The large seasonal distribution variance of the MSLS-val and SPED datasets may explain the significant improvement achieved by our method.

[0052] Ablation experiments compared with GeM verify the effectiveness of manually filling in the missing seasonal images. Figure 2The penultimate row shows the effect of using the pipeline without preprocessing the dataset, without using the seasonal transition model. Comparing the two shows that using the dataset enhanced by the seasonal transition module for feature extraction improves the accuracy and robustness of the algorithm.

[0053] In summary, the invention provides a dataset enhancement method for visual location recognition. It improves the scene coverage of the dataset and resists perceptual aliasing caused by environmental changes. In response to seasonal changes, the Pix2pix model is used to rely on Conditional GANs to perform image conversion, and a conversion model of the image between the four seasons is obtained. The trained seasonal conversion model is used to fill the training set used for feature extraction, and the processed dataset contains more useful seasonal information. Finally, more robust global image features are extracted. Experimental results show that the global features extracted by this method achieve better performance in the face of seasonal changes and even changes in lighting and weather, and can solve the problem of perceptual aliasing.

[0054] Although the present invention has been described in detail through the above preferred embodiments, it should be understood that the above description is not intended to limit the present invention. After reading the above description, various modifications and substitutions of the present invention will become apparent to those skilled in the art. Therefore, the scope of protection of the present invention should be defined by the appended claims.

[0055] Although the present invention has been described in detail through the above preferred embodiments, it should be understood that the above description is not intended to limit the present invention. After reading the above description, various modifications and substitutions of the present invention will become apparent to those skilled in the art. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A dataset enhancement method for visual location recognition, characterized in that: The following steps are involved: S1. Conditional GAN ​​is used to process the Nordland dataset containing the complete seasonal image distribution of all locations. The image season conversion is realized based on the Pix2pix model, and the image conversion ability between the four seasons of spring, summer, autumn and winter is obtained. A total of 12 different Pix2pix models between the four seasons are obtained; S2, using the 12 Pix2pix models obtained in step S1 to supplement the missing seasonal information of some locations in an existing GSV-Cities dataset, enhance the complete seasonal coverage of the dataset, and obtain a GSV-Cities dataset with more complete seasonal information; S3, inputting the image in the GSV-Cities dataset with more complete seasonal information obtained in step S2 into the feature extractor for training, and obtaining a trained image feature extraction model; the feature extraction process includes inputting the image into ResNet-50 to obtain local features of the image, and then aggregating the obtained local features using GeM to obtain the final global features; the entire feature extraction process is carried out in a fully supervised manner, using multiple similarity losses and online hard mining strategies; S4. Perform performance analysis on the trained image feature extraction model obtained in step S3 and compare it with several current VPR algorithms, including AVG, NetVLAD, CosPlace, and GeM. The comparative experiment is conducted on three existing Pitts250k-test, MSLS-val, and SPED datasets, using Recall@K (R@K) as the evaluation indicator of the experiment.

2. The method for enhancing a data set for visual location recognition according to claim 1, wherein: The image season conversion based on the Pix2pix model is a translation of the image, which realizes the conversion of the image from one domain to another, that is, domain migration.

3. The method for data set enhancement for visual location recognition according to claim 2, characterized in that: The Conditional GAN ​​is used to process the Nordland dataset containing the complete seasonal image distribution of all locations, wherein the Conditional GAN ​​model includes a generator and a discriminator. During the entire training process, the generator and the discriminator compete with each other until an equilibrium point is reached, at which time the optimal solution can be obtained; the Conditional GAN ​​model, whose generator adopts an improved "U-Net" architecture with skip connections added on the basis of the encoder-decoder, and the discriminator uses a convolutional PatchGAN classifier to judge whether each small block of the image is true or false.

4. The method for data set enhancement for visual location recognition according to claim 1, characterized in that: The feature extractor in step S3 uses the efficient and lightweight ResNet-50 to extract preliminary features, and the aggregation method uses GeM to aggregate local features.

5. The method for data set enhancement for visual location recognition according to claim 4, characterized in that: The image is input to the ResNet-50 backbone network and outputs a tensor Where H is the height, W is the width, H×W is the resolution, and C is the number of filters in the last convolutional layer of the backbone network; the specific form of GeM is where c∈{1,…,C}, is a set of feature maps; the parameter k can be set manually or learned.

6. The method for data set enhancement for visual location recognition according to claim 1, characterized in that: In step S3, the training is performed using a multi-similarity loss and an online hard mining strategy, wherein the multi-similarity loss, referred to as MS loss, combines an advanced pair weighting scheme, and the MS loss function is: Where M is the number of images in a batch; for each query image I q , P q Refers to the sample set that is directly opposite to the query. The images in the sample set that is directly opposite to the query represent the same location; N q Refers to the sample set that forms a negative pair with the query, and the images in the sample set that forms a negative pair with the query represent different locations; S qp and S qn Represent the cosine distance between the query and the positive and negative samples respectively; α, β and m are hyper parameters.

Images