A dataset augmentation method for visual position recognition

Abstract

The application relates to a data set enhancement method for visual position recognition. Firstly, a Nordland data set containing complete seasonal image distribution of all places is processed by using a Conditional GAN, and the conversion of image seasons is realized based on a Pix2pix model; then, 12 Pix2pix models obtained are used to supplement the missing seasonal information of some places in an existing GSV-Cities data set; images in the GSV-Cities data set with more complete seasonal information obtained are input into a feature extractor for training; and the performance of the trained image feature extraction model obtained is analyzed, and the image feature extraction model is compared with current several VPR algorithms. The comparison result shows that the method proposed in the application has more excellent performance in seasonal change and illumination and weather change.

Description

Technical Field

[0001] This invention relates to the fields of visual location recognition and image processing, and more specifically to a dataset augmentation method for visual location recognition. Background Technology

[0002] Visual place recognition (VPR) uses camera sensors on mobile devices, such as cars and mobile robots, to acquire raw images and perform matching calculations to determine whether certain images represent the same location. Therefore, VPR is essentially an image retrieval problem—searching for images in a database built from the global environment. However, in real-world environments, variations in lighting, weather, seasons, viewing angles, and perceptual aliasing pose significant challenges to VPR in practical applications.

[0003] As deep neural networks have become mainstream in image recognition and detection, achieving some state-of-the-art results, researchers are using unsupervised or weakly supervised methods to extract feature descriptors from images for location identification. Visual Representation Processing (VPR) uses feature extraction methods to transform images into a low-dimensional space, saving storage costs and ensuring geometric and even cross-domain invariance. Therefore, feature extraction methods are crucial for VPR when lighting and viewpoint change. However, some existing methods lack robustness in specific environments, such as when facing changing environmental conditions, and are susceptible to perceptual aliasing. To reduce the differences between different images of the same location caused by environmental changes, various techniques can be used to process images for better matching. These include lighting-invariant images, shadow removal, appearance change prediction, linear regression, and deep learning-based generative adversarial networks (GANs). Recent research trends focus on training deep architectures on different large-scale datasets to achieve robustness to both local and global features.

[0004] Current research on VPR (Video Feature Extraction) largely focuses on improving model architectures to extract more robust image descriptors, but neglects the limitations of datasets. Data augmentation can artificially introduce certain types of variations into the training dataset, making the training data more diverse, thereby enhancing the model's generalization performance and avoiding overfitting. Seasonal variation is a common environmental condition change, and many existing datasets contain limited seasonal information; images of the same location taken in different seasons are not comprehensive enough, and seasonal information is difficult to fully cover. Furthermore, current network architectures used for extracting image features do not efficiently and fully utilize the correlations between images from different seasons. Summary of the Invention

[0005] The purpose of this invention is to propose a dataset augmentation method for visual location recognition (VPR), primarily employing a Conditional GAN ​​to achieve seasonal transformation and enhance the seasonal coverage of the dataset. This invention trains a Conditional GAN ​​on a dataset with complete seasonal distributions for all locations to obtain the ability to transform images across the four seasons (spring, summer, autumn, and winter). Seasonal padding is applied to the training dataset used for feature extraction to ensure it covers sufficient seasonal information. A general neural network skeleton is used to obtain image descriptors. The backbone network initially extracts and then aggregates local features to obtain the final global image features. The correlation of seasonal image information is implicitly incorporated into the design of the loss function. During backpropagation of the image representation framework, seasonal information is fully utilized to improve the correlation between images of the same location in different seasons while maintaining the independence of images from different locations. Ultimately, this improves the robustness of VPR in changing seasons and overcomes perceptual aliasing caused by seasonal environmental changes.

[0006] The above objectives are achieved through the following technical solutions:

[0007] This invention proposes a dataset augmentation method for visual location recognition, comprising the following steps:

[0008] S1. The Nordland dataset containing complete seasonal image distributions of all locations is processed using Conditional GAN. Based on the Pix2pix model, the seasonal transformation of images is realized, and the ability of images to transform between the four seasons of spring, summer, autumn and winter is obtained. A total of 12 different Pix2pix models are obtained between the four seasons.

[0009] S2. Using the 12 Pix2pix models obtained in step S1, supplement the missing seasonal information of some locations in an existing GSV-Cities dataset, enhance the complete seasonal coverage of the dataset, and obtain a GSV-Cities dataset with more complete seasonal information.

[0010] S3. Input the images from the GSV-Cities dataset, which provides more complete seasonal information, obtained in step S2 into the feature extractor for training, resulting in a trained image feature extraction model. The feature extraction process includes inputting the images into a ResNet-50 to obtain local image features, and then using GeM aggregation to obtain the final global features. The entire feature extraction process is performed in a fully supervised manner, employing multi-SimilarityLoss and an online hard mining strategy.

[0011] S4. Performance analysis is performed on the image feature extraction model trained in step S3, comparing it with several advanced VPR algorithms, including AVG, NetVLAD, CosPlace, and GeM. Comparative experiments are conducted on three existing datasets: Pitts250k-test, MSLS-val, and SPED. These datasets have high sample similarity and include seasonal variations, allowing for the evaluation of VPR algorithm performance under extreme environmental changes. Recall@K (R@K) is used as the evaluation metric.

[0012] Furthermore, the seasonal transformation of images based on the Pix2pix model is a translation of the image, which can realize the transformation of the image from one domain to another, that is, domain migration. This invention realizes the seasonal transformation of images through the Pix2pix model.

[0013] Furthermore, the Nordland dataset, containing complete seasonal image distributions for all locations, is processed using Conditional GAN. The Conditional GAN ​​model comprises a generator and a discriminator. Throughout training, the generator and discriminator compete against each other until an equilibrium point is reached, at which point the optimal solution is obtained. The generator in this Conditional GAN ​​model employs an improved "U-Net" architecture that adds skip connections to the encoder-decoder, while the discriminator uses a convolutional PatchGAN classifier to determine whether each small patch of the image is real or fake.

[0014] Furthermore, in step S3, the backbone of the feature extractor uses the efficient and lightweight ResNet-50 to extract initial features, and the aggregation method uses GeM to aggregate local features. The image is output as a tensor through the backbone network. Where H is the height, W is the width, H×W is the resolution, and C is the number of filters in the last convolutional layer of the backbone network. The specific form of GeM is... Where c∈{1,…,C}, This is a set of feature maps. The parameter k can be set manually or learned.

[0015] Furthermore, the training process employs a multi-similarity loss (MS) loss method combined with an online hard mining strategy. The MS loss incorporates an advanced weighting scheme, and its function is as follows: Where M is the number of images in a batch, and for each batch, there is a query image I. q P q This refers to the set of samples that corresponds directly to the query (the images represent the same location), Nq This refers to the set of samples that form negative pairs with the query (images represent different locations). S qp and S qn α, β, and m represent the cosine distances between the query and the positive and negative pairs, respectively. α, β, and m are hyperparameters.

[0016] Beneficial effects:

[0017] 1. This invention implicitly incorporates the correlation of seasonal image information into the design of the loss function. During backpropagation of the image representation framework, it fully utilizes seasonal information to enhance the correlation between images of the same location in different seasons, while maintaining the independence of images from different locations. Ultimately, this improves the robustness of VPR (Visual Representation Profile) in changing seasons and overcomes perceptual aliasing caused by seasonal environmental changes.

[0018] 2. This invention uses the Nordland dataset, which includes seasonal variations, to train the pix2pix model. The images in the dataset exhibit distinct seasonal characteristics, as well as changes in lighting and weather, while also providing ground-based information. This dataset is commonly used to assess the performance of VPR algorithms under significant environmental changes. The training set used for extracting image features is the GSV-Cities dataset, which possesses rich geographical diversity and a wide range of time spans and geographical areas.

[0019] 3. The comparative experiments described in this invention include several advanced VPR algorithms, namely AVG, NetVLAD, CosPlace, and GeM. CosPlace utilizes data augmentation methods, including random adjustment of crop size and color jitter. Recall@K (R@K) is used as the evaluation metric. The experimental datasets used are Pitts250k-test, MSLS-val, and SPED. These datasets have high sample similarity and include seasonal variations, allowing for the evaluation of VPR algorithm performance under extreme environmental changes.

[0020] 4. This invention proposes a dataset augmentation method for visual location recognition, improving scene coverage and mitigating perceptual aliasing caused by environmental changes. The global features extracted by the proposed method achieve better performance in the face of seasonal changes, and even changes in lighting and weather.

[0021] 5. The method proposed in this invention can also be further extended to other challenging extreme conditions, such as extreme lighting changes (day-night matching), indoor scenes without texture or with repetitive structure, and dynamic changes over long periods of time. Attached Figure Description

[0022] Figure 1 This is a flowchart of the dataset augmentation method for visual location recognition according to the present invention;

[0023] Figure 2 This is the conditional adversarial network model of the present invention;

[0024] Figure 3 These are the experimental results of an embodiment of the present invention. Detailed Implementation

[0025] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a further detailed explanation of the dataset augmentation method for visual position recognition proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, intended only to facilitate and clearly illustrate the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes to aid those skilled in the art and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to the size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0026] like Figure 1 As shown, the dataset augmentation method for visual location recognition proposed in this invention includes the following steps:

[0027] S1. The Nordland dataset, which contains complete seasonal image distributions for all locations, is processed using Conditional GAN. Based on the Pix2pix model, the seasonal transformation of images is realized, and the ability of images to transform between the four seasons of spring, summer, autumn and winter is obtained. A total of 12 different Pix2pix models are obtained between the four seasons.

[0028] S2. Using the 12 Pix2pix models obtained in step S1, supplement the missing seasonal information for certain locations in an existing GSV-Cities dataset, enhancing the dataset's complete seasonal coverage. The GSV-Cities dataset covers 40 cities distributed across various continents, with a wide range of time spans and geographical locations.

[0029] S3. Following step S2, a more complete seasonal information GSV-Cities dataset was obtained. Images from this dataset were input into a feature extractor for training, resulting in a trained feature extraction model. The feature extraction process involves inputting images into a ResNet-50 to obtain local image features, and then using GeM aggregation to obtain the final global features. The entire feature extraction process is performed in a fully supervised manner, employing multi-Similarity Loss and an online hard mining strategy.

[0030] S4. Performance analysis is performed on the image feature extraction model trained in step S3, comparing it with several advanced VPR algorithms, including AVG, NetVLAD, CosPlace, and GeM. Comparative experiments are conducted on three existing datasets: Pitts250k-test, MSLS-val, and SPED. These datasets have high sample similarity and include seasonal variations, allowing for evaluation of the VPR algorithm's performance under extreme environmental changes. Recall@K (R@K) is used as the evaluation metric, representing the percentage of successful matches among the top K most similar images retrieved from the database. A successful match is defined as the physical distance between the two images being within a pre-set threshold. Here, we set the threshold to 25 meters.

[0031] The results show that the method proposed in this invention has superior performance under seasonal changes and changes in lighting and weather. Furthermore, the method proposed in this invention can also be transferred to other challenging situations, such as extreme lighting changes (day-night matching), scenes without texture or with repetitive structures (indoor scenes), and dynamic changes over long time scales. S1. This invention mainly considers the performance of image features in the face of seasonal changes, using Conditional GAN ​​for image processing and implementing image seasonal transformation based on the Pix2pix model. A dataset containing images of complete seasonal distributions for all locations is used for training to obtain the image transformation capability between the four seasons (spring, summer, autumn, and winter). A total of 12 different Pix2pix models are obtained, combining each of the four seasons in pairs.

[0032] The Pix2pix model described in step S1 is primarily designed to achieve seasonal image transformation capabilities. The Pix2pix model mainly addresses image translation, enabling the transformation of an image from one domain to another—a process known as domain migration. Examples include converting a simple sketch into a photograph, repairing images with missing pixels, and transitioning between day and night. This invention utilizes the Pix2pix model to achieve seasonal image transformation.

[0033] In Conditional GANs, for an image x in the source domain X and the target domain Y... i ,i=1,…,N,x∈X and y j j = 1, ..., N, y ∈ Y, with data distributions x ~ p data (x), y~p data (y). Additionally, the random noise variable z is z~p. z (z). The model consists of a generator and a discriminator, where the generator G:{x,z}→y transforms an image in the X domain to the Y domain, i.e., G(x,z)=y′. Therefore, the generator is essentially a mapping function, and the condition in the conditional adversarial network is the real image x in the source domain. The discriminator D compares and distinguishes between the real image x and the generated image y′, as well as between the images x and y in the source and target domains. D determines the quality of the image generated by the generator G and the degree to which the condition is met. If the discriminator is good enough, then D(x,y)=1 for the real image and D(x,y′)=0 for the generated image. However, the generator G wants the discriminator to misjudge and consider the generated image as real. Throughout the training process, the generator and discriminator compete against each other until an equilibrium point is reached, at which point the optimal solution is obtained.

[0034] For mapping functions, conditional adversarial loss is used:

[0035]

[0036] The goal is To ensure the correlation between the network input and output, i.e., the similarity between the generated image y′ and the original input image x, an L1 loss term is added. The L1 distance also reduces blurring. The L1 loss function is expressed as:

[0037]

[0038] The final loss function is

[0039] Loss = L(G,D,X,Y)

[0040] =L cGAN +λL L1

[0041] The generator employs an improved "U-Net" architecture that adds skip connections to the encoder-decoder, while the discriminator uses a convolutional PatchGAN classifier to determine whether each small patch of the image is real or fake. This invention trains the network using a dataset with distinct seasonal characteristics, enabling it to transform images between the four seasons: spring, summer, autumn, and winter.

[0042] The dataset containing complete seasonal distribution images of all locations mentioned in step S1 uses the Nordland dataset. This dataset was created by fixing cameras to train locomotives and taking photos at specific points. The shooting route is a 728km long railway track in northern Norway, and photos were taken four times in each of the four seasons. The images exhibit distinct seasonal characteristics and variations in lighting and weather. The images in the dataset primarily depict natural scenes, but also include a few cities and stations along the route, providing ground-level information. This dataset is commonly used to assess the performance of VPR algorithms under significant environmental changes. This invention uses this dataset, which includes seasonal variations, to train the pix2pix model.

[0043] The training set used for image feature extraction in step S2 utilizes the GSV-Cities dataset, which covers 40 cities across various continents, exhibiting rich geographical diversity. It contains over 67,000 locations, each photographed on 4–20 different dates, demonstrating a wide range of temporal and geographical scope. The GSV-Cities dataset, supplemented with a seasonal transition module, contains even more comprehensive seasonal information.

[0044] Step S3, which involves extracting global image features, uses the efficient and lightweight ResNet-50 to extract initial features, and GeM is used to aggregate local features. The image is output as a tensor through the backbone network. Where H is the height, W is the width, H×W is the resolution, and C is the number of filters in the last convolutional layer of the backbone network. After obtaining the feature map F, it needs to be aggregated into a more discriminative image feature representation through an aggregation layer. The aggregation layer can usually be accomplished by a pooling operation. This invention uses generalized average pooling (GeM) as the aggregation layer, which can better preserve spatial location information compared with traditional pooling methods. The specific form of GeM is as follows:

[0045]

[0046] Where c∈{1,…,C}, This is a set of feature maps. The parameter k can be set manually or learned.

[0047] Step S4 describes training the model in a fully supervised manner, employing Multi-Similarity Loss (MS Loss) and an online hard mining strategy. The MS loss incorporates an advanced pair-weighting scheme, and its function is as follows:

[0048]

[0049] Where M is the number of images in a batch, and for each batch, there is a query image I. q P qThis refers to the set of samples that corresponds directly to the query (the images represent the same location), N q This refers to the set of samples that form negative pairs with the query (images represent different locations). qp and S qn α, β, and m represent the cosine distances between the query and the positive and negative pairs, respectively. α, β, and m are hyperparameters.

[0050] The experimental part described in step S5 involved training 12 Pix2pix models on an NVIDIA RTX3090 using Tensorflow. A visual feature extraction framework was fine-tuned, using a pre-trained ResNet50 model as the backbone and GeM as the aggregation layer. The Pix2pix models were trained using the Adam optimizer with a learning rate of 2e-4 and an exponential decay rate of 0.5 for the first-moment estimation. When fine-tuning the visual feature extraction framework, each batch contained 100 different locations, with each location including images from four seasons. Stochastic gradient descent was used as the optimizer with a momentum of 0.9 and a decay rate of 0.001. The initial learning rate was 0.03, multiplied by 0.3 every 5 epochs, for a total of 30 epochs. All images were resized to 320×320 pixels.

[0051] The comparative experiments included several state-of-the-art VPR algorithms: AVG, NetVLAD, CosPlace, and GeM. CosPlace employed data augmentation methods, including random adjustment of crop size and color jitter. Recall@K (R@K) was used as the evaluation metric. It represents the percentage of successful matches among the top K most similar images retrieved from the database, with a successful match defined as a physical distance between the two images within a pre-defined threshold. Here, the threshold was set to 25 meters. The experimental datasets used were Pitts250k-test, MSLS-val, and SPED. These datasets have high sample similarity and include seasonal variations, allowing for the evaluation of VPR algorithm performance under extreme environmental conditions. Figure 2 The performance of four comparative methods and the proposed method on experimental datasets is shown, demonstrating the effectiveness in the Recall@K (R@K) case. Specifically, it outperforms other methods on the MSLS-val and SPED datasets, and also shows improvement over GeM on the Pitts250k-test dataset. The larger seasonal distribution variance of the MSLS-val and SPED datasets may explain the more significant improvement of our method.

[0052] Ablation experiments compared to GeM validated the effectiveness of manually filling in missing seasonal images. Figure 2The second-to-last line shows the effect of using the pipeline without preprocessing the dataset when the seasonal transformation model is not used. A comparison of the two shows that using the dataset enhanced by the seasonal transformation module for feature extraction improves the algorithm's accuracy and robustness.

[0053] In summary, this invention provides a dataset augmentation method for visual location recognition. It improves the scene coverage of the dataset and resists perceptual aliasing caused by environmental changes. For seasonal changes, it utilizes the Pix2pix model with Conditional GANs for image transformation, obtaining a transformation model for images across the four seasons. The trained seasonal transformation model is then used to fill in the training set for feature extraction, resulting in a dataset containing more useful seasonal information. Finally, more robust global image features are extracted. Experimental results show that the global features extracted by the proposed method achieve better performance in the face of seasonal changes, even changes in lighting and weather, and can solve the perceptual aliasing problem.

[0054] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

[0055] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A dataset augmentation method for visual location recognition, characterized in that, Includes the following steps: S1. The Nordland dataset containing complete seasonal image distributions of all locations is processed using Conditional GAN. Based on the Pix2pix model, the seasonal transformation of images is realized, and the ability of images to transform between the four seasons of spring, summer, autumn and winter is obtained. A total of 12 different Pix2pix models are obtained between the four seasons. S2. Using the 12 Pix2pix models obtained in step S1, supplement the missing seasonal information of an existing GSV-Cities dataset to enhance the complete seasonal coverage of the dataset and obtain a GSV-Cities dataset with more complete seasonal information. S3. Input the images from the GSV-Cities dataset with more complete seasonal information obtained in step S2 into the feature extractor for training to obtain a trained image feature extraction model. The feature extraction process includes inputting the image into ResNet-50 to obtain local image features, and then using GeM aggregation to obtain the final global features. The entire feature extraction process is carried out in a fully supervised manner, using multiple similarity loss and online hard mining strategies. S4. Perform performance analysis on the image feature extraction model trained in step S3, comparing it with current VPR algorithms, including AVG, NetVLAD, CosPlace, and GeM. Comparative experiments are conducted on three existing datasets: Pitts250k-test, MSLS-val, and SPED. As an evaluation metric for the experiment.

2. The dataset augmentation method for visual location recognition as described in claim 1, characterized in that, The image seasonal transformation based on the Pix2pix model is an image translation, realizing the transformation of the image from one domain to another, that is, domain migration.

3. The dataset augmentation method for visual location recognition as described in claim 2, characterized in that, The Nordland dataset, containing complete seasonal image distributions for all locations, is processed using Conditional GAN. The Conditional GAN ​​model consists of a generator and a discriminator. During training, the generator and discriminator compete against each other until an equilibrium point is reached, at which point the optimal solution is obtained. The generator in the Conditional GAN ​​model adopts an improved "U-Net" architecture that adds skip connections to the encoder-decoder, and the discriminator uses a convolutional PatchGAN classifier to determine whether each small patch of the image is real or fake.

4. The dataset augmentation method for visual location recognition as described in claim 1, characterized in that, The feature extractor described in step S3 uses the efficient and lightweight ResNet-50 to extract preliminary features, and the aggregation method uses GeM to aggregate local features.

5. The dataset augmentation method for visual location recognition as described in claim 4, characterized in that, An image is input to the ResNet-50 backbone network and outputs a tensor. ,in For high, For width, For resolution, This refers to the number of filters in the last convolutional layer of the backbone network; the specific form of GeM is... ,in , A set of feature maps; parameters It can be set manually or learned.

6. The dataset augmentation method for visual location recognition as described in claim 1, characterized in that, Step S3 describes training using a combination of multiple similarity loss and an online hard mining strategy. The multiple similarity loss, abbreviated as MS loss, incorporates an advanced weighting scheme. The MS loss function is as follows: in The number of images in a batch; for each query image in a batch , This refers to the set of samples that corresponds to the query, where the images in the set of samples that correspond to the query represent the same location. This refers to the set of samples that form negative pairs with the query, where the images in the set of samples that form negative pairs with the query represent different locations; and These represent the cosine distances between the query and the positive and negative pairs, respectively. , and This is a hyperparameter.

Images