An image-based GPS-free target localization method

Abstract

This invention belongs to the fields of machine vision and target localization technology, and discloses an image-based GPS-free target localization method. The method includes: establishing pre-processed acquired images, training a network, and outputting location information. This invention, through pure image recognition, eliminates the dependence of traditional positioning systems on GPS to locate the target's position. It has enormous application prospects in mobile robots and autonomous driving, while also emphasizing equipment safety in unmanned navigation, ensuring safe travel for users.

Description

Technical Field

[0001] This invention belongs to the fields of machine vision and target localization technology, and relates to an image-based target localization method without GPS. Background Technology

[0002] The Global Positioning System (GPS) is widely used for target positioning, but various problems remain. For example, in densely built-up areas, areas with aerial obstructions, or inclement weather, GPS signals may be inaccurate, making accurate target positioning difficult. Furthermore, as a commercial positioning system, GPS is easily tracked, compromising security. Image retrieval methods, as alternatives, often suffer from difficulties in matching feature values ​​due to varying image sizes. Therefore, designing an image-based, GPS-free target positioning method that eliminates dependence on GPS and adapts to multiple scales holds immense promise for applications in mobile robotics and autonomous driving. Summary of the Invention

[0003] (I) Purpose of the Invention

[0004] The purpose of this invention is to address the problems of limited application conditions, insufficient security, and low accuracy in existing target localization methods. Image retrieval methods based on Vector of Locally Aggregated Descriptors (VLAD) are difficult to handle the massive databases available today and struggle to address feature matching under different sizes and complex backgrounds. Therefore, this invention proposes a weakly supervised multi-scale NetVLAD, which uses a weakly supervised training method combined with a multi-scale attention mechanism to achieve object recognition and localization within target images.

[0005] (II) Technical Solution

[0006] To address the aforementioned technical problems, this invention provides an image-based GPS-free target localization method, comprising the following steps:

[0007] S1: Select the training dataset T;

[0008] S2: Input the training dataset T into the VGG16 convolutional network, perform multiple CNN convolutions, and obtain a feature map X of size W*H*D, where W, H, and D are the width, height, and length of the feature map;

[0009] S3: Input the feature map X to perform dimensionality reduction operation. Through three convolutions at different scales, obtain feature maps of the corresponding scales. Then, obtain channel attention through pooling and fully connected operations. Finally, fuse the channel attention with features of multiple scales to complete the multi-scale attention module and obtain the multi-scale attention mask.

[0010] S4: Combine the multi-scale attention module and the NetVLAD network to obtain the multi-scale NetVLAD network. Multiply the multi-scale attention mask obtained by the multi-scale attention module with the feature map to obtain the multi-scale VLAD vector.

[0011] S5: Normalize the VLAD vector using the L2 normalization algorithm to obtain a K*D dimensional VLAD feature vector; where K represents the number of feature points in the image and D represents the dimension of the feature vector.

[0012] S6: Reduce the dimensionality of the obtained K*D dimensional VLAD feature vector;

[0013] S7: Input the training dataset into the multi-scale NetVLAD network for training. Select the stochastic gradient descent algorithm to optimize the parameters of the multi-scale NetVLAD network and obtain the network model.

[0014] S8: Target image localization process: Input the target image into the network model obtained in the previous step, perform feature encoding, compare the feature encoding with the feature encoding in the training dataset, sort and output according to the similarity, and finally output the images with the highest similarity to obtain the location information of the target image.

[0015] In step S1, the triplet loss function is used to optimize the network. The training dataset is divided into anchor images and positive and negative sample images. The images of landmark buildings are used as anchor images. The RANSAC algorithm is used to filter the dataset. According to the number of feature point matching and sample constraints, the dataset is sorted from high to low and divided into positive samples and negative samples to form triplets, which are used as training datasets for input into the multi-scale NetVLAD network for training.

[0016] Step S1 includes the following sub-steps:

[0017] S1-1: Obtain a sufficient number of images with location information through the network and self-taken photos as a training dataset;

[0018] S1-2: Extract VLAD vectors from the images in the training dataset. Using images of landmark buildings as anchor images, select the top 10 images whose VLAD Euclidean distance to the anchor image is more than 255m away from the GPS coordinates as negative samples I. n Positive samples utilize the RANSAC algorithm based on SIFT feature points to match feature points between anchor images and sample images. The first four images with the most matching points and whose GPS coordinates are within 10m of the anchor image are selected as positive samples I. p A triplet is used as a training dataset T.

[0019] Step S3 includes the following sub-steps:

[0020] S3-1: Perform a 1*1*D / 4 convolution operation on the W*H*D feature map to obtain a W*H*D / 4 feature map X';

[0021] S3-2: Perform convolutions of the W*H*D / 4 feature map at three different scales (a, b, and c) to obtain three corresponding feature maps W. a W b W c The feature maps at these three scales are fused along the channel direction of their respective scales and the average value is taken to obtain the multi-scale feature X. m :

[0022] X m =[W a X′,W b X′,W c X′]

[0023] S3-3: Feed the multi-scale features into a multilayer perceptron consisting of two fully connected layers. First, compress the number of channels of the multi-scale features to half of their original size, and then restore them to their original size to obtain channel attention C.

[0024] C = W q (W p X m )

[0025] Among them, [W q W p ] represents the parameters of the convolutional kernels in the two fully connected layers, where the dimension of the channel attention C is D / 4;

[0026] S3-4: Channel attention and multi-scale fusion features are multiplied along the channels to obtain the optimized feature map, which is then fed into a 1*1 convolutional layer to obtain the multi-scale attention mask, i.e., spatial attention S, where W z Parameters for a 1x1 convolutional layer:

[0027] S = W z (CX m ).

[0028] Step S4 includes the following sub-steps:

[0029] S4-1: Perform L2 normalization on the W*H*D feature map X to obtain the normalized feature map X. l The multi-scale attention mask and feature map are multiplied together to obtain the optimized feature map X. s :

[0030] X s =X l *S

[0031] S4-2: When calculating the residual of the feature vector x at each position on the feature map X, it needs to be multiplied with the weight s at the corresponding position to obtain the feature vector matched to the multi-scale attention.

[0032] The formula for calculating the element V(k,j) in the k-th row and j-th column of the feature vector Xs is as follows:

[0033]

[0034] Among them, c k (j) represents the cluster centers obtained from image feature points through a 1*1*D / 4 convolution operation, with a total of k cluster centers. k (x i ) is the Softmax function, and its calculation method is as follows:

[0035]

[0036] S4-3: Accumulate the feature vector elements through the VLAD core layer to obtain the original VLAD vector.

[0037] In step S6, the PCA algorithm is used to reduce the dimensionality of the VLAD feature vector.

[0038] Step S6 includes the following sub-steps:

[0039] S6-1: Obtain the feature vector matrix S from the VLAD feature vectors in the training dataset. N×M :

[0040] S N×M = [S1; S2; ...; S N ]

[0041] Where M = K * D, N is the number of images in the training dataset, and S1, S2, ..., S... N VLAD feature vectors of images in the training dataset;

[0042] S6-2: For the characteristic matrix S N×M Find the covariance matrix C i,j Then, the eigenvalues ​​λ of the covariance matrix are obtained. i and eigenvector x i ;

[0043] S6-3: Sort the eigenvalues ​​in descending order, select the P largest eigenvalues, and then use their corresponding P eigenvectors as column vectors to form the eigenvector matrix H. M×P Finally, for the eigenvector matrix S N×M Dimensionality reduction is performed to obtain a VALD vector S' of dimension N×P;

[0044] S′=[S′1,S′2,…,S′ N ] = S N×M ·H M×P =S N×P .

[0045] In step S3-2, the scale values ​​a, b, and c are 3, 5, and 7 respectively.

[0046] In step S7, the learning rate is set to 0.0005, and the learning rate decays every 5 steps with a decay factor of 0.5.

[0047] (III) Beneficial Effects

[0048] The image-based, GPS-free target localization method provided by the above technical solution has the following beneficial effects:

[0049] 1) This invention uses the NetVLAD algorithm to complete the target localization of the image, which eliminates the dependence of traditional positioning systems on GPS and increases the universality and security of the positioning system.

[0050] 2) The dataset uses an improved triplet training method, which, compared to the traditional triplet training method, sets more specific constraints on positive and negative samples, thereby improving the convergence speed of the neural network.

[0051] 3) Employ a multi-scale attention module to form a new layer of weights, filter out the information that the neural network needs to focus on, improve the accuracy of network retrieval in complex contexts, and reduce the network's computational parameters.

[0052] 4) Employing multi-scale convolutional feature fusion, compared to the ordinary NetVLAD network, it can adapt to images with different field of view sizes, increasing the robustness of the entire neural network.

[0053] 5) The VLAD vector is reduced in dimensionality by using the PCA algorithm, which greatly reduces the difficulty of image retrieval. Attached Figure Description

[0054] Appendix Figure 1 This is a block diagram illustrating the principle of the method of the present invention;

[0055] Appendix Figure 2 This is a diagram of the multi-scale attention module structure.

[0056] Appendix Figure 3 This is a sample architecture diagram of a multi-scale attention module and a NetVLAD network. Detailed Implementation

[0057] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0058] Image-based GPS-free target localization methods, such as Figure 1 As shown, triplet constraints are set to filter the dataset, forming a triplet function as the training dataset. This function is then input into a VGG16 convolutional network to obtain feature map X. Feature map X is then simultaneously input into an L2 normalization module and a multi-scale attention module to obtain a feature map Xs with multi-scale attention fused. Xs is then input into a NetVLAD layer to obtain VLAD feature encoding. After PCA dimensionality reduction, the final image feature representation S is output. N×P Finally, the stochastic gradient descent algorithm is used to optimize the learning parameters in the network.

[0059] This invention includes the following steps:

[0060] (1) Select the training dataset and optimize the network using the triplet loss function. Divide the dataset into anchor images and positive and negative sample images. Using images of landmark buildings as anchor images, the RANSAC algorithm is used to select the dataset. Based on the number of feature point matches and sample constraints, the dataset is sorted from high to low and divided into positive and negative samples to form triplet pairs. These triplet pairs are then input into the multi-scale NetVLAD network for training to improve the network convergence speed. The specific steps are as follows:

[0061] (1-1) Obtain the training dataset.

[0062] (1-2) Extract VLAD vectors from the dataset images. Take the images of landmark buildings as anchor images I, and take the first m images with the closest VLAD Euclidean distance to the anchor images and a GPS coordinate distance of more than 255m as negative samples I. n Positive samples utilize the RANSAC algorithm based on SIFT feature points to match feature points between the anchor image and the sample image. The first n images with the most matching points and whose GPS coordinates are within 10m of the anchor image are selected as positive samples I. p A triplet is used as a training dataset T.

[0063] (2) Input the combined training dataset T into the VGG16 convolutional network and perform multiple CNN convolutions to obtain a feature map X of size W*H*D.

[0064] (3) Input the feature map X into the input for dimensionality reduction. Through convolution at three different scales, feature maps of corresponding scales are obtained. Then, channel attention is obtained through pooling and fully connected operations. The channel attention and features at multiple scales are then fused to complete the multi-scale attention module and obtain a multi-scale attention mask, such as... Figure 2 As shown, the specific steps are as follows:

[0065] (3-1) Perform a 1*1*D / 4 convolution operation on the W*H*D feature map to reduce the feature dimension and obtain a W*H*D / 4 feature map X'.

[0066] (3-2) Convolve the W*H*D / 4 feature map at three different scales: a, b, and c. The scale values ​​a, b, and c are typically 3, 5, and 7, respectively. Then, W... a W b W c The feature maps at multiple scales are fused along the channel direction and the average is taken to obtain the multi-scale feature X. m .

[0067] X m =[W a X′,W b X′,W c X′]

[0068] (3-3) The multi-scale capacitive features are fed into a multilayer perceptron consisting of two fully connected layers. In order to reduce the number of parameters, the number of channels is first compressed to half of the original size, and then the original size is restored. This channel attention C is used.

[0069] C = W q (W p X m )

[0070] Among them [W q W p ] represents the parameters of two fully connected layers, and C∈R / 4.

[0071] (3-4) The channel attention and multi-scale fusion features are multiplied along the channel to obtain the optimized feature map, which is then fed into a 1*1 convolutional layer to obtain the multi-scale attention mask, i.e., the spatial attention S, where W z Parameters for a 1x1 convolutional layer:

[0072] S = W z (CX m )

[0073] (4) Combine the multi-scale attention module and the NetVLAD network to obtain a multi-scale NetVLAD network. Multiply the multi-scale attention mask obtained by the multi-scale attention module with the feature map to obtain the multi-scale VLAD vector, such as... Figure 3 As shown, the specific steps are as follows:

[0074] (4-1) Perform L2 normalization on the feature map X of W*H*D to obtain the normalized feature map X. l The multi-scale attention mask and feature map are multiplied together to obtain the optimized feature map Xs.

[0075] X s =X l *S

[0076] (4-2) When calculating the residual for the feature vector x at each position on the feature map X, it is multiplied by the weight s at the corresponding position, and x i Each channel is multiplied by a weight s to obtain a feature vector matched to multi-scale attention. The formula for calculating the element V(k,j) in the k-th row and j-th column of the feature vector Xs is as follows:

[0077]

[0078] Where c k (j) represents the cluster centers obtained from image feature points through a 1*1 convolution operation, with a total of k cluster centers. k (x i ) is the Softmax function, and its calculation method is as follows:

[0079]

[0080] (4-3) Accumulate the feature vector elements through the VLAD core layer to obtain the original VLAD vector.

[0081] (5) Normalize the VLAD vectors using the L2 normalization algorithm to obtain K*D dimensional VLAD feature vectors.

[0082] (6) The obtained K*D VLAD feature vectors are subjected to dimensionality reduction using the PCA algorithm. The specific steps are as follows:

[0083] (6-1) Obtain the feature vector matrix S from the VLAD feature vectors in the database. N×M .

[0084] S N×M = [S1; S2; ...; S N ]

[0085] Where M = K * D, N is the number of images in the database, and Sk is the VLAD feature vector of the images in the database.

[0086] (6-2) For the characteristic matrix S N×M Find the covariance matrix C i,j Then, the eigenvalues ​​λ of the covariance matrix are obtained. i and eigenvector x i The eigenvalue λ i and eigenvector x i Arranged from smallest to largest, the eigenvalue matrix L is finally obtained. M×M and eigenvector matrix H M×M .

[0087] (6-3) Sort the eigenvalues ​​in descending order, select the P largest eigenvalues, and then use the corresponding P eigenvectors as column vectors to form the eigenvector matrix H. M×P Finally, for the eigenvector matrix S N×M Dimensionality reduction is performed.

[0088] S′=[S1′,S2′,…,S′ N ] = S N×M ·H M×P =S N×P

[0089] (7) Finally, the network model parameters are optimized using the stochastic gradient descent algorithm.

[0090] (8) Target image localization process: After the network training is completed, the target image is input into the training model, and feature encoding is performed. Then, the feature encoding is compared with the feature encoding in the database, and the images are sorted and output according to the similarity. Finally, the images with the highest similarity are output, and the location information of the target image can be obtained.

[0091] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image-based target localization method without GPS, characterized in that, Includes the following steps: S1: Select the training dataset T; S2: Input the training dataset T into the VGG16 convolutional network, perform multiple CNN convolutions, and obtain a feature map X of size W*H*D, where W, H, and D are the width, height, and length of the feature map; S3: Input the feature map X to perform dimensionality reduction operation. Through three convolutions at different scales, obtain feature maps of the corresponding scales. Then, obtain channel attention through pooling and fully connected operations. Finally, fuse the channel attention with features of multiple scales to complete the multi-scale attention module and obtain the multi-scale attention mask. S4: Combine the multi-scale attention module and the NetVLAD network to obtain the multi-scale NetVLAD network. Multiply the multi-scale attention mask obtained by the multi-scale attention module with the feature map to obtain the multi-scale VLAD vector. S5: Normalize the VLAD vector using the L2 normalization algorithm to obtain a K*D dimensional VLAD feature vector; where K represents the number of feature points in the image and D represents the dimension of the feature vector. S6: Reduce the dimensionality of the obtained K*D dimensional VLAD feature vector; S7: Input the training dataset into the multi-scale NetVLAD network for training. Select the stochastic gradient descent algorithm to optimize the parameters of the multi-scale NetVLAD network and obtain the network model. S8: Target image localization process: Input the target image into the network model obtained in the previous step, perform feature encoding, compare the feature encoding with the feature encoding in the training dataset, sort and output according to the similarity, and finally output the images with the highest similarity to obtain the location information of the target image. In step S1, the triplet loss function is used to optimize the network. The training dataset is divided into anchor images and positive and negative sample images. The images of landmark buildings are used as anchor images. The RANSAC algorithm is used to filter the dataset. According to the number of feature point matching and sample constraints, the dataset is sorted from high to low and divided into positive samples and negative samples to form triplets as training datasets, which are then input into the multi-scale NetVLAD network for training. Step S1 includes the following sub-steps: S1-1: Obtain a sufficient number of images with location information through the network and self-taken photos as a training dataset; S1-2: Extract VLAD vectors from the images in the training dataset. Using images of landmark buildings as anchor images, select the top 10 images whose VLAD Euclidean distance to the anchor image is more than 255m away from the GPS coordinates as negative samples I. n Positive samples utilize the RANSAC algorithm based on SIFT feature points to match feature points between anchor images and sample images. The first four images with the most matching points and whose GPS coordinates are within 10m of the anchor image are selected as positive samples I. p , using a triplet as a training dataset T; Step S3 includes the following sub-steps: S3-1: Perform a 1*1*D / 4 convolution operation on the W*H*D feature map to obtain a W*H*D / 4 feature map X'; S3-2: Perform convolutions of the W*H*D / 4 feature map at three different scales (a, b, and c) to obtain three corresponding feature maps W. a W b W c The feature maps at these three scales are fused along the channel direction of their respective scales and the average value is taken to obtain the multi-scale feature X. m : X m =[W a X′,W b X′,W c X′] S3-3: Feed the multi-scale features into a multilayer perceptron consisting of two fully connected layers. First, compress the number of channels of the multi-scale features to half of their original size, and then restore them to their original size to obtain channel attention C. C=W q (W p X m ) Among them, [W q W p ] represents the parameters of the convolutional kernels in the two fully connected layers, where the dimension of the channel attention C is D / 4; S3-4: Channel attention and multi-scale fusion features are multiplied along the channels to obtain the optimized feature map, which is then fed into a 1*1 convolutional layer to obtain the multi-scale attention mask, i.e., spatial attention S, where W z Parameters for a 1x1 convolutional layer: S=W z (CX m ) Step S4 includes the following sub-steps: S4-1: Perform L2 normalization on the W*H*D feature map X to obtain the normalized feature map X. l The multi-scale attention mask and feature map are multiplied together to obtain the optimized feature map X. s : X s =X l *S S4-2: When calculating the residual of the feature vector x at each position on the feature map X, it needs to be multiplied with the weight s at the corresponding position to obtain the feature vector matched to the multi-scale attention. The formula for calculating the element V(k,j) in the k-th row and j-th column of the feature vector Xs is as follows: Among them, c k (j) represents the cluster centers obtained from image feature points through a 1*1*D / 4 convolution operation, with a total of k cluster centers. k (x i ) is the Softmax function, and its calculation method is as follows: S4-3: Accumulate the feature vector elements through the VLAD core layer to obtain the original VLAD vector.

2. The image-based target localization method without GPS as described in claim 1, characterized in that, In step S6, the PCA algorithm is used to reduce the dimensionality of the VLAD feature vector.

3. The image-based target localization method without GPS as described in claim 2, characterized in that, Step S6 includes the following sub-steps: S6-1: Obtain the feature vector matrix S from the VLAD feature vectors in the training dataset. N’×M : S N’×M =[S1;S2;…;S N’ ] Where M = K * D, N' is the number of images in the training dataset, and S1, S2, ..., S... N’ VLAD feature vectors of images in the training dataset; S6-2: For the characteristic matrix S N’×M Find the covariance matrix C i,j Then, the eigenvalues ​​λ of the covariance matrix are obtained. i and eigenvector x i ; S6-3: Sort the eigenvalues ​​in descending order, select the P largest eigenvalues, and then use their corresponding P eigenvectors as column vectors to form the eigenvector matrix H. M×P Finally, for the eigenvector matrix S N’×M Dimension reduction is performed to obtain a VALD vector S' with dimension N'×P; S′=[S′1,S′2,…,S′ N’ ]=S N’×M ·H M×P =S N’×P 。 4. The image-based target localization method without GPS as described in claim 3, characterized in that, In step S3-2, the scale values ​​a, b, and c are 3, 5, and 7 respectively.

5. The image-based target localization method without GPS as described in claim 4, characterized in that, In step S7, the learning rate is set to 0.0005, and the learning rate decays every 5 steps with a decay factor of 0.

5.

6. An application of an image-based GPS-free target localization method based on any one of claims 1-5 in the fields of machine vision and target localization technology.

Images