NetVLAD image position identification method based on deep global feature and local feature joint learning

Abstract

The invention discloses a NetVLAD image position identification method based on deep global feature and local feature joint learning. In the offline stage, firstly, a plurality of reference points are deployed in a target area, and 90% and 10% are taken to divide a training set and a test set; then, learning local and global features by using hierarchical representation of a ResNet-50 convolutional neural network, and training the network by minimizing a cross entropy classification loss function of the local features and a triple loss function of the global features; in the online stage, a query picture is input into the network, global features of the query picture are extracted, similarity calculation is carried out on the global features of the query picture and global features of images in the offline database, similarity ranking is obtained, local features of the query picture are extracted, the similarity ranking is rearranged, and therefore the image which is most matched with the query picture is found. And finally, acquiring GPS information of the picture to realize position identification. According to the invention, local features are acquired by using shallow activation of a convolutional neural network trunk and similarity sequence rearrangement is guided, and a NetVLAD image position identification method based on deep global feature and local feature joint learning is provided, so that the system overhead is reduced, and the positioning precision is ensured at the same time.

Description

Technical Field

[0001] The present invention belongs to visual position recognition technology, and specifically relates to a NetVLAD image position recognition method based on joint learning of deep global features and local features. Background Art

[0002] Place recognition has received extensive attention from both academia and industry in recent years. For a query image, visual place recognition is to identify the location where the image was captured based on a set of previously collected images, possibly with some form of label information, such as text labels provided by people or GPS data captured by sensors. Visual place recognition is a core technology for many important applications such as simultaneous localization and mapping (SLAM), autonomous navigation, etc. However, place recognition has always been a challenge in computer vision and machine learning due to the large variations in the appearance of places, such as changes in viewpoint, seasons, lighting, weather, and irrelevant objects such as trees and cars.

[0003] If various types of label information are provided, location recognition can be transformed into an image retrieval problem or a classification problem. Popular works fit into the former context. This group of methods follows a very similar pipeline, first extracting local feature descriptors [scale-invariant feature transform (SIFT) features], then building a vocabulary on the local descriptors by clustering to form a global feature representation for image retrieval. Typical global feature representations include bag of words (BoW), vector of local aggregated descriptors (VLAD), Hamming embedding, and Fisher vector (FV). One of the drawbacks of the above methods is that they rely on holistic feature representations without explicitly distinguishing different regions of the image, and the feature representations may lack local information of the image. Therefore, some methods directly use local descriptors, which are retrieved or re-weighted before global feature creation to emphasize the most informative regions in the image. As a comparison, classification-based methods use feature representations derived from quantized local feature descriptors to train a classifier for each location.

[0004] Although VPR was formulated as an image retrieval task, there are specific challenges and use cases for recognizing places that distinguish it from other retrieval problems. These challenges are primarily related to the complexity of the scene and the dynamic nature of the world. First, images of places almost never have a single identifiable object in the foreground. Instead, they typically contain multiple visual elements. Many of these elements may not carry useful information about the place, and they may even occlude more useful objects in the background. Furthermore, the appearance of landscapes naturally changes over time, not only because of dynamic objects or physical changes, but also because of changes in lighting, weather, and seasons. Other challenges in VPR come from 1) the presence of recurring elements and architectural patterns that make different places look similar, and 2) the wide variety of angles from which a place can be viewed.

[0005] Therefore, in response to the above two challenges, the present invention proposes a NetVLAD image location recognition method based on joint learning of deep global features and local features. It proposes a carefully selected model that combines the global features of the NetVLAD generalized pooling layer of the deep activation aggregation of the convolutional neural network and the local features of the shallow activation aggregation of the convolutional neural network. By carefully balancing the gradient flow between the two heads, the entire network can be learned end-to-end using only image-level labels. We also introduce a local feature dimensionality reduction technique based on autoencoders and integrate it into the model to improve training efficiency and matching performance. Summary of the invention

[0006] The purpose of the present invention is to provide a NetVLAD image location recognition method based on joint learning of deep global features and local features. At the present stage, an image database is constructed, deep activations in the convolutional neural network backbone ResNet network are extracted, and a NetVLAD layer is added to aggregate the deep activations into a global feature. Local features are more suitable for encoding the intermediate layer of local information, which can be achieved by adopting an attention module, the goal of which is to predict which of the extracted local features are distinctive for the object of interest. In addition, in order to compactly represent hundreds to thousands of local features, we integrate a small convolutional autoencoder module, which is responsible for learning a suitable low-dimensional representation.

[0007] The present invention provides a NetVLAD image position recognition method based on joint learning of deep global features and local features. Figure 1 and Figure 2 The flow chart and system framework diagram of the present invention include the following steps:

[0008] Step 1: Deploy 9,500 reference points (Access Points, APs) with known locations in scenes such as downtown streets, denoted as APs 1 ,…,APm , the jth AP is denoted as AP j (j≤m, j is an integer), and mark the corresponding GPS location information on each image.

[0009] Step 2: Use a handheld camera to take a certain number of images (about 24) at each reference point, including horizontal and 45° upward directions. These images are used to build an image database, of which 90% of the images are used as training sets and 10% of the images are used as test sets. All training images and test images have a resolution of 640*480 pixels.

[0010] Step 3: Use NetVLAD to aggregate deep activations into global features. Figure 3 The following is a diagram of the NetVLAD network framework, which includes the following steps:

[0011] Step 3 (a) Use convolutional neural networks to obtain shallow activations and deep activations. Given an image, we apply the convolutional neural network backbone to obtain two feature maps: and denote shallower and deeper activations, respectively, where H, W, and C correspond to the height, width, and number of channels in each case. For a normal convolutional network, H D ≤H S ,W D ≤W S ,C D ≥C S ; Deeper layers have spatially smaller maps with more channels. Represents the features of the positions h, w in these maps. In our method, we pass these features through the ReLU nonlinearity to ensure that the obtained features are non-negative.

[0012] Step 3 (ii), construct NetVLAD global descriptor. To aggregate deep activations into a global feature, we use NetVLAD layer, a new generalized VLAD layer, which implements VLAD embedding and aggregation through differentiable operations, thus enabling end-to-end training of the network. In addition, NetVLAD provides more trainable parameters than VLAD, thus providing greater flexibility. Figure 3 A schematic diagram of the NetVLAD network structure is shown. In order to construct a layer that can be trained by back-propagation, the layer's operations need to be differentiable with respect to all its parameters and inputs. Therefore, the key challenge is to make the VLAD pool differentiable.

[0013] The source of discontinuity in VLAD is the descriptor x i For cluster center c k The hard assignment ak (x i ). To make this operation differentiable, we replace it with a soft assignment of descriptors to multiple clusters:

[0014]

[0015] The descriptor x i The weights are assigned to cluster c k The sizes are proportional to their proximity, but relative to the proximity of other cluster centers. The range is between 0 and 1, with the highest weights being those closest to the cluster center. α is a parameter (a positive constant) that controls the decay of the response with distance.

[0016] By expanding the square in equation (1), it is easy to see that Eliminating between the numerator and denominator, we obtain a soft assignment of the form:

[0017]

[0018] The vector W k =2αc k , scalar b k =-α||c k || 2 The final form of the NetVLAD layer is obtained by inserting the soft assignment (2) into the VLAD descriptor to obtain the NetVLAD descriptor:

[0019]

[0020] where {w k},{b k} and {c k} is a set of trainable parameters for each cluster k. Similar to the original VLAD descriptor, the NetVLAD layer transforms the descriptor x into i of Softly assign weights to cluster k, aggregating residuals (x i -c k ). However, unlike the original VLAD {c k}, the NetVLAD layer has three independent parameter sets {w k},{b k} and {c k}. This allows greater flexibility than the original VLAD.

[0021] Step 3 (iii) Use a weakly supervised ranking loss to learn a representation f that optimizes location recognition performance θThat is, for a given test query image q, the goal is to convert the database images from the near Ranked higher than all other distant images in the database I i In other words, we want the query q to be similar to the nearby images The Euclidean distance between Smaller than Database I i The distance from the middle to the far image, that is: For distance queries, all images on the map that are more than a certain distance away I i We will show how to translate this requirement into training triplets The ranking loss between .

[0022] From the dataset we created, we got a tuple where for each training query image q, we have a set of potential positive examples and a set of determined negative examples

[0023] The set of potential positive images contains at least one positive image that should match the query, but we do not know which one. To resolve this ambiguity, we propose to determine the best matching potential positive image

[0024]

[0025] For each training tuple The goal then becomes to learn an image representation f θ , so that the training query q is positive with the best matching potential The distance between Less than the query q and all negative images q j The distance between

[0026]

[0027] Based on this intuition, we define a weakly supervised ranking loss L θ For training tuples

[0028]

[0029] Where l is the hinge loss l(x) = max(x,0), and m is a constant parameter that gives the boundary. Note that the above formula is the negative image The loss is the sum of the individual losses of . For each negative, if the distance between the query and the negative is greater than the distance between the query and the best matching positive, the loss l is zero. Conversely, if the difference between the distance between the query image and the negative image and the distance between the query image and the best matching positive image is less than a bound, the loss value is proportional to the number of such negative images. Note that the above loss is related to the commonly used triplet loss, but is adapted to our weakly supervised scenario using a formula similar to that for multiple instance learning.

[0030] Step 4: Build a global feature and local feature joint learning framework based on convolutional neural network. Figure 4 The figure shows a joint learning framework of global and local features based on NetVLAD. The shallow activation of the convolutional neural network backbone is used to obtain local features, and the global and local features are combined for end-to-end learning.

[0031] Step 4 (a) Select relevant regions to match. This can be achieved by using an attention module M, whose goal is to predict which of the extracted local features are distinctive for the object of interest. This is performed using A = M (S), where M is a small convolutional network, Represents a map of attention scores associated with the features of S.

[0032] Step 4 (ii) Dimensionality reduction of local descriptors. In addition, since hundreds to thousands of local features are usually used, they must be represented compactly. To this end, we propose to integrate a small convolutional autoencoder (AE) module, which is responsible for learning a suitable low-dimensional representation. The local descriptor is obtained as L = T (S), where T is the encoding part of the autoencoder, corresponding to a 1×1 convolutional layer with C T Filter. Note that, in contrast to S, the local descriptor L is not constrained to be non-negative.

[0033] Step 4 (iii), construct local feature loss. To train local features, we use two losses. First, the mean squared error regression loss measures the ability of the autoencoder to reconstruct S. Denote S'=T'(L) as a reconstructed version of S with the same dimensions, where T' is a vector with C S 1×1 convolutional layer with 1×1 filters, followed by ReLU.

[0034]

[0035] Second, the cross-entropy classification loss encourages the attention module to select discriminative local features. This is achieved by first pooling the reconstructed features S· with the attention weights ah,w:

[0036]

[0037] Then use the standard softmax-cross-entropy loss:

[0038]

[0039] Where v i ,b i is the classifier weight and bias for class I, and k is the indicator of the ground truth class; this tends to make the attention weight of the discriminant features larger. The total loss is L g +λL r +βL a .

[0040] Step 4 (iv), control the gradient. Directly optimizing the above total loss leads to suboptimal results in experiments, because the reconstruction and attention loss terms severely perturb the hierarchical feature representations typically obtained when training deep models. In particular, both tend to make shallower features S more semantic and less localizable, and thus ultimately more sparse. Sparse features make it easier to optimize L r , and more semantic features may help optimize L a ; therefore, this will result in poor performance of local functions.

[0041] We stop at L r and L a This problem can be avoided by backpropagating the gradients to the backbone of the network, i.e. to S. This means that the backbone of the network is based only on L g The optimization is performed and will tend to produce the desired hierarchical feature representation.

[0042] Step 5: Input the test image into the trained model, match it with the database image, output the best matching image, and obtain the corresponding GPS location information.

[0043] Beneficial Effects

[0044] The present invention starts from image matching and image-based positioning. First, multiple reference points are deployed in the target area and 90% and 10% are taken to divide the training set and the test set; secondly, a NetVLAD position recognition framework based on deep global feature and local feature joint learning is constructed to learn the global features and local features of the image; finally, after the query image is input, the global features are used to perform preliminary sorting with the database image, and then the local features are used to rearrange the top results, thereby improving the accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the present invention;

[0046] Figure 2 This is a system framework diagram of the present invention;

[0047] Figure 3 This is the NetVLAD network framework diagram

[0048] Figure 4 Based on NetVLAD global feature and local feature joint learning framework Specific implementation plan

[0049] The purpose of the present invention is to provide a NetVLAD image location recognition method based on joint learning of deep global features and local features. At the present stage, an image database is constructed, deep activations in the convolutional neural network backbone ResNet network are extracted, and a NetVLAD layer is added to aggregate the deep activations into a global feature. Local features are more suitable for encoding the intermediate layer of local information, which can be achieved by adopting an attention module, the goal of which is to predict which of the extracted local features are distinctive for the object of interest. In addition, in order to compactly represent hundreds to thousands of local features, we integrate a small convolutional autoencoder module, which is responsible for learning a suitable low-dimensional representation.

[0050] The present invention provides a NetVLAD image position recognition method based on joint learning of deep global features and local features. Figure 1 and Figure 2 The flow chart and system framework diagram of the present invention include the following steps:

[0051] Step 1: Deploy 9,500 reference points (Access Points, APs) with known locations in scenes such as downtown streets, denoted as APs 1 ,…,AP m , the jth AP is denoted as AP j (j≤m, j is an integer), and mark the corresponding GPS location information on each image.

[0052] Step 2: Use a handheld camera to take a certain number of images (about 24) at each reference point, including horizontal and 45° upward directions. These images are used to build an image database, of which 90% of the images are used as training sets and 10% of the images are used as test sets. All training images and test images have a resolution of 640*480 pixels.

[0053] Step 3: Use NetVLAD to aggregate deep activations into global features. Figure 3 The following is a diagram of the NetVLAD network framework, which includes the following steps:

[0054] Step 3 (a) Use convolutional neural networks to obtain shallow activations and deep activations. Given an image, we apply the convolutional neural network backbone to obtain two feature maps: S∈RHs×Ws×Cs and denote shallower and deeper activations, respectively, where H, W, and C correspond to the height, width, and number of channels in each case. For a normal convolutional network, H D ≤H S ,W D ≤W S ,C D ≥C S ; Deeper layers have spatially smaller maps with more channels. Represents the features of the positions h, w in these maps. In our method, we pass these features through the ReLU nonlinearity to ensure that the obtained features are non-negative.

[0055] Step 3 (ii), construct NetVLAD global descriptor. To aggregate deep activations into a global feature, we use NetVLAD layer, a new generalized VLAD layer, which implements VLAD embedding and aggregation through differentiable operations, thus enabling end-to-end training of the network. In addition, NetVLAD provides more trainable parameters than VLAD, thus providing greater flexibility. Figure 3 A schematic diagram of the NetVLAD network structure is shown. In order to construct a layer that can be trained by back-propagation, the layer's operations need to be differentiable with respect to all its parameters and inputs. Therefore, the key challenge is to make the VLAD pool differentiable.

[0056] The source of discontinuity in VLAD is the descriptor x i For cluster center c k The hard assignment a k (x i ). To make this operation differentiable, we replace it with a soft assignment of descriptors to multiple clusters:

[0057]

[0058] The descriptor x i The weights are assigned to cluster c k The sizes are proportional to their proximity, but relative to the proximity of other cluster centers. The range is between 0 and 1, with the highest weights being those closest to the cluster center. α is a parameter (a positive constant) that controls the decay of the response with distance.

[0059] By expanding the square in equation (1), it is easy to see that Eliminating between the numerator and denominator, we obtain a soft assignment of the form:

[0060]

[0061] The vector W k =2αc k , scalar b k =-α||c k || 2 The final form of the NetVLAD layer is obtained by inserting the soft assignment (2) into the VLAD descriptor to obtain the NetVLAD descriptor:

[0062]

[0063] where {w k},{b k} and {c k} is a set of trainable parameters for each cluster k. Similar to the original VLAD descriptor, the NetVLAD layer transforms the descriptor x into i of Softly assign weights to cluster k, aggregating residuals (x i -c k ). However, unlike the original VLAD {c k}, the NetVLAD layer has three independent parameter sets {w k},{b k} and {c k}. This allows greater flexibility than the original VLAD.

[0064] Step 3 (iii) Use a weakly supervised ranking loss to learn a representation f that optimizes location recognition performance θ That is, for a given test query image q, the goal is to convert the database images from the near Ranked higher than all other distant images in the database I i In other words, we want the query q to be similar to the nearby images The Euclidean distance between Smaller than Database I i The distance from the middle to the far image, i.e.: d θ (q,I i* ) <d θ (q,I i ), for all images I on the map that are beyond a certain distance in the distance query i We will show how to translate this requirement into training triples {q,I i* ,I i}.

[0065] From the dataset we created, we got a tuple where for each training query image q, we have a set of potential positive examples and a set of determined negative examples

[0066] The set of potential positive images contains at least one positive image that should match the query, but we do not know which one. To resolve this ambiguity, we propose to determine the best matching potential positive image

[0067]

[0068] For each training tuple The goal then becomes to learn an image representation f θ , so that the training query q is positive with the best matching potential The distance between Less than the query q and all negative images q j The distance between

[0069]

[0070] Based on this intuition, we define a weakly supervised ranking loss L θ For training tuples

[0071]

[0072] Where l is the hinge loss l(x) = max(x,0), and m is a constant parameter that gives the boundary. Note that the above formula is the negative image The loss is the sum of the individual losses of . For each negative, if the distance between the query and the negative is greater than the distance between the query and the best matching positive, the loss l is zero. Conversely, if the difference between the distance between the query image and the negative image and the distance between the query image and the best matching positive image is less than a bound, the loss value is proportional to the number of such negative images. Note that the above loss is related to the commonly used triplet loss, but is adapted to our weakly supervised scenario using a formula similar to that for multiple instance learning.

[0073] Step 4: Build a global feature and local feature joint learning framework based on convolutional neural network. Use the shallow activation of the convolutional neural network backbone to obtain local features, and combine global features and local features for end-to-end learning.

[0074] Step 4 (a) Select relevant regions to match. This can be achieved by using an attention module M, whose goal is to predict which of the extracted local features are distinctive for the object of interest. This is performed using A = M (S), where M is a small convolutional network, Represents a map of attention scores associated with the features of S.

[0075] Step 4 (ii) Dimensionality reduction of local descriptors. In addition, since hundreds to thousands of local features are usually used, they must be represented compactly. To this end, we propose to integrate a small convolutional autoencoder (AE) module, which is responsible for learning a suitable low-dimensional representation. The local descriptor is obtained as L = T (S), where T is the encoding part of the autoencoder, corresponding to a 1×1 convolutional layer with C T Filter. Note that, in contrast to S, the local descriptor L is not constrained to be non-negative.

[0076] Step 4 (iii), construct local feature loss. To train local features, we use two losses. First, the mean squared error regression loss measures the ability of the autoencoder to reconstruct S. Denote S'=T'(L) as a reconstructed version of S with the same dimensions, where T' is a vector with C S 1×1 convolutional layer with 1×1 filters, followed by ReLU.

[0077]

[0078] Second, the cross-entropy classification loss encourages the attention module to select discriminative local features. This is achieved by first pooling the reconstructed features S· with the attention weights ah,w:

[0079]

[0080] Then use the standard softmax-cross-entropy loss:

[0081]

[0082] Where v i ,b i is the classifier weight and bias for class I, and k is the indicator of the ground truth class; this tends to make the attention weight of the discriminant features larger. The total loss is L g +λL r +βL a .

[0083] Step 4 (iv), control the gradient. Directly optimizing the above total loss leads to suboptimal results in experiments, because the reconstruction and attention loss terms severely perturb the hierarchical feature representations typically obtained when training deep models. In particular, both tend to make shallower features S more semantic and less localizable, and thus ultimately more sparse. Sparse features make it easier to optimize L r , and more semantic features may help optimize L a ; therefore, this will result in poor performance of local functions.

[0084] We stop at Lr and L a This problem can be avoided by backpropagating the gradients to the backbone of the network, i.e. to S. This means that the backbone of the network is based only on L g The optimization is performed and will tend to produce the desired hierarchical feature representation.

[0085] Step 5: Input the test image into the trained model, match it with the database image, output the best matching image, and obtain the corresponding GPS location information.

Claims

1. A NetVLAD image location recognition method based on joint learning of deep global features and local features, characterized in that The following steps are involved: Step 1: Deploy 9,500 reference points (Access Points, APs) with known locations in scenes such as downtown streets, denoted as AP1,…,AP m , the jth AP is denoted as AP j (j≤m, j is an integer), and mark the corresponding GPS location information on each image. Step 2: Use a handheld camera to take a certain number of images (about 24) at each reference point, including horizontal and 45° upward directions. These images are used to build an image database, of which 90% of the images are used as training sets and 10% of the images are used as test sets. All training images and test images have a resolution of 640*480 pixels. Step 3: Construct the NetVLAD pooling layer to aggregate deep activations into global features. Step 4: Construct a global feature and local feature joint learning framework based on convolutional neural network. Step 5: Input the test image into the trained model, match it with the database image, output the best matching image, and obtain the corresponding GPS location information.

2. The NetVLAD image position recognition method based on joint learning of deep global features and local features according to claim 1 is characterized in that The step three comprises the following steps: Step 3: Use NetVLAD to aggregate deep activations into global features. Figure 3 shows the NetVLAD network framework, which includes the following steps: Step 3 (a) Use convolutional neural networks to obtain shallow activations and deep activations. Given an image, we apply the convolutional neural network backbone to obtain two feature maps: S∈R Hs×Ws×Cs and denote shallower and deeper activations, respectively, where H, W, and C correspond to the height, width, and number of channels in each case. For a normal convolutional network, H D ≤H S ,W D ≤W S ,C D ≥C S ; Deeper layers have spatially smaller maps with more channels. Represents the features of the positions h, w in these maps. In our method, we pass these features through the ReLU nonlinearity to ensure that the obtained features are non-negative. Step 3 (ii), construct the NetVLAD global descriptor. In order to aggregate the deep activations into a global feature, we adopt the NetVLAD layer, a new generalized VLAD layer, which implements the embedding and aggregation of VLAD through differentiable operations, thereby enabling end-to-end training of the network. In addition, NetVLAD provides more trainable parameters than VLAD, providing greater flexibility. Figure 3 shows a schematic diagram of the NetVLAD network structure. In order to construct a layer that can be trained by back-propagation, the layer's operations need to be differentiable for all its parameters and inputs. Therefore, the key challenge is to make the VLAD pool differentiable. The source of discontinuity in VLAD is the descriptor x i For cluster center c k The hard assignment a k (x i ). To make this operation differentiable, we replace it with a soft assignment of descriptors to multiple clusters: The descriptor x i The weights are assigned to cluster c k The sizes are proportional to their proximity, but relative to the proximity of other cluster centers. The range is between 0 and 1, with the highest weights being those closest to the cluster center. α is a parameter (a positive constant) that controls the decay of the response with distance. By expanding the square in equation (1), it is easy to see that Eliminating between the numerator and denominator, we obtain a soft assignment of the form: The vector W k =2αc k , scalar b k =-α||c k || 2 The final form of the NetVLAD layer is obtained by inserting the soft assignment (2) into the VLAD descriptor to obtain the NetVLAD descriptor: where {w k },{b k } and {c k } is a set of trainable parameters for each cluster k. Similar to the original VLAD descriptor, the NetVLAD layer transforms the descriptor x into i of Softly assign weights to cluster k, aggregating residuals (x i -c k ) is the first-order statistic. However, unlike the original VLAD {c k }, the NetVLAD layer has three independent parameter sets {w k },{b k } and {c k }. This allows greater flexibility than the original VLAD. Step 3 (iii) Use a weakly supervised ranking loss to learn a representation f that optimizes location recognition performance θ That is, for a given test query image q, the goal is to convert the database images from the near Ranked higher than all other distant images in the database I i In other words, we want the query q to be similar to the nearby image I i* The Euclidean distance between Smaller than Database I i The distance from the middle to the far image, that is: For distance queries, all images on the map that are more than a certain distance away I i We will show how to translate this requirement into training triples {q,I i* ,I i }. From the dataset we created, we got a tuple where for each training query image q, we have a set of potential positive examples and a set of determined negative examples The set of potential positive images contains at least one positive image that should match the query, but we do not know which one. To resolve this ambiguity, we propose to determine the best matching potential positive image For each training tuple The goal then becomes to learn an image representation f θ , so that the training query q is positive with the best matching potential The distance between Less than the query q and all negative images q j The distance between Based on this intuition, we define a weakly supervised ranking loss L θ For training tuples Where l is the hinge loss l(x) = max(x,0), and m is a constant parameter that gives the boundary. Note that the above formula is the negative image The loss is the sum of the individual losses of . For each negative, if the distance between the query and the negative is greater than the distance between the query and the best matching positive, the loss l is zero. Conversely, if the difference between the distance between the query image and the negative image and the distance between the query image and the best matching positive image is less than a bound, the loss value is proportional to the number of such negative images. Note that the above loss is related to the commonly used triplet loss, but is adapted to our weakly supervised scenario using a formula similar to that for multiple instance learning.

3. The NetVLAD image position recognition method based on joint learning of deep global features and local features according to claim 1 is characterized in that The step 4 comprises the following steps: Step 4: Build a global feature and local feature joint learning framework based on convolutional neural network. As shown in Figure 4, it is a global feature and local feature joint learning framework based on NetVLAD. The shallow activation of the convolutional neural network backbone is used to obtain local features, and the global features and local features are combined for end-to-end learning. Step 4 (a) Select relevant regions to match. This can be achieved by using an attention module M, whose goal is to predict which of the extracted local features are distinctive for the object of interest. This is performed using A = M (S), where M is a small convolutional network, Represents a map of attention scores associated with the features of S. Step 4 (ii) Dimensionality reduction of local descriptors. In addition, since hundreds to thousands of local features are usually used, they must be represented compactly. To this end, we propose to integrate a small convolutional autoencoder (AE) module, which is responsible for learning a suitable low-dimensional representation. The local descriptor is obtained as L = T (S), where T is the encoding part of the autoencoder, corresponding to a 1×1 convolutional layer with C T Filter. Note that, in contrast to S, the local descriptor L is not constrained to be non-negative. Step 4 (iii), construct local feature loss. To train local features, we use two losses. First, the mean squared error regression loss measures the ability of the autoencoder to reconstruct S. Denote S'=T'(L) as a reconstructed version of S with the same dimensions, where T' is a vector with C S 1×1 convolutional layer with 1×1 filters, followed by ReLU. Second, the cross-entropy classification loss encourages the attention module to select discriminative local features. This is achieved by first pooling the reconstructed features S· with the attention weights ah,w: Then use the standard softmax-cross-entropy loss: Where v i ,b i is the classifier weight and bias for class I, and k is the indicator of the ground truth class; this tends to make the attention weight of the discriminant features larger. The total loss is L g +λL r +βL a . Step 4 (iv), control the gradient. Directly optimizing the above total loss leads to suboptimal results in experiments, because the reconstruction and attention loss terms severely perturb the hierarchical feature representations typically obtained when training deep models. In particular, both tend to make shallower features S more semantic and less localizable, and thus ultimately more sparse. Sparse features make it easier to optimize L r , and more semantic features may help optimize L a ; therefore, this will result in poor performance of local functions. We stop at L r and L a This problem can be avoided by backpropagating the gradients to the backbone of the network, i.e. to S. This means that the backbone of the network is based only on L g The optimization is performed and will tend to produce the desired hierarchical feature representation.

Images