A visual scene recognition method with spatial self-perception under self-attention

Abstract

The application relates to the technical field of robot autonomous navigation and visual recognition, and discloses a visual scene recognition method for spatial self-perception under self-attention, which is realized by adopting an S3VPR model composed of a ViTbackbone and a feature aggregation module Token Module and comprises the following steps: 1) for a given input image, extracting basic features by using the ViTbackbone to obtain a feature map, and then forming a new feature map; 2) performing a reshape operation on the new feature map to convert the two-dimensional feature map into a three-dimensional feature map; 3) using the feature aggregation module Token Module to first process the three-dimensional feature map along the spatial dimension to obtain a spatial interaction feature map, then processing the spatial interaction feature map along the channel dimension to obtain a channel interaction feature map, and further processing the channel interaction feature map to obtain a global descriptor; and 4) using a similarity algorithm to one-by-one compare the global descriptor of a query image with the global descriptors of all database images in a database to find all database images belonging to the same scene as the query image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot autonomous navigation and visual recognition, in particular to a visual scene recognition method of spatial self-perception under self-attention. BACKGROUND

[0002] Visual place recognition (VPR) technology can provide an effective solution for self-perception environment positioning and path planning of autonomous intelligent agents (such as drones and robots). In previous studies, scholars divided the VPR research into two categories: single-stage retrieval and two-stage retrieval. Single-stage retrieval fuses image features into global descriptors and filters out top-k candidate images based on Euclidean distance to draw conclusions; two-stage retrieval uses image local features and uses a re-ranking algorithm to re-rank the optimal candidate image to further clarify the conclusion.

[0003] Although two-stage retrieval can improve performance, it requires additional memory and time overhead. Numerous research works have shown that if the global descriptor generation algorithm of single-stage retrieval can effectively eliminate the interference of complex shooting environments and obtain a global descriptor with high robustness and visual invariance information, its performance will be even more efficient.

[0004] In recent years, most methods in the field of VPR tend to use models based on or involving self-attention mechanisms to extract basic features of images and use specific feature fusion algorithms to obtain highly robust descriptors from them. However, authors rarely pay attention to the strong discriminative visual attribute information possessed by image features themselves. SUMMARY

[0005] The present application aims to solve the deficiencies of the prior art and provide a visual scene recognition method of spatial self-perception under self-attention, which can aggregate a global descriptor with high robustness and visual invariance information from the extracted image basic features through a newly designed simple and efficient feature aggregation module Token Module.

[0006] The present application is implemented through the following technical solution: a visual scene recognition method of spatial self-perception under self-attention, which is implemented using an S3VPR model mainly composed of a ViT backbone and a feature aggregation module Token Module, including the following steps:

[0007] 1) For a given input image q, and q∈R 3×w×h (w, h respectively represent the width and height of the input image, preferably set w x h to 224 x 224, or other sizes) Use the ViT backbone to extract basic features to obtain a feature map F composed of wh+1 token vectorscls And F cls ∈R (wh+1)×c Then a new feature map F is formed. 2D And F 2D ∈R wh×c Where w and h represent the width and height of the input image, respectively, c represents the dimension of the token vector, and the input image q includes the query image and the database image;

[0008] 2) For the new feature map F 2D Perform a reshape operation to transform the 2D feature map into a 3D feature map F, where F ∈ R. w×h×c ;

[0009] 3) Using the newly designed feature aggregation module Token Module, the 3D feature map F is first processed along the spatial dimension to obtain the spatial interaction feature map F. si And F si ∈R w×h×c Then, from the channel dimension, the spatial interaction feature map F si The channel interaction feature map F is obtained through processing. ci And F ci ∈R w×h×c Then, further processing yields the global descriptor z;

[0010] The newly designed feature aggregation module Token Module is used to process the 3D feature map F along the spatial direction and the channel direction to enhance its ability to express the most significant features. Then, the feature aggregation component GeM head in Token Module is used to process it to obtain a robust global descriptor z for final discrimination.

[0011] 4) Use a similarity algorithm to compare the global descriptor z of the query image with the global descriptors z of all database images in the database to find all database images that belong to the same scene as the query image.

[0012] To further improve the spatial self-awareness visual scene recognition method under self-attention described in this invention, the following configuration is specifically adopted: the new feature map F 2D To discard feature map F cls The class token vector is obtained by taking only the Patch tokens vector set.

[0013] To further improve the visual scene recognition method for spatial self-awareness under self-attention described in this invention, the following setting is adopted: the similarity algorithm is cosine similarity (or other similarity algorithms such as Euclidean distance).

[0014] Further, in order to better realize the visual scene recognition method under self-attention and spatial self-awareness, the following setting mode is particularly used: the feature aggregation module Token Module includes a main part Token Block and a feature aggregation component GeM head, and the main part Token Block includes a Space interaction module for enhancing the expression ability of the salient features of the feature map from the spatial dimension and a Channel interaction module for enhancing the expression ability of the salient features of the feature map from the channel dimension.

[0015] Further, in order to better realize the visual scene recognition method under self-attention and spatial self-awareness, the following setting mode is particularly used: the Space interaction module includes a Space self-awareness unit and a Space fusion unit, and the process of the Space interaction module processing the three-dimensional feature map F from the spatial dimension includes:

[0016] 3.1.1) Based on the connection of a single pixel and other surrounding pixels in visual attributes, the Space self-awareness unit is first used to mine the visual structure attributes inside the three-dimensional feature map F, and a channel-type spatial self-awareness feature map (containing more rich visual information) F is calculated. hp hp ∈R w×h×c×p×p ; wherein p x p is the width and height size of the perception area.

[0017] 3.1.2) Then, the Space fusion unit is used to perform dimension reduction operation on the channel-type spatial self-awareness feature map F hp , so as to reduce the single-pixel perception domain.

[0018] 3.1.3) Finally, after step 3.1.2), the new three-dimensional feature map F sf after dimension reduction is fused with the three-dimensional feature map F through the mode of “residual connection”, and F sf ∈R w×h×c , so as to obtain a spatial interaction feature map F si , and F si ∈R w×h×c ; the new feature F si obtained has rich enhanced visual information while maintaining the inherent geometric structure attributes.

[0019] Further, in order to better realize the visual scene recognition method under self-attention and spatial self-awareness, the following setting mode is particularly used: the step 3.1.1) is specifically: ​

[0020] 3.1.1.1) After the space self-awareness unit fills the edges of the three-dimensional feature map F through the Pad operation, it uses the Expand operation to extract the surrounding area of size p x p (p is preferably set to 3) centered on each pixel channel by channel, to obtain a new feature F', and F' ∈ R w×h×c×p×p ;

[0021] 3.1.1.2) For the new feature F', the Hadamard product operation is used again to calculate the channel-wise spatial self-awareness feature map F of each pixel and its p x p surrounding area hp , that is:

[0022] F hp = Expand (Pad (F)) ⊙ Broadcast (F), F hp ∈ R w×h×c×p×p (1);

[0023] Where ⊙ represents the Hadamard product operation; Broadcast is a Python operation mechanism; during the Hadamard product calculation, Broadcast will automatically expand the three-dimensional feature map F to the same shape as the new feature F', that is, to get the broadcast feature F'' ∈ R w×h×c×p×p .

[0024] Further, in order to better realize the visual scene recognition method of spatial self-awareness under self-attention according to the present application, the following setting mode is particularly adopted: the space fusion unit re-encodes the high-dimensional channel-wise spatial self-awareness feature map F hp containing visual structure information into a new three-dimensional feature map F sf . The step 3.1.2) is specifically: first, the space fusion unit performs L2normalization operation on the channel-wise spatial self-awareness feature map F hp , and then uses GeMpooling to reduce its dimension, so that the size of the single-pixel perception domain is reduced from p x p to 1 x 1.

[0025] The design idea of "residual connection" is as follows: the new three-dimensional feature map F sf , although it contains richer image visual attribute information, but it will destroy the geometric structure of the original feature during spatial self-awareness calculation. Therefore, the present application selects to fuse the original feature through the method of residual connection based on F sf , in order to enhance its visual attributes while recovering its geometric structure information, so as to obtain the spatial interaction feature map F si .

[0026] Further to better realize the self-attention under the space self-perception visual scene recognition method of the application, the following setting mode is particularly used: in the main part Token Block, after the S3VPR captures the visual attribute information of the image features in the spatial dimension by using the Space interaction module, the Channel interaction module is used to further develop information interaction in the channel dimension. The Channel interaction module mainly consists of two fully connected layers with an activation function in the middle, and the processing process of the new features F si in the channel dimension includes:

[0027] 3.2.1) For the input spatial interaction feature map F si , a first fully connected layer is used to perform encoding operation, and the channel number of F si is reduced to 1 / r of the original;

[0028] 3.2.2) Then, the ReLU activation function is used to enhance the nonlinear expression of the encoded features;

[0029] 3.2.3) Finally, a second fully connected layer is used to perform decoding operation, while the dimension of the channel features is restored;

[0030] The Channel interaction module uses the following formula when processing the spatial interaction feature map F si in the channel dimension:

[0031] F ci =W2σ(W1F si +b1)+b2,F ci ∈R w×h×c (2);

[0032] In the formula, F ci is the channel interaction feature map, and σ represents the ReLU activation function; W1, W2 and b1, b2 are the weight parameters and the induction bias of the two fully connected layers, respectively; r is a scaling factor (variable, preferably set to 0.75).

[0033] Further to better realize the self-attention under the space self-perception visual scene recognition method of the application, the following setting mode is particularly used: the feature aggregation component GeM head includes a pooling layer for reducing the dimension of the input channel interaction feature map F ci , and a fully connected layer for transforming the reduced feature expression to obtain the global descriptor z, and z=GeMHead(F ci )=WGeM(F ci )+b,z∈Rd where W, b are the weight parameters and the inductive bias of the fully connected layer respectively; W e R c×d and d is the vector dimension of the global descriptor z (variable, preferably set to 4096).

[0034] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0035] (1) Experiments on multiple authoritative benchmarks (including four public datasets such as Tokyo24 / 7, MSLS-val, MSLS-challenge and Pitts30k-test) show that the S3VPR method of the present application outperforms the latest DHE-VPR (published in the top conference AAAI 2024) and ViT-R50-MSE (published in the top conference ICRA 2024) methods in 2024. At the same time, sufficient ablation experiments also show that the Token Module has good transferability and can be well combined with ViT or most CNN architectures.

[0036] (2) The S3VPR model used in the present application is a one-stage network, which has a light structure and low complexity, and can be trained end-to-end; the designed Token Module is "plug and play", has good transferability, and can be combined with other arbitrary models (including but not limited to CNN, Transformer or MLP) and play a role.

[0037] (3) The Token Module designed in the present application successively mines the strong discriminative visual attribute information possessed by the image features from the spatial and channel dimensions. This attribute information is globally related and retains the complete structural properties of the image, which can effectively cope with the influence of complex environmental changes on scene picture matching tasks. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the overall structure diagram of S3VPR.

[0039] Figure 2 is the structure diagram of Token Module. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below in conjunction with examples, but the embodiments of the present application are not limited thereto.

[0041] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0042] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly and specifically limited.

[0043] Noun explanation:

[0044] S3VPR model: the abbreviation of Space Self-awareness under Self-attention for Visual Place Recognition, which is a visual scene recognition method model under self-attention space self-awareness.

[0045] ViT backbone: refers to the backbone network of the type of ViT (the abbreviation of Vision Transformer).

[0046] Token Module: feature aggregation module.

[0047] Token vector: token is a feature form, and token vector refers to a one-dimensional feature vector obtained by flattening the token feature.

[0048] Patch token: token for feature processing.

[0049] Class token: token for classification.

[0050] Token Block: the main part of Token Module, used for spatial interaction and channel interaction operation respectively.

[0051] Space interaction: spatial interaction module.

[0052] Space self-awareness unit: spatial self-awareness unit.

[0053] Space fusion unit: spatial fusion unit.

[0054] Channel interaction: channel interaction module.

[0055] Expand: refers to the expansion operation.

[0056] Hadamard product operation: refers to Hadamard product, a kind of matrix operation, indicating that the corresponding elements of two matrices are multiplied.

[0057] L2 normalization operation: refers to L2 standardization operation.

[0058] GeM pooling: refers to GeM pooling method.

[0059] GeM head: feature aggregation component of Token Module, used for aggregation of complete feature map.

[0060] FC Layer: fully connected layer.

[0061] Embodiment 1:

[0062] The application designs a visual scene recognition method based on spatial self-awareness under self-attention, which can aggregate global descriptors with high robustness and visual invariance information from the extracted image basic features through a newly designed simple and efficient feature aggregation module Token Module, such as Figure 1 、 Figure 2 As shown, it is realized by using S3VPR model (as shown in Figure 1 ) mainly composed of ViT backbone and feature aggregation module Token Module, including the following steps:

[0063] 1) for the given input image q, and q∈R 3×w×h (w, h respectively represent the width and height of the input image, preferably set w×h as 224×224, also other sizes) basic features are extracted by using ViT backbone, to obtain a feature map F cls , and F cls ∈R (wh+1)×c , and then the Patch tokens vector set in the feature map F cls forms a new feature map F 2D, and F 2D ∈R wh×c , where w, h represent the width and height of the input image respectively, and c represents the dimension of the token vector, and the input image q includes the query image and the database image;

[0064] 2) The new feature map F 2D is reshaped from a two-dimensional feature map to a three-dimensional feature map F, and F w×h×c ∈R si ;

[0065] 3) The newly designed feature aggregation module Token Module is used to process the three-dimensional feature map F along the spatial dimension to obtain a spatial interaction feature map F si , and F w×h×c ∈R si , and then process the spatial interaction feature map F ci along the channel dimension to obtain a channel interaction feature map F ci , and F w×h×c ∈R 2D , so as to enhance the expression ability of the most significant features, and then further process to obtain a global descriptor z;

[0066] That is, the newly designed feature aggregation module Token Module is used to process the three-dimensional feature map F along the spatial direction and the channel direction, so as to enhance the expression ability of the most significant features, and then the robust global descriptor z used for final discrimination is obtained by processing the feature aggregation component GeM head in the Token Module.

[0067] 4) The global descriptor z of the query image and the global descriptor z of all database images in the database are compared one by one by using a similarity algorithm, and all database images belonging to the same scene as the query image are found.

[0068] Embodiment 2

[0069] This embodiment is further optimized on the basis of the above-mentioned embodiments, and the same as the foregoing technical solutions will not be repeated here. In combination with Figure 1 , Figure 2 , a better self-attention spatial self-perception visual scene recognition method is further provided, which is particularly configured as follows: the new feature map F cls is obtained by discarding the class token vector in the feature map F 2D and only taking the Patch token vector set.

[0070] Embodiment 3

[0071] The embodiment is further optimized on the basis of any of the above embodiments, and the same parts as the previous technical solutions will not be repeated here, in combination with Figure 1 , Figure 2 As shown in the above formula, further to better realize the visual scene recognition method under the self-attention and the space self-awareness, the similarity algorithm is cosine similarity (which can also be Euclidean distance or other similarity algorithms).

[0072] Embodiment 4:

[0073] The embodiment is further optimized on the basis of any of the above embodiments, and the same parts as the previous technical solutions will not be repeated here, in combination with Figure 1 , Figure 2 As shown in the above formula, further to better realize the visual scene recognition method under the self-attention and the space self-awareness, the similarity algorithm is cosine similarity (which can also be Euclidean distance or other similarity algorithms).

[0074] Embodiment 5:

[0075] The embodiment is further optimized on the basis of any of the above embodiments, and the same parts as the previous technical solutions will not be repeated here, in combination with Figure 1 , Figure 2 As shown in the above formula, further to better realize the visual scene recognition method under the self-attention and the space self-awareness, the similarity algorithm is cosine similarity (which can also be Euclidean distance or other similarity algorithms).

[0076] 3.1.1) Based on the connection of single pixel and other surrounding pixels in visual attributes, first use the Space self-awareness unit to mine the visual structure attributes inside the three-dimensional feature map F, and calculate the channel type space self-awareness feature map (containing more rich visual information features) F hp , and F hp ∈R w×h×c×p×p ; wherein p x p is the width and height size of the perception area.

[0077] 3.1.2) Then, using the Space fusion unit, the channel-wise space self-aware feature map F hp is obtained by performing a dimension reduction operation to reduce the single-pixel perception domain;

[0078] 3.1.3) Finally, after step 3.1.2), the new three-dimensional feature map F sf obtained by the dimension reduction operation is fused with the three-dimensional feature map F sf by means of "residual connection", and F w×h×c ∈R si , to obtain a space interaction feature map F si , and F w×h×c ∈R si ; the new feature F w×h×c×p×p obtained is rich in enhanced visual information while maintaining the inherent geometric structure property.

[0079] Embodiment 6:

[0080] This embodiment is further optimized on the basis of any of the above embodiments, and the same as the foregoing technical solutions will not be repeated here. In combination with the drawings, a better implementation of the self-attention space self-aware visual scene recognition method is further provided, in particular, the following setting method is adopted: Figure 1 、 Figure 2 The step 3.1.1) is specifically:

[0081] 3.1.1.1) After the three-dimensional feature map F is filled with edges by the Pad operation in the Space self-aware unit, the Expand operation is used to extract the surrounding area of each pixel with a size of p x p (p is preferably set to 3) as the center, to obtain a new feature F', and F' ∈R w×h×c×p×p ;

[0082] 3.1.1.2) For the new feature F', the Hadamard product operation is used to calculate the channel-wise space self-aware feature map F hp of each pixel and its p x p surrounding area, i.e.:

[0083] F hp = Expand(Pad(F)) ⊙ Broadcast(F), F hp ∈R w×h×c×p×p (1);

[0084] Wherein, ⊙ represents the Hadamard product operation; Broadcast is a Python operation mechanism; in the Hadamard product calculation process, Broadcast automatically expands the three-dimensional feature map F to the same shape as the new feature F', i.e. to obtain a broadcast feature F'' ∈Rw×h×c×p×p .

[0085] Embodiment 7:

[0086] This embodiment is further optimized on the basis of any of the above embodiments, and the same parts as the foregoing technical solutions will not be repeated here. In combination with Figure 1 , Figure 2 shown, further to better realize the visual scene recognition method of spatial self-perception under self-attention, especially the following setting method is adopted: the Space fusion unit re-encodes the high-dimensional channel type spatial self-perception feature map F hp into a new three-dimensional feature map F sf . The step 3.1.2) is specifically: first, the Space fusion unit performs L2 normalization operation on the channel type spatial self-perception feature map F hp , and then uses GeM pooling to reduce its dimension, so that the size of the single-pixel perception domain is reduced from p×p to 1×1.

[0087] The design idea of using "residual connection" is as follows: the new three-dimensional feature map F sf Although it contains richer image visual attribute information, it will destroy the geometric structure of the original feature when calculating spatial self-perception. Therefore, the present application selects to fuse the original feature through the method of residual connection on the basis of F sf , so as to enhance its visual attributes while recovering its geometric structure information, so as to obtain the spatial interaction feature map F si .

[0088] Embodiment 8:

[0089] This embodiment is further optimized on the basis of any of the above embodiments, and the same parts as the foregoing technical solutions will not be repeated here. In combination with Figure 1 , Figure 2 shown, further to better realize the visual scene recognition method of spatial self-perception under self-attention, especially the following setting method is adopted: in the main part Token Block, the present application uses the Space interaction module to make S3VPR capture the visual attribute information of image features in the spatial dimension, and then uses the Channel interaction module to further develop information interaction in the channel dimension. The Channel interaction module mainly consists of two fully connected layers with activation functions in the middle, and the processing process of the new feature F si in the channel dimension includes:

[0090] 3.2.1) for the input spatial interaction feature map Fsi First, use the first fully connected layer ( Figure 2 Encode it using Linear) and then use F si The number of channels is reduced to 1 / r of the original number;

[0091] 3.2.2) Then, the ReLU activation function is used to enhance the nonlinear representation of the encoded features;

[0092] 3.2.3) Finally, the decoding operation is performed using a second fully connected layer, while restoring the dimensions of the channel features;

[0093] The Channel Interaction module analyzes the spatial interaction feature map F along the channel dimension. si The following formula is used for processing:

[0094] F ci =W2σ(W1F si +b1)+b2,F ci ∈R w×h×c (2);

[0095] In the formula, F ci Here, σ represents the channel interaction feature map; W1,W2 and b1,b2 are the weight parameters and inductive bias of the two fully connected layers, respectively. r is a scaling factor (variable, preferably set to 0.75).

[0096] Example 9:

[0097] This embodiment is a further optimization based on any of the above embodiments. The similarities with the aforementioned technical solutions will not be repeated here. Figure 1 , Figure 2 As shown, to further better realize the visual scene recognition method with spatial self-awareness under self-attention described in this invention, the following configuration is specifically adopted: the feature aggregation component GeM head includes a channel interaction feature map F for input. ci Pooling layers for dimensionality reduction and fully connected layers for transforming the dimensionality-reduced feature representations to obtain the global descriptor z, where z = GeMHead(F ci ) = WGeM(F ci )+b,z∈R d Where W and b are the weight parameters and inductive bias of the fully connected layer, respectively; W∈R c×d d is the vector dimension of the global descriptor z (variable, preferably set to 4096).

[0098] Example 10:

[0099] A self-attention-based visual scene recognition method with spatial self-awareness utilizes a newly designed, simple, and efficient feature aggregation module, the Token Module, to aggregate highly robust global descriptors with visual invariance information from extracted basic image features. Figure 1 , Figure 2 As shown, the S3VPR model, mainly composed of the ViT backbone and the feature aggregation module Token Module, is adopted (e.g., Figure 1 The implementation (as shown) includes the following steps:

[0100] 1) For a given input image q, and q∈R 3×w×h (w and h represent the width and height of the input image, respectively. Ideally, w×h should be 224×224, but other sizes are also acceptable.) Basic features are extracted using the ViT backbone to obtain a feature map F composed of wh+1 token vectors. cls And F cls ∈R (wh+1)×c Then discard the feature map F. cls From the class token vectors, only the Patch tokens vector set is used to form a new feature map F. 2D And F 2D ∈R wh×c Where w and h represent the width and height of the input image, respectively, c represents the dimension of the token vector, and the input image q includes the query image and the database image.

[0101] 2) For the new feature map F 2D Perform a reshape operation to transform the 2D feature map into a 3D feature map F, where F ∈ R. w×h×c .

[0102] 3) Using the newly designed feature aggregation module Token Module, the 3D feature map F is first processed along the spatial dimension to obtain the spatial interaction feature map F. si And F si ∈R w×h×c Then, from the channel dimension, the spatial interaction feature map F si The channel interaction feature map F is obtained through processing. ci And F ci ∈R w×h×c To enhance the expressive power of its most salient features, further processing is then performed to obtain the global descriptor z;

[0103] The Token Module includes a Token Block and a GeMhead. The Token Block includes a Space interaction module and a Channel interaction module. The Space interaction module enhances the expression ability of the salient features of the feature map in the spatial dimension. The Channel interaction module enhances the expression ability of the salient features of the feature map in the channel dimension.

[0104] The Space interaction module includes a Space self-awareness unit and a Space fusion unit. The process of processing the three-dimensional feature map F in the spatial dimension by the Space interaction module includes:

[0105] 3.1.1) Based on the connection between a single pixel and other surrounding pixels in the visual attribute, the Space self-awareness unit is first used to mine the visual structure attribute inside the three-dimensional feature map F, and a channel-type spatial self-awareness feature map (containing richer visual information) F is calculated. hp hp ∈R w×h×c×p×p ; wherein p x p is the width and height size of the perception area; and specifically includes:

[0106] 3.1.1.1) After the Space self-awareness unit fills the edges of the three-dimensional feature map F through the Pad operation, the Expand operation is used to extract the surrounding area of each pixel as the center with a size of p x p (p is preferably set to 3) to obtain a new feature F', and F' ∈R w×h×c×p×p .

[0107] 3.1.1.2) For the new feature F', the Hadamard product operation is used to calculate the channel-type spatial self-awareness feature map F of each pixel and its p x p surrounding area. hp , that is:

[0108] F hp = Expand(Pad(F)) ⊙ Broadcast(F), F hp ∈R w×h×c×p×p (1);

[0109] ​Here, ⊙ represents the Hadamard product operation; Broadcast is a Python computation mechanism; during the Hadamard product calculation, Broadcast automatically expands the 3D feature map F to the same shape as the new feature F′, thus obtaining the broadcast feature F″∈R. w×h×c×p×p .

[0110] 3.1.2) Then, the Space Fusion unit is used to process the channel-type spatial self-sensing feature map F. hp Dimensionality reduction is performed to shrink the single-pixel receptive field. Specifically, the Space fusion unit first processes the channel-type spatial self-sensing feature map F. hp L2 normalization is performed, and then GeM pooling is used to reduce the dimensionality of the receptive field of a single pixel from p×p to 1×1.

[0111] 3.1.3) Finally, after step 3.1.2), the new dimensionality-reduced 3D feature map F is connected via "residual connection". sf It is fused with the 3D feature map F, and F sf ∈R w×h×c The spatial interaction feature map F is obtained. si And F si ∈R w×h×c ; This results in the new feature F si While incorporating enhanced visual information, it retains its inherent geometric structural properties.

[0112] The design concept using "residual connectivity" is as follows: New 3D feature map F sf Although it contains richer visual attribute information, it can disrupt the geometric structure of the original features during spatial self-perception calculation. Therefore, this invention chooses to perform calculations on F... sf Based on this, the original features are fused using residual connections to enhance their visual attributes while restoring their geometric structure information, thereby obtaining the spatial interaction feature map F. si .

[0113] In the main Token Block, this invention utilizes a Space interaction module to enable S3VPR to capture the visual attribute information of image features in the spatial dimension, and then uses a Channel interaction module to further conduct information interaction in the channel dimension. The Channel interaction module mainly consists of two fully connected layers with activation functions in between, and it applies new features F in the channel dimension. si The processing steps include:

[0114] 3.2.1) For the input spatial interaction feature map F si , first encode it using the first fully connected layer (Linear in the above code) and reduce the channel number of F si to 1 / r of the original; Figure 2

[0115] 3.2.2) Then, use the ReLU activation function to enhance the non-linear representation of the encoded features;

[0116] 3.2.3) Finally, use the second fully connected layer to perform the decoding operation while restoring the dimension of the channel features;

[0117] The Channel interaction module uses the following formula when processing the spatial interaction feature map F si in the channel dimension:

[0118] F ci = W2σ(W1F si + b1) + b2, F ci ∈ R w×h×c (2) ;

[0119] where F ci is the channel interaction feature map, σ represents the ReLU activation function; W1, W2 and b1, b2 are the weight parameters and the inductive bias of the two fully connected layers, respectively; r is the scaling factor (variable, preferably set to 0.75).

[0120] The feature aggregation component GeM head includes a pooling layer for reducing the dimension of the input channel interaction feature map F ci , and a fully connected layer for transforming the reduced feature representation to obtain the global descriptor z, and z = GeMHead (F ci ) = WGeM (F ci ) + b, z ∈ R d , where W, b are the weight parameters and the inductive bias of the fully connected layer, respectively; W ∈ R c×d , d is the vector dimension of the global descriptor z (variable, preferably set to 4096).

[0121] 4) Use similarity algorithms (cosine similarity, Euclidean distance, etc.) to compare the global descriptor z of the query image with the global descriptor z of all database images in the database one by one, and find all database images belonging to the same scene as the query image.

[0122] ​The above is only the preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification or equivalent change of the above embodiment according to the technical essence of the present application falls within the protection scope of the present application.

Claims

1. A visual scene recognition method based on spatial self-awareness under self-attention, characterized in that: The S3VPR model, mainly composed of ViTbackbone and the feature aggregation module TokenModule, is adopted and includes the following steps: 1) For a given input image q, use the ViT backbone to extract basic features, resulting in a feature map F composed of wh+1 token vectors. cls And F cls ∈R (wh+1)×c Then a new feature map F is formed. 2D And F 2D ∈R wh×c Where w and h represent the width and height of the input image, respectively, c represents the dimension of the token vector, and the input image q includes the query image and the database image; 2) For the new feature map F 2D Perform a reshape operation to transform the 2D feature map into a 3D feature map F, where F ∈ R. w ×h×c ; 3) The feature aggregation module TokenModule is used to process the 3D feature map F along the spatial dimension to obtain the spatial interaction feature map F. si And F si ∈R w×h×c Then, from the channel dimension, the spatial interaction feature map F si The channel interaction feature map F is obtained through processing. ci And F ci ∈R w×h×c Then, further processing is performed to obtain the global descriptor z; the feature aggregation module TokenModule includes a main part Token Block and a feature aggregation component GeM head. The main part TokenBlock includes a Space interaction module that enhances the expressive power of the salient features of the feature map from the spatial dimension and a Channel interaction module that enhances the expressive power of the salient features of the feature map from the channel dimension. The Space interaction module includes a Space self-awareness unit and a Space fusion unit, and the process by which the Space interaction module processes the three-dimensional feature map F in the spatial dimension includes: 3.1.1) Use Space self-awareness units to mine the visual structural properties within the 3D feature map F, and calculate the channel-type spatial self-awareness feature map F. hp And F hp ∈R w×h×c×p×p Where p×p represents the width and height dimensions of the sensing region; 3.1.2) Using Space Fusion units to process channel-type spatial self-sensing feature maps F hp Perform dimensionality reduction; 3.1.3) After step 3.1.2), the new 3D feature map F after dimensionality reduction is connected using the "residual connection" method. sf It is fused with the 3D feature map F, and F sf ∈R w×h×c The spatial interaction feature map F is obtained. si And F si ∈R w×h×c ; The Channel interaction module mainly consists of two fully connected layers with activation functions in between, and it also interacts with the spatial interaction feature map F in the channel dimension. si The processing steps include: 3.2.1) For the input spatial interaction feature map F si Encode it using the first fully connected layer, and then use F si The number of channels is reduced to 1 / r of the original number; 3.2.2) Use the ReLU activation function to enhance the nonlinear representation of encoded features; 3.2.3) Use a second fully connected layer to perform the decoding operation, while restoring the dimensions of the channel features; The Channel Interaction module analyzes the spatial interaction feature map F along the channel dimension. si The following formula is used for processing: F ci =W2σ(W1F si +b1)+b2,F ci ∈R w×h×c (2); In the formula, F ci Here, σ represents the channel interaction feature map; W1,W2 and b1,b2 are the weight parameters and inductive bias of the two fully connected layers, respectively. r is the scaling factor; 4) Use a similarity algorithm to compare the global descriptor z of the query image with the global descriptors z of all database images in the database to find all database images that belong to the same scene as the query image.

2. The visual scene recognition method under self-attention spatial self-perception according to claim 1, characterized in that: The new feature map F 2D To discard feature map F cls The class token vector is obtained by taking only the Patch tokens vector set.

3. A visual scene recognition method based on spatial self-awareness under self-attention as described in claim 1 or 2, characterized in that: The similarity algorithm is cosine similarity.

4. The visual scene recognition method under self-attention spatial self-perception according to claim 1, characterized in that: Step 3.1.1) specifically refers to: 3.1.1.1) The Space self-awareness unit fills the edges of the 3D feature map F using the Pad operation, and then uses the Expand operation to extract the surrounding region of size p×p centered on each pixel channel by channel to obtain a new feature F′, where F′∈R. w×h×c×p×p ; 3.1.1.2) For the new feature F′, the Hadamard product operation is used again to calculate the channel-based spatial self-aware feature map F of each pixel and its surrounding p×p region. hp ,Right now: F hp =Expand(Pad(F))⊙Broadcast(F),F hp ∈R w×h×c×p×p (1); Here, ⊙ represents the Hadamard product operation; Broadcast is a Python computation mechanism; during the Hadamard product calculation, Broadcast automatically expands the 3D feature map F to the same shape as the new feature F′, thus obtaining the broadcast feature F″∈R. w×h×c×p×p .

5. The visual scene recognition method under self-attention spatial self-perception according to claim 1, characterized in that: Step 3.1.2) specifically involves: the Space fusion unit processing the channel-type spatial self-sensing feature map F. hp Perform L2 normalization, and then use GeMpooling to reduce the dimensionality of the receptive field of a single pixel from p×p to 1×1.

6. The visual scene recognition method under self-attention spatial self-perception according to claim 1, characterized in that: The feature aggregation component GeM head includes F for input. ci Pooling layers for dimensionality reduction and fully connected layers for transforming the dimensionality-reduced feature representations to obtain the global descriptor z, where z = GeMHead(F ci ) = WGeM(F ci )+b,z∈R d Where W and b are the weight parameters and inductive bias of the fully connected layer, respectively; W∈R c×d d is the vector dimension of the global descriptor z.

Images