Spatial self-sensing visual scene recognition method under self-attention

Abstract

The invention relates to the technical field of robot autonomous navigation and visual identification, and discloses a spatial self-sensing visual scene identification method under self-attention, which is realized by adopting an S3VPR model mainly composed of two parts, namely a ViTbackbone and a feature aggregation module Token Module. Comprising the following steps: 1) for a given input image, extracting basic features by using ViTbackbone to obtain a feature map, and then forming a new feature map; 2) performing respe operation on the new feature map, and converting the new feature map from a two-dimensional feature map to a three-dimensional feature map; 3) using a feature aggregation module Token Module to process the three-dimensional feature map along a spatial dimension to obtain a spatial interaction feature map, then processing along a channel dimension to obtain a channel interaction feature map, and further processing to obtain a global descriptor; and 4) comparing the global descriptor of the query image with the global descriptors of all database images in the database one by one by utilizing a similarity algorithm, and finding out all database images belonging to the same scene as the query image.

Description

Technical Field

[0001] The present invention relates to the technical field of robot autonomous navigation and visual recognition, and in particular to a visual scene recognition method of spatial self-perception under self-attention. Background Art

[0002] Visual Place Recognition (VPR) technology can provide an effective solution for the self-perception environment positioning and path planning of autonomous intelligent agents (such as drones and robots). In previous studies, scholars divided the research on VPR into two categories: single-stage retrieval and dual-stage retrieval. Single-stage retrieval combines image features into global descriptors and selects the top-k candidate images based on Euclidean distance to draw conclusions; dual-stage retrieval uses local image features and uses a re-ranking algorithm to re-rank the optimal candidate images to further clarify the conclusions.

[0003] Although two-stage retrieval can improve performance, it requires additional memory and time overhead. Many research works have shown that if the global descriptor generation algorithm of single-stage retrieval can effectively eliminate the interference of complex shooting environments and obtain highly robust global descriptors with visual invariant information, its performance will be even more efficient.

[0004] In recent years, most methods in the field of VPR tend to use models based on or involving self-attention mechanisms to extract basic features of images and use specific feature fusion algorithms to obtain highly robust descriptors from them. However, authors rarely pay attention to the highly discriminative visual attribute information of image features themselves. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide a visual scene recognition method with spatial self-perception under self-attention. Through a newly designed simple and efficient feature aggregation module Token Module, a highly robust global descriptor with visual invariant information can be aggregated from the extracted basic features of the image.

[0006] The present invention is implemented by the following technical solution: a visual scene recognition method of spatial self-perception under self-attention is implemented by using an S3VPR model mainly composed of two parts: a ViT backbone and a feature aggregation module Token Module, and includes the following steps:

[0007] 1) For a given input image q, and q∈R 3×w×h (w,h represent the width and height of the input image respectively. It is preferred to set w×h to 224×224, but other sizes are also possible.) Use ViT backbone to extract basic features and obtain a feature map F consisting of wh+1 token vectors.cls , and F cls ∈R (wh+1)×c , and then form a new feature map F 2D , and F 2D ∈R wh×c , where w, h represent the width and height of the input image respectively, c represents the dimension of the token vector, and the input image q includes the query image and the database image;

[0008] 2) For the new feature map F 2D Perform a reshape operation to convert it from a two-dimensional feature map to a three-dimensional feature map F, and F∈R w×h×c ;

[0009] 3) Use the newly designed feature aggregation module Token Module to first process the three-dimensional feature map F along the spatial dimension to obtain the spatial interaction feature map F si , and F si ∈R w×h×c , and then the spatial interaction feature map F is transformed from the channel dimension si Processing is performed to obtain the channel interaction feature map F ci , and F ci ∈R w×h×c , and then further processed to obtain the global descriptor z;

[0010] That is, the newly designed feature aggregation module Token Module is used to process the three-dimensional feature map F in the spatial direction and the channel direction in order to enhance the expression ability of its most significant features, and then the feature aggregation component GeM head in the Token Module is further used to process it to obtain a robust global descriptor z for final discrimination;

[0011] 4) Use a similarity algorithm to compare the global descriptor z of the query image with the global descriptors z of all database images in the database one by one, and find all database images that belong to the same scene as the query image.

[0012] In order to better realize the visual scene recognition method of self-attention spatial self-perception described in the present invention, the following setting is particularly adopted: the new feature map F 2D To discard the feature map F cls The class token vector in is obtained by only taking the Patch tokens vector set.

[0013] In order to better implement the method for visual scene recognition based on spatial self-perception under self-attention described in the present invention, the following setting is particularly adopted: the similarity algorithm is cosine similarity (it can also be other similarity algorithms such as Euclidean distance).

[0014] In order to better realize the visual scene recognition method of spatial self-perception under self-attention described in the present invention, the following setting is particularly adopted: the feature aggregation module Token Module includes a main part Token Block and a feature aggregation component GeM head, and the main part Token Block includes a Space interaction module that enhances the expression ability of the significant features of the feature map from the spatial dimension and a Channel interaction module that enhances the expression ability of the significant features of the feature map from the channel dimension.

[0015] In order to better realize the visual scene recognition method of space self-awareness under self-attention described in the present invention, the following setting is particularly adopted: the Space interaction module includes a Space self-awareness unit and a Space fusion unit, and the process of the Space interaction module processing the three-dimensional feature map F in the spatial dimension includes:

[0016] 3.1.1) Based on the visual attribute relationship between a single pixel and other surrounding pixels, we first use the Space self-awareness unit to mine the visual structure attributes inside the 3D feature map F and calculate the channel-type spatial self-awareness feature map (features containing richer visual information) F hp , and F hp ∈R w×h×c×p×p ; Where p×p is the width and height of the perception area;

[0017] 3.1.2) Then, the spatial fusion unit is used to transform the channel-wise spatial self-aware feature map F hp Perform dimensionality reduction operations to reduce the single-pixel perception domain;

[0018] 3.1.3) Finally, after step 3.1.2), the new three-dimensional feature map F after dimensionality reduction is connected through the “residual connection” method. sf Fusion with the three-dimensional feature map F, and F sf ∈R w×h×c , and obtain the spatial interaction feature map F si , and F si ∈R w×h×c ; The new feature F si While embodying enhanced visual information, the intrinsic geometric structure properties are maintained.

[0019] In order to better implement the visual scene recognition method of spatial self-perception under self-attention described in the present invention, the following setting is particularly adopted: the step 3.1.1) is specifically:

[0020] 3.1.1.1) The Space self-awareness unit fills the edges of the three-dimensional feature map F through the Pad operation, and then uses the Expand operation to extract the surrounding area of ​​size p×p (p is preferably set to 3) centered on each pixel channel by channel to obtain a new feature F′, and F′∈R w×h×c×p×p ;

[0021] 3.1.1.2) For the new feature F′, use the Hadamard product operation to calculate the channel-wise spatial self-perception feature map F for each pixel and its p×p surrounding area hp ,Right now:

[0022] F hp =Expand(Pad(F))⊙Broadcast(F),F hp ∈R w×h×c×p×p (1);

[0023] Among them, ⊙ represents the Hadamard product operation; Broadcast is a Python operation mechanism; during the Hadamard product calculation process, Broadcast will automatically expand the three-dimensional feature map F to the same shape as the new feature F′, that is, obtain the broadcast feature F″∈R w×h×c×p×p .

[0024] In order to better realize the method of visual scene recognition based on spatial self-awareness under self-attention described in the present invention, the following setting is particularly adopted: the space fusion unit transforms the high-dimensional channel-type spatial self-awareness feature map F containing visual structure information into hp Re-encode into a new 3D feature map F sf The specific step 3.1.2) is as follows: First, the Spacefusion unit performs channel-wise spatial self-perception feature map F hp An L2normalization operation is performed, and then GeMpooling is used to reduce its dimension, so that the size of the single-pixel receptive field is reduced from p×p to 1×1.

[0025] The design idea of ​​using "residual connection" is as follows: the new three-dimensional feature map F sf Although it contains richer image visual attribute information, it will destroy the geometric structure of the original features during spatial self-perception calculation. sf Based on the residual connection method, the original features are fused to enhance its visual attributes while restoring its geometric structure information, thereby obtaining the spatial interaction feature map F si .

[0026] In order to better realize the method of visual scene recognition under self-attention space self-perception described in the present invention, the following setting is particularly adopted: in the main part Token Block, the present invention uses the Space interaction module to enable S3VPR to capture the visual attribute information of image features in the spatial dimension, and then uses the Channel interaction module to further carry out information interaction in its channel dimension. The Channel interaction module is mainly composed of two fully connected layers with an activation function in the middle, and the new feature F is processed in the channel dimension. si The processing process includes:

[0027] 3.2.1) For the input spatial interaction feature map F si , use the first fully connected layer to encode it, and F si The number of channels is reduced to 1 / r of the original;

[0028] 3.2.2) Then, the ReLU activation function is used to enhance the nonlinear expression of the encoded features;

[0029] 3.2.3) Finally, a second fully connected layer is used to perform the decoding operation and restore the dimension of the channel features;

[0030] The channel interaction module performs spatial interaction feature map F on the channel dimension. si The following formula is used for processing:

[0031] F ci =W 2 σ(W 1 F si +b 1 )+b 2 ,F ci ∈R w×h×c (2);

[0032] In the formula, F ci is the channel interaction feature map, σ represents the ReLU activation function; W 1 ,W 2 With b 1 ,b 2 They are the weight parameters and inductive bias of the two fully connected layers; r is the scaling factor (variable, preferably set to 0.75).

[0033] In order to better realize the method of visual scene recognition based on spatial self-awareness under self-attention described in the present invention, the following configuration is particularly adopted: the feature aggregation component GeM head includes a channel interaction feature map F for the input ciThe pooling layer for dimensionality reduction and the fully connected layer for transforming the feature expression after dimensionality reduction to obtain the global descriptor z, and z = GeMHead (F ci )=WGeM(F ci )+b,z∈R d , where W, b are the weight parameters and inductive bias of the fully connected layer respectively; W∈R c×d , d is the vector dimension of the global descriptor z (variable, preferably set to 4096).

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] (1) Experiments on multiple authoritative benchmarks (including four public datasets such as Tokyo24 / 7, MSLS-val, MSLS-challenge and Pitts30k-test) show that the S3VPR method of the present invention exceeds the latest DHE-VPR (published in the top conference AAAI 2024) and ViT-R50-MSE (published in the top conference ICRA 2024) methods in 2024. At the same time, sufficient ablation experiments also show that Token Module has good transferability and can be well combined with ViT or most CNN architectures.

[0036] (2) The S3VPR model adopted in the present invention is a one-stage network with a lightweight structure, low complexity, and end-to-end training. The designed Token Module is "plug-and-play" and has good portability. It can be combined with any other model (including but not limited to CNN, Transformer or MLP) to play a role.

[0037] (3) The Token Module designed in the present invention mines the highly discriminative visual attribute information of the image features themselves from the spatial and channel dimensions. This attribute information is globally associated and retains the complete structural properties of the image, which can effectively cope with the impact of complex environmental changes on scene image matching tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the overall structure diagram of S3VPR.

[0039] Figure 2 This is the structural diagram of the Token Module. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0043] Glossary:

[0044] S3VPR model: The abbreviation of Space Self-awareness under Self-attention for Visual Place Recognition, which is a model of visual scene recognition method based on spatial self-awareness under self-attention.

[0045] ViT backbone: refers to the ViT (abbreviation for Vision Transformer) type backbone network.

[0046] Token Module: feature aggregation module.

[0047] Token vector: A token is a feature form, and a token vector refers to a feature vector that flattens the token feature into one dimension.

[0048] Patch tokens: Tokens used for feature processing.

[0049] class token: a token used for classification.

[0050] Token Block: The main part of the Token Module, used for spatial interaction and channel interaction operations respectively.

[0051] Space interaction: space interaction module.

[0052] Space self-awareness unit: Space self-awareness unit.

[0053] Space fusion unit: space fusion unit.

[0054] Channel interaction: channel interaction module.

[0055] Expand: refers to the expansion operation.

[0056] Hadamard product operation: refers to the Hadamard product, a matrix operation that represents the multiplication of corresponding elements of two matrices.

[0057] L2 normalization operation: refers to the L2 standardization operation.

[0058] GeM pooling: refers to the GeM pooling method.

[0059] GeM head: The feature aggregation component of Token Module, used for aggregation of complete feature maps.

[0060] FC Layer: fully connected layer.

[0061] Embodiment 1:

[0062] The present invention designs a visual scene recognition method based on spatial self-awareness under self-attention. Through the newly designed simple and efficient feature aggregation module Token Module, a highly robust global descriptor with visual invariant information can be aggregated from the extracted basic features of the image, such as Figure 1 , Figure 2 As shown in the figure, the S3VPR model is mainly composed of two parts: ViT backbone and feature aggregation module Token Module (as shown in Figure 1 The implementation includes the following steps:

[0063] 1) For a given input image q, and q∈R 3×w×h (w,h represent the width and height of the input image respectively. It is preferred to set w×h to 224×224, but other sizes are also possible.) Use ViT backbone to extract basic features and obtain a feature map F consisting of wh+1 token vectors. cls , and F cls ∈R (wh+1)×c , and then take the feature map F cls The Patch tokens vector set in the new feature map F 2D, and F 2D ∈R wh×c , where w, h represent the width and height of the input image respectively, c represents the dimension of the token vector, and the input image q includes the query image and the database image;

[0064] 2) For the new feature map F 2D Perform a reshape operation to convert it from a two-dimensional feature map to a three-dimensional feature map F, and F∈R w×h×c ;

[0065] 3) Use the newly designed feature aggregation module Token Module to first process the three-dimensional feature map F along the spatial dimension to obtain the spatial interaction feature map F si , and F si ∈R w×h×c , and then the spatial interaction feature map F is transformed from the channel dimension si Processing is performed to obtain the channel interaction feature map F ci , and F ci ∈R w×h×c , to enhance the expressive power of its most significant features, and then further processed to obtain the global descriptor z;

[0066] That is, the newly designed feature aggregation module Token Module is used to process the three-dimensional feature map F in the spatial direction and the channel direction in order to enhance the expression ability of its most significant features, and then the feature aggregation component GeM head in the Token Module is further used to process it to obtain a robust global descriptor z for final discrimination;

[0067] 4) Use a similarity algorithm to compare the global descriptor z of the query image with the global descriptors z of all database images in the database one by one, and find all database images that belong to the same scene as the query image.

[0068] Embodiment 2:

[0069] This embodiment is further optimized on the basis of the above embodiment, and the same points as the above technical solution are not repeated here. Figure 1 , Figure 2 As shown, in order to better realize the visual scene recognition method of self-attention space self-perception described in the present invention, the following setting is particularly adopted: the new feature map F 2D To discard the feature map F cls The class token vector in is obtained by only taking the Patch tokens vector set.

[0070] Embodiment 3:

[0071] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 , Figure 2 As shown, in order to better realize the visual scene recognition method of spatial self-perception under self-attention described in the present invention, the following setting is particularly adopted: the similarity algorithm is cosine similarity (it can also be other similarity algorithms such as Euclidean distance).

[0072] Embodiment 4:

[0073] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 , Figure 2 As shown, in order to better realize the visual scene recognition method of spatial self-perception under self-attention described in the present invention, the following setting is particularly adopted: the feature aggregation module Token Module includes a main part Token Block and a feature aggregation component GeM head, and the main part TokenBlock includes a Space interaction module that enhances the expression ability of the significant features of the feature map from the spatial dimension and a Channel interaction module that enhances the expression ability of the significant features of the feature map from the channel dimension.

[0074] Embodiment 5:

[0075] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 , Figure 2 As shown, in order to better realize the visual scene recognition method of space self-awareness under self-attention described in the present invention, the following setting is particularly adopted: the Space interaction module includes a Space self-awareness unit and a Space fusion unit, and the process of the Space interaction module processing the three-dimensional feature map F in the spatial dimension includes:

[0076] 3.1.1) Based on the visual attribute relationship between a single pixel and other surrounding pixels, we first use the Space self-awareness unit to mine the visual structure attributes inside the 3D feature map F and calculate the channel-type spatial self-awareness feature map (features containing richer visual information) F hp , and F hp ∈R w×h×c×p×p ; Where p×p is the width and height of the perception area;

[0077] 3.1.2) Then, the spatial fusion unit is used to transform the channel-wise spatial self-aware feature map F hp Perform dimensionality reduction operations to reduce the single-pixel perception domain;

[0078] 3.1.3) Finally, after step 3.1.2), the new three-dimensional feature map F after dimensionality reduction is connected through the “residual connection” method. sf Fusion with the three-dimensional feature map F, and F sf ∈R w×h×c , and obtain the spatial interaction feature map F si , and F si ∈R w×h×c ; The new feature F si While embodying enhanced visual information, the intrinsic geometric structure properties are maintained.

[0079] Embodiment 6:

[0080] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 , Figure 2 As shown, in order to better implement the visual scene recognition method of spatial self-perception under self-attention described in the present invention, the following setting method is particularly adopted: the step 3.1.1) is specifically:

[0081] 3.1.1.1) The Space self-awareness unit fills the edges of the three-dimensional feature map F through the Pad operation, and then uses the Expand operation to extract the surrounding area of ​​size p×p (p is preferably set to 3) centered on each pixel channel by channel to obtain a new feature F′, and F′∈R w×h×c×p×p ;

[0082] 3.1.1.2) For the new feature F′, use the Hadamard product operation to calculate the channel-wise spatial self-perception feature map F for each pixel and its p×p surrounding area hp ,Right now:

[0083] F hp =Expand(Pad(F))⊙Broadcast(F),F hp ∈R w×h×c×p×p (1);

[0084] Among them, ⊙ represents the Hadamard product operation; Broadcast is a Python operation mechanism; during the Hadamard product calculation process, Broadcast will automatically expand the three-dimensional feature map F to the same shape as the new feature F′, that is, obtain the broadcast feature F″∈Rw×h×c×p×p .

[0085] Embodiment 7:

[0086] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 , Figure 2 As shown, in order to better realize the visual scene recognition method of space self-perception under self-attention described in the present invention, the following setting is particularly adopted: the space fusion unit transforms the high-dimensional channel-type space self-perception feature map F containing visual structure information into hp Re-encode into a new 3D feature map F sf The specific step 3.1.2) is as follows: First, the Space fusion unit performs channel-wise spatial self-perception feature map F hp An L2 normalization operation is performed, and then GeM pooling is used to reduce its dimension, so that the size of the single-pixel receptive field is reduced from p×p to 1×1.

[0087] The design idea of ​​using "residual connection" is as follows: the new three-dimensional feature map F sf Although it contains richer image visual attribute information, it will destroy the geometric structure of the original features during spatial self-perception calculation. sf Based on the residual connection method, the original features are fused to enhance its visual attributes while restoring its geometric structure information, thereby obtaining the spatial interaction feature map F si .

[0088] Embodiment 8:

[0089] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 , Figure 2 As shown, in order to better realize the method of visual scene recognition under self-attention space self-perception described in the present invention, the following setting is particularly adopted: in the main part Token Block, the present invention uses the Space interaction module to enable S3VPR to capture the visual attribute information of image features in the spatial dimension, and then uses the Channel interaction module to further carry out information interaction in its channel dimension. The Channel interaction module is mainly composed of two fully connected layers with an activation function in the middle, and the new feature F is activated in the channel dimension. si The processing process includes:

[0090] 3.2.1) For the input spatial interaction feature map Fsi , first use the first fully connected layer ( Figure 2 Linear) in the encoding operation, and F si The number of channels is reduced to 1 / r of the original;

[0091] 3.2.2) Then, the ReLU activation function is used to enhance the nonlinear expression of the encoded features;

[0092] 3.2.3) Finally, a second fully connected layer is used to perform the decoding operation and restore the dimension of the channel features;

[0093] The channel interaction module performs spatial interaction feature map F on the channel dimension. si The following formula is used for processing:

[0094] F ci =W 2 σ(W 1 F si +b 1 )+b 2 ,F ci ∈R w×h×c (2);

[0095] In the formula, F ci is the channel interaction feature map, σ represents the ReLU activation function; W 1 ,W 2 With b 1 ,b 2 They are the weight parameters and inductive bias of the two fully connected layers; r is the scaling factor (variable, preferably set to 0.75).

[0096] Embodiment 9:

[0097] This embodiment is further optimized on the basis of any of the above embodiments, and the same points as the above technical solutions are not repeated here. Figure 1 , Figure 2 As shown, in order to better implement the visual scene recognition method of spatial self-awareness under self-attention described in the present invention, the following setting is particularly adopted: the feature aggregation component GeM head includes a channel interaction feature map F for the input ci The pooling layer for dimensionality reduction and the fully connected layer for transforming the feature expression after dimensionality reduction to obtain the global descriptor z, and z = GeMHead (F ci )=WGeM(F ci )+b,z∈R d , where W, b are the weight parameters and inductive bias of the fully connected layer respectively; W∈R c×d, d is the vector dimension of the global descriptor z (variable, preferably set to 4096).

[0098] Embodiment 10:

[0099] A spatial self-aware visual scene recognition method under self-attention can aggregate a highly robust global descriptor with visual invariant information from the extracted basic image features through a newly designed simple and efficient feature aggregation module Token Module, such as Figure 1 , Figure 2 As shown in the figure, the S3VPR model is mainly composed of two parts: ViT backbone and feature aggregation module Token Module (as shown in Figure 1 The implementation includes the following steps:

[0100] 1) For a given input image q, and q∈R 3×w×h (w,h represent the width and height of the input image respectively. It is preferred to set w×h to 224×224, but other sizes are also possible.) Use ViT backbone to extract basic features and obtain a feature map F consisting of wh+1 token vectors. cls , and F cls ∈R (wh+1)×c , and then discard the feature map F cls The class token vector in the , only the Patch tokens vector set is taken to form a new feature map F 2D , and F 2D ∈R wh×c , where w, h represent the width and height of the input image respectively, c represents the dimension of the token vector, and the input image q includes the query image and the database image.

[0101] 2) For the new feature map F 2D Perform a reshape operation to convert it from a two-dimensional feature map to a three-dimensional feature map F, and F∈R w×h×c .

[0102] 3) Use the newly designed feature aggregation module Token Module to first process the three-dimensional feature map F along the spatial dimension to obtain the spatial interaction feature map F si , and F si ∈R w×h×c , and then the spatial interaction feature map F is transformed from the channel dimension si Processing is performed to obtain the channel interaction feature map F ci , and F ci ∈R w×h×c , to enhance the expressive power of its most significant features, and then further processed to obtain the global descriptor z;

[0103] Among them, the feature aggregation module Token Module includes the main part TokenBlock and the feature aggregation component GeMhead. The main part TokenBlock sequentially includes the Space interaction module for enhancing the expression ability of the significant features of the feature map from the spatial dimension and the Channel interaction module for enhancing the expression ability of the significant features of the feature map from the channel dimension, so as to enhance the expression ability of the significant features of the feature map from the spatial dimension and the channel dimension respectively.

[0104] The Space interaction module includes a Space self-awareness unit and a Space fusion unit, and the process of processing the three-dimensional feature map F by the Space interaction module in the spatial dimension includes:

[0105] 3.1.1) Based on the visual attribute relationship between a single pixel and other surrounding pixels, we first use the Space self-awareness unit to mine the visual structure attributes inside the 3D feature map F and calculate the channel-type spatial self-awareness feature map (features containing richer visual information) F hp , and F hp ∈R w×h×c×p×p ; Where p×p is the width and height of the perception area; specifically:

[0106] 3.1.1.1) The Space self-awareness unit fills the edges of the three-dimensional feature map F through the Pad operation, and then uses the Expand operation to extract the surrounding area of ​​size p×p (p is preferably set to 3) centered on each pixel channel by channel to obtain a new feature F′, and F′∈R w×h×c×p×p ;

[0107] 3.1.1.2) For the new feature F′, use the Hadamard product operation to calculate the channel-wise spatial self-perception feature map F for each pixel and its p×p surrounding area hp ,Right now:

[0108] F hp =Expand(Pad(F))⊙Broadcast(F),F hp ∈R w×h×c×p×p (1);

[0109] Among them, ⊙ represents the Hadamard product operation; Broadcast is a Python operation mechanism; during the Hadamard product calculation process, Broadcast will automatically expand the three-dimensional feature map F to the same shape as the new feature F′, that is, obtain the broadcast feature F″∈R w×h×c×p×p .

[0110] 3.1.2) Then, the spatial fusion unit is used to transform the channel-wise spatial self-aware feature map F hp Perform dimensionality reduction operations to reduce the single pixel perception domain. Specifically, first, the Space fusion unit performs dimensionality reduction on the channel-based spatial self-perception feature map F hp An L2 normalization operation is performed, and then GeM pooling is used to reduce its dimension, so that the size of the single-pixel receptive field is reduced from p×p to 1×1.

[0111] 3.1.3) Finally, after step 3.1.2), the new three-dimensional feature map F after dimensionality reduction is connected through the “residual connection” method. sf Fusion with the three-dimensional feature map F, and F sf ∈R w×h×c , and obtain the spatial interaction feature map F si , and F si ∈R w×h×c ; The new feature F si While embodying enhanced visual information, the intrinsic geometric structure properties are maintained.

[0112] The design idea of ​​using "residual connection" is as follows: the new three-dimensional feature map F sf Although it contains richer image visual attribute information, it will destroy the geometric structure of the original features during spatial self-perception calculation. sf Based on the residual connection method, the original features are fused to enhance its visual attributes while restoring its geometric structure information, thereby obtaining the spatial interaction feature map F si .

[0113] In the main part of Token Block, the present invention uses the Space interaction module to enable S3VPR to capture the visual attribute information of image features in the spatial dimension, and then uses the Channel interaction module to further carry out information interaction in its channel dimension. The Channel interaction module is mainly composed of two fully connected layers with an activation function in the middle, and the new feature F is activated in the channel dimension. si The processing process includes:

[0114] 3.2.1) For the input spatial interaction feature map F si , first use the first fully connected layer ( Figure 2 Linear) in the encoding operation, and F si The number of channels is reduced to 1 / r of the original;

[0115] 3.2.2) Then, the ReLU activation function is used to enhance the nonlinear expression of the encoded features;

[0116] 3.2.3) Finally, a second fully connected layer is used to perform the decoding operation and restore the dimension of the channel features;

[0117] The channel interaction module performs spatial interaction feature map F on the channel dimension. si The following formula is used for processing:

[0118] F ci =W 2 σ(W 1 F si +b 1 )+b 2 ,F ci ∈R w×h×c (2);

[0119] In the formula, F ci is the channel interaction feature map, σ represents the ReLU activation function; W 1 ,W 2 With b 1 ,b 2 They are the weight parameters and inductive bias of the two fully connected layers; r is the scaling factor (variable, preferably set to 0.75).

[0120] The feature aggregation component GeM head includes a channel interaction feature map F for the input ci The pooling layer for dimensionality reduction and the fully connected layer for transforming the feature expression after dimensionality reduction to obtain the global descriptor z, and z = GeMHead (F ci )=WGeM(F ci )+b,z∈R d , where W, b are the weight parameters and inductive bias of the fully connected layer respectively; W∈R c×d , d is the vector dimension of the global descriptor z (variable, preferably set to 4096).

[0121] 4) Use similarity algorithms (cosine similarity, Euclidean distance, etc.) to compare the global descriptor z of the query image with the global descriptors z of all database images in the database one by one, and find all database images that belong to the same scene as the query image.

[0122] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for visual scene recognition based on spatial self-awareness under self-attention, characterized by: The S3VPR model, which mainly consists of two parts, ViTbackbone and feature aggregation module Token Module, is used for implementation, including the following steps: 1) For a given input image q, use the ViT backbone to extract basic features and obtain a feature map F consisting of wh+1 token vectors cls , and F cls ∈R (wh+1)×c , and then form a new feature map F 2D , and F 2D ∈R wh×c , where w, h represent the width and height of the input image respectively, c represents the dimension of the token vector, and the input image q includes the query image and the database image; 2) For the new feature map F 2D Perform a reshape operation to convert it from a two-dimensional feature map to a three-dimensional feature map F, and F∈R w ×h×c ; 3) Use the feature aggregation module Token Module to first process the three-dimensional feature map F along the spatial dimension to obtain the spatial interaction feature map F si , and F si ∈R w×h×c , and then the spatial interaction feature map F is transformed from the channel dimension si Processing is performed to obtain the channel interaction feature map F ci , and F ci ∈R w×h×c , and then further processed to obtain the global descriptor z; 4) Use a similarity algorithm to compare the global descriptor z of the query image with the global descriptors z of all database images in the database one by one, and find all database images that belong to the same scene as the query image.

2. The method for visual scene recognition based on spatial self-perception under self-attention according to claim 1, characterized in that: The new feature map F 2D To discard the feature map F cls The class token vector in is obtained by only taking the Patch tokens vector set.

3. A method for visual scene recognition based on spatial self-perception under self-attention according to claim 1 or 2, characterized in that: The similarity algorithm is cosine similarity.

4. The method for visual scene recognition based on spatial self-perception under self-attention according to claim 1, characterized in that: The feature aggregation module Token Module includes a main part Token Block and a feature aggregation component GeM head. The main part TokenBlock includes a Spaceinteraction module that enhances the expression ability of the significant features of the feature map from the spatial dimension and a Channelinteraction module that enhances the expression ability of the significant features of the feature map from the channel dimension.

5. The method of visual scene recognition based on spatial self-perception under self-attention according to claim 4, characterized in that: The Space interaction module includes a Space self-awareness unit and a Space fusion unit, and the process of the Space interaction module processing the three-dimensional feature map F in the spatial dimension includes: 3.1.1) Use the Space self-awareness unit to mine the visual structure attributes inside the 3D feature map F and calculate the channel-based spatial self-awareness feature map F hp , and F hp ∈R w×h×c×p×p ; Where p×p is the width and height of the perception area; 3.1.2) Use Space fusion unit to transform channel-wise spatial self-aware feature map F hp Perform dimensionality reduction operations; 3.1.3) After step 3.1.2), the new three-dimensional feature map F after dimensionality reduction is connected by "residual connection" sf Fusion with the three-dimensional feature map F, and F sf ∈R w×h×c , and obtain the spatial interaction feature map F si , and F si ∈R w×h×c .

6. The method of visual scene recognition based on spatial self-perception under self-attention according to claim 5, characterized in that: The step 3.1.1) is specifically as follows: 3.1.1.1) The Space self-awareness unit fills the edges of the three-dimensional feature map F through the Pad operation, and then uses the Expand operation to extract the surrounding area of ​​size p×p centered on each pixel channel by channel to obtain a new feature F′, and F′∈R w×h×c×p×p ; 3.1.1.2) For the new feature F′, use the Hadamard product operation to calculate the channel-wise spatial self-perception feature map F for each pixel and its p×p surrounding area hp ,Right now: F hp =Expand(Pad(F))⊙Broadcast(F),F hp ∈R w×h×c×p×p (1); Among them, ⊙ represents the Hadamard product operation; Broadcast is a Python operation mechanism; during the Hadamard product calculation process, Broadcast will automatically expand the three-dimensional feature map F to the same shape as the new feature F′, that is, obtain the broadcast feature F″∈R w×h×c×p×p .

7. The method of visual scene recognition based on spatial self-perception under self-attention according to claim 5, characterized in that: The step 3.1.2) is specifically as follows: the space fusion unit performs channel-based spatial self-perception feature map F hp An L2normalization operation is performed, and then GeM pooling is used to reduce its dimension, so that the size of the single-pixel receptive field is reduced from p×p to 1×1.

8. The method of visual scene recognition based on spatial self-perception under self-attention according to claim 4, characterized in that: The Channel interaction module is mainly composed of two fully connected layers with an activation function in the middle, and the spatial interaction feature map F is si The processing process includes: 3.2.1) For the input spatial interaction feature map F si , use the first fully connected layer to encode it, and F si The number of channels is reduced to 1 / r of the original; 3.2.2) Use ReLU activation function to enhance the nonlinear expression of encoding features; 3.2.3) Use the second fully connected layer to perform the decoding operation and restore the dimension of the channel features; The channel interaction module performs spatial interaction feature map F on the channel dimension. si The following formula is used for processing: F ci =W2σ(W1F si +b1)+b2,F ci ∈R w×h×c (2); In the formula, F ci is the channel interaction feature map, σ represents the ReLU activation function; W1, W2 and b1, b2 are the weight parameters and inductive bias of the two fully connected layers respectively; r is the scaling factor.

9. The method of visual scene recognition based on spatial self-perception under self-attention according to claim 4, characterized in that: The feature aggregation component GeM head includes a F ci The pooling layer for dimensionality reduction and the fully connected layer for transforming the feature expression after dimensionality reduction to obtain the global descriptor z, and z = GeMHead (F ci )=WGeM(F ci )+b,z∈R d , where W, b are the weight parameters and inductive bias of the fully connected layer respectively; W∈R c×d , d is the vector dimension of the global descriptor z.

Images