A visual scene recognition method of domain-invariant information mining

Abstract

The application relates to the technical field of robot autonomous navigation and visual identification, and discloses a visual scene identification method based on domain-invariant information mining, which is realized by adopting a DIMNet model composed of a main network VGG-DIMM and an aggregation head NetVLAD and comprises the following steps: for a given input image, local feature descriptors of the input image are preliminarily extracted by using the VGG-DIMM; the local feature descriptors extracted by the VGG-DIMM are taken as input, and the original NetVLAD is used to aggregate the local feature descriptors into one-dimensional global feature descriptors F (I) ; the global feature descriptors F (I) of the query image and the global feature descriptors F (I) of all database images are compared one by one through a similarity algorithm, so that all database images belonging to the same scene as the query image are found; through the domain-invariant information mining module (DIMM) adopted, the interdependence between different spatial regions of feature maps is taken into account in the cascaded convolution unit group, so that the purpose of enhancing the attention of the DIMNet model to the domain-invariant static class object is achieved.

Description

Technical Field

[0001] This invention relates to the field of robot autonomous navigation and visual recognition technology, specifically, a visual scene recognition method based on domain-invariant information mining. Background Technology

[0002] Visual Place Recognition (VPR) is an indispensable key technology for smart city construction and national defense security. As a downstream task in image retrieval, it plays a crucial role in the navigation and positioning systems of robots and autonomous vehicles. The basic process of a VPR system is as follows: given a query image, it retrieves the most similar image from an existing geospatial reference image database and uses the retrieved image as the estimated geospatial location of the query image. However, due to the influence of different viewpoints and environments (such as changes in lighting and seasons), images of the same location exhibit significant stylistic differences, posing a considerable challenge to VPR tasks.

[0003] Therefore, extracting robust descriptors from scene images that can cope with complex environmental changes has become a key approach to improving the performance of VPR algorithms. However, previous algorithms emphasized the extraction and integration of general image features while neglecting the mining of salient features that play a crucial role in the VPR task. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies and provide a visual scene recognition method based on domain-invariant information mining. By employing a Domain-invariant Information Mining Module (DIMM) incorporating the interdependencies between different spatial regions of the feature map into cascaded convolutional unit groups, the DIMNet model achieves enhanced attention to domain-invariant static objects.

[0005] This invention is achieved through the following technical solution: a visual scene recognition method based on domain-invariant information mining, which is implemented using a DIMNet model composed of a backbone network VGG-DIMM and an aggregation head NetVLAD, and includes the following steps:

[0006] 1) For a given input image I, the local feature descriptors are initially extracted using the backbone network VGG-DIMM. and Where H represents height, W represents width, C represents number of channels, R represents a real number, and the input image I includes the query image and the database image;

[0007] 2) Local feature descriptors extracted from the backbone network VGG-DIMM As input, using the original aggregation header NetVLAD Aggregates into a one-dimensional global feature descriptor F (I) And F (I) ∈R D Where D = cluster_num × C, cluster_num is the number of aggregated clusters in NetVLAD (an optional parameter, preferably set to 64), and C is... The number of channels is expressed as follows:

[0008]

[0009] 3) Using a similarity algorithm, the global feature descriptor F of the query image is... (I) Global feature descriptor F for all database images (I) By comparing each image individually, all database images belonging to the same scene as the queried image can be identified.

[0010] To further improve the visual scene recognition method for mining domain-invariant information as described in this invention, the following setting is adopted: the similarity algorithm is any one of cosine similarity, Euclidean distance, etc.

[0011] To further improve the visual scene recognition method based on domain-invariant information mining as described in this invention, the following configuration is specifically adopted: Step 1) includes the following specific steps:

[0012] 1.1) Input image I is processed by VGG16 of the backbone network VGG-DIMM to obtain the basic feature map.

[0013] 1.2) After step 1.1), the DIMM of the backbone network VGG-DIMM first increases the capacity of feature channel information through a 3×3 convolution operation with a stride of 1, thus obtaining a new feature representation. Where r is the feature expansion rate (an optional parameter, preferably set to 4);

[0014] 1.3) In order to improve In DIMM, the capacity and richness of spatial information are enhanced by using a sliding window with k = 3 × 3 and stride = 1 in the new feature representation. The operation traverses along the width and height directions (defined as an Expand operation). Specifically, during this traversal, no convolution is performed; instead, the local receptive field region selected by the sliding window is copied and rearranged according to its original spatial structure to obtain the feature map. and The expression is as follows:

[0015]

[0016] In the formula, i and j represent the new feature representations, respectively. The pixel indices in the height and width dimensions are i = 1, 2, ..., H, j = 1, 2, ..., W; α×k+β is controlled by the size of the sliding window and is a parameter representing the unfolding of the local receptive field into a column. The index on the last dimension, where α = 0, 1, ..., k-1, β = 1, 2, ..., k;

[0017] 1.4) The attention mask output by the Weight-Block module of DIMM. (and (Same shape) Feedback to feature map In the middle, feature maps are mined. By extracting important domain-invariant information, we obtain feature representations with strong robustness and strong generalization ability. Right now:

[0018]

[0019] In the formula, This represents the attention mask output by the weight generation module, i.e., the weight descriptor; The symbol indicates element-wise multiplication;

[0020] 1.5) Steps 1.2) and 1.3) enrich the detailed information of the feature map from both channel and spatial dimensions, respectively. To obtain the correlation of spatial information in the feature map, this invention expresses the feature... In the last dimension k 2 The arithmetic addition operation is performed to fuse information from different local receptive fields, resulting in...

[0021]

[0022] 1.6) In addition, for the purpose of interactive feature map channel information, Further convolution operations with k = 3 × 3 and stride = 1 (k and stride are optional parameters, and can be set to other parameter values) yield the local feature descriptors of the final output of the backbone network VGG-DIMM.

[0023] To further improve the visual scene recognition method based on domain-invariant information mining as described in this invention, the following setting is specifically adopted: the attention mask The attention map attmap is obtained by processing the input attention map attmap into the weight generation module through a cascaded group of convolutional units. The attention map attmap is an initial tensor of all 1s with an initial shape of H×W×4.

[0024] To further improve the visual scene recognition method based on domain-invariant information mining as described in this invention, the following setting is specifically adopted: the attention mask The definition of is:

[0025]

[0026] To further improve the visual scene recognition method based on domain-invariant information mining as described in this invention, the following configuration is specifically adopted: the weight generation module includes N convolutional units, each of which consists of a convolutional layer Conv, a layer normalization layer LN, and an activation function GELU; wherein, the activation function GELU is defined as: In the formula, x represents the feature tensor that needs to be processed by GELU.

[0027] To further improve the visual scene recognition method for mining domain-invariant information as described in this invention, the following configuration is adopted: the backbone network VGG-DIMM is composed of Stage 1-Stage 5 of VGG16 and DIMM connected in series, and the Stage 5 part does not contain the last ReLU layer and subsequent layers in the original Stage 5.

[0028] The VGG portion of the backbone network VGG-DIMM can also be replaced with other existing network structures, including but not limited to CNN, Transformer, or MLP types; the aggregation head NetVLAD can also be replaced with other aggregation heads, including but not limited to GeM Head, MixVPR, etc.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] (1) The DIMNet model used in this invention can effectively mine domain-invariant information in scene images and enhance the model’s ability to distinguish between positive and negative samples. In terms of performance, it surpasses the DW-T and NocPlace algorithms proposed in 2024, and has reached the best level to date, especially on the TokyoTM benchmark.

[0031] (2) The DIMNet model proposed in this invention is a one-stage network with lightweight structure and low complexity, and can be trained end-to-end.

[0032] (3) The DIMM module designed in this invention is "plug and play" and has good transferability. It can be combined with other arbitrary models (including but not limited to CNN, Transformer or MLP networks) and play a role. Attached Figure Description

[0033] Figure 1 This is a diagram of the overall framework of DIMNet.

[0034] Figure 2 This is a structural diagram of a DIMM.

[0035] Figure 3 This is a schematic diagram of the Expand operation. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to embodiments, but the implementation of the present invention is not limited thereto.

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0039] Definitions:

[0040] DIMNet model: an abbreviation for Domain-invariant Information Mining Network model.

[0041] VGG-DIMM: This refers to the backbone network consisting of the VGG16 network (Stage 1 to Stage 5, excluding the last ReLU layer and subsequent layers in the original Stage 5) and the DIMM in series.

[0042] DIMM: Abbreviation for Domain-invariant Information Mining Module.

[0043] Conv: Convolutional layer.

[0044] Expand: to extend.

[0045] Sum: Add corresponding elements together.

[0046] NetVLAD: Aggregator Header, is a publicly available work.

[0047] Weight-Block: Weight generation module.

[0048] Attention Map: Attention map and attention map attmap refer to the same object.

[0049] First Layer: The first layer.

[0050] Middle Layer: The middle layer.

[0051] Last Layer: The final layer.

[0052] Example 1:

[0053] This invention designs a visual scene recognition method based on domain-invariant information mining. The method utilizes a Domain-invariant Information Mining Module (DIMM) to incorporate the interdependencies between different spatial regions of feature maps within cascaded convolutional units. This enhances the DIMNet model's focus on domain-invariant static objects. The DIMNet model is implemented using a backbone network VGG-DIMM and an aggregation head NetVLAD, and includes the following steps:

[0054] 1) For a given input image I, the local feature descriptors are initially extracted using the backbone network VGG-DIMM. and Where H represents height, W represents width, C represents number of channels, R represents a real number, and the input image I includes the query image and the database image;

[0055] 2) Local feature descriptors extracted from the backbone network VGG-DIMM As input, using the original aggregation header NetVLAD Aggregates into a one-dimensional global feature descriptor F (I) And F (I) ∈R D Where D = cluster_num × C, cluster_num is the number of aggregated clusters in NetVLAD (an optional parameter, preferably set to 64), and C is... The number of channels is expressed as follows:

[0056]

[0057] 3) Using a similarity algorithm, the global feature descriptor F of the query image is... (I) Global feature descriptor F for all database images (I) By comparing each image individually, all database images belonging to the same scene as the queried image can be identified.

[0058] Example 2:

[0059] This embodiment is a further optimization based on the above embodiment. The similarities with the aforementioned technical solutions will not be repeated here. In order to better realize the visual scene recognition method for mining domain invariant information according to the present invention, the following setting is adopted: the similarity algorithm is any one of cosine similarity, Euclidean distance, etc.

[0060] Example 3:

[0061] This embodiment is a further optimization based on any of the above embodiments. The similarities with the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the visual scene recognition method for mining domain-invariant information described in this invention, the following setting is specifically adopted: Step 1) includes the following specific steps:

[0062] 1.1) Input image I is processed by VGG16 of the backbone network VGG-DIMM to obtain the basic feature map.

[0063] 1.2) After step 1.1), the DIMM of the backbone network VGG-DIMM first increases the capacity of feature channel information through a 3×3 convolution operation with a stride of 1, thus obtaining a new feature representation. Where r is the feature expansion rate (an optional parameter, preferably set to 4);

[0064] 1.3) In order to improve In DIMM, the capacity and richness of spatial information are enhanced by using a sliding window with k = 3 × 3 and stride = 1 in the new feature representation. The operation traverses along the width and height directions (defined as an Expand operation). Specifically, during this traversal, no convolution is performed; instead, the local receptive field region selected by the sliding window is copied and rearranged according to its original spatial structure to obtain the feature map. and The expression is as follows:

[0065]

[0066] In the formula, i and j represent the new feature representations, respectively. The pixel indices in the height and width dimensions are i = 1, 2, ..., H, j = 1, 2, ..., W; α×k+β is controlled by the size of the sliding window and is a parameter representing the unfolding of the local receptive field into a column. The index on the last dimension, where α = 0, 1, ..., k-1, β = 1, 2, ..., k; the principle of the Expand operation is as follows: Figure 3 As shown.

[0067] 1.4) The attention mask output by the Weight-Block module of DIMM. (and (Same shape) Feedback to feature map In the middle, feature maps are mined. By extracting important domain-invariant information, we obtain feature representations with strong robustness and strong generalization ability. Right now:

[0068]

[0069] In the formula, This represents the attention mask output by the weight generation module, i.e., the weight descriptor; The symbol indicates element-wise multiplication;

[0070] 1.5) Steps 1.2) and 1.3) enrich the detailed information of the feature map from both channel and spatial dimensions, respectively. To obtain the correlation of spatial information in the feature map, this invention expresses the feature... In the last dimension k 2 The arithmetic addition operation is performed to fuse information from different local receptive fields, resulting in...

[0071]

[0072] 1.6) In addition, for the purpose of interactive feature map channel information, Further convolution operations with k = 3 × 3 and stride = 1 (k and stride are optional parameters, and can be set to other parameter values) yield the local feature descriptors of the final output of the backbone network VGG-DIMM.

[0073] Example 4:

[0074] This embodiment is a further optimization based on any of the above embodiments. The similarities with the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the visual scene recognition method for mining domain-invariant information described in this invention, the following setting is specifically adopted: the attention mask... The attention map attmap is obtained by processing the input attention map attmap into the weight generation module through a cascaded group of convolutional units. The attention map attmap is an initial tensor of all 1s with an initial shape of H×W×4.

[0075] Example 5:

[0076] This embodiment is a further optimization based on any of the above embodiments. The similarities with the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the visual scene recognition method for mining domain-invariant information described in this invention, the following setting is specifically adopted: the attention mask... The definition of is:

[0077]

[0078] Example 6:

[0079] This embodiment is a further optimization based on any of the above embodiments. The similarities with the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the visual scene recognition method for mining domain-invariant information described in this invention, the following configuration is specifically adopted: The weight generation module includes N convolutional units, each convolutional unit consisting of a convolutional layer Conv, a layer normalization layer LN, and an activation function GELU; wherein, the activation function GELU is defined as: In the formula, x represents the feature tensor that needs to be processed by GELU.

[0080] Example 7:

[0081] This embodiment is a further optimization based on any of the above embodiments. The similarities with the aforementioned technical solutions will not be repeated here. In order to better realize the visual scene recognition method for mining domain invariant information as described in this invention, the following setting is adopted: the backbone network VGG-DIMM is composed of Stage 1-Stage 5 of VGG16 and DIMM in series, and the Stage 5 part does not contain the last ReLU layer and subsequent layers in the original Stage 5.

[0082] The VGG portion of the backbone network VGG-DIMM can also be replaced with other existing network structures, including but not limited to CNN, Transformer, or MLP types; the aggregation head NetVLAD can also be replaced with other aggregation heads, including but not limited to GeM Head, MixVPR, etc.

[0083] Example 8:

[0084] A visual scene recognition method based on domain-invariant information mining, combined with Figures 1-3 As shown, the DIMNet model is implemented using a backbone network VGG-DIMM (primarily used for feature mining throughout the network) and an aggregation head NetVLAD, and includes the following steps:

[0085] Step 1: For a given input image I, use the backbone network VGG-DIMM to initially extract its local feature descriptors. and The expression is as follows:

[0086]

[0087] Where H represents height, W represents width, C represents number of channels, R represents a real number, and the input image I includes the query image and the database image.

[0088] The backbone network VGG-DIMM consists of VGG16 (Stage 1-Stage 5) and DIMMs connected in series, such as... Figure 1 As shown. Specifically, to avoid affecting the distribution of the processed data, Stage 5 of the VGG16 in this invention does not include the last ReLU layer and subsequent layers of its original structure. The initial design intent of this invention is: VGG16 extracts basic features from the input image, and DIMM completes the mining of salient information from these basic features.

[0089] The backbone network VGG-DIMM initially extracts local feature descriptors from the input image I. The process includes the following steps:

[0090] Step 1.1: Input image I is processed using VGG16 to extract the basic feature map.

[0091] Step 1.2: After Step 1.1, DIMM (as a plug-and-play feature enhancement module) first increases the capacity of feature channel information through a 3×3 convolution operation with a stride of 1, resulting in a new feature representation. Where r is the feature expansion rate (an optional parameter, preferably set to 4).

[0092] Step 1.3: In order to improve In DIMM, to enhance the capacity and richness of spatial information, a sliding window with k = 3 × 3 and stride = 1 is designed for new feature representation. The process iterates along the width and height directions (defined as an Expand operation); specifically, this process does not perform convolution operations, but simply copies the local receptive field region selected by the sliding window and rearranges it according to its original spatial structure to obtain the feature map. and The expression is as follows:

[0093]

[0094] In the formula, i and j represent the new feature representations, respectively. The pixel indices in the height and width dimensions are i = 1, 2, ..., H, j = 1, 2, ..., W; α×k+β is controlled by the size of the sliding window and is a parameter representing the unfolding of the local receptive field into a column. The index on the last dimension, where α = 0, 1, ..., k-1, β = 1, 2, ..., k; the principle of the Expand operation is as follows: Figure 3 As shown.

[0095] Step 1.4: Apply the attention mask output by the Weight-Block module of DIMM. (and (Same shape) Feedback to feature map In the middle, feature maps are mined. By extracting important domain-invariant information, we obtain feature representations with strong robustness and strong generalization ability. Right now:

[0096]

[0097] In the formula, This represents the attention mask output by the weight generation module, i.e., the weight descriptor; The symbol indicates that corresponding elements are multiplied.

[0098] like Figure 2 As shown, the attention map attmap (an initial tensor of shape H×W×4 filled with all 1s) is the input to the weight generation module, and it is transformed into an attention mask after being processed by a cascaded group of convolutional units. Deeper levels can be acquired through subsequent iterative training. This involves identifying the interrelationships between different spatial regions and recalibrating the salient information that needs attention. (Attention mask) It can be defined as:

[0099]

[0100] In the weight generation module, each layer captures the features of the input data. As the number of layers increases, the DIMM can learn more complex and advanced data representations. It should be noted that, except for the first and last layers which need to match the channels and dimensions of the external feature map, the intermediate layers (N convolutional units) of the weight generation module have the same structure (N is an optional parameter, set to 4 in this example). Specifically, each convolutional unit in the weight generation module consists of a convolutional layer Conv (optional, the convolutional parameters used in this example are: k = 3 × 3, stride = 1, dimension L = 4, corresponding to...). Figure 2 The algorithm consists of the symbol L), layer normalization LN, and the activation function GELU. Layer normalization outperforms batch normalization in mini-batch training and is more flexible. The activation function GELU has zero centrality, which helps alleviate the gradient vanishing problem and provides better gradient information. Its definition is:

[0101]

[0102] In the formula, x represents the feature tensor that needs to be processed by GELU.

[0103] In summary, during training, the attention map is iteratively processed by the weight generation module, allowing it to be adaptively obtained during the learning process. Weights of different local features are assigned, and the function can implicitly capture... By analyzing the semantic relationships between different spatial regions, information about buildings in an image can be extracted.

[0104] Step 1.5: Steps 1.2 and 1.3 enrich the detailed information of the feature map from both channel and spatial dimensions, respectively. To obtain the correlation of spatial information in the feature map, this invention expresses the feature... In the last dimension k 2 The arithmetic addition operation is performed to fuse information from different local receptive fields, resulting in... In addition, for the purpose of interactive feature map channel information, Further convolution operations with k = 3 × 3 and stride = 1 (k and stride are optional parameters, and can be set to other parameter values) yield the local feature descriptors of the final output of the backbone network VGG-DIMM.

[0105] Step 2: Using the local feature descriptors extracted from the backbone network VGG-DIMM As input, using the original aggregation header NetVLAD Aggregates into a one-dimensional global feature descriptor F (I) And F (I) ∈R D Where D = cluster_num × C, cluster_num is the number of aggregated clusters in NetVLAD (an optional parameter, preferably set to 64), and C is... The number of channels is expressed as follows:

[0106]

[0107] Step 3: First, perform the operations in Step 1-Step 2 on both the query image and the database images to obtain different global feature descriptors for discrimination. Then, using cosine similarity (or Euclidean distance, or other similarity calculation methods), compare the global feature descriptor of the query image with the global feature descriptors of all database images to find all database images belonging to the same scene as the query image.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A visual scene recognition method based on domain-invariant information mining, characterized in that: The DIMNet model is implemented by using a backbone network VGG-DIMM and an aggregation head NetVLAD, and includes the following steps: 1) For a given input image The local feature descriptors were initially extracted using the backbone network VGG-DIMM. ,and , The specific steps include the following: 1.1) Input image The basic feature map is obtained by extracting VGG16 from the backbone network VGG-DIMM. ; 1.2) After step 1.1), the DIMM of the backbone network VGG-DIMM first increases the capacity of feature channel information through a 3×3 convolution operation with a stride of 1, thus obtaining a new feature representation. ,in, For feature expansion rate; 1.3) In DIMM, adopt =3×3、 A sliding window of value 1 in the new feature representation The feature map is obtained by traversing along the width and height directions. During the traversal, no convolution operation is performed; instead, the local receptive field region selected by the sliding window is copied and rearranged according to its original spatial structure. ,and The expression is as follows: ； In the formula, Each represents a new feature representation. Pixel indexes in the height and width dimensions, i.e. , ; The size of the sliding window controls the parameters that expand the local receptive fields into a column, representing... The index on the last dimension, where , ; 1.4) The attention mask output by the DIMM weight generation module. Feedback to feature map In this process, feature representations with strong robustness and strong generalization ability are obtained. ,Right now: ； In the formula, This represents the attention mask output by the weight generation module, i.e., the weight descriptor; This indicates element-wise multiplication; 1.5) Expressing features In the last dimension Performing arithmetic addition on the above, we get ; 1.6) Further convolution operations yield the local feature descriptors of the final output of the backbone network VGG-DIMM. ;in, H Indicates altitude, W Indicates width, C Indicates the number of channels. R Represent real numbers, input image Includes query images and database images; 2) Local feature descriptors extracted from the backbone network VGG-DIMM As input, using the original aggregation header NetVLAD Aggregates into a one-dimensional global feature descriptor ,and ,in , It is an aggregation in NetVLAD Quantity, for The number of channels is expressed as follows: ； 3) Using a similarity algorithm, the global feature descriptor of the query image is obtained. Global feature descriptors for all database images By comparing them one by one, all database images belonging to the same scene as the queried image can be found; The backbone network VGG-DIMM is composed of Stage 1-Stage 5 of VGG16 and DIMM connected in series, and the Stage 5 part does not contain the last ReLU layer and subsequent layers in the original Stage 5.

2. The visual scene recognition method based on domain-invariant information mining according to claim 1, characterized in that: The similarity algorithm is either cosine similarity or Euclidean distance.

3. The visual scene recognition method based on domain-invariant information mining according to claim 1, characterized in that: The attention mask The attention map is input into the weight generation module. The attention map is obtained after processing by cascaded convolutional units. To initialize the shape as H × W A tensor of 4 × 1s.

4. The visual scene recognition method based on domain-invariant information mining according to claim 1, characterized in that: The attention mask The definition of is: 。 5. The visual scene recognition method based on domain-invariant information mining according to claim 1, characterized in that: The weight generation module includes N Each convolutional unit consists of a convolutional layer (Conv), a layer normalized layer (LN), and an activation function (GELU); the activation function GELU is defined as follows: In the formula, This represents the feature tensor that needs to be processed by GELU.

Images