Scene recognition method based on multi-modal feature and graph attention mechanism

Abstract

The application belongs to the technical field of scene recognition, and discloses a scene recognition method based on multi-modal features and graph attention mechanism. First, a double-branch feature extraction module is used to capture RGB features and depth features respectively, then a shared feature fusion module based on Transformer is designed to fully interact with the detailed information and structural information to construct robust multi-modal features, and a NetVLAD layer is used for pooling to obtain a global scene descriptor. Considering that the context information between image sequences has a good discriminative effect on scene recognition, a graph attention aggregation module based on time constraint is also designed to integrate the spatial information and temporal information of the descriptor to further refine the scene descriptor. The application can effectively solve the problems of view angle, light or seasonal change in the scene recognition task, and serve the application scenarios of unmanned driving and large-scale visual positioning.

Description

Technical Field

[0001] This invention relates to the field of scene recognition technology, and in particular to a scene recognition method based on multimodal features and graph attention mechanism. Background Technology

[0002] Visual scene recognition refers to determining whether a location has been visited before based on the content contained in a scene image. It plays a crucial role in autonomous driving and robot navigation, as it perceives and understands scene information in the environment, ensuring that vehicles or robots can navigate and drive accurately and safely. Visual scene recognition faces two prominent challenges: First, changes in scene appearance, primarily due to day-night cycles, seasonal changes, and lighting variations. These changes result in inconsistent feature extraction for the same scene, sometimes leading to robot recognition failures. Second, changes in perspective. When a camera captures an image from different angles, the resulting scene images of the same location will show significant deviations, greatly complicating scene recognition. Current methods typically convert visual images into visual scene features for image retrieval, but these are highly sensitive to changes in appearance such as day / night cycles, lighting, and seasonal changes, and provide only limited information. Depth images, on the other hand, possess stable structural information and are very robust to changes in appearance and perspective, but they offer less semantic information.

[0003] The paper "Seqnet: Learning descriptors for sequence-based hierarchical place recognition," by S. Garg and M. Milford, in IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 4305–4312, 2021, proposes to extract scene descriptors by aggregating sequence information from images to avoid perceptual aliasing. However, sequence-to-sequence matching requires more memory and longer matching time than image-to-image matching. The paper "Multires-netvlad: Augmenting place recognition training with low-resolution imagery," by A. Khaliq, M. Milford, and S. Garg, in IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3882–3889, 2022, proposes to use low-dimensional image pyramids to capture more detailed semantic information. While achieving high recognition accuracy, the algorithm's performance significantly degrades when the environment becomes very large (e.g., day-night cycles) and the scene changes drastically. The paper "Look no deeper: Recognizing places from opposing viewpoints under varying scene appearance using single-view depthestimation," by S. Garg, M. Babu, T. Dharmairi, S. Hausler, N. Suenderhauf, S. Kumar, T. Drummond, and M. Milford, in the 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 4916–4923, proposes a two-stage retrieval strategy. First, it uses common keypoint extraction algorithms for cross-matching to obtain the first-stage search results. Second, it uses depth disparity information to filter out unreasonable keypoints for a second-stage re-ranking, yielding the final search results. This approach achieves good results under opposing viewpoints, but it is highly dependent on the first-stage results. If the first-stage search results are poor, the second-stage method does not significantly improve recognition accuracy.

[0004] Therefore, in order to obtain a robust scene descriptor, this invention considers fusing RGB image information and depth information for scene recognition. Summary of the Invention

[0005] This invention proposes a scene recognition method based on multimodal features and graph attention mechanism, which resists changes in the outer ring of the environment and camera perspective, and improves the accuracy of visual scene recognition.

[0006] The technical solution of this invention is as follows: A scene recognition method based on multimodal features and graph attention mechanism, which obtains RGB feature maps and depth feature maps through a dual-path feature extraction network; the dual-path feature extraction network is divided into an RGB branch and a depth branch; a multimodal feature fusion network fuses the RGB feature maps and depth feature maps of each layer to obtain multimodal features; in the depth branch, the RGB image is depth estimated to obtain a depth image, and the depth image is processed by a depth image feature extractor to obtain depth feature maps of each layer; in the RGB branch, the RGB image is extracted by the first layer of the RGB image feature extractor to obtain the first layer RGB feature map; the input of the remaining layers of the RGB image feature extractor is the multimodal features obtained in the multimodal feature fusion network, and the output is the corresponding layer RGB feature map; the output features of the last layer of the RGB image feature extractor are obtained by NetVLAD pooling to obtain the scene descriptor D. f The system employs triplet loss in metric learning to update the deep image feature extractor and the RGB image feature extractor; it refines the scene descriptor using time-constrained graph attention; and it obtains the refined descriptor by calculating the cosine similarity between the refined descriptor corresponding to the query image and the refined descriptor corresponding to the database image.

[0007] The RGB image feature extractor is a VGG16 network with the last pooling layer and the last fully connected layer removed; given an input RGB image I R Let the output of each layer be denoted as i = 1, 2, 3, 4, representing the RGB feature map of the i-th layer.

[0008] The depth image feature extractor is the first four layers of a ResNet18 network; given an input depth image I D The output is i = 1, 2, 3, 4, representing the depth feature map of the i-th layer.

[0009] The multimodal feature fusion network is a Transformer-based multimodal feature fusion network, which includes a noise filtering module and a multimodal feature fusion module;

[0010] Deep feature map The input is fed into the noise filtering module, where it undergoes max pooling, average pooling, and min pooling operations to obtain F. DMax F DAvg F DMin After F DMax FDAvg Subtract F element by element DMin The results are then concatenated along the channel dimension, and weight factors are obtained through normalization. Finally, a deep feature map is generated. Multiplying the result by the weight factor matrix yields the filtered, clean depth feature map; the calculation formula is expressed as:

[0011]

[0012] Where σ represents the sigmoid activation function and Conv represents a 7×7 convolution; the clean depth feature map and RGB feature map are input into the multimodal feature fusion module; following the Patch Embedding operation process in the standard Transformer, the input clean depth feature map and RGB feature map are segmented into non-overlapping small blocks, projected onto the C′ dimension, and positional encoding is added to obtain RGB feature sequence vectors respectively. Pure deep feature sequence vector Where C′=P 2 ×C, N=HW / P, where N represents the number of small blocks and P represents the size of the small blocks; given RGB feature sequence vector A uniform grid point p is generated by downsampling and used as a reference point. Where N G = N / r, where r represents the downsampling factor; the reference point values ​​are linearly projected onto a one-dimensional coordinate system and normalized to the range (-1, +1), where -1 represents the leftmost point and +1 represents the rightmost point; the query vector Q is obtained through linear calculation. R =Z R ×W q ,in It is the projection matrix; Q R The input is fed into the offset network Offset, which consists of depthwise convolution, GELU activation function, and 1×1 convolution, to obtain a one-dimensional offset Δp = Offset(Q). R );

[0013] The deformable key-value vector and attribute-value vector can then be expressed as:

[0014]

[0015] in, φ() represents the bilinear interpolation operation. The projection matrix is ​​represented as follows; multi-head attention with m heads is represented as:

[0016]

[0017] Where d = C′ / m represents the dimension of each head; then the effective depth features captured by the Transformer are expressed as:

[0018]

[0019] MHSA stands for Multi-head Attention, and MLP stands for Multilayer Perceptron.

[0020] Finally, the RGB features and the effective depth features are added pixel by pixel to obtain the multimodal features.

[0021] The time-constrained graph attention refinement scenario descriptor is specifically as follows:

[0022] First, construct an undirected complete graph model:

[0023]

[0024] Where V represents the set of nodes, d′ represents the node vector dimension, which is used to generate the scene descriptor D. f As nodes in a complete graph model; E represents the edges between nodes, calculated using Euclidean distance, e ij =||v i -v j ||2, representing node v i With v j Spatial relationship; T represents node v i With v j The time difference; using normalization operations to obtain soft time labels. Define an attention matrix, introduce soft time labels, and normalize it using the L1 norm, expressed by the formula:

[0025]

[0026] Where η is a hyperparameter; then, the original scene descriptor and attention matrix are used to obtain a refined descriptor through linear computation, expressed by the following formula:

[0027]

[0028] Where θ is the equilibrium parameter.

[0029] The triplet consists of an anchor point, a positive sample, and a negative sample. The objective of the triplet loss is to minimize the distance between the anchor point and the positive sample, and to maximize the distance between the anchor point and the negative sample. The calculation formula is as follows:

[0030] L = max(||X) a -X p ||2-||X a -X n||2+ω,0) (8)

[0031] Among them, X a Indicates the anchor point, X p X represents a positive sample. n ω represents the negative sample and ω represents the phase angle.

[0032] The beneficial effects of this invention are as follows: Addressing the problem of decreased accuracy in visual scene recognition methods due to changes in scene appearance and viewing angle, this invention designs a new baseline for visual scene recognition by fusing RGB and depth information to extract robust multimodal descriptors to resist unstable factors in the scene. First, a dual-branch feature extraction module captures RGB and depth features separately. Then, a Transformer-based shared feature fusion module is designed to fully interact with detail and structural information to construct robust multimodal features. Finally, NetVLAD layer pooling is used to obtain the global scene descriptor. Considering the good discriminative effect of contextual information between image sequences on scene recognition, this invention also designs a time-constrained graph attention aggregation module to integrate the spatial and temporal information of the descriptor to further refine the scene descriptor. This invention can effectively solve problems such as viewing angle, lighting, or seasonal changes in scene recognition tasks, serving applications such as autonomous driving and large-scale visual positioning. Attached Figure Description

[0033] Figure 1 The flowchart shows a scene recognition method based on multimodal features and graph attention mechanism.

[0034] Figure 2 This is a schematic diagram of a dual-path feature extraction network;

[0035] Figure 3 This is a schematic diagram of the noise filtering module;

[0036] Figure 4 This is a schematic diagram of a multimodal feature fusion module;

[0037] Figure 5(a) shows the standard graph attention node update method; Figure 5(b) shows the graph attention node update method based on time constraints. Detailed Implementation

[0038] Figure 1 This is the main flowchart of the technical solution of this invention. For example... Figure 1 As shown, the visual scene recognition algorithm based on a graph attention mechanism with multimodal features and temporal constraints proposed in this invention includes the following steps:

[0039] Step (1): Obtain RGB features and depth features using a dual-path feature extraction network:

[0040] Dual-path feature extraction network architecture such as Figure 2 As shown. For the RGB branch, a VGG16 network is used as the encoder. The VGG16 network is known for its relatively simple structure and powerful representation capabilities, extracting image features by stacking multiple convolutional and pooling layers. This invention removes the last pooling and fully connected layers of the VGG16 network, which retains the VGG16 network's ability to mine spatial information, allowing the encoder to better capture the image's content information, while also reducing the number of network parameters and improving the feature extraction speed. That is, given an input RGB image I... R Let its output be the feature map. (i = 1, 2, 3, 4) represents the RGB feature map of the i-th layer. For the depth branch, this invention selects the first four layers of the ResNet18 network as the depth encoder. The ResNet18 network is a deep convolutional neural network with residual connections, which can effectively solve the gradient vanishing problem in the training of deep networks and shows good performance when processing networks with a large number of layers. Depth images contain less information than RGB images, and are prone to gradient vanishing during gradient backpropagation in deep networks. By utilizing the first four layers of ResNet18, the depth encoder can both suppress gradient vanishing and fully extract the geometric structure information in the depth image, thereby better capturing the spatial features of the image. That is, given an input depth image I... D Its output (i = 1, 2, 3, 4) represents the depth feature map of the i-th layer.

[0041] Step (2), implementation of multimodal feature fusion network:

[0042] This invention designs a Transformer-based shared multimodal feature fusion mechanism SFFM (SharedFeature Fusion Module) to fuse RGB features and depth features, thereby extracting robust scene features. The mechanism consists of two parts: a noise filtering module and a multimodal feature fusion module.

[0043] Noise filtering module such as Figure 3 As shown. Depth feature map F is obtained by performing max pooling, average pooling, and min pooling operations respectively. DMax F DAvg F DMin After F DMax F DAvg Subtract F respectively DMin The results are concatted along the channel dimension, then normalized to obtain weight factors, and finally the original depth features are obtained. Multiplying by the weighting factor yields the filtered, clean feature map. The calculation formula is expressed as:

[0044]

[0045] Here, σ represents the sigmoid activation function, and Conv represents a 7×7 convolution. The theoretical basis for introducing minimum pooling is that if there is noise in the pixels of an image, since the convolution operation is linear, both max pooling and minimum pooling will be affected by the noise. Subtracting minimum pooling can obtain clean data at the lowest cost. On the other hand, for data without noise, subtracting minimum pooling will weaken high-response regions, allowing the network to still extract more effective spatial information from the remaining regions during subsequent training.

[0046] Shared feature fusion modules, such as Figure 4 As shown. Using RGB features and depth features after noise filtering module As input, following the PatchEmbedding operation in the standard Transformer, the input is divided into non-overlapping patches, projected onto the C′ dimension, and positional encoding is added to obtain a sequence vector. Where C′=P 2 ×C, N = HW / P, where N represents the number of patches and P represents the size of a patch. Given an expanded RGB feature map. Uniform grid points are generated through downsampling. As a reference point, where N G = N / r, where r represents the downsampling factor. The value of the reference point is linearly projected onto a one-dimensional coordinate system and normalized to the range (-1, +1), where -1 represents the leftmost point and +1 represents the rightmost point. To obtain the offset of the reference point, the unfolded RGB feature map Z... R The query vector Q is obtained through linear calculation. R =Z R ×W q ,in It is the projection matrix. Then, Q... R The output is fed into the offset network to obtain a one-dimensional offset Δp = Offset(Q). R The network incorporates a predefined factor s to prevent excessively large offsets Δp, where the offset network consists of depthwise convolutions, GELU activation functions, and 1×1 convolutions. The deformable key vectors and attribute value vectors can then be expressed as:

[0047]

[0048] in, φ() represents the bilinear interpolation operation. Let represent the projection matrix. A multi-head attention mechanism with m heads can be represented as:

[0049]

[0050] Where d = C′ / m represents the dimension of each head. Therefore, the effective depth features captured by the Transformer can be expressed as:

[0051]

[0052] Here, MHSA stands for Multi-Head Attention, and MLP stands for Multilayer Perceptron. Finally, the RGB features are pixel-wise added to the effective depth features to obtain the multimodal features. The output of the last layer of the network yields the complete multimodal features, which are then pooled using NetVLAD to obtain the scene descriptor D. f .

[0053] Step (3), implementation of time-constrained graph attention aggregation:

[0054] First, construct an undirected complete graph model:

[0055]

[0056] Where V represents the set of nodes, d′ represents the node vector dimension. This module uses the multimodal descriptor D generated above. f As nodes in a graph model, E represents the edge between nodes, calculated using Euclidean distance, i.e., e^(-1 / 2). ij =||v i -v j ||2, representing node v i With v j Spatial relationship. T represents node v. i With v j The time difference. To compare the time differences of the descriptors, this module uses a normalization operation to obtain soft time labels, i.e. Furthermore, to represent the correlation between different descriptors, this module defines an attention matrix, introduces soft time labels, and uses the L1 norm for normalization, expressed by the formula:

[0057]

[0058] In this context, η is a hyperparameter. Then, the original node descriptors and attention matrix are used to obtain refined descriptors through linear computation, expressed as:

[0059]

[0060] Where θ is the balancing parameter. As can be seen from equation (6), the introduction of soft time tags can filter out nodes with large time differences, avoiding perceptual aliasing. As can be seen from equation (7), by aggregating the feature information of neighboring nodes, the original node contains richer information and is more robust. Finally, this invention obtains the optimal retrieval result by calculating the cosine similarity between the query image and the image descriptors in the database.

[0061] This invention is achieved through Figure 1 The flowchart shown demonstrates visual positioning, and the specific implementations are achieved through... Figure 2 , Figure 3 , Figure 4 The network diagram shown in Figure 5 is used to implement a robust scene descriptor, enabling accurate scene retrieval.

[0062] To verify the algorithm's ability to achieve high localization accuracy and address various challenges in visual scene recognition tasks, this invention was tested on the Oxford RobotCar and Mapillary Street Level Sequences datasets. In the Oxford RobotCar dataset, which shows day and night variations, the algorithm achieved accuracies of 0.936, 0.980, and 0.996 on Recall@1, Recall@5, and Recall@20, respectively, demonstrating its effectiveness in handling day-night variations. In the summer scene, the algorithm achieved accuracies of 0.930, 0.968, and 0.982 on Recall@1, Recall@5, and Recall@20, respectively, indicating its effectiveness in handling shadow conditions. In the seasonally varying Mapillary Street Level Sequences dataset, the algorithm achieved localization accuracies of 0.623, 0.751, and 0.829 in the San Francisco scene, demonstrating its effectiveness in handling seasonal variations.

Claims

1. A scene recognition method based on multimodal features and graph attention mechanism, characterized in that, RGB feature maps and depth feature maps are obtained through a dual-path feature extraction network; the dual-path feature extraction network is divided into an RGB branch and a depth branch; The multimodal feature fusion network fuses the RGB feature maps and depth feature maps of each layer to obtain multimodal features. In the depth branch, the RGB image is depth estimated to obtain a depth image, and the depth image is then processed by a depth image feature extractor to obtain depth feature maps for each layer. In the RGB branch, the RGB image is extracted by the first layer of the RGB image feature extractor to obtain the first layer RGB feature map. The inputs of the remaining layers of the RGB image feature extractor are the multimodal features obtained in the multimodal feature fusion network, and the output is the corresponding layer RGB feature map. The output features of the last layer of the RGB image feature extractor are pooled by NetVLAD to obtain the scene descriptor. The triplet loss in metric learning is used to update the deep image feature extractor and the RGB image feature extractor; the scene descriptor is refined based on time-constrained graph attention to obtain the refined descriptor; the best retrieval result is obtained by calculating the cosine similarity between the refined descriptor corresponding to the query image and the refined descriptor corresponding to the database image. The time-constrained graph attention refinement scenario descriptor is specifically as follows: First, construct an undirected complete graph model: （5） in, Represents a set of nodes. , Representing the node vector dimension, in the generated scene descriptor As a node in a complete graph model; The edges between nodes are represented using Euclidean distance. , representing a node and Spatial relationships; Represents a node and The time difference; using normalization operations to obtain soft time labels. Define an attention matrix, introduce soft time labels, and normalize it using the L1 norm, expressed as follows: （6） in, These are hyperparameters; subsequently, the original scene descriptor and attention matrix are used to obtain a refined descriptor through linear computation, expressed by the following formula: （7） in, It is a balance parameter.

2. The scene recognition method based on multimodal features and graph attention mechanism according to claim 1, characterized in that, The RGB image feature extractor is a VGG16 network with the last pooling layer and the last fully connected layer removed; given an input RGB image... Let the output of each layer be denoted as , indicating the first Layer RGB feature map.

3. The scene recognition method based on multimodal features and graph attention mechanism according to claim 1, characterized in that, The depth image feature extractor is the first four layers of a ResNet18 network; given an input depth image The output is , indicating the first Layer depth feature map.

4. The scene recognition method based on multimodal features and graph attention mechanism according to claim 3, characterized in that, The multimodal feature fusion network is a Transformer-based multimodal feature fusion network, which includes a noise filtering module and a multimodal feature fusion module; Deep feature map The input is fed into the noise filtering module and undergoes max pooling, average pooling, and min pooling operations respectively to obtain... , , ,after , Subtract element by element The results are then concatenated along the channel dimension, and weight factors are obtained through normalization. Finally, a deep feature map is generated. Multiplying the result by the weight factor matrix yields the filtered, clean depth feature map; the calculation formula is expressed as: （1） in, This represents the sigmoid activation function. express Convolution; the clean depth feature map and RGB feature map are input into the multimodal feature fusion module; following the Patch Embedding operation process in the standard Transformer, the input clean depth feature map and RGB feature map are segmented into non-overlapping small blocks and projected onto... Dimension, add positional encoding, and obtain RGB feature sequence vectors respectively. Pure deep feature sequence vector ,in , , Indicates the number of small pieces. Indicates the size of the small patch; given an RGB feature sequence vector. Uniform grid points are generated through downsampling. As a reference point, ,in , This represents the downsampling factor; the value of the reference point is linearly projected onto a one-dimensional coordinate system and normalized to standardize the value. Between, among This represents the leftmost point. The rightmost point is represented; the query vector is obtained through linear calculation. ,in It is the projection matrix; The input is processed by depthwise convolution, GELU activation function, Offset network composed of convolutions In the middle, we obtain the one-dimensional offset. ; The deformable key-value vector and attribute-value vector can then be expressed as: （2） in, , This represents the bilinear interpolation operation. Represents the projection matrix; has Multi-head attention is represented as follows: （3） in, Let represent the dimension of each head; then the effective depth features captured by the Transformer are represented as: （4） in, This indicates the attention of the bulls. This represents a multilayer perceptron; Finally, the RGB features and the effective depth features are added pixel by pixel to obtain the multimodal features.

5. The scene recognition method based on multimodal features and graph attention mechanism according to claim 4, characterized in that, The triplet consists of an anchor point, a positive sample, and a negative sample. The objective of the triplet loss is to minimize the distance between the anchor point and the positive sample, and to maximize the distance between the anchor point and the negative sample. The calculation formula is as follows: （8） in, Indicates the anchor point. Indicates a positive sample. Indicates a negative sample. This indicates the phase angle.

Images