A scene recognition method based on multimodal information and global attention mechanism
By combining RGB and depth images, using a dual-channel deep neural network model with global attention mechanism, the problem of insufficient accuracy of single-modal data recognition in the existing technology is solved, and the accuracy of scene recognition and parallel computing capabilities are improved.
Patent Information
- Application Number
- CN202111592561.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-23
AI Technical Summary
Existing scene recognition technologies rely on single modal data, making it difficult to effectively identify complex spatial layouts and images with small differences within and outside the class, resulting in low recognition accuracy.
Using scene recognition method based on multimodal information and global attention mechanism, RGB images and depth images are combined, and the graph embedded network and global attention coding network are used to build an end-to-end dual-channel deep neural network model, using lateral connections and the cross entropy loss function with weights to improve feature extraction and parallel computing capabilities.
It improves the accuracy of multimodal scene recognition, effectively utilizes the complementarity of RGB and depth images, solves the problem of category imbalance, and enhances the parallel computing power and interpretability of the network.
Smart Images

Figure CN114266938B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing technology, and specifically relates to a scene recognition method based on multimodal information and a global attention mechanism. Background Art
[0002] Currently, scene recognition technology has become an important branch of computer vision, and is mainly used in image retrieval, robot navigation, intelligent video surveillance, autonomous driving, and disaster detection. With the revival of deep neural networks and the emergence of large data sets, the performance of scene recognition has been significantly improved. However, due to the large number of targets in scene images, complex spatial layout, large intra-class differences and small inter-class differences, scene recognition based solely on RGB image data is still far behind human discrimination ability.
[0003] With the rapid development of depth sensors, researchers have discovered that RGB images and depth images are highly complementary. Therefore, a large number of scene recognition methods that combine RGB images and depth images have emerged. Studies have found that multimodal scene recognition methods based on RGB images and depth images have obvious advantages over methods using single-modal data.
[0004] In recent years, researchers have found that when humans recognize things, they focus on key and useful information and pay less attention to useless information. Based on the characteristics of human visual mechanisms, researchers have proposed attention mechanism architectures. However, in the current image recognition field, most attention mechanism architectures are implemented by connecting attention mechanism modules through convolutional neural networks. Summary of the invention
[0005] In order to solve the above problems, the present invention provides a scene recognition method based on multimodal information and a global attention mechanism, which sends RGB images and depth images into the network together, further explores the relationship between the two, and makes good use of the complementarity of RGB images and depth images; in addition, the present invention proposes a global attention mechanism system that does not require a convolutional neural network through a graph embedding network and a global attention encoding network, which not only increases the feature extraction capability but also maintains the characteristics of parallel computing.
[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:
[0007] The present invention is a scene recognition method based on multimodal information and a global attention mechanism, and the scene recognition method comprises the following steps:
[0008] Step (1): Select RGB images and depth images of multiple scenes from the multimodal scene database, re-encode the depth image using three channels, pair the encoded depth image (hereinafter referred to as HHA image) with the RGB image, and then divide the paired images into training set and test set according to corresponding proportions.
[0009] Step (2): Construct an end-to-end trainable dual-channel deep neural network model (hereinafter referred to as SR-MGA) that combines the global attention mechanism and multimodal information. SR-MGA includes a graph embedding network, a global attention encoding network, a feature fusion network and a classification network. The dual-channel structure is the same, both consisting of a graph embedding network and a global attention encoding network. The paired RGB image and HHA image obtained in step (1.1) are respectively input into the graph embedding network to obtain the corresponding RGB image block sequence and HHA image block sequence. The two block sequences are then respectively input into the global attention encoding network for learning, wherein a lateral connection is added between the dual-channel global attention encoding networks, and then the learned RGB image features and HHA image features are sent to the feature fusion network for splicing to obtain fused features, and finally the features are sent to the classification network.
[0010] Step (3): The training set divided in step (1) is fed into the deep neural network for training. During training, the class imbalance problem is solved by using a weighted cross entropy loss function.
[0011] Step (4): When recognizing scene images, the paired RGB image and HHA image obtained in step (1) are input into the SR-MGA network model to obtain the predicted probability value of each scene category in the multiple scene categories corresponding to each paired multimodal image. If it is consistent with the true category, it means that the prediction is correct, and finally the classification accuracy of the scene image is obtained. The classification accuracy is the ratio of the number of correct predictions to the total number of predictions.
[0012] A further improvement of the present invention is that the three channels mentioned in step (1) respectively refer to the horizontal parallax, the ground height and the inclination angle of the local surface of the pixel and the inferred gravity direction.
[0013] A further improvement of the present invention is that the paired RGB image and HHA image are converted into corresponding 1D RGB image block sequences and 1D HHA image block sequences respectively through the graph embedding network of SR-MGA. The graph embedding network is composed of a convolutional layer. Specifically, the input 2D image is denoted as x∈R H×W×C , where H and W are the width and height of the image, respectively, and C is the number of channels of the image. The image is divided into image blocks of size P×P, and we get Image co-cutting is N = HW / P 2image blocks, N is also the length of the image block sequence. Then it is mapped to a dimension of size D through linear transformation. In order to facilitate the subsequent acquisition of image features, a learnable category vector X is added to the two-dimensional matrix class . You also need to add the position code E pos To encode the position information, finally the image block sequence z0 is obtained.
[0014] The expression of the above steps is:
[0015]
[0016] A further improvement of the present invention is that the global attention encoding network of the SR-MGA in step (2) is composed of 12 identical global attention encoding modules. The global attention encoding module is composed of two residual blocks connected in series. The first residual block is composed of a layer normalization, three fully connected layers, a self-attention mechanism and a fully connected layer. The second residual block is composed of a layer normalization, two fully connected layers, and two feature loss layers. In order to further solve the problem of network overfitting, a feature loss layer is added at the connection jump of the two residual blocks.
[0017] A further improvement of the present invention is that a lateral connection is added between the two channels of the SR-MGA in step (2), and the output of the Nth global attention encoding module in the global attention encoding network corresponding to the HHA image is added to the input of the N+1th global attention encoding module in the global attention encoding network corresponding to the RGB image.
[0018] A further improvement of the present invention is that during training in step (3), the problem of class imbalance is solved by using a weighted cross entropy loss function to increase the weight of a small number of classes and reduce the weight of a large number of classes. The weight calculation formula is:
[0019]
[0020] a is the a-th scene category, N n is the number of images of the nth scene category. The weighted cross entropy loss function is:
[0021]
[0022] The beneficial effects of the present invention are: by using three channels to encode the depth image to obtain the HHA image, further obtaining rich information similar to the grayscale image from the depth image, inputting the paired RGB image and HHA image into the image embedding network, respectively converting them into image block sequences that are conducive to the learning of the global attention encoding network, and paying attention to information from different areas at different positions through the global attention encoding network and lateral connections; and further mining the complementary relationship between the RGB image and the depth image, solving the problem of uneven category distribution through the weighted cross entropy loss function, increasing the weight of the small number of categories, and reducing the weight of the large number of categories; while abandoning the tradition of the convolutional neural network connection attention mechanism, the module improves the parallel computing capability and interpretability of the network. In general, the SR-MGA proposed by the present invention improves the accuracy of multimodal scene recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a structural flow chart of the scene recognition method of the present invention.
[0024] Figure 2 is the input data of the method of the present invention, wherein Figure 2 (a) is an RGB image, Figure 2 (b) is the depth image, Figure 2 (c) is the HHA image.
[0025] Figure 3 It is a schematic diagram of the global attention encoding network structure of the present invention. DETAILED DESCRIPTION
[0026] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.
[0027] like Figure 1 The present invention provides a scene recognition method based on multimodal information and global attention mechanism, comprising the following steps:
[0028] Step 1: Select RGB images and depth images of multiple scenes from the multimodal scene database, re-encode the depth image using three channels, where the three channels refer to the horizontal disparity, the ground height, and the inclination of the local surface of the pixel to the inferred gravity direction, pair the encoded depth image with the RGB image, and then divide the paired images into training and test sets according to the corresponding ratio. Figure 2 As shown in (a), the depth image is Figure 2 As shown in (b), the depth image is referred to as the HHA image below. Figure 2 (c)
[0029] Step 2: Construct an end-to-end trainable dual-channel deep neural network model that combines global attention mechanism and multimodal information, hereinafter referred to as SR-MGA. SR-MGA includes a graph embedding network, a global attention encoding network, a feature fusion network and a classification network. The dual-channel structure is the same, both consisting of a graph embedding network and a global attention encoding network. The paired RGB image and HHA image obtained in step 1 are respectively input into the graph embedding network to obtain the corresponding RGB image block sequence and HHA image block sequence, and then the two block sequences are respectively input into the global attention encoding network for learning, wherein a lateral connection is added between the dual-channel global attention encoding networks, and then the learned RGB image features and HHA image features are sent to the feature fusion network for splicing to obtain fused features, and finally the features are sent to the classification network.
[0030] The paired RGB images and HHA images are converted into corresponding 1D RGB image block sequences and 1D HHA image block sequences respectively through the graph embedding network of SR-MGA. The graph embedding network consists of a convolutional layer. Specifically, the input 2D image is denoted as x∈R H×W×C , where H and W are the width and height of the image, both of which are 224, and C is the number of channels of the image, which is 3. The image is divided into image blocks of size P×P, where P is 16 in this embodiment, and we get Image co-cutting is N = HW / P 2 image blocks, N is also the length of the image block sequence, which is 196. Then it is mapped to a dimension of size D through linear transformation, and D is set to 768. In order to facilitate the subsequent acquisition of image features, the learnable category vector X is added to the two-dimensional matrix class . You also need to add the position code E pos To encode the position information, finally the image block sequence z0 is obtained.
[0031] The expression of the above steps is:
[0032]
[0033] Next, the RGB image block sequence and the HHA image block sequence are respectively sent to the global attention encoding network composed of 12 identical global attention encoding modules, such as Figure 3As shown in Figure 1, the global attention encoding module consists of two residual blocks connected in series. The first residual block consists of a layer normalization, three fully connected layers, a self-attention mechanism, and a fully connected layer. The second residual block consists of a layer normalization, two fully connected layers, and two feature loss layers. In order to further solve the problem of network overfitting, a feature loss layer is added at the connection jump between the two residual blocks.
[0034] In order to better utilize the complementarity between RGB images and depth images, a lateral connection is added between the two channels of SR-MGA. The output of the Nth global attention encoding module in the global attention encoding network corresponding to the HHA image is added to the input of the N+1th global attention encoding module in the global attention encoding network corresponding to the RGB image, and N is set to 10.
[0035] Step 3: Send the training set divided in step 1 to the deep neural network for training. During training, the problem of class imbalance is solved by using a weighted cross entropy loss function.
[0036] The weight calculation formula is:
[0037]
[0038] a is the a-th scene category, N n is the number of images of the nth scene category. The weighted cross entropy loss function is:
[0039]
[0040] Step 4: When recognizing scene images, the paired RGB images and HHA images obtained in step 1 are input into the SR-MGA network model to obtain the predicted probability value of each scene category in multiple scene categories corresponding to each paired multimodal image. If it is consistent with the true category, it means that the prediction is correct. Finally, the classification accuracy of the scene image is obtained. The classification accuracy is the ratio of the number of correct predictions to the total number of predictions.
[0041] The present invention effectively utilizes the complementarity between RGB images and depth images. By performing global attention monitoring on RGB images and depth images, corresponding learnable category vectors are obtained respectively, thereby performing scene classification. While improving the feature extraction capability and maintaining the characteristics of parallel computing, the present invention also improves the recognition rate of multimodal scene recognition.
[0042] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims
1. A scene recognition method based on multimodal information and global attention mechanism, characterized by: The scene recognition method comprises the following steps: Step 1: Select RGB images and depth images of multiple scenes from the multimodal scene database, re-encode the depth image using three channels, pair the encoded depth image with the RGB image, and then divide the paired images into training set and test set according to the corresponding ratio; Step 2: Build an end-to-end trainable dual-channel deep neural network model that combines global attention mechanism and multimodal information; Step 3: Send the training set divided in step 1 to the dual-channel deep neural network in step 2 for training; Step 4: Identify scene images: Input the paired RGB image and HHA image obtained in step 1 into the dual-channel deep neural network model in step 2, and obtain the predicted probability value of each scene category in the multiple scene categories corresponding to each paired multimodal image. If it is consistent with the true category, it means that the prediction is correct, and finally the classification accuracy of the scene image is obtained, where: The dual-channel deep neural network model in step 2 includes a graph embedding network, a global attention encoding network, a feature fusion network and a classification network, wherein both channels are composed of a graph embedding network and a global attention encoding network, and the construction process of the dual-channel deep neural network model is: Step 2-1: Input the paired RGB image and depth image in step 1 into the image embedding network respectively to obtain the corresponding RGB image block sequence and depth image block sequence; Step 2-2: Input the RGB image block sequence and the depth image block sequence obtained in step 2-1 into the global attention encoding network for learning; Step 2-3: Send the learned RGB image features and depth image features to the feature fusion network for splicing to obtain fusion features, and finally send the features to the classification network; In step 2-2, a lateral connection is added between the dual-channel global attention encoding networks, specifically: the output of the Nth global attention encoding module in the global attention encoding network corresponding to the depth image is added to the input of the N+1th global attention encoding module in the global attention encoding network corresponding to the RGB image.
2. A scene recognition method based on multimodal information and global attention mechanism according to claim 1, characterized in that: The global attention encoding network in step 2 is composed of 12 identical global attention encoding modules, each of which is composed of two residual blocks connected in series. The first residual block consists of a layer normalization, three fully connected layers, a self-attention mechanism and a fully connected layer. The second residual block consists of a layer normalization, two fully connected layers, and two feature loss layers. A feature loss layer is added at the connection jump between the two residual blocks.
3. A scene recognition method based on multimodal information and global attention mechanism according to claim 1, characterized in that: When the training set divided in step 1 is sent to the dual-channel deep neural network in step 2 for training in step 3, the class imbalance problem is solved by using a weighted cross entropy loss function to increase the weight of a small number of classes and reduce the weight of a large number of classes. The weight calculation formula is: a is the a-th scene category, N n is the number of images of the nth scene category; The weighted cross entropy loss function is:
4. The scene recognition method based on multimodal information and global attention mechanism according to claim 1, characterized in that: The three channels in step 1 refer to the horizontal disparity, the ground height, and the inclination of the local surface of the pixel to the inferred gravity direction.
