RGB-D feature fusion method based on attention mechanism
By processing RGB and Depth feature maps through local and cross-modal attention mechanisms, the problems of low accuracy and efficiency in existing RGB-D fusion methods are solved, achieving more efficient information fusion and visual task completion.
Patent Information
- Application Number
- CN202411027014.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-08
AI Technical Summary
Existing RGB-D fusion methods are not accurate or time-efficient in vision tasks, and they ignore the differences in format and features between RGB and Depth image data.
An attention-based approach is adopted to perform local attention-guided channel feature selection on RGB and Depth feature maps respectively, use a cross-modal attention module to embed cross-modal feature correlations, and perform information fusion through the cross-modal attention module.
It improves the accuracy and time efficiency of RGB-D fusion tasks, makes full use of RGB and Depth image information, and has a simple and effective network structure.
Smart Images

Figure CN118918426A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to an RGB-D feature fusion method based on an attention mechanism. Background Art
[0002] Multimodal fusion has always been a research direction that has attracted much attention in the field of computer vision. Due to the limitations of hardware devices or data collection methods, the information captured by a single type of image is not enough to fully represent the imaging scene. Compared with a single image, the processed fused image has better scene representation and visual perception capabilities, which is convenient for subsequent practical visual applications, such as target detection, tracking, semantic segmentation, scene understanding, etc. RGB-D fusion is the most common image fusion, which not only obtains the color and edge perception capabilities of the RGB camera, but also obtains the distance perception capabilities of the depth camera.
[0003] Most of the existing RGB-D fusion methods simply process the two input images first, then directly splice the channels and input them into a single-branch neural network encoder or transformer for processing. However, this ignores the differences in the data formats and image features of the two images, and the accuracy is not high. Although some methods set independent branches for RGB and Depth images, the fusion module stacks many sub-modules, which is not time-efficient. Summary of the invention
[0004] The purpose of the present invention is to provide an RGB-D feature fusion method based on the attention mechanism to solve the technical problems of low accuracy and low time efficiency in completing visual tasks in existing methods.
[0005] In order to solve the above technical problems, the specific technical solution of the RGB-D feature fusion method based on the attention mechanism of the present invention is as follows:
[0006] A RGB-D feature fusion method based on an attention mechanism, characterized by comprising the following steps:
[0007] Step 1: Perform local attention-guided channel feature selection on the RGB feature map and the depth feature map respectively;
[0008] Step 2: Use the cross-modal attention module to perform cross-modal attention-guided feature correlation embedding; Step 3: Calculate the final fusion result.
[0009] Furthermore, the step 1 includes the following specific steps:
[0010] Step 1.1: For the input RGB image, use the compression and excitation network, i.e., the SE module, to perform channel feature selection;
[0011] Step 1.2: For the input depth image, the spatial pyramid attention network, namely the SPA module, is used for channel feature selection. The proposed spatial pyramid structure adaptively pools the input feature map to three different scales: fine average pooling is used to capture more feature representation and structural information, coarse average pooling aims to strike a balance between structural information and structural regularization, and 1×1 average pooling is the traditional global average pooling GAP. These three outputs are then upsampled to the same spatial resolution as the fine representation and combined using weighted summation;
[0012] Step 1.3: Determine whether the feature maps obtained in step 1.1 and step 1.2 have the same number of channels. If they are the same, add them directly. If they are different, expand the Depth branch to the same number of channels as the RGB branch through a convolution with a convolution kernel of 1 and then add them together to obtain the result of channel feature selection guided by local attention, which is used as the input of step 2.
[0013] Furthermore, the step 1.1 includes the following steps:
[0014] First, the spatial feature encoding on each channel is compressed into a global feature, which is implemented by global average pooling. The output dimension is 1×1×C. In the excitation operation, a fully connected layer FC1 is used to reduce the number of channels, followed by a ReLU activation function, and then a fully connected layer FC2 is used to increase the number of channels. Finally, a Sigmoid activation function is used, and the output dimension is also 1×1×C. Finally, the weighted operation is to multiply the channel weights, and the result dimension is W×H×C.
[0015] Furthermore, the step 1.2 includes the following steps:
[0016] Assume that the input of the feature map is x∈R W×H×C , ρ(·) represents the adaptive average pooling, ω∈R 3 is a trainable parameter, and the output of the spatial pyramid structure component is expressed as:
[0017] S=ω T [ρ fine (x)ρ coarse (x)ρ global (x)]
[0018] According to the actual training situation, the pyramid scale is 147, which can take into account both time efficiency and training effect. Since the three feature maps obtained after the maximum average pooling are of different sizes, the small feature maps must be upsampled to the most refined size, and then weighted summed, and then the excitation operation is performed. First, a fully connected layer FC1 is used to reduce the number of channels, followed by a ReLU activation function, and then a fully connected layer FC2 is used to increase the number of channels. Finally, a Sigmoid activation function is used, and the output dimension is 7×7×C. The result of the excitation operation is upsampled to H×W×C and multiplied with the original depth feature map to obtain the result of this step.
[0019] Furthermore, the step 2 comprises the following steps:
[0020] Step 2.1: Define cross-modal self-similarity;
[0021] Step 2.2: Feature embedding;
[0022] Step 2.3: Splitting and mixing of multimodal information;
[0023] Step 2.4: Representation learning in subspace.
[0024] Furthermore, the step 2.1 includes the following steps:
[0025] Assuming that RGB and Depth features are seamlessly embedded in Key and Query, the cross-modal self-similarity between pixel (i0, j0) and other pixels (i, j) is defined as:
[0026]
[0027] Among them, Kr n,i,j Represents pixel key RGB The nth eigenvalue of n,i0,j0 , Kd n,i,j , Qd n,i0,j0 Represents pixel queues RGB ,Key Depth ,Queue Depth The nth eigenvalue of .
[0028] Furthermore, the step 2.2 includes the following steps:
[0029] The cross-modal attention module CMA has three input features, namely RGB features, Depth features and the preliminary fusion features obtained in step 1. Before subsequent processing, these three input features must first be embedded into the vector space K, Q, V.
[0030]
[0031] Further, the step 2.3 includes the following steps:
[0032] Channel mixing for K and Q sequences:
[0033] KQ=cat(K RGB ,Q RGB ,K Depth ,Q Depth )
[0034] KQ=View (b,N,-1) (Trans (2,3) (View (b,N,2,channels / 2) (KQ)))
[0035] K1,Q1,K2,Q2=Split(KQ)
[0036] The obtained mixed features evenly mix the features from RGB and Depth inputs, which can effectively represent various information of the image.
[0037] Further, the step 2.4 includes the following steps:
[0038] Calculate the cross-modal similarity and embed the calculation results into the fused feature value: By calculating the dot product K1, Q1 and K2, Q2, we get two representations of subspaces W1 and W2. Then, W1 and W2 embed the information into V1 and V2 through dot products. The fused feature Fused2 is the concatenation of V1 and V2 in the channel dimension, and then a simple convolution plus normalization is performed to get the result:
[0039]
[0040] Fused2=BN(conv(cat(W1·V1,W2·V2)))
[0041] Furthermore, the step 3 comprises the following steps:
[0042] According to the result Fused1 obtained in step 1 and the result Fused2 obtained in step 2, they are directly added to obtain the final result Fused after fusion of this method. Fused2 is regarded as a correction of Fused1. The result obtained by adding the two fused results has a stronger ability to represent the image than the single representation of any result. The obtained result is further added and fused with the backbone network and used as the input of the multi-layer decoder.
[0043] The RGB-D feature fusion method based on the attention mechanism of the present invention has the following advantages: The RGB-D feature fusion method based on the attention mechanism of the present invention combines the channel feature selection guided by the dual-branch local attention, the feature correlation embedding guided by the cross-modal attention, and the module fusion result obtained by combining the above two methods, which can improve the accuracy of the RGB-D fusion task while taking into account the time efficiency, has good application prospects, and has proved its effectiveness in some tasks. The method of the present invention can make full use of the information contained in the RGB image and the depth image, and has a higher accuracy rate in completing computer vision tasks than the existing methods, and the network structure is simpler and the time efficiency is better. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 is a basic flow chart of the present invention;
[0045] Figure 2 Schematic diagram of channel feature selection guided by local attention;
[0046] Figure 3 Schematic diagram of feature correlation embedding guided by cross-modal attention. DETAILED DESCRIPTION
[0047] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of an RGB-D feature fusion method based on the attention mechanism of the present invention in conjunction with the accompanying drawings.
[0048] like Figure 1 As shown in the figure, the RGB-D feature fusion method based on the attention mechanism proposed in the present invention mainly includes two parts: channel feature selection guided by local attention and feature correlation embedding guided by cross-modal attention. First, the input RGB and Depth images are subjected to channel feature selection guided by local attention to obtain Fused1, and then Fused1 obtained in the previous step and the two original feature maps are used to perform feature correlation embedding guided by cross-modal attention to obtain the result Fused2. The results obtained in the above two steps are directly added to obtain the final result. The obtained result can also be used to correct the backbone network as the input of the subsequent decoder.
[0049] like Figure 2 As shown, an RGB-D feature fusion method based on an attention mechanism of the present invention comprises the following steps:
[0050] Step 1: Perform local attention-guided channel feature selection on the RGB feature map and the depth feature map respectively;
[0051] Step 1.1: For the input RGB image, use the compression and excitation network (SE module) to perform channel feature selection. Through this mechanism, we can learn to use global information to selectively emphasize informative features and suppress less useful features. The essence of the SE module is to set a trainable weight coefficient for each channel of the image, specifically including compression, excitation, and weighting.
[0052] In order to extract the relationship between channels, the spatial features on each channel must first be encoded (compressed) into a global feature, which is implemented using global average pooling, with an output dimension of 1×1×C. In the excitation operation, a fully connected layer FC1 is used to reduce the number of channels, followed by a ReLU activation function, and then a fully connected layer FC2 is used to increase the number of channels, and finally a Sigmoid activation function is used, and the output dimension is also 1×1×C. Finally, weighting is to multiply the channel weights, and the result dimension is W×H×C.
[0053] Step 1.2: For the input depth image, a spatial pyramid attention network (SPA module) is used for channel feature selection. The proposed spatial pyramid structure adaptively pools the input feature map to three different scales: fine average pooling is used to capture more feature representation and structural information, coarse average pooling aims to strike a balance between structural information and structural regularization, and 1×1 average pooling is the traditional global average pooling (GAP), which provides strong structural regularization. These three outputs are then upsampled to the same spatial resolution as the fine representation and combined using weighted summation. This small learnable combination enables the network to adaptively adjust the importance of these three components in each attention block. Through the spatial pyramid structure we designed, we aggregate local and global contexts in an integrated way and significantly improve the feature representation capability.
[0054] Specifically, we assume that the input feature map is x∈R W×H×C , ρ(·) represents the adaptive average pooling, ω∈R 3 As a trainable parameter, the output of the spatial pyramid structure component can be expressed as:
[0055] S=ω T [ρ fine (x)ρ coarse (x)ρ global (x)]
[0056] According to the actual training situation, the pyramid scale is 147, which can take into account both time efficiency and training effect. Since the three feature maps obtained after the maximum average pooling are of different sizes, the small feature maps should be upsampled to the most refined size and then weighted summed. Then the excitation operation is performed. First, a fully connected layer FC1 is used to reduce the number of channels, followed by a ReLU activation function, and then a fully connected layer FC2 is used to increase the number of channels. Finally, a Sigmoid activation function is used, and the output dimension is 7×7×C. The result of the excitation operation is upsampled to H×W×C and multiplied with the original deep feature map to obtain the result of this step.
[0057] Step 1.3: Determine whether the feature maps obtained in step 1.1 and step 1.2 have the same number of channels. If they are the same, add them directly. If they are different (generally, the RGB image has more channels), expand the Depth branch to the same number of channels as the RGB branch through a convolution with a convolution kernel of 1 and then add them together to obtain the result of channel feature selection guided by local attention, which is used as the input of step 2. Performing feature selection on the RGB and Depth branches separately and then adding them together can maximize the feature independence of each branch.
[0058] Step 2: Use the cross-modal attention module to perform cross-modal attention-guided feature correlation embedding; Figure 3 As shown, step 2 includes:
[0059] Step 2.1: Define cross-modal self-similarity: Assuming that RGB and Depth features are seamlessly embedded in Key and Query, the cross-modal self-similarity between pixel (i0, j0) and other pixels (i, j) can be defined as:
[0060]
[0061] Among them, Kr n,i,j Represents pixel key RGB The nth eigenvalue of n,i0,j0 , Kd n,i,j , Qd n,i0,j0 Represents pixel queues RGB ,Key Depth ,Queue Depth The nth eigenvalue of .
[0062] Step 2.2: Feature Embedding: The cross-modal attention module (CMA) has three input features, namely RGB features, Depth features and the preliminary fusion features obtained in step 1. Before subsequent processing, these three input features must be embedded into the vector space K, Q, V.
[0063]
[0064]
[0065] Step 2.3: Splitting and mixing of multimodal information. K and Q in step 2.2 only contain the features of their respective images. In order to learn features from multiple subspaces and mix the information of RGB input and Depth input to improve the generalization representation ability of the network, it is necessary to mix the channels of K and Q sequences:
[0066] KQ=cat(K RGB ,Q RGB ,K Depth ,Q Depth )
[0067] KQ=View (b,N,-1) (Trans (2,3) (View (b,N,2,channels / 2) (KQ)))
[0068] K1,Q1,K2,Q2=Split(KQ)
[0069] The obtained mixed features evenly mix the features from RGB and Depth inputs, which can effectively represent various information of the image.
[0070] Step 2.4: Representation learning in subspace: Calculate cross-modal similarity and embed the calculation results into the fused feature value. Specifically, by calculating the dot product K1, Q1 and K2, Q2, two representations of subspaces W1 and W2 are obtained. Then, W1 and W2 embed information into V1 and V2 through dot products. The fused feature Fused2 is the concatenation of V1 and V2 in the channel dimension, and then a simple convolution plus normalization is performed to obtain the result:
[0071]
[0072] Fused2=BN(conv(cat(W1·V1,W2·V2)))
[0073] Step 3: Calculate the final fusion result;
[0074] According to the result Fused1 obtained in step 1 and the result Fused2 obtained in step 2, they are directly added to obtain the final result Fused after fusion by this method. In this way, Fused2 can be regarded as a correction of Fused1. The result obtained by adding the two fused results has a stronger ability to represent the image than the single representation of any result. The obtained result can be further added and fused with the backbone network and used as the input of the multi-layer decoder.
[0075] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A RGB-D feature fusion method based on attention mechanism, characterized in that: The steps include: Step 1: Perform local attention-guided channel feature selection on the RGB feature map and the depth feature map respectively; Step 2: Use the cross-modal attention module to perform cross-modal attention-guided feature correlation embedding; Step 3: Calculate the final fusion result.
2. The RGB-D feature fusion method based on the attention mechanism according to claim 1, characterized in that: The step 1 comprises the following specific steps: Step 1.1: For the input RGB image, use the compression and excitation network, i.e., the SE module, to perform channel feature selection; Step 1.2: For the input depth image, the spatial pyramid attention network, namely the SPA module, is used for channel feature selection. The proposed spatial pyramid structure adaptively pools the input feature map to three different scales: fine average pooling is used to capture more feature representation and structural information, coarse average pooling aims to strike a balance between structural information and structural regularization, and 1×1 average pooling is the traditional global average pooling GAP. These three outputs are then upsampled to the same spatial resolution as the fine representation and combined using weighted summation; Step 1.3: Determine whether the feature maps obtained in step 1.1 and step 1.2 have the same number of channels. If they are the same, add them directly. If they are different, expand the Depth branch to the same number of channels as the RGB branch through a convolution with a convolution kernel of 1 and then add them together to obtain the result of channel feature selection guided by local attention, which is used as the input of step 2.
3. The RGB-D feature fusion method based on the attention mechanism according to claim 2, characterized in that: The step 1.1 comprises the following steps: First, the spatial feature encoding on each channel is compressed into a global feature, which is implemented by global average pooling. The output dimension is 1×1×C. In the excitation operation, a fully connected layer FC1 is used to reduce the number of channels, followed by a ReLU activation function, and then a fully connected layer FC2 is used to increase the number of channels. Finally, a Sigmoid activation function is used, and the output dimension is also 1×1×C. Finally, the weighted operation is to multiply the channel weights, and the result dimension is W×H×C.
4. The RGB-D feature fusion method based on the attention mechanism according to claim 2, characterized in that: The step 1.2 comprises the following steps: Assume that the input of the feature map is x∈R W×H×C , ρ(·) represents the adaptive average pooling, ω∈R 3 is a trainable parameter, and the output of the spatial pyramid structure component is expressed as: S=ω T [r fine (x) p coarse (x) p global (x)] According to the actual training situation, the pyramid scale is 147, which can take into account both time efficiency and training effect. Since the three feature maps obtained after the maximum average pooling are of different sizes, the small feature maps must be upsampled to the most refined size, and then weighted summed, and then the excitation operation is performed. First, a fully connected layer FC1 is used to reduce the number of channels, followed by a ReLU activation function, and then a fully connected layer FC2 is used to increase the number of channels. Finally, a Sigmoid activation function is used, and the output dimension is 7×7×C. The result of the excitation operation is upsampled to H×W×C and multiplied with the original depth feature map to obtain the result of this step.
5. The RGB-D feature fusion method based on the attention mechanism according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: Define cross-modal self-similarity; Step 2.2: Feature embedding; Step 2.3: Splitting and mixing of multimodal information; Step 2.4: Representation learning in subspace.
6. The RGB-D feature fusion method based on the attention mechanism according to claim 5, characterized in that: The step 2.1 comprises the following steps: Assuming that RGB and Depth features are seamlessly embedded in Key and Query, the cross-modal self-similarity between pixel (i0, j0) and other pixels (i, j) is defined as: Among them Kr n,i,j Represents pixel key RGB The nth eigenvalue of n,i0,j0 , Kd n,i,j , Qd n,i0,j0 Represents pixel queues RGB ,Key Depth ,Queue Depth The nth eigenvalue of .
7. The RGB-D feature fusion method based on the attention mechanism according to claim 5, characterized in that: The step 2.2 comprises the following steps: The cross-modal attention module CMA has three input features, namely RGB features, Depth features and the preliminary fusion features obtained in step 1. Before subsequent processing, these three input features must first be embedded into the vector space K, Q, V.
8. The RGB-D feature fusion method based on the attention mechanism according to claim 5, characterized in that: The step 2.3 comprises the following steps: Channel mixing for K and Q sequences: KQ=cat(K RGB ,Q RGB ,K Depth ,Q Depth ) KQ=View (b,N,-1) (Trans (2,3) (View (b,N,2,channels / 2) (KQ))) K1,Q1,K2,Q2=Split(KQ) The obtained mixed features evenly mix the features from RGB and Depth inputs, which can effectively represent various information of the image.
9. The RGB-D feature fusion method based on the attention mechanism according to claim 5, characterized in that: The step 2.4 comprises the following steps: Calculate the cross-modal similarity and embed the calculation results into the fused feature value: By calculating the dot product K1, Q1 and K2, Q2, we get two representations of subspaces W1 and W2. Then, W1 and W2 embed the information into V1 and V2 through dot products. The fused feature Fused2 is the concatenation of V1 and V2 in the channel dimension, and then a simple convolution plus normalization is performed to get the result: Fused2=BN(conv(cat(W1·V1,W2·V2))).
10. The RGB-D feature fusion method based on the attention mechanism according to claim 1, characterized in that: The step 3 comprises the following steps: According to the result Fused1 obtained in step 1 and the result Fused2 obtained in step 2, they are directly added to obtain the final result Fused after fusion of this method. Fused2 is regarded as a correction of Fused1. The result obtained by adding the two fused results has a stronger ability to represent the image than the single representation of any result. The obtained result is further added and fused with the backbone network and used as the input of the multi-layer decoder.
Citation Information
Cited By
Weld joint identifying and tracking method based on multi-modal information, automatic welding device and computer equipment
CN119681385A
A welding seam recognition and tracking method based on multi-modal information, an automatic welding device and a computer device
CN119681385B
Defect detection method and device based on improved YOLO model
CN121169795A
