Visual saliency target detection method based on RGB-D image

Abstract

The invention relates to the technical field of computer vision, and discloses a visual saliency target detection method based on an RGB-D image, and the method comprises the following steps: S1, carrying out the channel alignment and spatial calibration of a mutual relation between corresponding features of different modals through an MI module, and obtaining a cross-modal representation; s2, then, an IGF module is added to the tail end of the Swin Transform encoder, and the IGF module is added to the tail end of the Swin Transform encoder; s3, the cross-modal fusion feature # imgabs0 # is subjected to channel cross fusion through a CCT module, and relevant information about significance is further obtained; and S4, finally, carrying out modeling on the remote dependency of the fusion features from a global perspective by using a decoding block based on Swin-Transform. According to the visual saliency target detection method based on the RGB-D image, by designing a modal interaction module and a channel cross fusion mechanism, the fusion effect of multi-modal features and the fine modeling capability of multi-scale features are fundamentally improved, and therefore the problems of saliency target detection precision and robustness which are difficult to overcome in the prior art are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for detecting visually significant objects based on RGB-D images. Background Art

[0002] Salient object detection technology can be widely used in computer vision tasks such as image retrieval, video segmentation, target recognition, target tracking, etc., providing more accurate support for intelligent visual processing. It can be used for salient object detection in scenarios such as public safety, autonomous driving, and intelligent monitoring. Salient object detection focuses on locating and segmenting the most visually attractive objects and areas, and is also widely used in the preliminary preprocessing stages of various other computer vision tasks.

[0003] However, the existing technologies in the field of RGB-D salient object detection have the following technical problems: most methods fail to fully utilize the complementary information between RGB images and depth maps, and insufficiently process the interference information in low-quality depth maps and complex scenes, resulting in a decrease in the accuracy of salient detection. In addition, the direct fusion of multimodal features often introduces redundant or noisy information, ignoring the differences between different modalities, which has a negative impact on the quality of the final saliency map. In the processing of multi-scale features, existing methods usually adopt simple aggregation strategies and lack modeling of the global correlation of salient features, which easily causes the loss of semantic information or blurred boundaries. The above defects are mainly due to the deficiencies of existing models in information interaction, feature integration and global modeling capabilities. Therefore, a visual salient object detection method based on RGB-D images is proposed to solve the above problems. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides a visually salient target detection method based on RGB-D images, which has the advantages of fundamentally improving the fusion effect of multi-modal features and the fine modeling capability of multi-scale features by designing a modal interaction module and a channel cross-fusion mechanism, and solves the problems of accuracy and robustness of salient target detection that are difficult to overcome in the prior art.

[0006] (II) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solution: a method for detecting visually significant objects based on RGB-D images, comprising the following steps:

[0008] S1. First, the MI module is used to align the channels and calibrate the spatial relationships between the corresponding features of different modalities to obtain cross-modal representation.

[0009] S2, then add the IGF module at the end of the Swin Transformer encoder;

[0010] S3, then cross-modal fusion features The CCT module is used to realize cross-channel fusion to further obtain relevant information about saliency;

[0011] S4. Finally, a Swin-Transformer-based decoding block is used to model the long-range dependencies of the fused features from a global perspective.

[0012] Preferably, the S4 is specifically a channel cross-fusion feature generated by the CCF module and the upsampled features of the previous decoding stage It is fed into two cascaded Swin-Transformer modules to model the global relationship.

[0013] Preferably,

[0014] Among them, Cat represents the concatenation operation in the feature dimension, Liner is the linear layer, ST represents two cascaded Swin-Transformer blocks, and Exp is the operation of converting the features back to the spatial resolution. Finally, at the end of the decoder, the saliency map S of the original resolution is obtained. out .

[0015] Preferably, the MI module in S1 is a modal interaction module, the MI module is used to fuse and enhance RGB features and depth features, and the MI module consists of a cross-modal feature enhancement unit (RDEN), a self-modal refinement fusion unit (SMRF) and a fusion unit (RF).

[0016] Preferably, the cross-modal feature enhancement unit utilizes the complementary correspondence between the RGB image and the depth image to enhance the depth feature using the semantic information extracted from the RGB features, while introducing the extracted spatial structure information in the depth data.

[0017] Preferably, after passing through the RDEN module, the self-modal refinement fusion unit outputs complementary features of modality enhancement, the self-modal refinement fusion unit contains rich spatial and channel information representing salient objects, and the self-modal refinement fusion unit integrates SA and CA into a 3D attention tensor of a spatial channel to suppress background noise, enhances robustness in a parallel manner, and reduces computational complexity at the same time, and condenses morphological features from a single modality in this new spatial channel 3D attention method to make important clues more prominent.

[0018] Preferably, the fusion module uses 3D attention tensor and refines and fuses different modal features through residual connection to finally obtain the output of MI in,

[0019]

[0020] Preferably, the CCF module is specifically a channel cross-fusion module, and the CCF module is composed of three units, specifically including: a multi-scale feature embedding unit, a multi-head channel cross-attention unit and a multi-layer perceptron (MLP) unit.

[0021] Preferably, the IGF module can determine the contribution of complementary information of different modalities in the process of cross-modal information interaction.

[0022] (III) Beneficial effects

[0023] Compared with the prior art, the present invention provides a method for detecting visually significant objects based on RGB-D images, which has the following beneficial effects:

[0024] 1. This visual salient object detection method based on RGB-D images uses a channel-wise cross-fusion transformer network (MICCFNet) with modal interaction for salient object detection (RGB-D SOD). Through modal interaction and cross-channel fusion networks, the mutual relationship between modalities is considered and the predicted salient map is decoded from the channel perspective to reduce semantic differences. The Swin Transformer is introduced as the backbone network to make full use of its global modeling ability, which makes up for the deficiency of traditional CNN methods in capturing long-range dependencies, thereby improving the feature representation ability of the model. The strategy of enhancing and selectively fusing low-quality depth map features is innovatively designed, which fundamentally reduces the negative impact of low-quality modalities on the overall performance and improves the applicability of the model in complex scenes.

[0025] 2. This visual salient object detection method based on RGB-D images proposes a modal interaction module (MI) to address the problem of insufficient utilization of inter-modal feature complementarity in RGB-D salient object detection. Through the design of cross-modal feature enhancement, self-modal refinement fusion and feature fusion sub-modules, it dynamically balances the feature contributions of RGB and depth maps, effectively alleviating the problems caused by poor depth map quality or complex scene interference, and improving the expressiveness of fused features.

[0026] 3. This visual salient object detection method based on RGB-D images designs a channel cross fusion module (CCF) to address the problems of noise interference and semantic information loss that may occur in the process of multi-scale feature aggregation. This module enhances the correlation between features of different scales, refines and calibrates multi-scale features, and significantly improves the global consistency and boundary clarity of the saliency map through the combination of multi-scale feature embedding, multi-head channel cross attention and multi-layer perceptron.

[0027] 4. To address the problem of insufficient extraction of high-level semantic information, this visually salient object detection method based on RGB-D images designs an important gated fusion module (IGF). This module enhances the discrimination and diversity of high-level features through dynamic weight control, effectively improving the model's ability to detect salient objects in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the overall model architecture of MICCFNet of the present invention;

[0029] Figure 2 This is a schematic diagram of the MI module structure in the present invention;

[0030] Figure 3 This is a schematic diagram of the CCF module structure in the present invention;

[0031] Figure 4 It is a schematic diagram of the IGF module structure in the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the embodiments of the present invention and the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See also Figure 1-4 , a visually salient object detection method based on RGB-D images, comprising the following steps:

[0034] S1. First, the MI module is used to align the channels and calibrate the spatial relationships between the corresponding features of different modalities to obtain cross-modal representation.

[0035] S2, then add the IGF module at the end of the Swin Transformer encoder;

[0036] S3, then cross-modal fusion features The CCT module is used to realize cross-channel fusion to further obtain relevant information about saliency;

[0037] S4. Finally, the Swin-Transformer-based decoding block is used to model the long-range dependency of the fused features from a global perspective, specifically the channel cross-fusion features generated by the CCF module. and the upsampled features of the previous decoding stage It is fed into two cascaded Swin-Transformer modules to model the global relationship.

[0038] Specifically, Figure 1 As shown in the figure, the network as a whole follows the encoder-decoder structure. The left branches are the feature encoders of RGB images and depth images respectively, both of which use the shared weight Swin-Transformer model. The middle branch is the cross-modal information integration part, and the rightmost branch is the bottom-up decoding process.

[0039] Specifically,

[0040] Among them, Cat represents the concatenation operation in the feature dimension, Liner is the linear layer, ST represents two cascaded Swin-Transformer blocks, and Exp is the operation of converting the features back to the spatial resolution. Finally, at the end of the decoder, the saliency map S of the original resolution is obtained. out .

[0041] Specifically, the MI module in S1 is a modal interaction module. The MI module is used to fuse and enhance RGB features and depth features. The MI module consists of a cross-modal feature enhancement unit (RDEN), a self-modal refinement fusion unit (SMRF) and a fusion unit (RF).

[0042] Furthermore, the cross-modal feature enhancement unit utilizes the complementary correspondence between the RGB image and the depth image, and extracts semantic information from the RGB features to enhance the depth features, while introducing the extracted spatial structure information in the depth data. This unit utilizes the complementary correspondence between the RGB image and the depth image, and extracts semantic information from the RGB features to enhance the depth features, while introducing the extracted spatial structure information in the depth data, such as Figure 2 As shown in RDEN in Figure 2, the multi-level features from the backbone network We use the spatial attention mechanism to calculate their common spatial attention map, use the spatial attention map as the weight of the RGB feature and the deep feature respectively, and then use the residual connection to retain the original feature information. Subsequently, the channel attention mechanism is used to calculate the channel attention map respectively. Finally, the channel attention map is multiplied with the original features extracted by the backbone to obtain the enhanced cross-modal fusion feature. The specific process is as follows:

[0043]

[0044]

[0045]

[0046] Among them, “×” represents element-by-element multiplication, “+” represents element-by-element addition, CA(·) represents the channel attention mechanism, SA(·) represents the spatial attention mechanism, and the calculation process of the spatial attention mechanism and the channel attention mechanism is defined as:

[0047] SA(x)=Sigmoidd(Conv3(CGMP(x)))

[0048] CA(x)=Sigmoid(Conv1(GMP(x)))

[0049] Among them, CGMP(·) is the global maximum pooling operation in the channel direction, GMP(·) represents the global maximum pooling operation, Conv3(·) represents the convolution operation using a 3×3 convolution kernel, Conv1(·) represents the convolution operation using a 1×1 convolution kernel, and Sigmoid(·) represents the Sigmoid activation function.

[0050] Furthermore, after passing through the RDEN module, the self-modal refinement fusion unit outputs modality-enhanced complementary features. The self-modal refinement fusion unit contains rich spatial and channel information representing salient objects. The self-modal refinement fusion unit integrates SA and CA into a 3D attention tensor of a spatial channel to suppress background noise, enhances robustness in a parallel manner, and reduces computational complexity. This new spatial channel 3D attention method condenses morphological features from a single modality to make important clues more prominent.

[0051] Further, such as Figure 2 As shown in SERF, first calculate the original features extracted by the backbone network CA and The SA is obtained to obtain the corresponding spatial attention map and channel attention map, and then they are directly fused to the attention space features through matrix multiplication to generate a 3D attention tensor. This process can be described as follows:

[0052]

[0053] in, represents matrix multiplication, CA(·) represents the channel attention mechanism, and SA(·) represents the spatial attention mechanism.

[0054] Furthermore, the fusion module uses the 3D attention tensor and refines and fuses the features of different modalities through residual connections to finally obtain the output of MI in,

[0055]

[0056] Specifically, the CCF module is a channel cross-fusion module. The CCF module consists of three units, including: a multi-scale feature embedding unit, a multi-head channel cross-attention unit and a multi-layer perceptron (MLP) unit.

[0057] Furthermore, the multi-scale feature embedding unit introduces a channel embedding unit, which aims to enrich the content representation of the image by combining block-based convolutional embedding with learnable position embedding. (i=1, 2, 3, 4) is used as input, and a tokenization operation is first performed. The features are reorganized into a series of flattened 2D patches, and different block sizes (P, P / 2, P / 4, P / 8) are used respectively to map these patches to the same area in the encoder features to form representations of four scales. During the whole process, we retain the original channel dimension, and then connect the tokens of these four layers together. In short, the purpose is to retain the channel information through blocks of different scales, and connect the processed features together to form a tag sequence with multi-scale information, which helps the model better capture the multi-scale features of the image. We label the four-layer features T i (i=1, 2, 3, 4) are connected together as keys and values.

[0058] Furthermore, the multi-head cross attention unit inputs the tokens into the multi-head channel cross attention unit, and then passes through a multi-layer perceptron (MLP) with a residual structure to encode channels and dependencies using multi-scale features to refine the output features of the MI fusion modules at different stages. The multi-head cross attention unit contains 5 inputs, namely four tags T i As a query, the concatenated T ∑ =Concat(T1, T2, T3, T4) as key and value.

[0059]

[0060] in, is the weight of different inputs, d is the sequence length (total number of patches), C i is the channel dimension of the RGB-D modality fusion feature. In our implementation, C1=192, C2=384, C3=768, C4=768.

[0061] Instance normalization is used to normalize the similarity matrix of each instance in the similarity graph, which helps to achieve smooth gradient propagation. In the case of N-head attention, the output after multi-head cross attention is calculated as follows:

[0062]

[0063] Where N is the number of heads. After that, MLP and residual operator are applied and the output is obtained as follows:

[0064]

[0065] Finally, The four outputs are concat- ed with the decoder features of the corresponding stage to predict the final salient target result.

[0066] Specifically, the IGF module can determine the contribution of complementary information of different modalities in the process of cross-modal information interaction. First, the highest level single modal features (i.e. and Features) generated by splicing The output of the MI module of the fourth stage The feature concatenation is performed through the convolution layer to obtain the encoding fusion feature matrix H. Finally, the important weight values ​​that can be learned are used to perform gated selection with the two modal features respectively, and finally the final result is The whole process is as follows:

[0067]

[0068] in, and are the outputs of the corresponding modalities through the convolutional layer, and W is the learnable important weight, which measures the importance of high-level salient features in the fusion process. Specifically, we first concatenate the two features (i.e. and ), then apply convolution to reduce the number of channels and produce feature U. Next, we use channel attention with sigmoid activation function to obtain the importance map

[0069]

[0070] Where CA is the channel attention and σ is the sigmoid activation function.

[0071] In summary, this method of visual salient object detection based on RGB-D images uses a channel-wise cross-fusion transformer network with modal interaction (MICCFNet) for salient object detection (RGB-DSOD). Through modal interaction and cross-channel fusion networks, the mutual relationship between modalities is considered and the predicted salient map is decoded from the channel perspective to reduce semantic differences. The Swin Transformer is introduced as the backbone network to make full use of its global modeling ability, which makes up for the deficiency of traditional CNN methods in capturing long-range dependencies, thereby improving the feature representation ability of the model. An innovative strategy for enhancing and selectively fusing low-quality depth map features is designed, which fundamentally reduces the negative impact of low-quality modalities on the overall performance and improves the applicability of the model in complex scenes. In view of the problem of insufficient utilization of inter-modal feature complementarity in RGB-D salient object detection, a modal interaction module (MI) is proposed. Through the design of cross-modal feature enhancement, self-modal refinement fusion and feature fusion sub-modules, the feature contribution of RGB and depth maps is dynamically balanced, which effectively alleviates the problems caused by poor depth map quality or complex scene interference and improves the expression ability of fused features.

[0072] Moreover, in order to address the problems of noise interference and semantic information loss that may occur in the process of multi-scale feature aggregation, a channel cross-fusion module (CCF) is designed. This module enhances the correlation between features of different scales through the combination of multi-scale feature embedding, multi-head channel cross-attention and multi-layer perceptron, refines and calibrates multi-scale features, and significantly improves the global consistency and boundary clarity of the saliency map. In order to address the problem of insufficient extraction of high-level semantic information, an important gated fusion module (IGF) is designed. This module enhances the discriminative ability and diversity of high-level features through dynamic weight control, effectively improving the model's ability to detect salient targets in complex scenes, and solving the problems of salient target detection accuracy and robustness that are difficult to overcome in existing technologies.

[0073] The relevant modules involved in this system are all hardware system modules or functional modules that combine computer software programs or protocols with hardware in the prior art. The computer software programs or protocols involved in the functional modules are themselves technologies that are well known to those skilled in the art and are not improvements of this system. The improvements of this system are the interaction or connection relationships between the modules, that is, improvements to the overall structure of the system to solve the corresponding technical problems to be solved by this system.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A visually salient object detection method based on RGB-D images, characterized in that: The following steps are involved: S1. First, the MI module is used to align the channels and calibrate the spatial relationships between the corresponding features of different modalities to obtain cross-modal representation. S2, then add the IGF module at the end of the Swin Transformer encoder; S3, then cross-modal fusion features The CCT module is used to realize cross-channel fusion to further obtain relevant information about saliency; S4. Finally, a Swin-Transformer-based decoding block is used to model the long-range dependencies of the fused features from a global perspective.

2. The method for detecting visually significant objects based on RGB-D images according to claim 1, characterized in that: S4 is specifically the channel cross-fusion feature generated by the CCF module and the upsampled features of the previous decoding stage It is fed into two cascaded Swin-Transformer modules to model the global relationship.

3. The method for detecting visually significant objects based on RGB-D images according to claim 2, characterized in that: Among them, Cat represents the concatenation operation in the feature dimension, Liner is the linear layer, ST represents two cascaded Swin-Transformer blocks, and Exp is the operation of converting the features back to the spatial resolution. Finally, at the end of the decoder, the saliency map S of the original resolution is obtained. out .

4. The method for detecting visually significant objects based on RGB-D images according to claim 1, characterized in that: The MI module in S1 is a modal interaction module, which is used to fuse and enhance RGB features and depth features. The MI module consists of a cross-modal feature enhancement unit (RDEN), a self-modal refinement fusion unit (SMRF) and a fusion unit (RF).

5. The method for detecting visually significant objects based on RGB-D images according to claim 4, characterized in that: The cross-modal feature enhancement unit utilizes the complementary correspondence between the RGB image and the depth image, and enhances the depth feature with the semantic information extracted from the RGB feature, while introducing the extracted spatial structure information in the depth data.

6. The method for detecting visually significant objects based on RGB-D images according to claim 4, characterized in that: After passing through the RDEN module, the self-modal refinement fusion unit outputs complementary features of modality enhancement. The self-modal refinement fusion unit contains rich spatial and channel information representing salient objects. The self-modal refinement fusion unit integrates SA and CA into a 3D attention tensor of a spatial channel to suppress background noise, enhances robustness in a parallel manner, and reduces computational complexity. This new spatial channel 3D attention method condenses morphological features from a single modality to make important clues more prominent.

7. The method for detecting visually significant objects based on RGB-D images according to claim 4, characterized in that: The fusion module uses 3D attention tensors and refines and fuses different modal features through residual connections to finally obtain the output of MI. in, 8. The method for detecting visually significant objects based on RGB-D images according to claim 2, characterized in that: The CCF module is specifically a channel cross-fusion module, which consists of three units, including: a multi-scale feature embedding unit, a multi-head channel cross-attention unit and a multi-layer perceptron (MLP) unit.

9. The method for detecting visually significant objects based on RGB-D images according to claim 1, characterized in that: The IGF module can determine the contribution of complementary information of different modalities in the process of cross-modal information interaction.

Images