Method for detecting salient objects from rgb-d images based on cross-modal interaction and correction

Abstract

The present application relates to a kind of RGB-D image saliency target detection method based on cross-modal interaction and revision, comprising:1, in the encoding stage, color image encoder and depth map encoder respectively extract the features of color image modal and depth map modal, the high-level features of color image modal and depth map modal are gradually guided to be integrated unit and are cross-modal interaction and obtain RGB-D feature;2, feature revision intermediate structure is revised to the feature of color image modal, depth map modal and RGB-D modal obtained in the encoding stage in self-modal and cross-modal;3, in the decoding stage, color image modal and depth map modal are decoded respectively, and each level decoding feature is sent into importance gate fusion unit and is fused decoding, to complete the decoding of RGB-D modal, obtain the final saliency map.The present application respectively in different stages carries out interaction and revision to feature, realizes the more comprehensive fusion of two kinds of modal and the extraction of complementary information.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for RGB-D image salient object detection based on cross-modal interaction and correction. BACKGROUND

[0002] When people watch images, they are involuntarily attracted by some objects or regions in the image, which is mainly caused by the human visual attention mechanism. These objects are called salient objects. In the field of computer vision, to simulate this scheme, salient object detection, which automatically locates the most visually attractive objects or regions in a scene, has been successfully applied to many tasks, such as video segmentation, image retrieval, and quality assessment. In fact, the natural binocular structure of human beings can also perceive the depth of field of the scene, thereby producing stereoscopic perception. The depth / disparity map is used to express this depth relationship in the form of an image. In recent years, the development and popularity of depth sensors, especially the rise of affordable and portable consumer depth cameras, have further promoted the application of RGB-D images. For RGB-D images, the color image contains rich detail and appearance information (e.g., color, texture, structure, etc.). While the depth map provides some valuable complementary information (e.g., shape, surface normal, internal consistency, etc.). In recent years, more and more research has focused on introducing depth cues in the salient object detection task to effectively suppress background interference in complex scenes, thereby further fully highlighting the foreground salient region. Past research can be roughly divided into three structures according to the number of encoders and decoders: single-stream structure, double-stream structure, and triple-stream structure. For single-stream structure, Zhao et al. adopted a single-stream encoder to make full use of the representation ability of a pre-trained network and proposed a real-time and robust salient detection model. Zhang et al. proposed the first uncertainty-aware RGB-D salient object detection model based on a variable auto-encoder. For double-stream structure, Li et al. adopted a late feature fusion strategy in the decoding stage to combine the features of the high-level color image and depth image two independent branches to generate cross-channel representations. Zhai et al. designed a novel cascaded refinement network using multi-modal and multi-level features, and the color image and depth image features can be fused in a complementary manner. For triple-stream structure, Fan et al. designed a gating mechanism to filter out low-quality depth images using the decoding results of the color image, depth image, and RGB-D branch.

[0003] During the implementation of the present application, the inventors found that the prior art at least has the following disadvantages and deficiencies: (1) For single-flow structure, the prior art ignores the difference between color map and depth map modalities, and lacks comprehensive cross-modality interaction. (2) Although the dual-flow structure and the triple-flow structure consider the difference between the two modalities, the dual-flow structure does not fully interact the two modalities, and it is difficult to fully utilize the complementary information; the triple-flow structure closely interacts the two modalities, but brings greater parameter quantity and computation, and has certain redundancy. SUMMARY

[0004] In view of the defects in the prior art, the purpose of the present application is to effectively obtain and utilize the cross-modality information of color and depth, improve the fusion effect of the two modalities, and design a new network for cross-modality interaction and correction of RGB-D image saliency target detection, which fully utilizes the cross-modality information and obtains better detection effect.

[0005] To achieve the above purpose, the technical solution adopted by the present application is:

[0006] The present application proposes a network based on cross-modality interaction and correction (CIR-Net). The network structure is a form between two-flow and three-flow networks, where the third branch, i.e. the RGB-D flow, is formed by interacting the high-level features learned by the color map and depth map branch networks, which can reduce the network parameters and better establish the RGB-D features. For this purpose, the present application designs the following components: 1) progressive attention guidance integration unit and importance gated fusion unit, which realize comprehensive interaction of cross-modality at the encoding end and decoding end, respectively. 2) feature correction middleware structure including self-modality attention correction unit and cross-modality weighted correction unit, which corrects the multi-modality encoder features by encoding the self-modality attention tensor and cross-modality context dependency.

[0007] A cross-modality interaction and correction network, comprising a progressive attention guidance integration unit, an importance gated fusion unit, a feature correction middleware structure, a color map encoder, a depth map encoder, a color map decoder and a depth map decoder; the feature correction middleware structure comprises a self-modality attention correction unit and a cross-modality weighted correction unit.

[0008] The color image encoder and the depth image encoder are respectively used for extracting features of a color image modality and a depth image modality; the progressive attention guided integration unit is used for cross-modal interaction of high-level features of the color image modality and the depth image modality to obtain RGB-D features; the feature correction middleware structure is used for self-modal and cross-modal correction of features of the color image, the depth image and the RGB-D obtained in the encoding stage; the color image decoder and the depth image decoder are respectively used for decoding the color image modality and the depth image modality, and the decoded features of each level are sent to the importance gated fusion unit for fusion decoding, so that the decoding of the RGB-D modality is completed, and a final saliency map is obtained.

[0009] An RGB-D image saliency target detection method based on cross-modal interaction and correction, using the above cross-modal interaction and correction network, comprising the following steps:

[0010] Step 1, in the encoding stage, the color image encoder and the depth image encoder extract features of a color image modality and a depth image modality respectively, and high-level features of the color image modality and the depth image modality are cross-modal interacted by the progressive attention guided integration unit to obtain RGB-D features;

[0011] Step 2, the feature correction middleware structure corrects the features of the color image modality, the depth image modality and the RGB-D modality obtained in the encoding stage in a self-modal and cross-modal manner;

[0012] Step 3, in the decoding stage, the color image modality and the depth image modality are respectively decoded, and the decoded features of each level are sent to the importance gated fusion unit for fusion decoding, so that the decoding of the RGB-D modality is completed, and a final saliency map is obtained.

[0013] In step 1, the feature combination and forward propagation start from the third layer:

[0014]

[0015] wherein, represents the RGB-D modality feature obtained after fusion of the i th encoding level, and respectively represent the color image feature and the depth image feature of the i th encoding level, represents that the channel concatenation operation is performed on , and conv represents a convolution layer. Then, in order to highlight the complementary information and suppress irrelevant redundancy in cross-level and cross-channel fusion, the spatial attention map generated by the upper level RGB-D feature is used to progressively guide the feature fusion of the current level. Therefore, the final RGB-D features of the fourth level and the fifth level are updated as:

[0016]

[0017]

[0018] where, is the updated RGB-D feature, is the element-wise multiplication, A i-1 denotes the i-1-th level attention map, SA is the spatial attention operation, and denotes the down-sampling operation.

[0019] In step 2, the self-modality attention modification unit is used to suppress background noise and highlight important clues from the perspective of the modality itself in a new spatial channel 3D attention manner;

[0020] Three encoder branches and The output features of the three encoder branches are embedded into the self-modality attention modification unit. First, the channel attention and spatial attention of the input features in the parallel structure are calculated to obtain the corresponding spatial attention map and channel attention map, then they are directly fused into the attention space through matrix multiplication to generate a three-dimensional attention tensor. This process is described as:

[0021]

[0022] where, 3D denotes the three-dimensional attention tensor, denotes the feature of each modality of the top-level encoding layer, mod e {r, d, rgbd}, r, d, rgbd represent color image modality, depth image modality and RGB-D modality respectively, SA and CA represent spatial attention operation and channel attention operation respectively, denotes matrix multiplication. The 3D attention tensor and the residual connection are used to refine the features of each modality itself:

[0023]

[0024] where is the element-wise multiplication.

[0025] In step 2, the cross-modality weighted modification unit further captures the long-range dependency between multiple modalities. The output features of the self-modality attention modification unit are taken as input, where C, H and W represent the number of channels, height and width of the feature map respectively. First, the number of channels is halved using a convolutional layer, and different modalities are mapped to a unified feature space, which is represented as:

[0026]

[0027]

[0028]

[0029]

[0030] where F θ , F ξ , F ψ represents the mapped feature, W θ , W ξ , and W ψ denote the learnable embedding weights of the convolutional layer.

[0031] Then, the correlation between color image features and depth image features and the self-correlation of RGB-D features are calculated in a pixel-wise manner:

[0032]

[0033]

[0034] where is the matrix multiplication, and softmax is the Softmax activation function. The common response between color image modalities and depth image modalities is emphasized, while The dependence on the RGB-D modalities themselves is modeled.

[0035] Finally, the correlation information mapped to the RGB-D modalities collectively generates cross-modal global dependency weights to refine the original input features:

[0036]

[0037] where mod ∈ {r, d, rgbd}, is the element-wise multiplication, and the operation R reshapes the feature dimension to

[0038] In step 3, single-modal features of the same level will flow to the corresponding level of the RGB-D branch to learn more comprehensive cross-modal decoding features. First, the color image decoding features and the depth image decoding features are fused with the corresponding skip-connection encoding features through two convolutional layers, thereby obtaining the fused decoding features. Then, the fused decoding features are channel-parallel to obtain the RGB-D branch decoding features H i . Finally, the features of the previous fusion unit and the RGB-D branch decoding features H i are combined into the output of the current fusion unit through learnable importance weights:

[0039]

[0040] where, Let P represent the output feature of the fusion unit in the (i+1)th layer, where i∈{5,4,…,1}, and ↑ denotes the upsampling operation. i It is a learnable importance weight graph;

[0041] H i and The channels are concatenated in parallel, and then a 1×1 convolution is applied to reduce the number of channels, producing feature U. i Next, a channel attention mechanism with a sigmoid activation function is used to obtain the importance weight map.

[0042]

[0043] Where CA represents the channel attention operation, and σ represents the sigmoid activation function; finally, for Convolution and activation operations are performed to obtain a saliency map.

[0044] The beneficial effects of this invention are as follows: This invention explores a new fusion mode for different modalities and proposes a network based on cross-modal interaction and correction. The invention designs four units in the network (a progressive attention-guided integration unit, a self-modal attention correction unit, a cross-modal weighted correction unit, and an importance-gated fusion unit) to interact and correct features at different stages, achieving a more comprehensive fusion of the two modalities and the extraction of complementary information.

[0045] The method of this invention has achieved competitive performance on multiple public benchmark datasets.

[0046] (1) This invention proposes an end-to-end cross-modal interaction and correction network, which fully captures and utilizes cross-modal information in an interactive and corrective manner for salient target detection in RGB-D images.

[0047] (2) The present invention designs a progressive attention-guided integration unit and an importance-gated fusion unit to realize cross-modal integrated interaction at the encoding end and the decoding end, respectively. It also designs a correction middleware structure including a self-modal attention correction unit and a cross-modal weighted correction unit, and corrects the multimodal encoder features by encoding the self-modal attention tensor and the cross-modal context dependency. Attached Figure Description

[0048] The present invention includes the following figures:

[0049] Figure 1 A schematic diagram of the overall architecture of the cross-modal interactive correction network of the present invention.

[0050] Figure 2 A schematic diagram illustrating a visualization example of RGB-D salient target detection in this invention. DETAILED DESCRIPTION

[0051] The application will be further described in detail below in combination with the drawings.

[0052] Figure 1 The overall architecture diagram of the network for cross-modal interaction and correction is shown in the figure. The network structure is between the double-flow and the three-flow. In the encoding stage, the progressive attention guided integration unit obtains the RGB-D feature by integrating the high-level features of two modalities (color image modality and depth image modality). The three encoding features are corrected by the self-modal and cross-modal correction intermediate structure, and then are decoded in three ways. The color image modality and the depth image modality not only need to be decoded respectively, but also the decoding features corresponding to each level need to be further fused and decoded by the importance gated fusion unit, so as to complete the decoding of the RGB-D modality, and then obtain the final saliency map.

[0053] ATTACHMENT Figure 2 The visual example diagram of the present application technology is shown. The first column is the color image, the second column is the depth image, the third column is the true value map of the RGB-D saliency target detection, and the fourth column is the saliency map predicted by the present application. It can be seen from the result that the present application method realizes better visual effect in many challenging scenes. Under the conditions of multi-target, complex background, low contrast and the like, the present application can extract effective information in two modalities and obtain accurate prediction results.

[0054] The present application explores an RGB-D image saliency target detection method based on cross-modal interaction and correction, and proposes a cross-modal interaction and correction network. The network realizes the effective fusion and extraction of complementary information of two modalities by designing the cross-modal interaction and correction unit at the encoding end, the intermediate, and the decoding end respectively.

[0055] 1. Technical route

[0056] The cross-modal interaction correction network designed by the application comprises a progressive attention guide integration unit, an importance gate fusion unit, and a feature correction intermediate structure comprising a self-modal attention correction unit and a cross-modal weighting correction unit. The network adopts an encoder-decoder architecture, and the overall structure is between a double-flow structure and a three-flow structure, and is more inclined to the three-flow structure, but the calculation amount at the encoding end is lower than that of the traditional three-flow structure. Specifically, in the encoding stage, the color image encoder and the depth image encoder extract the features of the color image modality and the depth image modality respectively, and the high-level features of the two modalities are subjected to cross-modal interaction through the progressive attention guide integration unit to obtain RGB-D features; the encoding features of the three modalities obtained in the encoding stage are subjected to self-modal and cross-modal correction through the feature correction intermediate structure; in the decoding stage, the color image modality and the depth image modality not only need to be decoded respectively, but also the decoding features corresponding to each level of them need to be further fused and decoded through the importance gate fusion unit, so as to complete the decoding of the RGB-D modality and obtain the final saliency map.

[0057] 2. Progressive attention guide integration unit

[0058] Considering the complementarity and diversity of different modalities, effective cross-modal information interaction plays a crucial role in the RGB-D saliency object detection task. In order to effectively integrate and generate RGB-D features based on multi-layer color image and depth image features in the encoding stage, the application proposes a progressive attention guide integration unit in the encoding stage.

[0059] Specifically, in order to effectively integrate RGB-D features in the encoding stage, the application considers two aspects when designing the progressive attention guide integration unit: (1) sufficient multi-level information fusion; (2) effective feature selection and highlighting. For the former, considering that different levels of features contain different information, have different scales, receptive fields and contents in the encoding stage. Therefore, the progressive cross-layer fusion strategy is designed to obtain a more comprehensive RGB-D representation in a coarse-to-fine manner. For the latter, although the encoding features contain rich multi-level information, the commonly used fusion strategies (such as parallel and convolution) may introduce information redundancy and easily confuse the feature representation. Therefore, in the aspect of feature selection and enhancement, the application introduces a spatial attention mechanism to guide the cross-level and cross-modal feature fusion by highlighting complementary information and suppressing irrelevant redundancy.

[0060] Firstly, considering that shallow features often contain more background noise, and high-level features contain clear information of salient objects but lack details, the application selects to combine high-level RGB features and depth features to generate initial cross-channel features, and feature combination and forward propagation starts from the third layer:

[0061]

[0062] where, denotes the RGB-D modality feature obtained after fusion of the i-th encoding level, and denote the color map feature and the depth map feature of the i-th encoding level, respectively, denotes the channel concatenation operation, and conv denotes the convolution layer. Then, in order to highlight the complementary information and suppress irrelevant redundancy in cross-level and cross-channel fusion, the present application utilizes the spatial attention map generated by the upper-level RGB-D feature to gradually guide the feature fusion of the current level. Therefore, the final RGB-D features of the 4th and 5th levels are updated as:

[0063]

[0064]

[0065] where, is the updated RGB-D feature, is the element-wise multiplication, and A i-1 denotes the attention map of the i-1th level, SA is the spatial attention operation, and denotes the down-sampling operation.

[0066] Considering the inaccurate attention results that may occur in some challenging cases, the present application adopts a residual connection in the formula. In general, the progressive attention-guided integration unit proposed by the present application not only integrates different modality information, but also encodes the features of different levels in a progressive attention weighting manner, thereby generating RGB-D encoding features.

[0067] 3. Feature correction intermediate structure

[0068] In order to transfer more effective encoding features to the decoding end, the present application inserts a feature correction intermediate structure as a connecting link between the encoding end and the decoding end, and corrects the encoding characteristics from the perspective of self-modality and cross-modality. For the design of the correction intermediate structure, the present application considers two aspects: 1) the encoding features of each modality contain rich spatial and channel information, and direct transmission may increase the difficulty of learning effective feature representation. Therefore, the present application designs a self-modality attention correction unit to suppress background noise and highlight important clues from the perspective of the modality itself; 2) considering the strong correlation and complementarity between different modalities, in which the color map modality contains the graph-background color contrast, and the depth map modality contains the internal consistency, the present application designs a cross-modality weighting correction unit to capture the long-term correlation of multiple modalities and extract cross-modality features from a global perspective.

[0069] ​1) Self-modal attention modification unit: After feature encoding, the obtained color map, depth map and RGB-D encoded features contain rich information representing salient objects. However, there will be redundancy in single modal information. In addition, indiscriminate information transmission may increase the difficulty of feature learning, and even pollute the inference of the subsequent decoding process. Therefore, the present application designs a self-modal attention modification unit in the modification intermediate to suppress background noise and highlight important clues from the perspective of the modal itself in a new spatial channel 3D attention way.

[0070] The output features of the three encoder branches (i.e. and ) are embedded into the self-modal attention modification unit. First, the channel attention and spatial attention of the input features in the parallel structure are calculated, so as to obtain the corresponding spatial attention map and channel attention map. Then, they are directly fused into the attention space through matrix multiplication to generate a three-dimensional attention tensor. This process can be described as:

[0071]

[0072] where A 3D represents a three-dimensional attention tensor, represents the features of each modality of the top encoding layer, mod∈{r,d,rgbd}, r, d, rgbd represent color map modality, depth map modality and RGB-D modality respectively, SA and CA represent spatial attention and channel attention operations respectively, and represents matrix multiplication. Using the 3D attention tensor, the present application refines the features of each modality itself through the 3D attention tensor and the residual connection:

[0073]

[0074] where is the element-wise multiplication.

[0075] 2) Cross-modal weighting modification unit: the self-modal attention modification unit modifies the encoded features in each modality, but does not fully utilize the strong correlation and complementarity between different modalities. The present application proposes a novel cross-modal weighting modification unit in the second stage of the modification intermediate to further capture the long-range dependencies between multi-modalities.

[0076] The output features of the self-modal attention modification unit are taken as input, where C, H and W represent the number of channels, height and width of the feature map respectively. First, the present application uses a convolutional layer to halve the number of channels and map different modalities to a unified feature space, which can be represented as:

[0077]

[0078]

[0079]

[0080]

[0081] where F θ , F ξ , F ψ represents the mapped features, W θ , W ξ , and W ψ denote the learnable embedding weights of the convolutional layers. Then, similar to the dot-product attention operation, the correlation between color image features and depth map features and the self-correlation of RGB-D features are computed in a pixel-wise manner:

[0082]

[0083]

[0084] where is the matrix multiplication, the superscript T denotes the matrix transpose, and softmax is the Softmax activation function. emphasizes the common responses between color image and depth map modalities, while models the dependency on the RGB-D modality itself. Finally, the correlation information mapped to the RGB-D modality collectively generates the cross-modal global dependency weights to refine the original input features:

[0085]

[0086] where mod ∈ {r, d, rgbd}, is the element-wise multiplication, and the operation R reshapes the feature dimension to

[0087] 4、Importance gate fusion unit

[0088] Cross-modal information interaction is essential for RGB-D salient object detection task. Existing methods usually only interact in the encoder or decoder stage, which is insufficient. In fact, the encoder and decoder play different roles in feature learning, where the encoder focuses more on the extraction of general-purpose features, while the decoder emphasizes the learning of saliency-related features. Therefore, in addition to cross-modal feature integration in the encoder stage, the present invention also performs cross-modal information interaction in the decoder to obtain discriminative saliency prediction features. Considering that the decoding features of the color image branch and the depth image branch can further provide effective guiding information (such as sharp edges, internal consistency, etc.) for the RGB-D branch, the present invention designs a convergent aggregation structure throughout the decoding stage.

[0089] Specifically, monomodal features of the same level will flow to the corresponding level of the RGB-D branch to learn more comprehensive cross-modal decoding features. First, the color image decoding features and the depth image decoding features are fused with the corresponding skip connection encoding features through two convolutional layers to obtain fused decoding features. Then, the fused decoding features are channel-parallel to obtain the RGB-D branch decoding features H i . Finally, the features of the previous fusion unit and the RGB-D branch decoding features H i are combined into the output of the current fusion unit by a learnable importance weight:

[0090]

[0091] where, represents the output feature of the i+1th fusion unit, i∈{5,4,…,1},↑ represents the up-sampling operation, P i is a learnable importance weight map that can measure the importance of RGB-D decoding features in the fusion process. Specifically, the present invention first channels the two features (i.e., H i and ) in parallel, then applies a 1×1 convolution to reduce the number of channels, resulting in feature U i . Next, a channel attention mechanism with a Sigmoid activation function is used to obtain the importance weight map

[0092]

[0093] where CA represents the channel attention operation, and σ represents the Sigmoid activation function. The importance map can explicitly measure the contribution of the complementary information of different modalities at the i-th decoding stage. Finally, the is subjected to convolution and activation operations to obtain the saliency map.

[0094] The term explanation: RGB-D image contains two modalities of color and depth information. RGB-D image saliency detection is to detect the most concerned target or region in the RGB-D image data. Single-flow, double-flow and triple-flow structure respectively refer to using one, two and three codec structures in the network to process multi-modal information.

[0095] The above embodiments are only used to illustrate the present application, but not to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and all equivalent technical solutions also belong to the protection scope of the present application.

[0096] The contents not described in detail in the specification belong to the prior art known to those skilled in the art.

Claims

1. A method for salient object detection of RGB-D images based on cross-modal interaction and correction, characterized in that, The method comprises the following steps: Step 1, in the encoding stage, the color image encoder and the depth image encoder extract the features of the color image modal and the depth image modal respectively, the high-level features of the color image modal and the depth image modal are integrated by the progressive attention guide integration unit to obtain the RGB-D features through cross-modal interaction; Step 2, the feature correction middleware structure corrects the features of the color image modal, the depth image modal and the RGB-D modal obtained in the encoding stage in the self-modal and cross-modal manner; Step 3, in the decoding stage, the color image modal and the depth image modal are decoded respectively, and the decoded features of each level are sent to the importance gate fusion unit for fusion decoding, so that the decoding of the RGB-D modal is completed, and the final saliency map is obtained; In step 1, the feature combination and forward propagation start from the third layer: ; wherein, RGB-D features obtained after fusing the first and represent color map features and depth map features of the first represents performing a channel concatenation operation on and represents a convolutional layer; then, in order to highlight complementary information and suppress irrelevant redundancy in cross-level and cross-channel fusion, a spatial attention map generated using the RGB-D features of the previous level is used to progressively guide the feature fusion of the current level; thus, the final RGB-D features of the fourth and fifth levels are updated as follows:​​​ ； wherein, is the updated RGB-D feature, is an element-wise multiplication, denotes the attention map of the level, is a spatial attention operation, denotes a down-sampling operation.

2. The RGB-D image saliency target detection method based on cross-modal interaction and correction according to claim 1, wherein: In step 2, the self-modal attention correction unit is used to suppress background noise, and a new spatial channel 3D attention manner is used to highlight important clues from the perspective of the modal itself; three encoder branches and The output features of the three encoder branches are embedded into the self-modal attention revision unit; first, the channel attention and spatial attention of the input features in the parallel structure are calculated, thereby obtaining the corresponding spatial attention map and channel attention map, then they are directly fused into the attention space through matrix multiplication to generate a three-dimensional attention tensor; this process is described as: ； wherein, represents a three-dimensional attention tensor, denotes the features of each modality of the top-level encoding layer, denotes the color map modality, the depth map modality, and the RGB-D modality, respectively, and denotes a spatial attention operation and a channel attention operation, respectively, denotes a matrix multiplication; using the 3D attention tensor and a residual connection to distill the own features of each modality: ； wherein is an element-wise multiplication. 3.The RGB-D salient object detection method based on cross-modal interaction and revision of claim 2, wherein: In step 2, the cross-modal weighted correction unit further captures the long-range dependency relationship between the multi-modal; output features of the self-modal attention correction unit as input, where and denote the channel, height and width of the feature map, respectively; first, the number of channels is halved using a convolutional layer, and different modalities are mapped to a unified feature space, which is denoted as: ； wherein, represents the features obtained after mapping, and denotes the learnable embedding weights of the convolutional layer; Then, the correlation between the color image features and the depth image features and the self-correlation of the RGB-D features are calculated in a pixel-level manner: ； wherein is a matrix multiplication, the superscript denotes a matrix transpose, is a Softmax activation function; emphasizes the common response between the color image modality and the depth map modality, while models the dependency on the RGB-D modality itself; Finally, the correlation information mapped to the RGB-D modal generates the cross-modal global dependency weight together, so as to refine the original input features: ； wherein is an element-wise multiplication, operation re-shape the feature dimension to .

4. The RGB-D salient object detection method based on cross-modal interaction and revision of claim 1, wherein: In step 3, the single-modal features of the same level will flow to the corresponding level of the RGB-D branch to learn more comprehensive cross-modal decoding features; first, the color image decoding features and the depth image decoding features are fused with the corresponding jump connection encoding features through two convolution layers, so as to obtain the fused decoding features; Then, the fused decoded features are channel concatenated to obtain RGB-D branch decoded features Finally, the features from the previous fusion unit and the RGB-D branch decoded features are combined into the current fusion unit output by learnable importance weights: ； wherein, denotes the layer's fusion unit output feature, denotes an up-sampling operation, is a learnable importance weight map; Will and channels are concatenated and then a 1x1 convolution is applied to reduce the number of channels, resulting in feature ; Next, a channel attention mechanism with a sigmoid activation function is used to obtain an importance weight map : ； wherein denotes a channel attention operation, denotes a Sigmoid activation function; finally, the convolution and activation operations are performed to obtain the saliency map.

Images