A multi-modal salient object detection method based on cross-modal uncertainty region correction

Abstract

The application discloses a kind of based on multi-modal saliency target detection method of cross-modal uncertain area correction, cross-modal feature enhancement module utilizes attention mechanism to enhance the effective information in shallow feature, while inhibiting interference information;According to the cross-modal feature correction module of uncertain area perception, in the process of deep single-modal feature extraction, the uncertain area existing in the deep feature of two modalities is cross-modally corrected by the way of bidirectional interaction, and more discriminative single-modal feature is obtained;According to the multiscale cross-modal feature fusion module, the visible light image feature and the depth image feature are cross-modally fused to fully exploit the multiscale context information and complementary information in the two modalities;According to the discriminative cross-modal feature, the final saliency map is predicted, and the saliency prediction map is obtained;Supervised learning model is used to obtain network model parameters for saliency prediction map. Complete and fine saliency prediction map is obtained.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multimodal saliency target detection method based on cross-modal uncertainty region correction. Background Technology

[0002] Saliency object detection aims to simulate the human visual system in segmenting the most visually appealing objects in a scene. In recent years, saliency object detection has been widely applied in many computer vision fields, such as object recognition, video segmentation, person re-identification, semantic segmentation, and image quality assessment, and has significant research value and broad application potential.

[0003] Early methods for salient object detection based on visible light depth images primarily relied on handcrafted features and different prior information. However, handcrafted features have limited representational power and lack guidance from deep semantic information, making accurate detection of salient objects difficult and encountering performance bottlenecks. Given the powerful representational capabilities of deep convolutional neural networks, they have been successfully applied to salient object detection and have experienced rapid development.

[0004] Despite the impressive detection results achieved by convolutional neural networks, these methods still face several challenges. During the imaging process of different modalities, limitations imposed by imaging conditions (such as low light, haze, and other special environments) or imaging techniques (such as low-resolution cameras and interference from external equipment) can cause noise to affect the imaging sensor of a particular modality, resulting in low-quality images. When these low-quality depth images are used for salient object detection, the low-quality images inevitably introduce noise, thereby reducing the discriminative power of the fused features and degrading the model's performance.

[0005] To alleviate the aforementioned problems, several saliency detection methods based on image quality issues have been proposed. Most existing methods primarily address the performance degradation caused by image quality through feature selection. In addition, there are approaches that address this issue from the perspective of image enhancement. These methods estimate a new depth image, then extract and fuse features from both the estimated and original depth images to enhance the depth image and resolve potential image quality problems.

[0006] However, the above methods only consider low-quality depth images, and when visible light images have low visual quality, the above models cannot perform good detection and segmentation of salient targets. Summary of the Invention

[0007] In view of the above problems, this invention proposes a method based on cross-modal uncertainty region correction.

[0008] According to one aspect of the present invention, a multimodal saliency target detection method based on cross-modal uncertainty region correction is provided, the detection method comprising:

[0009] The multimodal image salient target detection method based on cross-modal uncertainty region correction simultaneously considers low-quality visible light images and low-quality depth images.

[0010] According to the cross-modal feature enhancement module, the attention mechanism is used to enhance the effective information in the shallow features and suppress the interfering information.

[0011] According to the cross-modal feature correction module for uncertainty region perception, during the single-modal feature extraction process, cross-modal correction is performed on the uncertainty region of deep features of two modes through bidirectional information interaction to obtain more discriminative single-modal features.

[0012] The multi-scale cross-modal feature fusion module described above performs cross-modal fusion on the extracted visible light image features and depth image features to fully mine the multi-scale contextual information and complementary information in the two modal features;

[0013] Based on the discriminative cross-modal fusion features described above, the saliency map is decoded step by step to predict the final saliency prediction result.

[0014] The network model parameters are obtained by using a supervised learning model on the saliency prediction map.

[0015] According to the cross-modal feature enhancement module, shallow features primarily provide rich appearance information to help the network refine the boundaries of salient targets. However, as mentioned earlier, input visible light images and depth images sometimes have low visual quality, which results in shallow features extracted from low-quality input images containing a large amount of interfering information, further reducing the discriminative power of cross-modal fusion features. Therefore, to alleviate this problem, an attention mechanism is used to enhance the effective information in shallow features while suppressing interfering information.

[0016] According to the uncertain region-aware cross-modal feature correction module, deep features primarily provide semantic information, helping the network model better locate salient targets. However, due to interference information in low-quality input images, some uncertain regions inevitably exist in deep features. Within these uncertain regions, features from one modality typically possess strong discriminative power, while features from another modality contain interference information. Therefore, we can use information interaction to correct features with strong discriminative power that contain interference information, mitigating the impact of interference information on the saliency prediction of uncertain regions. Thus, our designed uncertain region-aware cross-modal feature correction module performs cross-modal correction on uncertain regions present in deep features from both modalities, obtaining more discriminative single-modal features and improving the model's accuracy in detecting salient targets in uncertain regions.

[0017] According to the multi-scale cross-modal feature fusion module, one of the challenges in salient target detection lies in the diversity of the scale, shape, and location of salient targets. Fully leveraging the multi-scale cross-modal complementary information between visible light and depth images helps address the problem of the size and dimension diversity of salient targets in salient target detection. Therefore, we utilize the proposed multi-scale cross-modal feature fusion module to fully mine the multi-scale cross-modal complementary information in visible light and depth features.

[0018] The saliency prediction map is obtained using a supervised learning model to obtain network model parameters, specifically including:

[0019] On the training dataset, a supervised learning model is used to train the algorithm network end-to-end on the predicted saliency map, resulting in the network model parameters:

[0020] On the training dataset, a supervised learning mechanism is employed to calculate the loss function L between the saliency map prediction results and the ground truth values ​​in the network model. joint :

[0021] L joint (S,G)=l bce (S,G)+l iou (S,G)

[0022] Among them l bce and l iou These are the cross-entropy loss function and the cross-union ratio (CUI) boundary loss function, respectively; the total loss function is set as follows:

[0023]

[0024] The multimodal saliency target detection method based on cross-modal uncertainty region correction provided by this invention trains the algorithm end-to-end. After training the overall saliency detection network, the model parameters are obtained. In order to avoid overfitting of the training dataset, the visible light images and depth images in the dataset are horizontally flipped and randomly cropped for data augmentation. Attached Figure Description

[0025] Figure 1 This is a flowchart of a multimodal saliency target detection method based on cross-modal uncertainty region correction disclosed in this invention;

[0026] Figure 2 This is a network diagram of the algorithm for a multimodal saliency target detection method based on cross-modal uncertainty region correction proposed in this invention.

[0027] Figure 3 This is a framework diagram of the cross-modal feature enhancement module proposed in this invention;

[0028] Figure 4 This is a framework diagram of the uncertain region cross-modal feature correction module proposed in this invention;

[0029] Figure 5 This is a framework diagram of the bidirectional cross-modal feature interaction submodule and feature selection submodule proposed in this invention.

[0030] Figure 6 This is a framework diagram of the multi-scale cross-modal fusion module proposed in this invention.

[0031] Figure 7 This is a simulation diagram of the evaluation results proposed in this invention. Detailed Implementation

[0032] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0033] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.

[0034] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0035] like Figure 1As shown, a multimodal image salient target detection method based on cross-modal uncertainty region correction includes the following steps:

[0036] Specifically, the multimodal image salient target detection network based on cross-modal uncertainty region correction includes three key modules: a bi-branch feature extraction network, a cross-modal feature enhancement module, a cross-modal feature correction module for uncertainty region awareness, and a multi-scale cross-modal fusion module.

[0037] Step 1: Two-Branch Feature Extraction Network. The two-branch feature extraction network includes one branch related to visible light features, one branch related to depth features, two cross-modal feature enhancement modules, and three cross-modal feature correction modules for uncertainty region awareness. Both main branches use the VGG-16 network for feature extraction. In any branch, the input of the intermediate layer is the output of the corresponding previous cross-modal feature enhancement module or the cross-modal feature correction module for uncertainty region awareness. The output features of the five visible light / depth VGG layers are represented as r. i d i (i = 1, 2, 3, 4, 5), where i represents the corresponding feature level. r and d represent the visible light modality and depth modality data, respectively. The cross-modal feature enhancement module and the cross-modal feature correction module for uncertain region perception are represented as follows: and

[0038] Step 2: First, the cross-modal feature enhancement module is used to enhance the effective information in the shallow features while suppressing interference information, thus obtaining the enhanced visible light features. and depth features The specific process is as follows:

[0039] Step 1: First, a spatial attention mechanism is used to select and enhance the features of the two modalities in the spatial dimension. Then, a channel attention mechanism is used on the enhanced unimodal features to select important features in the channel dimension, further suppressing interfering information in the unimodal features. Specifically, given a unimodal RGB feature r at a specific level... i and depth features d i (i = 1, 2) We first compute their shared spatial attention map SA i :

[0040]

[0041] in, This represents the element-wise multiplication operation. W sa (*) indicates the spatial weight generation function:

[0042]

[0043] Here, Avg(*) represents the average pooling operation along the channel, and Max(*) represents the max pooling operation along the channel dimension. Next, the shared spatial attention map is used as weights for the RGB and depth features to select discriminative features for enhancement while suppressing interfering information, as shown below:

[0044]

[0045]

[0046] Then, for the features enhanced in the spatial dimension and Perform channel attention separately, generating channel attention maps to select important unimodal features along the channel dimension. and Calculate using the following formulas respectively:

[0047]

[0048]

[0049] Step 2: Cross-modal feature correction for uncertain regions. Deep features primarily provide semantic information, helping the network model better locate salient targets. However, due to interference from low-quality input images, some uncertain regions inevitably exist in deep features. Within these uncertain regions, features from one modality typically have strong discriminative power, while features from another modality contain interference. Therefore, we can use information interaction to correct features with strong discriminative power that contain interference, mitigating the impact of interference on the saliency prediction of uncertain regions. The proposed cross-modal feature correction module for uncertain regions includes a bidirectional cross-modal feature interaction submodule and a feature selection submodule. Specifically, the bidirectional cross-modal feature interaction submodule performs cross-modal correction on uncertain region features present in the two modalities of deep features, while the feature selection submodule selects features that better correct the uncertain region from the corrected features as the features for that uncertain region.

[0050] Specifically, the proposed cross-modal correction module for uncertainty region perception first establishes the interaction relationship between RGB and depth features by calculating the shared information and uncertainty region information of deep RGB features and depth features in different local regions. Then, it obtains the uncertainty region features in the corresponding modality by constructing weights for the uncertainty regions of RGB and depth features. Since we obtain the uncertainty regions existing in the deep features of two modalities and their features in the corresponding modal regions, we cannot determine in advance which modality's uncertainty region features are more discriminative. Therefore, we first use the proposed bidirectional cross-modal feature interaction submodule to correct the uncertainty region features of the two modalities through bidirectional interaction. Then, a designed feature selection submodule selects features that better correct the uncertainty regions, and then enhances the original RGB and depth features respectively. Detailed steps are as follows:

[0051] Step 1: First, use the feature transformation function to transform the deep-level RGB features r i and depth features d i (i = 3, 4, 5) are mapped to the same feature space, that is:

[0052] R i =Conv(r i ,σ 1 ),

[0053] D i =Conv(d i ,σ 1 ),

[0054] Wherein, Conv(*,σ 1 ) represents a 1×1 convolution operation, which serves as the feature transformation function, and its corresponding parameter is σ. 1 R i and D i This represents the features after mapping.

[0055] Step 2: Utilize the mapped features R i and D i The following formula is used to calculate the region features shared by deep-level RGB features and depth features, and then to calculate the features of the uncertain regions of each modality.

[0056]

[0057] Among them, F co This represents the shared information between RGB features and depth features. This represents information about uncertain regions within the RGB feature. It can reflect information about uncertain regions in depth features.

[0058] Step 3: Obtain the spatial weights of uncertain regions in the deep features of the two modalities through spatial attention operations, and then use these weights to select the features of the corresponding region of the other modality, i.e.:

[0059]

[0060]

[0061] in, This refers to the spatial attention weight generation function described in the cross-modal feature enhancement module. This indicates the discriminability of uncertain regions in depth features within the corresponding RGB features. This represents the discriminability of uncertain regions in RGB features within their corresponding depth features. Simultaneously, we obtain the features of uncertain regions in both RGB and depth features through similar operations. and Right now:

[0062]

[0063]

[0064] Step 4: Utilize the bidirectional cross-modal feature interaction submodule to process the obtained... and and Cross-modal correction is performed using a bidirectional interactive approach. In the bidirectional cross-modal feature interaction submodule, we primarily use cross-attention operations for cross-modal feature correction. Specifically, using... and For example, firstly, similar to self-attention, 1×1 convolution and reshape operations are used to transform the input features from... Convert to and After that Matrix multiplication and normalization operations are performed on its transpose and then used to calculate and generate the position correlation matrix. Then, using this location correlation matrix to... Perform correction. The entire process can be represented as:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Where Nor(*) represents normalizing the values ​​in the channel correlation matrix to [0,1], and Reshape(*) represents reshaping the channel correlation matrix to [0,1]. Convert from size C1×H×W to C1×HW. Conv(*,σ) 2 ) and Conv(*,σ 3 ) represents two 1×1 convolutional layers and their parameters σ. 2 and σ 3 ReLU(*) represents the ReLU activation function. Since we are using a two-way information exchange method, therefore, for and and and Applying the same correction process as above, we obtain and

[0071] Step 5: Due to the uncertainty inherent in bidirectional correction—that is, it's uncertain which correction method yields more discriminative features—we designed a feature selection submodule after obtaining cross-modal interactive correction to select features corrected for the uncertain region. Taking RGB feature correction as an example, firstly, in the proposed feature selection module, we select features corrected by both methods for the uncertain region. and Dimensionality reduction is achieved using 1×1 convolutions. Next, global average pooling and global max pooling operations are used to compress the vectors into one-dimensional feature vectors. Finally, feature weights W are generated using two cascaded fully connected layers and a sigmoid function operation. i ∈R C This is done to select the more discriminative features. Therefore, the feature selection submodule uses the following formula, utilizing weight W... i This allows us to select features that provide more accurate correction for uncertain regions. The mathematical expression for this process is:

[0072]

[0073]

[0074]

[0075] Where σ(*) represents the Sigmoid function, GAP(*) represents the global average pooling operation, GMP(*) represents the global max pooling operation, and FC(*,γ) represents the global max pooling operation. ω ) represents the fully connected layer and its corresponding parameter γ. ω We followed the same approach using the corrected features. and Obtain the region-corrected features from the depth features

[0076] Therefore, after the above operations, the corrected RGB features and depth features are respectively... and The mathematical expression for this process is:

[0077]

[0078]

[0079] Step 3: Multi-scale Cross-modal Fusion Module. One of the challenges in salient object detection lies in the diversity of scale, shape, and location of salient objects. Fully leveraging the multi-scale cross-modal complementary information between RGB and depth images helps address the problem of size and dimension diversity of salient objects in salient object detection. Therefore, this chapter proposes a multi-scale cross-modal feature fusion module. This module fully leverages the multi-scale cross-modal complementary information in RGB and depth features. Considering that dilated convolution-based multi-scale feature pooling modules can effectively extract multi-scale features, this chapter proposes a dilated convolution-based multi-scale feature extraction module (Re_ASPP) with residual connections, based on ASPP, and applies it to the cross-modal fusion feature module to extract multi-scale features from single-modal features before cross-modal fusion. Re_ASPP employs four branches, each using dilated convolutional layers with different dilatations to extract single-modal features at different scales. Meanwhile, considering the different receptive fields of single-modal features at different levels, as shown in Table 4.1, the dilation rate of the Re_ASPP module dilated convolution varies for different levels. Furthermore, residual connections are used in Re_ASPP to ensure gradient stability. The mathematical expression for Re_ASPP is as follows:

[0080]

[0081] Here, δ(*) and Cat(*) represent the ReLU activation function and the cascade operation, respectively. This represents convolutional layers with different hole ratios and their corresponding parameters. This represents a convolutional layer with a 1×1 kernel and its corresponding parameters.

[0082] Therefore, in the multi-scale cross-modal feature fusion module, we first obtain the multi-scale features of each single modality through the Re_ASPP module, then use multiplication operations to mine the common information of the two modalities to supplement and enhance the original single-modal features, and finally use concatenation operations to obtain the cross-modal fused features. The specific calculation process is as follows:

[0083]

[0084]

[0085] Wherein, Conv(*,σ 5 ) and Conv(*,σ 6 ) represents two convolutional layers and their parameters σ. 5 and σ 6 .

[0086] Finally, the cross-modal fusion features are enhanced in the channel dimension through a channel attention mechanism, namely:

[0087]

[0088] Among them, W ca (*) represents the channel attention weight generation function mentioned above. This represents the output characteristics of the multi-scale cross-modal fusion module.

[0089] Step 4: Saliency Prediction. To obtain more detailed information, multimodal features from adjacent levels are also integrated for saliency prediction. This fully explores and utilizes the complementary relationships between multi-level features, improving the performance of the salient object detection algorithm. A total of five saliency prediction results were generated in the above process:

[0090]

[0091]

[0092] Among them, S i This represents the saliency map generated for each level, Conv(*,θ) s ) represents a 1×1 convolutional layer. Cat(*) and Up(*) represent cascaded feature mapping and bilinear interpolation upsampling operations along the channel dimension, respectively.

[0093] On the training dataset, a supervised learning mechanism is employed to calculate the loss function L between the saliency map prediction results and the ground truth values ​​in the network model. joint :

[0094]

[0095] Among them l bce (*) and l iou (*) represent the cross-entropy loss function and the boundary loss function based on the cross-union ratio, respectively. Their definitions are as follows:

[0096]

[0097]

[0098] Where G(m,n)∈{0,1} is the true pixel label. P(m,n)∈{0,1} is the probability of predicting each pixel in the saliency map. W and H represent the width and height of the input image, respectively.

[0099] The technical effects of the present invention will be further explained below with reference to simulation experiments:

[0100] 1. Simulation conditions: All simulation experiments were conducted on the operating system Ubuntu 16.04.5, the hardware environment was an Nvidia GeForce GTX 1080Ti GPU, and the PyTorch deep learning framework was used.

[0101] 2. Simulation content and result analysis:

[0102] Simulation 1

[0103] The present invention and existing saliency detection methods based on visible light depth images were tested on six public visible light depth image saliency detection datasets: DUT-RGBD, NJU2K, NLPR, LFSD, RGBD135, and STERE. Some experimental results were compared intuitively.

[0104] Compared to existing technologies, this invention achieves better detection results for low-to-medium visual quality images from input visible light-depth images. This is due to the cross-modal feature enhancement module, which enhances effective information in shallow features and suppresses interfering information. Furthermore, the cross-modal feature correction module for uncertain region perception effectively corrects uncertain regions in the two modalities of deep features through information exchange via the cross-modal feature bidirectional interaction module, resulting in more discriminative single-modal features and improving the accuracy of uncertain region saliency prediction. In addition, the invention fully captures and integrates multi-scale, multi-modal complementary information in visible light and depth images, allowing for a comprehensive combination of the two cues and leveraging their respective advantages. This enables better segmentation of small and multiple targets in complex scenes, while also providing more complete saliency detection results for multi-target images. Simulation results are evaluated as follows: Figure 6 As shown:

[0105] Among them, (a) RGB image; (b) Depth image; (c) MMCI prediction result; (d) TANet prediction result; (e) DMRA prediction result; (f) CPFP prediction result; (g) ICNet prediction result; (h) CPFP prediction result; (i) S2MA prediction result; (j) D3Net prediction result; (k) A2dele prediction result; (l) ASIFNet prediction result; (m) SSF prediction result; (n) DRLF prediction result; (o) DQSD prediction result; (p) CCAFNet prediction result; (q) JL-DCF prediction result; (r) DFMNet prediction result; (s) Ours prediction result; (t) ground truth. Figure 6 It can be seen that the saliency map for RGB-D image prediction produced by this invention is more complete overall and more detailed, which fully demonstrates the effectiveness and superiority of the method of this invention.

[0106] Simulation 2

[0107] The results of saliency detection experiments conducted on six public RGB-D image saliency detection datasets—DUT-RGBD, NJU2K, NLPR, RGBD135, LFSD, and STERE—compared with existing multimodal saliency detection methods based on RGB-D images were obtained. The simulation results were objectively evaluated using recognized evaluation metrics, and are shown in Table 1.

[0108] in:

[0109] F β This represents the maximum value of the weighted harmonic sum of precision and recall.

[0110] E m This means combining local pixel values ​​with image-level means to jointly evaluate the similarity between the prediction and the ground truth.

[0111] S α This indicates the structural similarity between object perception and region perception in the predictions;

[0112] Metric MMCI DMRA D3Net ICNet A2dele S2MA JL-DCF SSF DQSD CCAFNet DFMNet TMFNet OURS Fβ 0.852 0.886 0.9 0.891 0.873 0.889 0.912 0.896 0.9 0.91 0.913 0.882 0.925 Em 0.915 0.927 0.936 0.926 0.916 0.93 0.949 0.935 0.936 0.943 0.949 0.91 0.953 Sα 0.858 0.886 0.9 0.894 0.869 0.894 0.91 0.899 0.899 0.909 0.912 0.91 0.918 MAE 0.079 0.051 0.046 0.052 0.051 0.053 0.038 0.043 0.05 0.037 0.039 0.041 0.034 Fβ 0.815 0.88 0.897 0.908 0.88 0.902 0.915 0.896 0.898 0.908 0.912 0.867 0.910 Em 0.913 0.947 0.953 0.952 0.945 0.953 0.963 0.953 0.952 0.956 0.961 0.944 0.956 Sα 0.856 0.899 0.912 0.923 0.896 0.915 0.926 0.914 0.916 0.921 0.923 0.921 0.924 MAE 0.059 0.031 0.03 0.028 0.028 0.03 0.024 0.026 0.029 0.026 0.026 0.027 0.024 Fβ 0.767 0.898 0.793 0.85 0.892 0.901 0.878 0.924 0.827 0.913 0.747 - 0.926 Em 0.859 0.933 0.829 0.899 0.93 0.937 0.92 0.951 0.878 0.943 0.844 - 0.950 Sα 0.791 0.889 0.773 0.852 0.885 0.903 0.881 0.915 0.845 0.903 0.791 - 0.917 MAE 0.113 0.048 0.098 0.072 0.042 0.043 0.055 0.033 0.072 0.037 0.092 - 0.034 Fβ 0.863 0.857 0.891 0.898 0.879 0.882 0.898 0.89 0.886 0.887 0.904 - 0.906 Em 0.927 0.916 0.938 0.942 0.928 0.932 0.942 0.936 0.935 0.934 0.948 - 0.946 Sα 0.873 0.845 0.899 0.903 0.879 0.89 0.9 0.893 0.892 0.892 0.908 - 0.904 MAE 0.068 0.063 0.046 0.045 0.045 0.051 0.042 0.044 0.051 0.044 0.04 - 0.037 Fβ 0.771 0.856 0.81 0.871 0.835 0.835 0.839 0.866 0.847 0.832 0.866 0.846 0.872 Em 0.839 0.9 0.862 0.903 0.879 0.873 0.879 0.9 0.878 0.876 0.902 0.865 0.909 Sα 0.787 0.847 0.825 0.878 0.836 0.837 0.833 0.859 0.851 0.826 0.87 0.849 0.873 MAE 0.132 0.075 0.095 0.071 0.074 0.094 0.084 0.066 0.085 0.087 0.068 0.084 0.062 Fβ 0.822 0.888 0.885 0.913 0.867 0.935 0.917 0.883 0.927 0.937 0.932 0.892 0.944 Em 0.928 0.945 0.946 0.96 0.923 0.973 0.96 0.941 0.973 0.977 0.973 0.968 0.982 Sα 0.848 0.901 0.898 0.92 0.885 0.941 0.924 0.905 0.935 0.938 0.938 0.936 0.944 MAE 0.065 0.029 0.031 0.027 0.028 0.021 0.021 0.025 0.021 0.018 0.019 0.021 0.016

[0113] MAE represents the average absolute difference between normalized predictions.

[0114] F β E m S α Higher values ​​are better, and lower values ​​for MAE are better. Table 1 shows that the present invention has a more accurate saliency segmentation capability for RGB-D images, fully demonstrating the effectiveness and superiority of the method.

[0115] The embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments, and various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

[0116] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for multi-modal salient object detection based on cross-modal uncertainty region correction, characterized in that, The method comprises the following steps: The multi-modal salient object detection model based on cross-modal uncertain area correction fully considers the influence of low-quality input images on the performance of the salient object detection model; The cross-modal feature enhancement module enhances the effective information in the shallow features by using an attention mechanism, while suppressing the interference information; The cross-modal feature correction module with uncertain area perception performs cross-modal correction on the uncertain areas existing in the deep features of the two modalities through bidirectional information interaction during the extraction of the deep single-modal features, so as to obtain more discriminative single-modal features and improve the accuracy of the model in detecting the salient objects in the uncertain areas. The method comprises the following steps: First, based on the RGB features mapped to the same space... and depth features Calculate the region features shared by deep-level RGB features and depth features, and then calculate the features of the uncertain regions of the two modalities. and ; Then the spatial weights of the uncertain regions in the deep features of the two modalities are obtained respectively through a spatial attention operation, and then the features of the corresponding regions of the other modality are selected by using the weights and At the same time, the features of the uncertain regions in the RGB features and the depth features are obtained and ​ Then the obtained and , and are respectively corrected by the cross-modal correction in the form of bidirectional interaction, and the cross-modal feature correction is performed by the cross-attention operation, and finally the corrected features are , which are corrected by the bidirectional information interaction, so that and , and and and apply the same correction process to obtain , and ; Finally, the features after the bidirectional cross-modal feature interaction submodule are selected according to the feature selection submodule, and the specific steps are as follows: First, the features of the uncertain region after correction by the two methods and Using 1 dimensional feature vector by using global average pooling operation and global maximum pooling operation respectively, and finally, the feature weights are generated by two cascaded fully connected layers and a Sigmoid function operation respectively According to the weight, the more accurate features for correcting the uncertain region are selected and Finally, the corrected RGB features and depth features are obtained as and ; The multi-scale cross-modal feature fusion module fully excavates and captures the complementary information in the visible light image and the depth image and the multi-scale context information of the cross-modal fusion features; a multi-scale feature extraction module based on hole convolution with residual connection is used to extract the multi-scale features before cross-modal fusion, and then the encoded features are decoded by the decoder; The fused features are decoded and predicted in a step-by-step manner to obtain the saliency prediction map; The joint cross-entropy loss and the loss function based on the intersection over union are used to further train the network to obtain a more complete salient object; The supervised learning model is used on the saliency prediction map to obtain the network model parameters.

2. The multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, characterized in that, The cross-modal feature enhancement module enhances the effective information in the shallow features by using an attention mechanism, while suppressing the interference information, and specifically includes: Firstly, the spatial attention mechanism is used to select and enhance the features of two modalities in the spatial dimension. Firstly, the common spatial attention map of the two modalities is calculated Then, the discriminative features are enhanced according to the common spatial attention map, and the features enhanced in the spatial dimension are obtained and ; Then a channel attention map is generated through a channel attention mechanism and Through the channel attention map, enhancement in the channel dimension is realized, interference information contained therein is suppressed, and the final enhanced single-modal RGB feature is obtained and single-modal depth feature .

3. The multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, characterized in that, The cross-modal feature correction module with uncertain area perception performs cross-modal correction on the uncertain areas existing in the deep features of the two modalities through information interaction, so as to obtain more discriminative single-modal features and improve the accuracy of the model in detecting the salient objects in the uncertain areas.

4. The multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, characterized in that, According to the multi-scale cross-modal feature fusion module, the complementary information in the visible light image and the depth image and the multi-scale context information are fully excavated, specifically including: Firstly, the Re_ASPP module is used to obtain multi-scale features of single modal, and then the multiplication operation is used to mine the common information of two modalities to supplement and enhance the original single modal features, and finally the cross-modal fusion features are obtained through splicing operation ; Then the cross-modal fusion features are enhanced in the channel dimension by a channel attention mechanism to obtain output features of the multi-scale cross-modal fusion block .

5. The multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, characterized in that, According to the fused cross-modal features, the final fused feature prediction final saliency map is obtained in a step-by-step manner in the decoding stage, and the saliency prediction map is obtained, specifically including: For the fusion feature through one The final saliency map is obtained by a convolution layer and a Sigmoid function .

6. The multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, characterized in that, The supervised learning model is used on the saliency prediction map to obtain the network model parameters, specifically including: On the training data set, the supervised learning model is used on the predicted saliency map to complete the algorithm network training end-to-end to obtain the network model parameters: On the training data set, the loss function of the significant map prediction result and the true value in the network model is calculated by using a supervised learning mechanism : wherein and are cross-entropy loss function and IoU-based loss function, respectively; The definitions of the two are as follows: wherein each pixel label that is a true value; probability of each pixel of the predicted saliency map; W is the width of the input image and H is the height of the input image.

7. The multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, characterized in that, The multi-modal image salient object detection network based on cross-modal uncertain area correction comprises three modules: a double-branch feature extraction network, a cross-modal feature enhancement module, a cross-modal feature correction module with uncertain area perception, and a multi-scale cross-modal fusion module.

Images