A multi-modal salient object detection method based on feature pre-correction and fusion

Abstract

The application discloses a kind of based on feature first correction and then multi-modal saliency target detection method of fusion, including two-step sample selection strategy obtain high-quality and with visible light image foreground alignment depth image as visible light image generation depth image network training target, send into visible light image generation depth image network and obtain high-quality pseudo depth image;Depth feature correction module is corrected to original depth image and obtains the depth feature after correction;Multi-modal multi-scale fusion module is extracted to visible light image feature and the depth image feature of getting correction is cross-modal fusion, and the complementary features of two kinds of modalities are fully mined;Using shallow feature injection module, shallow detail information is supplemented to deep feature to obtain more discriminative feature;Discriminative cross-modal feature predicts final saliency map, and obtains saliency prediction map;Using supervised learning model obtains network model parameter, and obtains complete and fine saliency prediction map.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multimodal saliency target detection method based on feature correction followed by fusion. Background Technology

[0002] Salient object detection aims to highlight and segment the most visually appealing objects or regions in natural images. As an important image preprocessing method, it has been applied in many computer vision fields in recent years, such as weakly supervised image semantic segmentation, visual tracking, object recognition, image retrieval, and video compression.

[0003] Early methods for salient object detection based on visible light depth images primarily involved designing handcrafted features and utilizing different prior information. However, these traditional methods encountered performance bottlenecks due to limited feature representation capabilities. Given the powerful representational capabilities of deep convolutional neural networks, they have been successfully applied to salient object detection, achieving substantial performance improvements.

[0004] Despite the impressive detection results achieved by convolutional neural networks, these methods still face several challenges. During the imaging process of different image modalities, limitations in the imaging environment (such as low light, haze, etc.) or imaging techniques (such as low-resolution cameras and external interference) can lead to noise affecting the depth image sensor, resulting in low-quality depth images. When these low-quality depth images are used for salient object detection, the noise inevitably introduced reduces the discriminative power of the fused features, impacting model performance.

[0005] To alleviate the aforementioned problems, several saliency detection methods based on image quality issues have been proposed. Most existing methods primarily address depth quality issues through feature-level selection. In addition, there are approaches that address the problem from the perspective of image enhancement. These methods estimate a new depth image, then extract and fuse features from both the estimated and original depth images to enhance the depth image and resolve potential image quality problems.

[0006] However, this method only considers low-quality depth images, and when the foreground of the depth image and the visible light image are inconsistent, the model cannot perform good detection and segmentation of salient targets in such cases. Summary of the Invention

[0007] To overcome the shortcomings of the existing technology, the present invention aims to propose a two-stage method for multimodal saliency target detection based on feature correction followed by fusion, which addresses image quality issues. The detection method includes:

[0008] The two-stage multimodal image salient target detection method based on feature correction and fusion simultaneously considers low-quality depth images and depth images that are inconsistent with the foreground of visible light images.

[0009] In the image generation stage, a high-quality depth image that is consistent with the foreground of the visible light image is selected according to the two-step sample selection strategy described above as the supervision information of the visible light image to generate a depth image network, and then fed into the visible light image to generate a depth image network to obtain a high-quality pseudo depth image.

[0010] In the saliency inference stage, according to the depth feature correction module, the original depth image is corrected using the generated high-quality pseudo-depth image to obtain the corrected depth features;

[0011] During the saliency reasoning stage, important shallow detail information is supplemented into deep features by the shallow feature injection module to obtain more discriminative single-modal features.

[0012] In the saliency reasoning stage, the extracted visible light image features and the corrected depth image features are fused across modes according to the multimodal multiscale fusion module to fully explore the complementary features and multiscale contextual information of the two modes.

[0013] Based on the discriminative cross-modal features described above, the saliency map is predicted step by step through decoding to obtain the final saliency prediction result;

[0014] The network model parameters are obtained by using a supervised learning model on the saliency prediction map.

[0015] In the image generation stage, the specific steps for generating high-quality pseudo-depth images using the two-step sample selection strategy are as follows:

[0016] The saliency map and ground truth are predicted from the input visible light depth image pairs using depth image prediction.

[0017] The intersection-union ratio is used to select deep samples D that contain more saliency information. a ;

[0018] Specifically, we first train an encoder-decoder network to predict the depth saliency map P. d Here, the VGG network is used as the encoder, and UNet is used as the decoder. Then, for each original depth image, we compute the predicted saliency map. and its corresponding truth value GT k Crossover ratio between Ultimately, we selected depth images containing more salient information to construct a new depth image set D. a ,Right now

[0019] Then, for D a For each depth image in the dataset, we calculate the positive positivity rate of the depth saliency map. The positive rate of the saliency map predicted by the corresponding visible light image. Right now:

[0020]

[0021] Finally, we derived our data from the deep sample set D. a Choose those The depth images are used as the final depth sample set D. b Furthermore, we select D from the original visible light depth image pairs. b The visible light image corresponding to the medium-depth image is used to obtain a new visible light depth image set RD.

[0022] The visible light images in the RD (Representational Depth Image) set are used as input to the image generation network, and the depth images in the RD set are used as supervision information for the image generation network. After retraining the image generation network, all visible light images contained in the original visible light depth image set are fed into the image generation network to generate corresponding high-quality pseudo-depth images with consistent foreground.

[0023] During the saliency inference phase, according to the depth feature correction module, the extracted pseudo-depth features are first used to reweight the original depth features in the channel dimension to suppress unreliable information. Based on the obtained channel-by-channel weights, the initially corrected depth features are obtained. Then pseudo-depth features The initial depth features are further corrected using a spatial attention mechanism. Final pseudo-depth features Further embedded into enhanced deep features In the process, the final corrected depth feature d is obtained. i .

[0024] In the saliency inference stage, according to the aforementioned shallow feature injection module, to reduce computational complexity, we only perform cross-modal feature fusion in the last three layers. However, this results in the loss of important details in the shallow layers, which is crucial for refining the boundaries of the saliency target. Therefore, we learn the weights of deep features to select important details from the shallow layers and supplement them into the deep features, obtaining enhanced and more discriminative single-modal visible light features and depth features.

[0025] In the saliency inference stage, based on the aforementioned multimodal multiscale fusion module, the features of the two modalities are fused using a channel-by-channel weighting mechanism to fully explore the complementary information of the two modalities. Considering that cross-modal features at different scales contribute differently to the final saliency prediction, the initially fused cross-modal features are re-weighted and fused using a multi-scale attention mechanism to obtain the final, more discriminative cross-modal features for subsequent saliency prediction.

[0026] The saliency prediction map is obtained using a supervised learning model to obtain network model parameters, specifically including:

[0027] On the training dataset, a supervised learning model is used to train the algorithm network end-to-end on the predicted saliency map, resulting in the network model parameters:

[0028] For the region consistency-aware loss function, we fully consider the consistency of the foreground saliency region and the background region. By calculating the false shadow regions in the foreground saliency region and the false sun regions in the background region, we achieve region consistency by reducing the differences between each pixel and its surrounding pixels in these two regions. This promotes obtaining a more complete saliency target while reducing interference from the background region.

[0029] On the training dataset, a supervised learning mechanism is employed to calculate the loss function L between the saliency map prediction results and the ground truth values ​​in the network model. joint :

[0030]

[0031] in and These are the cross-entropy loss function and the cross-union ratio (CUI) boundary loss function, respectively; the hyperparameter λ is set to 15. The total loss function is set as follows:

[0032]

[0033] The salient target detection method based on feature correction and fusion provided by this invention trains the algorithm end-to-end. After training the overall salient detection network, the model parameters are obtained. In order to avoid overfitting of the training dataset, the visible light images and depth images in the dataset are horizontally flipped and randomly cropped for data augmentation.

[0034] Compared to existing technologies, this invention demonstrates superior detection performance for depth images with low to medium visual quality and inconsistent foreground in visible light images. The high-quality pseudo-depth image generation and feature-corrected-then-fused network in this invention enable more complete and precise saliency detection even in depth images with low visual quality and inconsistent foreground, thus making the target saliency results closer to the manually labeled ground truth map. Furthermore, the invention's thorough mining of complementary information and capture of multi-scale contextual information in visible light depth images allows for a full combination of these two cues, leveraging their respective advantages. This results in better segmentation of small and multiple targets in complex scenes, while also achieving more complete saliency detection results for multi-target images. Attached Figure Description

[0035] Figure 1 This is a flowchart of a salient target detection method based on feature correction followed by fusion disclosed in this invention;

[0036] Figure 2 This is a schematic diagram of the selection process of the two-step sample selection strategy proposed in this invention;

[0037] Figure 3 This is a network diagram of the algorithm for a salient target detection method based on feature correction followed by fusion proposed in this invention.

[0038] Figure 4 This is a framework diagram of the depth feature correction module proposed in this invention;

[0039] Figure 5 This is a framework diagram of the shallow feature injection module proposed in this invention;

[0040] Figure 6 This is a framework diagram of the multimodal and multiscale fusion module proposed in this invention.

[0041] Figure 7 This is a simulation diagram of the evaluation results proposed in this invention. Detailed Implementation

[0042] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0043] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.

[0044] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0045] like Figure 1 As shown, a multimodal image salient target detection method based on feature correction followed by fusion includes the following steps:

[0046] (1) The proposed two-stage visible light depth image saliency detection model consists of an image generation stage and a saliency inference stage. In the image generation stage, a high-quality pseudo-depth image with consistent foreground is generated using the visible light image through an image generation network. Specifically, a two-step sample selection strategy (TSS) is applied to select high-quality depth images with consistent foreground from the original visible light-depth image as supervision information for training the image generation network. In the saliency inference stage, a feature-corrected and then fused network is constructed to perform saliency target detection. In the network, the generated pseudo-depth images are first used to correct the unreliable information contained in the original depth image, and then the cross-modal complementary information in the visible light image and the depth image is captured to perform saliency prediction. This process is implemented through a depth feature correction module, a shallow feature injection module, and a multimodal multi-scale fusion module. Moreover, the feature-corrected and then fused network is trained by combining cross-entropy loss, cross-union ratio-based loss function, and a newly constructed region consistency-aware loss function. The specific process will be described below.

[0047] (2) Image generation stage:

[0048] First, a two-step sample selection strategy is used to select depth images with high visual quality and foreground consistency as training supervision information for the image generation network. Specific details are as follows:

[0049] Step 1: Select depth samples D containing more saliency information from the input visible light depth image pairs using the IoUs between the saliency map predicted by the depth images and the ground truth. a ;

[0050] Specifically, first, an encoder-decoder network is trained to predict the depth saliency map P. d Here, the VGG network is used as the encoder, and UNet is used as the decoder. Then, for each original depth image, a predicted saliency map is computed. and its corresponding truth value GT k Crossover ratio between Ultimately, we selected depth images containing more salient information to construct a new depth image set D. a ,Right now

[0051]

[0052] Step 2: Calculate the positive rate (TP) of their predicted significance plot. d From the newly constructed depth image set D a Select high-quality depth images with consistent foregrounds to obtain the final depth image set D. b As supervisory information for image generation networks.

[0053] Specifically, the visible light image is first fed into the pre-trained saliency target detection network mentioned in the previous step to predict the visible light saliency map P. r Then, for D a For each depth image in the dataset, calculate the positive positivity rate of the depth saliency map. The positive rate of the saliency map predicted by the corresponding visible light image. Right now:

[0054]

[0055] Finally, from the depth sample set D a Choose those The depth images are used as the final depth sample set D. b Further select D from the original visible light depth image pairs. b The visible light image corresponding to the medium-depth image is used to obtain a new visible light depth image set RD.

[0056] For the visible light depth image set RD constructed above, the image generation network is retrained. The visible light images in RD serve as the input to the image generation network, and the depth images in RD serve as the supervision information for the image generation network. After retraining, all visible light images contained in the original visible light depth image set are fed into the image generation network to generate corresponding high-quality, foreground-consistent pseudo-depth images. These newly generated pseudo-depth images are used to correct the original depth images in the subsequent saliency inference stage. Compared to the original depth images, the pseudo-depth images generated by this method have higher visual quality and better foreground consistency with the corresponding visible light images.

[0057] (3) Significance reasoning stage

[0058] In the saliency inference stage, the original depth features are first corrected using a generated pseudo-depth image. Then, the corrected depth features and visible light features are fused to better capture cross-modal complementary information for saliency prediction. Specifically, the network that first corrects and then fuses features includes three key modules: a feature correction module, a shallow feature injection module, and a multimodal multi-scale fusion module.

[0059] First, a VGG-16-based three-stream coding network is used to simultaneously extract hierarchical features from the visible light image, the original depth image, and the pseudo-depth image, denoted as r. i , and 'i' represents the hierarchical index. Step 1: First, the feature correction module is used to correct the pseudo-depth features. With the help of correcting those original depth features Obtain the corrected depth features d i The specific process is as follows:

[0060] Step 1: First, utilize the extracted pseudo-depth features. Reweight the original depth features in the channel dimension To suppress unreliable information. First, the original deep features... and pseudo-depth features The concatenation is performed using convolution and global average pooling operations to learn channel-wise weights, i.e.:

[0061]

[0062] Where GAP(*) represents the global average pooling operation. σ(*) represents the Sigmoid function. W i Represents the original depth features of the i-th layer The channel-wise weights are calculated. Conv3 represents a 3×3 convolutional layer. Based on the obtained channel-wise weights, the depth features are initially corrected. It can be obtained through the following formula:

[0063]

[0064] Step 2: Then pseudo-depth features The initial depth features are further corrected using a spatial attention mechanism. Consistency of the saliency objective content has been achieved:

[0065]

[0066] Here W sa (*) represents the spatial weight generation function:

[0067]

[0068] Here, Avg(*) represents the average pooling operation along the channel, and Max(*) represents the maximum pooling operation along the channel dimension.

[0069] Step 3: Final pseudo-depth features Further embedded into enhanced deep features In the process, the final corrected depth feature d is obtained.i .

[0070]

[0071] Step Two: Shallow Feature Injection. Cross-modal feature fusion is only performed in the last three layers to reduce computational complexity. However, this results in the loss of some important shallow-level details, which are crucial for refining the boundaries of salient targets. Therefore, a shallow feature injection module is designed to inject important detailed features from the first two layers into the third layer. Taking visible light features as an example, the specific details are as follows:

[0072] Step 1: The first two single-mode visible light features are first spliced ​​together to obtain the fused feature r of the first two layers. sl :

[0073] r sl =Conv3(Cat(Conv3(Down(r) 1 ),ε),r 2 ),β),

[0074] Down(*) represents a downsampling operation. Conv3(*,ε) and Conv3(*,β) represent two 3×3 convolutional layers with parameters ε and β, respectively.

[0075] Step 2: The same spatial attention mechanism mentioned in the feature correction module is applied to the third layer r of visible light features. 3 To obtain the corresponding weights in order to fuse features r from the first two layers sl Valuable details are then selected. Finally, the selected shallow visible light features are further incorporated into the third layer of visible light features to obtain enhanced visible light characteristics.

[0076]

[0077] here This represents the enhanced visible light features of the third layer. Conv3(*,γ) represents a 3×3 convolutional layer with parameter γ. Similarly, the enhanced depth features of the third layer are obtained using the same process.

[0078] Step 3: Multimodal and Multiscale Fusion Module. Simple fusion strategies cannot fully extract complementary information from visible light depth image pairs. Therefore, this module was designed to better extract complementary information from the two modalities and enhance cross-modal fusion features from a multi-scale perspective to achieve better saliency detection results. The specific implementation is as follows:

[0079] Step 1: Single-mode visible light characteristics r i and single-modal depth features d iFirst, the data is concatenated and then fed into convolutional layers to learn channel-wise weights, which can be calculated using the following formula:

[0080]

[0081]

[0082] Here, i = 3, 4, 5. Specifically, when i = 3, r 3 and d 3 Enhanced single-modal features are obtained through a shallow feature injection module. and F i Indicates r i and d i The splicing characteristics. This represents a convolutional block containing two 3×3 convolutional layers. MLP(*,η) and MLP(*,ψ) represent two fully connected layers. GAP(*) and GMP(*) represent global average pooling and global max pooling operations, respectively. Representing a series of channel-wise weights, the fused features are obtained using the following formula:

[0083]

[0084] This represents the initial fused features of the i-th layer. 1 represents the sum of the initial features and the initial features of the i-th layer. Vectors of the same size that are all equal to 1.

[0085] Step 2: Four parallel convolutions with different kernel sizes are used in the initial feature fusion. The above yields four multi-scale features:

[0086]

[0087] Here Conv m (*,θ m θ represents a convolutional layer of size m × m (m = 3, 5, 7, 9). m For their parameters. These represent features of different sizes in the i-th layer.

[0088] Considering that features at different scales contribute differently to the final saliency prediction, a scale-aware attention mechanism is constructed to adaptively fuse multi-scale features. Specifically, the multi-scale features are first fed into several SE blocks to obtain their scale attention weights. Then, the multi-scale weight vector W is obtained through the Softmax function. i Finally, the multi-scale features are reweighted by the obtained weight vector to achieve the enhanced cross-modal feature F. iThe entire process can be expressed by the following formula:

[0089]

[0090]

[0091]

[0092] Step 4: Reasoning about the saliency map. Saliency map S (t) (t = 3, 4, 5) are obtained from the side output features of each decoder through a 1×1 convolutional layer and a sigmoid function.

[0093] Step 5: Train the algorithm network to obtain model parameters

[0094] On the training dataset, a supervised learning model was used to train the saliency map of the predictions in step four. The algorithm network was trained end-to-end using the cross-entropy loss function, the IoU boundary loss function, and the region consistency-aware loss function to obtain the network model parameters.

[0095] The specific implementation of the region consistency perception loss function is as follows: First, calculate the false negative region FN of the foreground saliency region and the false positive region FP of the background region.

[0096]

[0097]

[0098] Here, S represents the predicted saliency map and G represents the corresponding ground truth. |·| represents the absolute value of a numerical value. For a given FN and FP, the region consistency-aware loss function... The following formula can be used to calculate and thus enhance the local consistency of the foreground saliency region and the consistency of the background region by reducing the difference between each pixel value and the surrounding pixel values:

[0099]

[0100] Here Avg spa (*) represents a 7×7 average pooling operation along the spatial dimension. H and W represent the height and width of the saliency map, respectively. 1 represents a vector of all 1s with the same size as the dimension G. The constructed region consistency-aware loss function helps to maintain the integrity of the saliency objective and suppress background interference in the saliency map.

[0101] On the training dataset, a supervised learning mechanism is employed to calculate the loss function L between the saliency map prediction results and the ground truth values ​​in the network model. joint :

[0102]

[0103] in and These are the cross-entropy loss function and the boundary loss function based on the cross-union ratio, respectively. Their definitions are as follows:

[0104]

[0105]

[0106] Where G(m,n)∈{0,1} is the true pixel label. P(m,n)∈{0,1} is the probability of predicting each pixel in the saliency map. W and H represent the width and height of the input image, respectively.

[0107] The technical effects of the present invention will be further explained below with reference to simulation experiments:

[0108] 1. Simulation conditions: All simulation experiments were conducted on the operating system Ubuntu 16.04.5, the hardware environment was an Nvidia GeForce GTX 1080Ti GPU, and the PyTorch deep learning framework was used.

[0109] 2. Simulation content and result analysis:

[0110] Simulation 1

[0111] The present invention and existing saliency detection methods based on visible light depth images were tested on six public visible light depth image saliency detection datasets: DUT-RGBD, NJU2K, NLPR, LFSD, RGBD135, and STERE. Some experimental results were compared intuitively.

[0112] Compared to existing technologies, this invention demonstrates superior detection performance for visible light depth images with low to medium visual quality and inconsistent foregrounds. Thanks to the combination of high-quality pseudo-depth image generation and a feature-corrected-then-fused network, this invention achieves more complete and refined saliency detection even in depth images with low visual quality and inconsistent foregrounds, resulting in target saliency results closer to manually calibrated ground truth maps. Furthermore, by fully exploiting complementary information and capturing multi-scale contextual information in visible light depth images, this invention effectively combines two cues, leveraging their respective advantages. This allows for better segmentation of small and multiple targets in complex scenes, while also achieving more complete saliency detection results for multi-target images. Simulation results are evaluated as follows: Figure 6 As shown:

[0113] Among them, (a) RGB image; (b) Depth image; (c) MMCI prediction result; (d) TANet prediction result; (e) CPFP prediction result; (f) DMRA prediction result; (g) D3Net prediction result; (h) SSF prediction result; (i) ICNet prediction result; (j) A2dele prediction result; (k) S2MA prediction result; (l) DRLF prediction result; (m) FRDT prediction result; (n) CMWNet prediction result; (o) CCAFNet prediction result; (p) JL-DCF prediction result; (q) DQSD prediction result; (r) DFMNet prediction result; (s) FCFNet (OURs) prediction result; (t) ground truth. Figure 6 It can be seen that the saliency map for RGB-D image prediction produced by this invention is more complete overall and more detailed, which fully demonstrates the effectiveness and superiority of the method of this invention.

[0114] Simulation 2

[0115] The results of saliency detection experiments conducted on six public RGB-D image saliency detection datasets—DUT-RGBD, NJU2K, NLPR, RGBD135, LFSD, and STERE—compared with existing multimodal saliency detection methods based on RGB-D images were obtained. The simulation results were objectively evaluated using recognized evaluation metrics, and are shown in Table 1.

[0116]

[0117]

[0118] in:

[0119] F β This represents the maximum value of the weighted harmonic sum of precision and recall.

[0120] E m This means combining local pixel values ​​with image-level means to jointly evaluate the similarity between the prediction and the ground truth.

[0121] S α This indicates the structural similarity between object perception and region perception in the predictions;

[0122] MAE represents the average absolute difference between normalized predictions.

[0123] F β E m S α Higher values ​​are better, and lower values ​​for MAE are better. Table 1 shows that the present invention has a more accurate saliency segmentation capability for RGB-D images, fully demonstrating the effectiveness and superiority of the method.

[0124] The embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments, and various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

[0125] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal saliency target detection method based on feature-based correction followed by fusion, characterized in that, The detection method includes: In the image generation stage, a two-step sample selection strategy is used to select high-quality depth images with consistent foreground from the input visible light-depth image pairs as supervision information for the image generation network, and the image generation network is retrained. Then, all visible light images from the original input visible light-depth image pairs are input into the trained image generation network to obtain high-quality pseudo depth images with consistent foreground. During the saliency reasoning stage, the deep feature correction module uses the extracted pseudo-deep features to correct the unreliable information contained in the original deep features; The depth feature correction module fully utilizes the generated high-quality pseudo-depth image with consistent foreground to correct unreliable information contained in the original depth image, specifically including: First, utilize the extracted pseudo-depth features Reweight the original depth features at the channel dimension. This further suppresses features contained in the original depth features. Unreliable information in; Pseudo-depth features The initial depth features are further corrected using a spatial attention mechanism. This achieves consistency in the content of the saliency objective: Final pseudo-depth features Further embedded into enhanced deep features In the process, the final corrected depth features are obtained. ; During the saliency inference stage, the shallow feature injection module uses the deep feature generation weights to select important shallow detail features and then supplements them into the deep features for more accurate saliency prediction. In the saliency reasoning stage, the multimodal multiscale fusion module fully mines and captures complementary information and multiscale contextual information of cross-modal fusion features in visible light images and depth images; an attention mechanism is used to integrate cross-modal fusion features to obtain the final fusion features, and then the decoder decodes the encoded features. The multimodal, multi-scale fusion module fully mines complementary information and multi-scale contextual information in visible light and depth images, specifically including: Single-modal RGB features and single-modal depth features First, the data is concatenated and then fed into convolutional layers to learn channel-wise weights. Based on the learned channel-by-channel weights, the initial fused features are obtained. : Four parallel convolutions with different kernel sizes were used in the initial fusion feature set. The above yields four multi-scale features: ， Represents a size of Convolutional layers of varying sizes, m=3,5,7,9. For their parameters, Features representing different sizes of the i-th layer; The fused features are decoded and predicted step by step to obtain the final saliency map; Based on the region consistency-aware loss function, which considers both the local region consistency of the foreground salient target and the consistency of the background region, the network is further trained using a combination of cross-entropy loss and a loss function based on intersection-union ratio to obtain a more complete salient target. The parameters of the network model are obtained by using a supervised learning model on the saliency prediction map; The shallow injection module uses weights generated from deep features to select important detail features from the shallow layer, and then injects the selected features into the third layer to obtain enhanced RGB features. Based on this process, the enhanced depth features of the third layer are obtained. .

2. The multimodal saliency target detection method based on feature correction followed by fusion according to claim 1, characterized in that, In the image generation stage, a two-step sample selection strategy is used to obtain high-quality depth images with consistent foreground. These depth images are then used as supervision information for the image generation network, resulting in a retrained network. The visible light image from the original input visible light-depth image pair is fed into the trained network to obtain a high-quality pseudo-depth image with consistent foreground. Specifically, this includes: The depth image from the original visible light depth image pair is fed into a pre-trained saliency prediction network to obtain the depth saliency map. Calculate the intersection-union ratio (IUU) between them and their corresponding true values. Select from the depth images and construct a new depth image set. .

3. The multimodal saliency target detection method based on feature correction followed by fusion according to claim 2, characterized in that, from We selected high-quality depth images with consistent foreground and further calculated the positive positivity rates of the RGB image saliency prediction results and the depth image saliency prediction results separately: ， for Select from the depth images to construct a new depth sample set. As the final depth sample set At the same time, select the corresponding RGB images to further obtain new visible light depth image pairs RD; For all visible light depth images in the RD, the image generation network is retrained using them. The visible light images in the RD are used as input, and the depth images are used as supervision information for the image generation network. After training, all visible light images in the original input visible light depth images are fed into the retrained image generation network to obtain the corresponding high-quality pseudo-depth images with consistent foreground.

4. The multimodal saliency target detection method based on feature correction followed by fusion according to claim 1, characterized in that, Considering that features at different scales contribute differently to the final saliency prediction, a scale-aware attention mechanism is constructed to adaptively fuse multi-scale features. First, the multi-scale features are fed into several SE blocks to obtain their scale attention weights. The values ​​are m=3, 5, 7, 9. Then, the multi-scale weight vector is obtained through the Softmax function. Finally, the multi-scale features are reweighted by the obtained weight vector to achieve the enhanced cross-modal features. .

5. The multimodal saliency target detection method based on feature-based correction and fusion according to claim 1, characterized in that, The region consistency perception loss function, by simultaneously considering the local region consistency of the foreground saliency target and the background region consistency, further obtains a more complete foreground and achieves better suppression of background interference. Specifically, it includes: First, calculate the false shadow region within the foreground salience target area. FP represents the false positive region in the background region, S represents the predicted saliency map, and G represents the true value. For a given FN and FP, the region consistency-aware loss function is... It can be calculated using the following formula: ， ， here Represents along spatial dimensions Average pooling, where H and W represent the height and width of the saliency map respectively, and 1 represents a vector of all 1s with the same dimension as G, enhances the local consistency of the foreground saliency region and the consistency of the background region by reducing the difference between each pixel and its surrounding pixels.

6. The multimodal saliency target detection method based on feature correction followed by fusion according to claim 1, characterized in that, The fused cross-modal features are fused in a stepwise manner during the decoding stage to obtain the final saliency map of the fused feature prediction. The specific steps to obtain the saliency prediction map include: For the fusion feature through a A convolutional layer and a sigmoid function yield the final saliency map. .

7. The multimodal saliency target detection method based on feature correction followed by fusion according to claim 1, characterized in that, The saliency prediction map uses a supervised learning model to obtain network model parameters, specifically including: On the training dataset, a supervised learning model is used to train the algorithm network end-to-end on the predicted saliency map, resulting in the network model parameters: On the training dataset, a supervised learning mechanism is employed to calculate the loss function between the saliency map prediction results and the ground truth values ​​in the network model. : ， in and These are the cross-entropy loss function and the cross-union ratio-based loss function, respectively. The definitions of the two are as follows: in The label for each pixel that represents the true value; This is used to predict the probability of each pixel in the saliency map; W is the width of the input image, and H is the height of the input image.

Images