RGB-D saliency target detection method in multi-target scene

Abstract

The invention discloses an RGB-D saliency target detection method in a multi-target scene, and relates to the technical field of image processing. According to the method, firstly, an estimated depth map is introduced, the contribution of a depth mode to saliency map prediction is improved in combination with an original depth map, and utilization of depth map space information is improved; secondly, designing a saliency detection module capable of being combined with multi-scale features, enhancing the consistency of each mode for distinguishing a saliency target, and reducing the saliency difference before multi-mode information fusion; and finally, designing a specific fusion module for aligning the significance of the RGB and the depth mode.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an RGB-D salient target detection method in a multi-target scene. Background Art

[0002] Salient object detection is a computer vision task that simulates the human visual attention mechanism and is used to accurately segment the most visually attractive objects in a scene pixel by pixel. Salient object detection has been widely used in satellite remote sensing, target detection, video segmentation, target tracking, smart devices, weather forecasting, portrait photography and other fields.

[0003] Initially, salient object detection used only RGB images as input. However, the ability of a single sensor to obtain information in complex scenes is limited, which will lead to a decrease in detection accuracy. Therefore, depth maps are introduced as auxiliary information in SOD. Current RGB-D salient object detection methods are mostly based on two-stream models and have achieved good performance in single-target scenarios. However, in multi-target scenarios, there is inconsistent saliency between the RGB modality and the depth modality, which will destroy the subsequent fusion results. In scenes where there is only a single object in the RGB and depth maps and there is a significant difference in the depth of the foreground and background, current RGB-D SOD methods can generate accurate saliency maps. However, in scenes with multiple similar objects, although the depth map has highlighted obvious salient objects, the salient objects predicted by current methods all contain a part of non-salient objects. In addition, when the salient objects predicted by the RGB or depth modality are accurate, some erroneous non-salient objects can be eliminated by the fusion block. However, when the saliency results are heavily dependent on a certain modality and there are obvious errors in the saliency of this modality, or when there are obvious errors in the saliency of both modalities, no fusion method can eliminate these errors. This saliency inconsistency often occurs in multi-target scenes, greatly reducing the accuracy of salient target detection.

[0004] There are three main reasons for the inconsistent saliency: First, there are a large number of low-quality depth maps in the current public datasets. These depth maps have problems such as missing object parts, object edge artifacts, and confusing depth values. The models trained using these low-quality depth maps rely more on the RGB modality to predict the saliency map, weakening the utilization of the spatial information of the depth map, which is crucial for the task of salient object detection. In addition, the state-of-the-art RGB-based depth estimation methods can generate accurate depth maps in most cases, but have errors in estimating the depth of white objects, thin objects, and objects with surface reflections. The artifacts, missing depth values, and confusion of depth values ​​in the depth map make the depth modality unreliable, causing the results to rely more on the RGB modality. Second, in multi-target scenes, the saliency of the RGB modality and the depth modality are usually different. The RGB modality pays more attention to the unified whole of color prominence, and the depth modality pays more attention to objects close to the front. How to eliminate the difference and extract the saliency of each modality is a challenging problem. Third, the complementarity of RGB and depth information is still a problem worthy of in-depth exploration. In the fusion stage, there is still a difference in the saliency of the RGB modality and the depth modality. When both modalities have high confidence in non-salient objects, it is difficult for existing fusion methods to eliminate non-salient objects. Therefore, how to adjust and fuse the saliency of the two modalities is a challenge.

[0005] The current RGB-D salient object detection model cannot accurately extract salient objects in multi-object scenarios, resulting in the salient prediction results often containing non-salient objects. This is because the salient results are more dependent on the RGB modality and the saliency of the RGB modality and the depth modality are quite different before fusion.

[0006] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0007] The purpose of the present invention is to provide an RGB-D salient target detection method in a multi-target scene. First, an estimated depth map is introduced, and the original depth map is combined to jointly improve the contribution of the depth modality to the prediction of the saliency map, and improve the utilization of the spatial information of the depth map; secondly, a saliency detection module that can combine multi-scale features is designed to enhance the consistency of each modality in distinguishing salient targets and reduce the saliency difference before multi-modal information fusion; finally, a specific fusion module is designed to align the saliency of RGB and depth modalities to solve the technical problems raised in the background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a RGB-D salient target detection method in a multi-target scene, comprising at least the following steps:

[0009] S1: Estimate the depth map. Use an open source model based on RGB to estimate the depth map. Send all RGB images in the RGB-D public dataset into the network to generate an estimated depth map.

[0010] S2: Set the model architecture, design the model according to the architecture, and sequentially design the depth recovery module, the saliency detection module and the specificity fusion module in the architecture. The depth recovery module uses the estimated depth map and the original depth map to correct the inaccurate depth. The state saliency detection module is used to combine the features of adjacent layers, improve the detection capability of salient objects of different scales, significantly enhance the robustness and reliability of the features of the current layer, and through the guidance of deep semantics, the multi-scale features quickly focus on the salient objects, and through the supervision of the RGB modality and the depth modality, adjust the saliency of the two modalities to maintain consistent saliency. The specificity fusion module is used to adjust the saliency of the modality to make it consistent and finally fuse the features;

[0011] S3: Design and apply loss function;

[0012] S4: The RGB and depth images in the RGB-D training set used for model training and the estimated depth map generated by S1 corresponding to the training set are cropped to a size of 352×352 and then fed into this model. The model is trained using a public training method to generate predicted salient objects.

[0013] S5: After the model training is completed, the public RGB-D dataset and the estimated depth map and predicted depth map generated by S1 are fed into the model. The saliency map generated by the last layer of the fusion modality is the model prediction result. The effectiveness of the prediction results of this model is verified using the four public evaluation indicators.

[0014] Furthermore, the model in S2 is composed of three independent ResNet-based encoders, which are used to extract RGB features, original depth features and estimated depth features at five scales, and load ResNet pre-trained weights. According to the model architecture diagram of the present invention, three ResNet networks are used as encoding layers, and decoding networks of RGB modality, depth modality and fusion modality are designed to restore the image to the original size.

[0015] Furthermore, the application of the model in S2 comprises at least the following steps:

[0016] S211: First, the extracted original depth features and estimated depth features of each scale are fused through a depth recovery module to obtain a recovered depth feature;

[0017] S212: Then, the RGB features and the restored depth features of adjacent scales are respectively sent to the modality saliency detection module to enhance the saliency of each modality;

[0018] S213: After obtaining reliable RGB and depth information, fuse them through a specific fusion module to obtain fusion features;

[0019] S214: Finally, the extracted multimodal features are sent to the decoder of the specific modality to generate saliency maps of the RGB modality, depth modality, and fusion modality, and supervise the losses of the three modalities.

[0020] Furthermore, the application of the depth recovery module includes at least the following steps:

[0021] S221: First, the residual dual-channel attention is used to optimize the salient target depth values ​​of the estimated depth features and the original depth features, thereby enhancing the importance of the single-peak features in the channel dimension and suppressing the interference caused by inaccurate depth values;

[0022] S222: For the original deep features, global average pooling and global maximum pooling with parallel structures are used to better suppress irrelevant information in shallow features and capture global contextual relationships in deep features;

[0023] S223: The two weighted features generated by S222 and S221 are multiplied by the original depth features respectively, and the two optimized features are connected along the channel dimension;

[0024] S224: Then, a 3×3 convolution layer is used to compress the feature to the original size, and it is added element by element with the original depth feature, and each depth feature is refined through the residual connection to obtain the enhanced original depth feature, and the enhanced estimated depth feature can be obtained;

[0025] S225: Next, the importance of the unimodal features in the channel dimension is re-weighted using the enhanced estimated depth features, and unreliable depth values ​​in the improved original depth features are reduced;

[0026] S226: Subsequently, a global average pooling method is used to adjust the contribution of the enhanced original depth features and the enhanced estimated depth features to the restored depth, and the weighted features are concatenated along the channel dimension to obtain a mixed depth feature. Considering the significant foreground and background differences in the enhanced estimated depth map, spatial attention is used to enhance the saliency of the mixed depth feature.

[0027] S227: Finally, the mixed depth features are combined with the enhanced estimated depth map to further enrich the details and obtain the final depth features.

[0028] Further, the modal saliency detection module includes a three-branch structure, and the three-branch structure includes a current layer feature enhancement branch and two spatial attention guided feature selection branches;

[0029] The modal saliency detection module connecting the first layer features with the second layer features and connecting the fourth layer features with the fifth layer features respectively adopts two branches;

[0030] The modal saliency detection module that connects the first layer features and the second layer features adopts the current layer feature enhancement branch and the adjacent next layer spatial attention guided feature selection branch;

[0031] The modal saliency detection module connecting the fourth and fifth layer features adopts the current layer feature enhancement branch and the adjacent previous layer spatial attention guided feature selection branch;

[0032] The modality saliency detection module further includes a weighted context enhancement feature block, which is used to obtain richer current layer features without increasing the number of parameters.

[0033] Furthermore, the application of the modality saliency detection module includes at least the following steps:

[0034] S231: Four parallel branches are designed, each using a different dilated convolution layer, using the method of gradually enhancing the receptive field of multi-scale objects, and then connecting them. The dilated convolution hyperparameters d of the four parallel branches are 1, 2, 3, and 4 respectively;

[0035] S232: Then, the current layer features are adaptively refined using subtle channel attention and spatial attention to further enhance their performance. Considering the different scales of features in adjacent layers, the features of the previous layer are downsampled and the features of the next layer are upsampled. The deeper features contain richer semantic information, which can help quickly identify the salient objects in each modality. The spatial attention of the deep features is used to guide the shallow features to pay more attention to the salient objects.

[0036] S233: Finally, the three branches are connected with the original features of the current layer to obtain consistent saliency features.

[0037] Furthermore, the application of the specific fusion module comprises at least the following steps:

[0038] S241: First, channel attention and spatial attention are used to refine each modality feature, and a cross-modal attention block is constructed to promote the mutual learning of RGB features and deep features, thereby achieving mutual complementation of modality information;

[0039] S2411: The cross-modal attention block explores the interrelationships between different modalities from two perspectives: channel and space. First, it is refined by channel attention enhancement;

[0040] S2412: then multiplied by another modal feature;

[0041] S2413: Next, the features obtained in the previous step guide the learning of the salient target of the modality by the features of another modality through spatial attention;

[0042] S2414: Finally, the obtained features, the enhanced features of each modality, and the original features are fused by element addition to generate RGB and depth features containing complementary information, and the RGB and depth features with complementary information are further fused by element multiplication and addition to eliminate the inconsistency of saliency;

[0043] S242: Finally, the fusion features of this layer are connected in series with the fusion features obtained from the previous layer to obtain the final fusion features.

[0044] Furthermore, the S3 at least includes the following steps:

[0045] Use binary cross entropy loss and intersection-over-union loss to supervise the saliency maps of RGB modality, depth modality and fusion modality;

[0046] The RGB modality and depth modality losses are defined as follows:

[0047] L mod =L bce (GT, P mod )+L Iou (GT, P mod )

[0048] Where L mod represents the loss of RGB modality and depth modality, L bce represents the binary cross entropy loss, L Iou represents the intersection-over-union loss, GT represents the label, and mod represents the saliency map of the last layer of the RGB modality and the depth modality in the decoder;

[0049] For the fusion modality, the saliency maps of the last two decoder layers are supervised, and the fusion modality loss is defined as follows:

[0050] L fusion =L bce (GT, P i )+L Iou (GT, P i )

[0051] Among them, P i Represents the saliency map of the last two layers of the fused modality;

[0052] Finally, the total loss is:

[0053] L total =L RGB +L depth+L fusion .

[0054] Further, the four evaluation indicators in S5 include F-measure, mean absolute error, S-measure and E-measure;

[0055] The F-measure is F β , the F β It is a region-based similarity index that takes into account the precision P and recall R, β 2 The value of is 0.3;

[0056] The mathematical formula is expressed as:

[0057]

[0058] The mean absolute error refers to the degree of approximation between the saliency map S and the true value label G at the pixel level, and its mathematical expression formula is:

[0059]

[0060] The S-measure is S m , the S m To evaluate the object perception of saliency maps (S 0 ) and region awareness (S r )The similarity between the structure and the label, the value of α is set to 0.5;

[0061] The S m The mathematical formula is:

[0062] S m =α·S 0 +(1-α)·S r

[0063] The E-measure is E m , the E m To evaluate image-level statistics and local pixel matching information, the E m The mathematical expression is:

[0064]

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] The present invention is reasonably designed and expanded based on the convolutional neural network with multi-stream structure;

[0067] First, the model innovation in the present invention introduces an estimated depth map to improve the quality of the depth map, solving the problem that the original depth map is of poor quality and causes the saliency results to rely more on RGB images;

[0068] Then, multi-layer features are extracted from the RGB image and the adjusted depth map respectively, and the multi-layer adjacent features of each modality are combined. The deep semantic information is used to guide the attention of shallow features to shift to the salient targets layer by layer, breaking the limitation of direct fusion after extracting multi-modal features by the previous backbone network.

[0069] At the same time, it combines style transfer and contrastive learning methods, uses style transfer to generate more diverse and rich samples for contrastive learning, and then trains 3DCNN, which enhances the robustness of the model for different populations and different data sets, and has better performance and generalization ability than other excellent remote physiological measurement methods;

[0070] Compared with other models, the model in the present invention has better practicality in terms of performance and parameter quantity. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0072] Figure 1 Schematic diagram of the RGB-D salient object detection model architecture of the present invention;

[0073] Figure 2 Schematic diagram of the depth recovery module of the present invention;

[0074] Figure 3 Schematic diagram of the modal saliency detection module of the present invention;

[0075] Figure 4 Schematic diagram of the specific fusion module of the present invention;

[0076] Figure 5 A visual comparison diagram of the depth map restored by the depth restoration module provided by the present invention, the original depth map and the estimated depth map;

[0077] Figure 6 A comparison chart of the visualization results of salient object detection of different models provided by the present invention on 9 public RGB-D datasets. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0079] A RGB-D salient object detection method in a multi-object scene includes at least the following steps:

[0080] S1: Estimate the depth map. Use an open source model based on RGB depth map estimation, and send all RGB images in the RGB-D public dataset into the network to generate an estimated depth map. The public RGB-D datasets required for the training and testing stages of this embodiment include NJU2K, NLPR, STERE, DES, SSD, SIP, LFSD, DUT-RGBD, and COME15K.

[0081] S2: Set the model architecture and design the model according to the architecture. In the architecture, the depth recovery module, the saliency detection module and the specificity fusion module are designed in sequence. The depth recovery module uses the estimated depth map and the original depth map to correct the inaccurate depth. The state saliency detection module is used to combine the features of adjacent layers, improve the detection ability of salient objects of different scales, and significantly enhance the robustness and reliability of the current layer features. Through the guidance of deep semantics, multi-scale features quickly focus on salient objects, and through the supervision of RGB modality and depth modality, adjust the saliency of the two modalities to maintain consistent saliency. The specificity fusion module is used to adjust the saliency of the modality to keep it consistent and finally fuse the features.

[0082] S3: Design and apply loss function;

[0083] S4: The RGB and depth images in the RGB-D training set used for model training and the estimated depth map generated by S1 corresponding to the training set are cropped to a size of 352×352 and then sent to this model. The model is trained using a public training method to generate predicted salient targets; the RGB-D training set required for this embodiment includes 1485 samples of NJU2K-train and 700 samples of NLPR-train.

[0084] S5: After the model training is completed, the public RGB-D dataset and the estimated depth map and predicted depth map generated by S1 are fed into the model. The saliency map generated by the last layer of the fusion modality is the model prediction result. The effectiveness of the prediction results of this model is verified using the four public evaluation indicators.

[0085] 1. Overall model architecture

[0086] The schematic diagram of the RGB-D salient object detection model architecture designed by the present invention to solve the saliency inconsistency problem is as follows: Figure 1shown.

[0087] The model in S2 consists of three independent ResNet-based encoders, which are used to extract RGB features, original depth features and estimated depth features at five scales, and load ResNet pre-trained weights. According to the model architecture diagram of the present invention, three ResNet networks are used as encoding layers, and decoding networks of RGB modality, depth modality and fusion modality are designed to restore the image to the original size.

[0088] The application of the model in S2 includes at least the following steps:

[0089] First, the original depth features and estimated depth features extracted at each scale are fused through the depth recovery module to obtain the restored depth features. Then, the RGB features and restored depth features of adjacent scales are sent to the modality saliency detection module respectively to enhance the saliency of each modality. After obtaining reliable RGB and depth information, they are fused through the specific fusion module to obtain the fused features. Finally, the extracted multimodal features are sent to the decoder of the specific modality to generate the saliency maps of the RGB modality, depth modality and fused modality, and supervise the losses of the three modalities.

[0090] 2. Deep recovery module

[0091] The original depth map contains wrong depth values, such as artifacts, depth loss and depth inconsistency. Current depth estimation methods achieve good results on object outlines, but do not work well on white objects. Therefore, the present invention proposes a depth recovery block that uses the estimated depth map and the original depth map to correct the inaccurate depth.

[0092] Figure 2The schematic diagram of the depth correction module proposed in the present invention is shown. First, we use residual dual-channel attention to optimize the salient target depth values ​​of the estimated depth features and the original depth features. This method enhances the importance of unimodal features in the channel dimension and suppresses the interference caused by inaccurate depth values. For the original depth features, we use global average pooling and global maximum pooling with parallel structures to better suppress irrelevant information in shallow features and capture global contextual relationships in deep features. The two weighted features generated by the above operations are multiplied with the original depth features respectively, and the two optimized features are connected along the channel dimension. Then, a 3×3 convolution layer is used to compress the features to the original size and add them element-by-element to the original depth features. Each depth feature is refined by residual connection to obtain the enhanced original depth features. The enhanced estimated depth features can be obtained by the same operation as above. Next, we use the enhanced estimated depth features to reweight the importance of unimodal features in the channel dimension and reduce the unreliable depth values ​​in the improved original depth features. Subsequently, the global average pooling method is used to adjust the contribution of the enhanced original depth features and the enhanced estimated depth features to the restored depth. We concatenate the weighted features along the channel dimension to obtain the hybrid depth features. Considering the significant foreground-background difference in the enhanced estimated depth map, we use spatial attention to enhance the saliency of the hybrid depth features. Finally, we combine the hybrid depth features with the enhanced estimated depth map to further enrich the details and obtain the final depth features. In order to overcome the limitations of the original depth map and the estimated depth map, we combine the original depth map and the estimated depth map for depth recovery. Figure 5 As shown in Figure 1, we compare the depth map of the dog restored by the depth recovery module with the original depth map and the estimated depth map. Rows 1-3 show the missing depth values ​​of the white objects in the estimated depth map, but our depth map is able to combine the information of the original depth to alleviate this problem. Rows 4-6 show the problems of missing depth values, artifacts, and confusion of depth values ​​in the original depth map, respectively. This module can effectively integrate this information from the estimated depth map to obtain good results.

[0093] 3. Modal saliency detection module

[0094] The multimodal features obtained after encoding contain rich object information. However, these features have obvious differences in saliency. Directly aligning multimodal features will result in some erroneous information not being eliminated, which limits the accuracy of the fusion results. The rich adjacent semantic information enables multi-scale features to quickly locate salient objects. Before fusing multimodal features, we design a modality saliency detection block.

[0095] Figure 3The schematic diagram of the modal saliency detection module proposed in the present invention is shown in FIG. The specific structure of the module is as follows: Figure 3 As shown on the left, it has a three-branch structure, including a current-layer feature enhancement branch and two spatial attention-guided feature selection branches.

[0096] The modal saliency detection modules connecting the first and second layer features and connecting the fourth and fifth layer features respectively adopt two branch structures.

[0097] The module connecting the first and second layer features contains the feature enhancement branch of the current layer and the adjacent spatial attention-guided feature selection branch of the next layer;

[0098] The module connecting the fourth and fifth layer features contains the feature enhancement branch of the current layer and the spatial attention guided feature selection branch of the adjacent previous layer;

[0099] Since multi-scale features have the same fixed receptive field, they only provide object information to the decoder in a limited domain. In order to enhance the current layer features, such as Figure 3 As shown on the right, we design a weighted context-enhanced feature block to obtain richer current layer features without increasing the number of parameters.

[0100] Specifically, similar to existing methods, we design four parallel branches, each using a different dilated convolutional layer. However, instead of directly connecting four different receptive fields, we adopt a method of gradually enhancing the receptive fields of multi-scale objects and then connecting them. The dilated convolution hyperparameters d of the four parallel branches are 1, 2, 3, and 4, respectively. Subsequently, we use subtle channel attention and spatial attention to adaptively refine the current layer features to further enhance their performance. Considering the different scales of features in adjacent layers, the features of the previous layer are downsampled and the features of the next layer are upsampled. Deeper features contain richer semantic information and can help quickly identify salient objects in each modality. We use the spatial attention of deep features to guide shallow features to pay more attention to salient objects. Finally, the three branches are connected with the original features of the current layer to obtain consistent salient features.

[0101] In summary, the modal saliency detection module combines features from adjacent layers to improve the detection capability of salient objects of different scales and significantly enhances the robustness and reliability of features in the current layer. Guided by deep semantics, multi-scale features quickly focus on salient objects. By supervising the RGB modality and the depth modality, the saliency of the two modalities is adjusted to maintain consistent saliency.

[0102] 4. Specific fusion module

[0103] The fusion of RGB and depth features faces two key issues. The first issue is to complement each other with scene content information. RGB features and depth features are interested in different information of objects, and we need to consider how to make them complement each other. The second issue is modality mismatch, which requires addressing the modality difference between RGB and depth modalities. The saliency of RGB is usually attracted by brightly colored objects, while the saliency of depth is usually concentrated on the nearest objects. Therefore, adjusting the saliency of the modality is a crucial issue. Based on the above views, we designed a specific fusion module.

[0104] The application of the specific fusion module comprises at least the following steps:

[0105] like Figure 4 As shown on the left, we first use channel attention and spatial attention to refine each modal feature. In order to achieve the mutual complementation of modal information, such as Figure 4 As shown on the right, we construct a cross-modal attention block (CMA) to promote the mutual learning of RGB features and deep features. Specifically, CMA explores the relationship between different modalities from two perspectives: channel and space. First, it is refined through channel attention enhancement and then multiplied with the features of another modality. Next, the features obtained in the previous step guide the learning of the saliency target of the modality by the features of the other modality through spatial attention. Finally, we fuse the obtained features, the enhanced features of each modality, and the original features through element-wise addition to generate RGB and deep features containing complementary information. The RGB and deep features with complementary information are further fused through element-wise multiplication and addition to eliminate the inconsistency of saliency. Finally, we concatenate the fused features of this layer with the fused features obtained from the previous layer to obtain the final fused features.

[0106] S3 includes at least the following steps:

[0107] Use binary cross entropy loss and intersection-over-union loss to supervise the saliency maps of RGB modality, depth modality and fusion modality;

[0108] The RGB modality and depth modality losses are defined as follows:

[0109] L mod =L bce (GT, P mod )+L Iou (GT, P mod )

[0110] Where L mod represents the loss of RGB modality and depth modality, L bce represents the binary cross entropy loss, L Iourepresents the intersection-over-union loss, GT represents the label, and mod represents the saliency map of the last layer of the RGB modality and the depth modality in the decoder;

[0111] For the fusion modality, the saliency maps of the last two decoder layers are supervised, and the fusion modality loss is defined as follows:

[0112] L fusion =L bce (GT, P i )+L Iou (GT, P i )

[0113] Among them, P i Represents the saliency map of the last two layers of the fused modality;

[0114] Finally, the total loss is:

[0115] L total =L RGB +L depth +L fusion .

[0116] The four evaluation indicators in S5 include F-measure, mean absolute error, S-measure, and E-measure;

[0117] F-measure is F β , F β It is a region-based similarity index that takes into account the precision P and recall R, β 2 The value of is 0.3;

[0118] The mathematical formula is:

[0119]

[0120] The mean absolute error (M) refers to the degree of approximation between the saliency map S and the true value label G at the pixel level. Its mathematical expression is:

[0121]

[0122] S-measure is S m , S m To evaluate the object perception of saliency maps (S 0 ) and region awareness (S r )The similarity between the structure and the label, the value of α is set to 0.5;

[0123] S m The mathematical formula is:

[0124] Sm =α·S 0 +(1-α)·S r

[0125] E-measure is E m , E m To evaluate image-level statistics and local pixel matching information, E m The mathematical expression is:

[0126]

[0127] Table 1 Comparison results of four evaluation indicators of the present invention on 9 public data sets (bold represents the best)

[0128]

[0129]

[0130] The present invention uses 9 publicly recognized RGB-D datasets to train and verify the effectiveness of the model and method proposed in the present invention. All compared models use public source code and the same training and testing methods as this method. The training set uses 1485 samples contained in NJU2K-train and 700 samples contained in NLPR-train, and the remaining datasets are used to verify the effectiveness of the present invention. When measuring the performance of the experimental test method, the following indicators are used to evaluate the measurement model: S-measure, F-measure, E-measure, and mean absolute error (MAE).

[0131] The method proposed in this invention is compared with many excellent salient object detection methods. The verification results on 9 data sets are shown in Table 1. The visual comparison results are shown in Figure 6 It can be seen that in the multi-target scenario, the model proposed in the present invention can better detect salient targets without being disturbed by the background, and has better accuracy and robustness. It is worth emphasizing that on the largest validation dataset CME15K, our model shows excellent performance, which shows that the model has excellent generalization performance. All experiments show that the model proposed in the present invention achieves SOTA performance in comparing salient target detection methods.

[0132] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A RGB-D salient object detection method in a multi-object scene, characterized by: At least the following steps are included: S1: Estimate the depth map. Use an open source model based on RGB to estimate the depth map. Send all RGB images in the RGB-D public dataset into the network to generate an estimated depth map. S2: Set the model architecture, design the model according to the architecture, and sequentially design the depth recovery module, the saliency detection module and the specificity fusion module in the architecture. The depth recovery module uses the estimated depth map and the original depth map to correct the inaccurate depth. The state saliency detection module is used to combine the features of adjacent layers, improve the detection capability of salient objects of different scales, significantly enhance the robustness and reliability of the features of the current layer, and through the guidance of deep semantics, the multi-scale features quickly focus on the salient objects, and through the supervision of the RGB modality and the depth modality, adjust the saliency of the two modalities to maintain consistent saliency. The specificity fusion module is used to adjust the saliency of the modality to make it consistent and finally fuse the features; S3: Design and apply loss function; S4: The RGB and depth images in the RGB-D training set used for model training and the estimated depth map generated by S1 corresponding to the training set are cropped to a size of 352×352 and then fed into this model. The model is trained using a public training method to generate predicted salient objects. S5: After the model training is completed, the public RGB-D dataset and the estimated depth map and predicted depth map generated by S1 are fed into the model. The saliency map generated by the last layer of the fusion modality is the model prediction result. The effectiveness of the prediction results of this model is verified using the four public evaluation indicators.

2. The RGB-D salient target detection method in a multi-target scene according to claim 1, characterized in that: The model in S2 is composed of three independent ResNet-based encoders, which are used to extract RGB features, original depth features and estimated depth features at five scales, and load ResNet pre-trained weights. According to the model architecture diagram of the present invention, three ResNet networks are used as encoding layers, and decoding networks of RGB modality, depth modality and fusion modality are designed to restore the image to the original size.

3. The RGB-D salient target detection method in a multi-target scene according to claim 2, characterized in that: The application of the model in S2 comprises at least the following steps: S211: First, the extracted original depth features and estimated depth features of each scale are fused through a depth recovery module to obtain a recovered depth feature; S212: Then, the RGB features and the restored depth features of adjacent scales are respectively sent to the modality saliency detection module to enhance the saliency of each modality; S213: After obtaining reliable RGB and depth information, fuse them through a specific fusion module to obtain fusion features; S214: Finally, the extracted multimodal features are sent to the decoder of the specific modality to generate saliency maps of the RGB modality, depth modality, and fusion modality, and supervise the losses of the three modalities.

4. The RGB-D salient target detection method in a multi-target scene according to claim 1, characterized in that: The application of the depth recovery module comprises at least the following steps: S221: First, the residual dual-channel attention is used to optimize the salient target depth values ​​of the estimated depth features and the original depth features, thereby enhancing the importance of the single-peak features in the channel dimension and suppressing the interference caused by inaccurate depth values; S222: For the original deep features, global average pooling and global maximum pooling with parallel structures are used to better suppress irrelevant information in shallow features and capture global contextual relationships in deep features; S223: The two weighted features generated by S222 and S221 are multiplied by the original depth features respectively, and the two optimized features are connected along the channel dimension; S224: Then, a 3×3 convolution layer is used to compress the feature to the original size, and it is added element by element with the original depth feature, and each depth feature is refined through the residual connection to obtain the enhanced original depth feature, and the enhanced estimated depth feature can be obtained; S225: Next, the importance of the unimodal features in the channel dimension is re-weighted using the enhanced estimated depth features, and unreliable depth values ​​in the improved original depth features are reduced; S226: Subsequently, a global average pooling method is used to adjust the contribution of the enhanced original depth features and the enhanced estimated depth features to the restored depth, and the weighted features are concatenated along the channel dimension to obtain a mixed depth feature. Considering the significant foreground and background differences in the enhanced estimated depth map, spatial attention is used to enhance the saliency of the mixed depth feature. S227: Finally, the mixed depth features are combined with the enhanced estimated depth map to further enrich the details and obtain the final depth features.

5. The RGB-D salient target detection method in a multi-target scene according to claim 1, characterized in that: The modal saliency detection module includes a three-branch structure, wherein the three-branch structure includes a current layer feature enhancement branch and two spatial attention guided feature selection branches; The modal saliency detection module connecting the first layer features with the second layer features and connecting the fourth layer features with the fifth layer features respectively adopts two branches; The modal saliency detection module that connects the first layer features and the second layer features adopts the current layer feature enhancement branch and the adjacent next layer spatial attention guided feature selection branch; The modal saliency detection module connecting the fourth and fifth layer features adopts the current layer feature enhancement branch and the adjacent previous layer spatial attention guided feature selection branch; The modality saliency detection module further includes a weighted context enhancement feature block, which is used to obtain richer current layer features without increasing the number of parameters.

6. The RGB-D salient target detection method in a multi-target scene according to claim 5, characterized in that: The application of the modality saliency detection module comprises at least the following steps: S231: Four parallel branches are designed, each using a different dilated convolution layer, using the method of gradually enhancing the receptive field of multi-scale objects, and then connecting them. The dilated convolution hyperparameters d of the four parallel branches are 1, 2, 3, and 4 respectively; S232: Then, the current layer features are adaptively refined using subtle channel attention and spatial attention to further enhance their performance. Considering the different scales of features in adjacent layers, the features of the previous layer are downsampled and the features of the next layer are upsampled. The deeper features contain richer semantic information, which can help quickly identify the salient objects in each modality. The spatial attention of the deep features is used to guide the shallow features to pay more attention to the salient objects. S233: Finally, the three branches are connected with the original features of the current layer to obtain consistent saliency features.

7. The RGB-D salient target detection method in a multi-target scene according to claim 1, characterized in that: The application of the specific fusion module comprises at least the following steps: S241: First, channel attention and spatial attention are used to refine each modality feature, and a cross-modal attention block is constructed to promote the mutual learning of RGB features and deep features, thereby achieving mutual complementation of modality information; S2411: The cross-modal attention block explores the interrelationships between different modalities from two perspectives: channel and space. First, it is refined by channel attention enhancement; S2412: then multiplied by another modal feature; S2413: Next, the features obtained in the previous step guide the learning of the salient target of the modality by the features of another modality through spatial attention; S2414: Finally, the obtained features, the enhanced features of each modality, and the original features are fused by element addition to generate RGB and depth features containing complementary information, and the RGB and depth features with complementary information are further fused by element multiplication and addition to eliminate the inconsistency of saliency; S242: Finally, the fusion features of this layer are connected in series with the fusion features obtained from the previous layer to obtain the final fusion features.

8. The RGB-D salient target detection method in a multi-target scene according to claim 1, characterized in that: The S3 at least includes the following steps: Use binary cross entropy loss and intersection-over-union loss to supervise the saliency maps of RGB modality, depth modality and fusion modality; The RGB modality and depth modality losses are defined as follows: IT mod =L bce (GT,P mod )+L Iou (GT,P mod ) Where L mod represents the loss of RGB modality and depth modality, L bce represents the binary cross entropy loss, L Iou represents the intersection-over-union loss, GT represents the label, and mod represents the saliency map of the last layer of the RGB modality and the depth modality in the decoder; For the fusion modality, the saliency maps of the last two decoder layers are supervised, and the fusion modality loss is defined as follows: IT fusion =L bce (GT,P i )+L Iou (GT,P i ) Among them, P i Represents the saliency map of the last two layers of the fused modality; Finally, the total loss is: L total =L RGB +L depth +L fusion .

9. The RGB-D salient target detection method in a multi-target scene according to claim 1, characterized in that: The four evaluation indicators in S5 include F-measure, mean absolute error, S-measure and E-measure; The F-measure is F β , the F β It is a region-based similarity index that takes into account the precision P and recall R, β 2 The value of is 0.3; The mathematical formula is expressed as: The mean absolute error refers to the degree of approximation between the saliency map S and the true value label G at the pixel level, and its mathematical expression formula is: The S-measure is S m , the S m To evaluate the object perception (S0) and region perception (S r )The similarity between the structure and the label, the value of α is set to 0.5; The S m The mathematical formula is: S m =α·S0+(1-α)·S r The E-measure is E m , the E m To evaluate image-level statistics and local pixel matching information, the E m The mathematical expression is:

Images