Salient target detection methods for RGB-D images

Abstract

The present invention aims to provide a salient target detection method for RGB-D images, comprising: sampling RGB image and depth map samples at a specified number to form sampling data, and preprocessing the sampling data using data augmentation techniques to obtain data to be processed; in the cross-modal fusion of each layer in the feature extraction stage, using an attention mechanism to infer salient regions to determine the saliency degree of different regions, and fusing at each layer to obtain low-level features and high-level features; performing concatenation and convolution operations on the high-level features and the low-level features to obtain joint features of RGB features and depth features; dynamically allocating the weights of RGB image and depth map features according to the joint features of RGB features and depth features to obtain a salient target detection model; and realizing the salient target detection result according to the salient target detection model. The method described in this invention can detect salient targets well and handle scenes with low depth map contrast.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and RGB-D salient object detection, and particularly to a method for improving the performance of an RGB-D salient object detection system in the case of low-quality depth maps. Background Technology

[0002] In machine vision tasks, salient object detection (SOD) is a method that simulates the human visual perception system to locate the most attractive objects in a scene. Currently, salient object detection plays a crucial role in many real-world applications, such as semantic segmentation, object tracking, and action recognition. With advancements in sensing technology, depth sensors can effectively capture rich spatial depth information in a scene and generate depth maps, potentially improving the performance of salient object detection.

[0003] Currently, salient object detection in RGB-D images combining RGB and depth maps has received widespread attention and research. However, current RGB-D salient object detection methods are mainly based on two-stream models and multi-scale fusion strategies, utilizing depth information for cross-modal fusion, while neglecting the impact of depth map quality issues on salient map prediction in challenging environments such as foreground-background similarity and low contrast. Furthermore, the fusion process ignores the fact that different regions in the RGB and depth maps have varying importance, causing unnecessary additional information to act as noise in the network, making it difficult to generate accurate salient masks.

[0004] Therefore, a salient target detection method that can provide higher quality salientity map prediction results urgently needs to be studied. Summary of the Invention

[0005] The purpose of this invention is to provide a salient target detection method for RGB-D images, which can improve the performance of RGB-D salient target detection systems under low-quality depth map conditions, thereby solving at least one technical problem in the prior art.

[0006] The technical solution of this invention is:

[0007] A salient object detection method for RGB-D images, comprising:

[0008] Sample RGB and depth images of a specified number to form sampled data, and preprocess the sampled data using data augmentation techniques to obtain data to be processed;

[0009] In the cross-modal fusion of each layer in the feature extraction stage, attention mechanism is used to infer salient regions in order to determine the saliency of different regions. The fusion of each layer yields low-level features and high-level features.

[0010] In feature guidance, the high-level features and the low-level features are concatenated and convolved to obtain joint features of RGB features and depth features;

[0011] The weights of the RGB and depth map features are dynamically assigned based on the joint features of the RGB and depth features to obtain a salient target detection model.

[0012] The salient target detection results are achieved based on the salient target detection model.

[0013] The phrase "in the cross-modal fusion of each layer during the feature extraction stage, an attention mechanism is used to infer salient regions to determine the saliency of different regions, and the fusion of each layer yields high-level and low-level features" includes:

[0014] Collect a specified number of RGB and depth map samples, and process the RGB and depth map samples using data augmentation techniques;

[0015] The feature extraction network extracts features from the input RGB image and depth image samples; the feature extraction network uses an attention mechanism to optimize the features across modalities, and uses the optimized features as inputs to the next layer.

[0016] Low-level features are obtained by selecting specific level features and fusing them together; high-level features are obtained by fusing other level features and the output features of the dilated spatial convolution pooling pyramid.

[0017] The feature extraction network includes two independent VGG16 branches;

[0018] Each VGG16 branch is divided into 5 sub-layers;

[0019] The output of the fifth sublayer of VGG16 passes through a dilated spatial convolution pooling pyramid, which consists of four dilated convolutions with kernel sizes and dilation rates of [k=1, d=1; k=3, d=6; k=3, d=12; k=3, d=18]; where k represents the kernel size and d represents the dilation rate of the dilated convolution.

[0020] The phrase "using an attention mechanism for cross-modal feature optimization, and using each optimized feature as input to the next layer" includes:

[0021] The features of corresponding sub-layers in the two independent VGG16 branches within the feature extraction network are used as input for feature optimization, and spatial attention scores are obtained to obtain the spatial attention weight matrix S. * (S R S D The details are as follows:

[0022] S *=σ(F 1×1 (F 3×3 (F));

[0023] Where F represents RGB features or depth features; F 3×3 3×3 convolution for RGB features or depth features; F 1×1 σ is a 1×1 convolution of RGB features or depth features; σ is the Sigmoid function;

[0024] Based on the spatial attention score, the spatial attention weight matrix is ​​multiplied and summed element-wise to obtain the features of each level after spatial attention processing, as shown in the following formula:

[0025]

[0026] Among them, F * For the corresponding RGB feature or depth feature, input S * For spatial attention weight matrices of RGB features or depth features; These are features calculated using spatial attention; S R S is the spatial attention weight matrix for RGB features; D The spatial attention weight matrix for deep features;

[0027] A feature of C×H×W First, feed them into a 1×1 convolution; then, perform a spatial global average pooling (AvgPool) and a max pooling (MaxPool) to obtain two C×1×1 channel descriptions.

[0028] The two channel descriptions are summed and then passed through a Sigmoid activation function to obtain the channel attention weight coefficient C. * (C R C D ), C * C is the channel attention weight coefficient for RGB features. R Or the channel attention weight coefficient C of deep features D .

[0029] Through C * and Features obtained after channel attention processing

[0030]

[0031] The result The output obtained after the self-attention mechanism is optimized by features and used as the input of the next sub-layer.

[0032] Will Expanding in the third dimension, the expanded features are then processed by W. Q W K W V The linear matrix yields the Q, K, and V matrices;

[0033] We can obtain the following formula As input for the next sub-layer:

[0034]

[0035] Where, d K Let K represent the dimension; S be the Softmax function; M is the inverse operation of expanding the computed features, and then... Perform residual connection.

[0036] The phrase "selecting specific level features to obtain low-level features through feature fusion; and obtaining high-level features through feature fusion of other level features and the output features of dilated spatial convolutional pooling pyramids" includes:

[0037] The feature extraction network has two independent VGG16 branches: an RGB branch and a depth branch.

[0038] The VGG16 branch includes: 5 sub-layers and 1 void space convolutional pyramid;

[0039] The output features of sub-layers 1, 2, and 3 are used as low-level features for feature fusion, while the outputs of sub-layers 4 and 5, as well as the outputs of dilated spatial convolutional pooling pyramids, are used as high-level features for feature fusion.

[0040] After feature fusion, the low-level features and high-level features corresponding to the RGB branch and the depth branch are obtained respectively.

[0041] The phrase "concatenating and convolving the high-level features and the low-level features to obtain joint features of RGB features and depth features" includes:

[0042] By using high-level features to guide low-level features, the salient regions of low-level features are identified while preserving edge details, resulting in guided low-level features.

[0043] The salient regions of the guided low-level features and the high-level features are joined to obtain the final RGB features and depth map features.

[0044] The final RGB features and depth map features are concatenated to obtain the joint features of RGB features and depth features.

[0045] The step of guiding low-level features by high-level features includes:

[0046] High-level features are upsampled and preserved, and then concatenated with low-level features to obtain concatenated features;

[0047] The concatenated features are then passed through a convolutional layer to obtain features that combine edge refinement and distinct salient regions.

[0048] The guided low-level features are concatenated with the high-level features again and fed into a convolution to obtain the final features corresponding to RGB and depth, respectively. The final features are then concatenated into joint features along the channel dimension.

[0049] The phrase "dynamically assigning weights to the RGB and depth map features based on the joint features of the RGB and depth features to obtain a salient target detection model" includes:

[0050] The joint features are weighted to obtain the contribution weights of RGB features and depth features to the final prediction;

[0051] The contribution weights are multiplied by the joint features, and the result is obtained by a decoder to obtain the final salient prediction map.

[0052] The loss function is calculated by comparing the salient predicted value map with the label ground truth, and the feature extraction network is updated using gradient descent. When the loss value of the feature extraction network converges, the salient target detection model is obtained.

[0053] For the joint feature [F R ;F D Perform convolution, normalization, and ReLU activation;

[0054] After 4×4 average pooling, the joint features are flattened in the channel dimension;

[0055] We obtain two weight values ​​P R P D The vector W formed P , respectively representing F in the joint features R F D The contribution of the input feature matrix F is obtained using the following formula. All :

[0056] F All =[ F R×P R ;F D ×P D ]

[0057] F All The value is input into the decoder, and the final saliency map prediction value I is output. pred ;

[0058] And, sum the joint feature [F R ; F D in the channel dimension to obtain a single-channel matrix P s ; and obtain the distance matrix D s :

[0059] D s = |P s - [F R ; F D |;

[0060] Respectively obtain the distance average values in two ranges where the number of channels is 0 ≤ k < C and C ≤ k < 2C, and obtain the first average distance value D R and the second average distance value D D ;

[0061] Obtain the two elements of the true value W P corresponding to the vector W GT through the following formula:

[0062]

[0063] Use the L1 loss function to calculate the L1 distance between W<http: / / www.example.com / P and W GT , and calculate the loss value L wam through the following formula:

[0064]

[0065] where, i represents the number of weight values; the final features F R , F D .

[0066] The "updating the feature extraction network using the gradient descent method" includes:

[0067] Use a hybrid loss function to supervise the training of the salient object detection model, including the binary cross-entropy loss function, the intersection over union loss function, and the L wam loss function;

[0068] The hybrid loss function is designed as follows:

[0069] L = λ1L bce + λ2L iou + λ3L wam ;

[0070] where, L bce is the binary cross-entropy loss function, L iou is the intersection over union loss function, and λ1, λ2, and λ3 respectively represent the weighting coefficients;

[0071] The binary cross-entropy loss function is defined as follows:

[0072]

[0073] The crossover ratio loss function is defined as follows:

[0074]

[0075] Among them, I gt For truth labels, I gt The value of I is {0, 1}, where 0 represents a non-significant region and 1 represents a significant region; pred The predicted value; (x, y) represents the pixel position; I gt (x, y) represents the pixel value in the x-th row and y-th column of the truth label; I pred (x, y) represents the pixel value in the x-th row and y-th column of the predicted value label.

[0076] The beneficial effects of the present invention include at least the following:

[0077] The method described in this invention uses an attention mechanism to infer salient regions during cross-modal fusion at each layer in the feature extraction stage to determine the saliency of different regions. Simultaneously, convolution operations are performed on high-level and low-level features to identify salient regions of low-level features while preserving edge details. Furthermore, after performing a connection operation on the final features of the two branches, the weights of the RGB image and depth map features are dynamically assigned. The method described in this invention can effectively detect salient targets and handle scenarios with low depth map contrast well. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the overall architecture of the salient target detection model in the method described in this invention;

[0079] Figure 2 This is a flowchart of the target detection model training process in the method described in this invention;

[0080] Figure 3 This is a schematic diagram of feature optimization in the method described in this invention;

[0081] Figure 4 This is a schematic diagram of weight allocation in the method described in this invention;

[0082] Figure 5 This is a practical application effect diagram of the salient target detection model in the method described in this invention;

[0083] Figure 6 This is an illustration of another practical application of the salient target detection model in the method described in this invention. Detailed Implementation

[0084] The present application will now be further described with reference to the accompanying drawings.

[0085] To address the problems in existing technologies, this invention provides a salient object detection model that uses an attention mechanism to determine salient regions and performs multi-scale fusion during the feature extraction stage, and a method for detecting low-quality depth maps using this salient object detection model, thereby improving the quality of RGB-D based salient object detection. Simultaneously, at the output end, a dynamic weight allocation step is designed to determine the contribution of RGB image and depth map features to the salient map prediction. The method described in this invention will be explained in detail below through specific embodiments.

[0086] Specific Implementation Example I:

[0087] This embodiment is applicable to all current dual-stream salient target detection models. The following example uses two independent VGG16 branches as the backbone network.

[0088] This embodiment presents the overall network architecture and training flowchart of a salient object detection method for RGB-D images; as follows: Figure 1 The entire network architecture of the method described in this embodiment can be considered as an end-to-end iterative training architecture, which mainly consists of a feature extraction network, a feature optimization step, a feature fusion step, and a weight allocation step.

[0089] First, this invention samples a batch of data according to a specified quantity. Data augmentation techniques, such as flipping, translation, and color enhancement, are applied to the RGB and depth map samples in this batch of data. Second, in the feature extraction network, two independent VGG16 branches extract features from the input RGB and depth map batches. The feature extraction network has a total of 5 sub-layers and a dilated spatial convolutional pooling pyramid. At this stage, each sub-layer performs cross-modal fusion through a feature optimization step. In the feature optimization step, an attention mechanism is used to optimize the features across modalities, obtaining robust features at each level. The optimized features are then used as inputs for the next layer. Third, after passing through the feature extraction network, features at levels 1, 2, 3, 4, and 5 are obtained respectively.

[0090] In this embodiment, the features at levels 1, 2, and 3 are fused in the feature fusion step to obtain low-level features, and the features at levels 4 and 5 and the output of the dilated spatial convolutional pooling pyramid are fused in the feature fusion step to obtain high-level features. Then, the high-level features guide the low-level features to determine the salient regions of the low-level features and preserve edge details. Next, the guided features of the two branches and the high-level features are connected to obtain the final RGB features and depth map features. The final features are then concatenated to obtain the joint features of the RGB features and depth features.

[0091] The joint features described above are weighted to obtain the contribution weights of RGB features and depth features to the final prediction. The contribution weights are then multiplied with the joint features. Finally, the decoder obtains the final salient prediction map. The salient prediction map and the ground truth label are compared to calculate the loss function and the network is updated using gradient descent. When the network loss value converges, the training is complete, and the model can be used independently for salient object detection tasks.

[0092] The feature optimization steps in this embodiment are as follows: Figure 3 As shown, the feature extraction network consists of two independent VGG16 branches. One branch extracts feature values ​​from the RGB image, and the other branch extracts feature values ​​from the depth map. Each branch is divided into five sub-layers, each with convolution and pooling operations allocated according to the VGG16 standard. Furthermore, after feature extraction, each sub-layer undergoes a feature optimization step for bidirectional interaction, such as... Figure 3 As shown; in the last sub-layer, i.e., the 5th sub-layer, to obtain rich contextual information, a dilated spatial convolutional pooling pyramid is adopted. The kernel size and dilation rate of the four-layer convolutional combination of the dilated spatial convolutional pooling pyramid are represented as [k=1, d=1; k=3, d=6; k=3, d=12; k=3, d=18], where k represents the kernel size and d represents the dilation rate of the dilated convolution. The feature optimization step uses an attention model to find the most important parts in the network for processing.

[0093] from Figure 3 As can be seen, the input to the feature optimization step is the features of the corresponding sub-layers in the two branches. After the RGB features and depth map features enter the feature optimization step, the spatial attention score is calculated, resulting in a spatial attention weight matrix S of size 1×H×W. * (S R S D The details are as follows:

[0094] S * =σ(F 1×1 (F 3×3 (F)) (1)

[0095] The input is a C×H×W RGB feature or depth feature F (where C is the number of channels, H is the number of rows in the matrix, and W is the number of columns in the matrix), where F... 3×3 For a 3×3 convolution, F 1×1 It is a 1×1 convolution, and σ is the Sigmoid function.

[0096] Based on the spatial attention score, bidirectional interaction is performed using formula (2) to obtain the features after spatial attention calculation:

[0097]

[0098] Among them, F * For the corresponding RGB features and depth features as input, S * The spatial attention calculation result for the corresponding branch (the value can be S). R S D ), These are features calculated using spatial attention. in For the corresponding RGB features, (For the corresponding depth features).

[0099] Furthermore, the features after spatial attention calculation proceed to the next stage, where channel attention calculation is performed to measure the importance of features in different channels. Using formula (3), the channel attention is calculated to obtain C, which has a size of 1×1×C. * (C R C D ), where C * C is the channel attention weight coefficient for RGB features. R Or the channel attention weight coefficient C of deep features D :

[0100]

[0101] The input is a C×H×W feature. First, feed them into a 1×1 convolution F. 1×1 Then, a spatial global average pooling (AvgPool) and a max pooling (MaxPool) are performed to obtain two C×1×1 channel descriptions. The two channel descriptions are then added together and passed through a sigmoid activation function σ to obtain the channel attention weight coefficients C. * (C R C D ).

[0102] Furthermore, using formula (4) to convert C * and The features obtained by element-wise multiplication are those processed by channel attention.

[0103]

[0104] Finally, the result will be After passing through the self-attention mechanism, the final output after the feature optimization step is obtained, which is also the input of the next sub-layer. During the self-attention process, firstly... Expanding on the third dimension, which is about to The size changes from C×H×W to N×C, where N is the product of H and W; then, the expanded features are processed by W respectively. Q W K W V The linear matrix yields the Q, K, and V matrices; then, the output matrix is ​​obtained using formula (5). As input for the next sub-layer:

[0105]

[0106] Where, d K Let K represent the dimension, S be the Softmax function, and M be the operation that transforms the computed features from N×C to C×H×W, then... Perform residual connection.

[0107] The feature extraction network in this embodiment includes five sub-layers and one dilated spatial convolutional pooling pyramid. Specifically, the output features of sub-layers 1, 2, and 3 are fused as low-level features in the feature fusion step. Similarly, the outputs of sub-layers 4 and 5, as well as the output of the dilated spatial convolutional pooling pyramid, are fused as high-level features in the feature fusion step. The feature fusion step mainly consists of a 1×1 convolution and an upsampling function, ensuring that the three input feature matrices are of the same size. After fusion, the low- and high-level features of the RGB branch are obtained, denoted as F. LR F HR The low- and high-level features of deep branches are denoted as F. LD F HD .

[0108] Furthermore, in the process of guiding low-level features through high-level features, the high-level feature F is first... HR F HD Upsample and hold, then concatenate them into [F LR ;F HR ]、[F LD ;F HD The [;] symbol represents a concatenation operation, which passes the concatenated features through a convolutional layer to obtain a feature F that combines edge refinement and clear salient regions. LRO F LDO ;

[0109] Furthermore, the guided features are again related to F HR F HD Connect into [F LRO ;F HR ]、[F LDO ;F HD The data is fed into a convolutional operation to obtain the final RGB and depth features F. R F DFurthermore, the final features of the two branches are concatenated along the channel dimension as [F]. R ;F D The combined features are fed into the weight allocation step for weight calculation.

[0110] The weight allocation steps are as follows: Figure 4 As shown; in the weight allocation step, the joint feature [F] is... R ;F D The process involves convolution, normalization, and ReLU activation; followed by 4×4 average pooling, and then flattening the features along the channel dimension; finally, a linear function and a sigmoid function are applied to obtain a result consisting of two weight values ​​P. R P D The vector W formed P Its value lies in the interval (0,1), representing F respectively. R F D The contribution of the input feature matrix F of the decoder is then calculated using formula (6). All :

[0111] F All =[F R ×P R ;F D ×P D (6)

[0112] Finally, F All The input is fed into the decoder, where the decoding stage mainly consists of a convolution, an upsampling operation, and a sigmoid operation. The output is the final saliency map prediction value I. pred .

[0113] To optimize the weight allocation process, a corresponding loss function was introduced. A vector W was generated to provide ground truth values ​​to support the calculation of the loss value. P The corresponding truth value W GT First, regarding [F] R ;F D Summing along the channel dimension and then applying minimum-maximum normalization yields a single-channel matrix P. s Next, P is calculated using formula (7). s With [F R ;F D The absolute distances of each channel are used to obtain the distance matrix D. s :

[0114] D s =|P s -[F R ;F D ]| (7)

[0115] Furthermore, the resulting distance matrix Ds With a size of 2C×H×W, we calculate the average values within the ranges of the number of channels 0≤k<C and C≤k<2C respectively to obtain the average distance value D R and D D . Finally, we obtain the two elements of W through formula (8) GT :

[0116]

[0117] Next, the L1 loss function is used to calculate the L1 distance between W P and W GT , and the loss value L is calculated through formula (9) wam :

[0118]

[0119] The salient object detection model of this embodiment adopts an end-to-end iterative training method. The update strategy of this model is to update by the backpropagation gradient descent method according to the loss function until the loss function reaches a convergence state. In this embodiment, a hybrid loss function is preferably used to supervise the model training, including the binary cross-entropy (BCE) loss function, the intersection over union (IOU) loss function, and the L wam loss function. The IOU loss function focuses on the global structure, making the salient object have a clear boundary. Specifically, the hybrid loss function is designed as follows:

[0120] L = λ1I bce + λ2L iou + λ3L wam (10)

[0121] where L bce is the binary cross-entropy loss function, L iou is the intersection over union loss function, and λ1, λ2, and λ3 represent the weighting coefficients. Among them, the BCE loss function is the most classical loss function in the saliency detection task and is used to calculate the loss of each pixel independently. The IOU loss function focuses on the global structure, making the salient object have a clear boundary. In this model, the two are defined as follows:

[0122]

[0123]

[0124] In the above formula, I gt is the ground truth label, and the value of I gt is {0, 1}, where 0 represents the non-salient region and 1 represents the salient region; I pred is the predicted value, and I predThe value takes a probability value between [0,1]; where (x,y) represents the probability value when the pixel position is such that I gt (x, y) represents the pixel value in the x-th row and y-th column of the truth label, and similarly, I pred The same applies to (x, y). Thus, the technical solution of this invention has been fully described.

[0125] Verification process:

[0126] To further verify the effectiveness of the method described in this invention, experimental verification was conducted on the above-mentioned RGB-D salient target detection model, using the following... Figure 5-6 The visual representation methods were tested; among them, Figure 5 and Figure 6 The images are presented in several ways. In each image, RGB represents the RGB image, Depth represents the depth map, GT represents the ground truth value, and Ours represents the result obtained by the model. The image scenes are divided into small objects, multiple objects, high depth map contrast, and low depth map contrast from top to bottom.

[0127] according to Figure 5 6. Analysis of experimental results shows that the salient target detection method for RGB-D images proposed in this invention can detect salient targets well and can handle scenes with low depth map contrast well.

[0128] The above descriptions only cover a few specific embodiments of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention. The above-mentioned serial numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

Claims

1. A method for salient object detection for RGB-D images, characterized in that, The method comprises the following steps: Sampling RGB image and depth image samples according to a specified number to form sampling data, and pre-processing the sampling data by using a data enhancement technique to obtain to-be-processed data; In the cross-modal fusion of each layer of the feature extraction stage, an attention mechanism is used to infer a salient region to determine the saliency degree of different regions, and each level of fusion obtains low-level features and high-level features; In feature guidance, the high-level features and the low-level features are spliced and convoluted to obtain joint features of the RGB features and the depth features; According to the joint features of the RGB features and the depth features, the weight of the RGB image and the depth image features is dynamically allocated to obtain a salient target detection model; According to the salient target detection model, a salient target detection result is realized; According to the joint features of the RGB features and the depth features, the weight of the RGB image and the depth image features is dynamically allocated to obtain a salient target detection model, which comprises: The joint features are subjected to weight allocation to obtain the contribution weight of the RGB features and the depth features to the final prediction; The contribution weight is subjected to multiplication operation with the joint features, and a decoder is used to obtain a final salient prediction image; The salient prediction image is subjected to loss function calculation with a label true value, and a gradient descent method is used to update the feature extraction network, and when the feature extraction network loss value reaches convergence, the salient target detection model is obtained; convolving, normalizing and Relu function activation on the joint features [F R ;F D ] After 4*4 average pooling, the joint feature is flattened in the channel dimension; We obtain two weight values ​​P R P D The vector W formed P , respectively representing F in the joint features R F D The contribution of the input feature matrix F is obtained using the following formula. All : F All values into the decoder, outputting the final saliency map prediction value I pred ; Furthermore, regarding the joint feature [F] R ;F D Summing along the channel dimension yields a single-channel matrix P. s The distance matrix D is obtained using the following formula. s : ； respectively, to obtain a first average distance value D R and a second average distance value D D ; The vector W is obtained by the following equation P The corresponding true value W GT of the two elements: ； The L1 loss function is used to calculate W P and the L1 distance of W GT , and the loss value is calculated by the following formula : ； Wherein, i represents the number of weights; FR and FD are respectively the final features corresponding to the RGB image and the depth image; The gradient descent method is used to update the feature extraction network, which comprises: The mixed loss function is used to supervise the training of the salient object detection model, including a binary cross-entropy loss function, a symmetric cross ratio loss function, loss function; The hybrid loss function is designed as follows: ； wherein, is a binary cross-entropy loss function, is a Jaccard loss function, respectively denote weighting coefficients; The definition of the binary cross-entropy loss function is: ; The anharmonic ratio loss function is defined as: ； wherein, is a true value label, is a value of {0, 1}, wherein 0 represents a non-significant region, and 1 represents a significant region; is a predicted value; (x, y) represents a pixel point position; represents a pixel point value in the xth row and yth column among the true value label; represents a pixel point value in the xth row and yth column among the predicted value label. 2.The method for salient object detection for RGB-D images according to claim 1, characterized in that, In the cross-modal fusion of each layer of the feature extraction stage, an attention mechanism is used to infer a salient region to determine the saliency degree of different regions, and each level of fusion obtains high-level features and low-level features, which comprises: RGB image and depth image samples are collected according to a specified number, and the RGB image and depth image samples are processed by using a data enhancement technique; The feature extraction network extracts features from the input RGB image and depth image samples; the attention mechanism is used in the feature extraction network to optimize the features in a cross-modal manner, and the optimized features are respectively used as the input of the lower layer; A specific level of feature is selected to obtain low-level features through feature fusion; other levels of features and the output features of the dilated spatial convolution pyramid are obtained as high-level features through feature fusion.

3. The salient object detection method for RGB-D images according to claim 2, wherein: The feature extraction network comprises two independent VGG16 branches; Each VGG16 branch is divided into five sub-layers; The output of the fifth sub-layer of the VGG16 is subjected to dilated spatial convolution pyramid, which is composed of four dilated convolutions, and the convolution kernel size and the dilated rate are [k=1, d=1; k=3, d=6; k=3, d=12; k=3, d=18]; wherein, k represents the size of the convolution kernel, and d represents the dilated rate of the dilated convolution. 4.The method for salient object detection for RGB-D images according to claim 3, characterized in that, The cross-modal optimization of features is performed by using the attention mechanism, and the optimized features are respectively taken as the input of the lower layer, including: The features of the corresponding sublayers in the two independent VGG16 branches in the feature extraction network are taken as the input of feature optimization, and a spatial attention score is obtained to obtain a spatial attention weight matrix , as follows: ； wherein F is RGB feature or depth feature; 3x3 convolution for RGB feature or depth feature; 1x1 convolution for RGB feature or depth feature; is a Sigmoid function; Based on the spatial attention score, the spatial attention weight matrix is multiplied and added element by element to obtain the features of each level after spatial attention processing, as follows: ; wherein, is input corresponding RGB feature or depth feature; is spatial attention weight matrix of RGB feature or depth feature; is feature after spatial attention calculation; is spatial attention weight matrix of RGB feature; is spatial attention weight matrix of depth feature; A feature of C×H×W First, each of them is fed into a 1×1 convolution; then, a spatial global average pooling is performed on each. and max pooling We obtain two C×1×1 channel descriptions; The two obtained channel descriptions are added and then passed through a Sigmoid activation function to obtain a channel attention weight coefficient wherein, the channel attention weight coefficient of the RGB feature or the channel attention weight coefficient of the depth feature by and obtained feature after channel attention processing : ； The obtained After the self-attention mechanism, the output optimized by the characteristics is obtained as the input of the next sub-layer. And will In the third dimension, the unfolded features are respectively passed through W Q , W K , W V Linear matrix to get Q, K, V matrix; is obtained using the formula Input for next sublayer: )； where d K represents the dimension of K; S is a Softmax function; M is an operation that expands the calculated features, and then combines them with performs residual connection. 5.The method for salient object detection for RGB-D images according to claim 2, characterized in that, The selected specific level features are fused to obtain low-level features, and the other level features and the output features of the empty space convolution pyramid are fused to obtain high-level features, including: Two independent VGG16 branches are set in the feature extraction network, which are RGB branch and depth branch respectively; The VGG16 branch includes 5 sub-layers and 1 empty space convolution pyramid; The output features of the 1st, 2nd and 3rd sub-layers are fused as low-level features, and the output of the 4th and 5th sub-layers and the output of the empty space convolution pyramid are fused as high-level features; After feature fusion, the low-level features and high-level features corresponding to the RGB branch and the depth branch are obtained respectively. 6.The method for salient object detection for RGB-D images according to claim 1, characterized in that, The high-level features and the low-level features are spliced and convolved to obtain the joint features of RGB features and depth features, including: The low-level features are guided by the high-level features to determine the salient regions and retain the edge details, obtaining guided low-level features; The salient regions of the guided low-level features and the high-level features are connected to obtain the final RGB features and depth map features; The final RGB features and depth map features are spliced to obtain the joint features of RGB features and depth features.

7. The method for salient object detection for RGB-D images according to claim 6, characterized in that, The step of guiding the low-level features by the high-level features includes: The high-level features are upsampled and connected with the low-level features to obtain the connected features; The connected features are convolved to obtain features with edge refinement and salient regions; The guided low-level features are connected with the high-level features again and sent to convolution to obtain the final features corresponding to RGB and depth respectively, and the final features are connected in the channel dimension to obtain the joint features.

Images