Lightweight double-flow cross-modal interaction RGB-D saliency target detection method

Abstract

The invention relates to a lightweight double-flow cross-modal interaction RGB-D saliency target detection method, which comprises the following steps: S1, data preparation: obtaining an RGB-D data set of a task for training and testing, taking a part of an NJU2K data set and a part of an NLPR data set as a training set, and taking the training set as a training set; taking the rest of the NJU2K data set, the rest of the NLPR data set, the rest of the SIP data set, the rest of the STERE data set and the rest of the SSD data set as test sets; s2, constructing a network model: S21, constructing a feature extraction backbone network of a decoder, S22, constructing a self-adaptive cross-modal fusion module (ACM), and S23, constructing a multi-scale cavity attention module (MSA); the method comprises the following steps: S24, setting a decoder, S25, calculating a loss function, and S26, evaluating indexes; the method has the advantages that high calculation efficiency can still be kept under the condition of low model complexity, and meanwhile high generalization and accuracy are achieved when multiple types of scenes are processed.

Description

Technical Field

[0001] The present invention relates to computer vision and image processing technology, and in particular to a lightweight two-stream cross-modal interactive RGB-D salient object detection method. Background Art

[0002] In recent years, the emergence of smart cars, smart robots, and various smart devices has led to increasing attention for the application of computer vision on mobile devices. While existing RGB-D salient object detection (SOD) methods have achieved remarkable results by striving to improve model performance, they inevitably require a large number of parameters and high computational cost, making their deployment on mobile devices extremely difficult. Numerous researchers have devoted themselves to the study of real-world applications of RGB-D salient object detection (SOD), actively attempting to optimize and innovate traditional model architectures to meet the diverse needs of different scenarios. Traditional backbone networks, such as VGG and ResNet, have demonstrated excellent performance in tasks such as image feature extraction due to their complex and deep structures. However, with the continuous advancement of deep learning technology, these traditional networks face challenges, particularly in resource-constrained real-world applications, such as high computational complexity and limitations on mobile devices. Consequently, lightweight backbone networks such as ShuffleNet, MobileNet, and EfficientNet have become a research hotspot. These networks offer high computational accuracy, a small number of parameters, and a small number of convolutional layers and filters. Compared to traditional convolutional neural networks, they can significantly reduce computational effort, making them ideal for processing limited computing resources. However, when a lightweight backbone network is used for feature extraction, a certain degree of information loss is inevitable due to the simplification of the network structure, resulting in a decrease in feature representation capability.

[0003] Therefore, it is necessary to design a lightweight model to achieve efficient and accurate RGB-D salient object detection while minimizing information loss. Summary of the Invention

[0004] In view of the above problems, the purpose of this paper is to provide a lightweight two-stream cross modal interaction network (LSMNet) for RGB-D salient object detection. LSMNet uses a lightweight network to extract features and conduct effective cross-modal interaction and integration. This method can compensate for the information loss caused by the lightweight network and effectively aggregate the salient features contained in RGB and depth images to improve the performance of RGB-D salient object detection models.

[0005] The present invention provides a lightweight two-stream cross-modal interactive RGB-D salient object detection method, comprising the following steps:

[0006] Step S1: Data preparation,

[0007] Obtain the RGB-D dataset for this task for training and testing. Part of the NJU2K dataset and part of the NLPR dataset are used as training sets, and the remaining NJU2K dataset, NLPR dataset, SIP dataset, STERE dataset, and SSD dataset are used as test sets.

[0008] Step S2: Constructing a network model, including: Step S21: Constructing a feature extraction backbone network of the decoder, Step S22: Adaptive cross-modal fusion module ACM, Step S23: Multi-scale void attention module MSA; Step S24: Decoder, Step S25: Loss function calculation, Step S26: Evaluation indicators.

[0009] As a preferred embodiment of the present invention, step S1 further includes the following steps:

[0010] Step S11: The RGB-D dataset includes RGB images, depth images, and manually annotated true value maps;

[0011] Step S12: First adjust the resolution of the RGB image and depth image to 256×256 respectively, and then use them as the input of the backbone network.

[0012] As a preference of the present invention, step S2 further includes the following steps:

[0013] Step S21: Decoder feature extraction backbone network construction,

[0014] The MobileNetV3 network pre-trained on the ImageNet dataset is used to improve the MobileNetV3-large network and the MobileNetV3-small network. These networks are used as the backbone networks of the RGB stream and the depth stream, respectively, to extract image features of the RGB and depth modalities.

[0015] In the RGB stream branch, the MobileNetV3-large network starts with a convolutional block, followed by 15 bottleneck blocks. The 15 bottleneck blocks are connected on both sides by 1×1 convolutional connections to form an expansion layer and a projection layer, respectively. A depthwise separable convolution operation is used in the middle. The input and output of the bottleneck block are connected by a residual connection. The SE module is added to the residual connection to extract more representative features. The ReLU function is used as the activation function in the first half of the network layer, and the h-swish function is used as the activation function in the second half of the network.

[0016] In the depth stream branch, based on the original MobileNetV3-Small network, the basic bottleneck block is retained and a bottleneck block from the deep layer is added to the shallow layer to extract more underlying features for edge supervision. The number of extended layers and output channels of the MobileNetV3-Small network is also modified. Specifically, a MobileNetV3-Small network with ten bottleneck blocks is created. Similarly, the bottleneck block uses residual connections to speed up convergence while using ReLU functions and h-swish functions as activation functions to maintain the integrity and accuracy of information.

[0017] As a preference of the present invention, step S2 further includes the following steps:

[0018] Step S22: Adaptive cross-modal fusion module ACM,

[0019] The ACM (Adaptive Cross-Model Fusion) module is used to fuse RGB image features and depth image features. The ACM module is divided into a channel expansion layer, a feature fusion layer, and a channel projection layer. In the channel expansion layer, the features are expanded to a high dimension through 1×1 convolution so that the RGB stream branch and the depth stream branch have the same number of channels. In the feature fusion layer, a lightweight method is used to connect the features for adaptive fusion and filtering. Finally, in the channel projection layer, 1×1 convolution is used to project the features to a low dimension.

[0020] As a preferred embodiment of the present invention, step S22 further includes the following steps:

[0021] Step S221: In the channel expansion layer, the features of the RGB image and the depth image are extracted respectively through the lightweight backbone network and Afterwards, 1×1 convolution, batch normalization, and ReLU activation function operations are performed on the input RGB image features and depth image features respectively, so that the outputs of the RGB stream branch and the depth stream branch have the same dimension for subsequent feature fusion and processing. The specific process is as follows:

[0022]

[0023] Among them, BN represents the batch normalization operation, ReLU represents the activation function, and the RGB stream branch and the depth stream branch features after channel expansion are used and Indicates that if there are features of the previous layer, then the features of the previous layer also need to be subjected to 1×1 convolution, batch normalization, and ReLU operations accordingly, and the number of its channels is adjusted to be the same as the number of channels after the RGB stream branch and the depth stream branch channel are expanded. This process is expressed as:

[0024]

[0025] Where pre_conv and bil represent 1×1 convolution and bilinear interpolation operations, pre_fea represents the result of the previous layer feature operation, conact represents feature fusion, f_conv represents the dimensionality reduction operation through convolution, Fr and Fd represent the results of the feature fusion operation of RGB features and depth features with the processed previous layer features respectively;

[0026] Step S222: In the feature fusion layer, the fused features Fr and Fd are subjected to global average pooling and flattening operations respectively to obtain a vector F containing global information of RGB features and depth features. r_global and F d_global , which can be expressed as:

[0027] F r_global =global_avg_pool(Fr).flatten(start_dim=1)

[0028] F d_global =global_avg_pool(Fd).flatten(start_dim=1)

[0029] The global eigenvector F r_global and F d_global Splicing, input to the fully connected layer to calculate the fusion weight fusion_weights. Assuming that the weight matrix of the fully connected layer is W, the bias is b, and the activation function is σ, the calculation of the fusion weight can be expressed as follows:

[0030] fusion_weights=σ([F r_global ; F d_global ]W+b)

[0031] Among them [F r_global ; F d_global ] indicates splicing on the feature dimension, σ indicates the Softmax function, which is used to normalize the output to between 0 and 1, and obtain two weight values, which correspond to the weights of RGB features and depth features respectively. The view operation is used to adjust its shape to obtain the weights of RGB features and depth features respectively, as shown in the formula:

[0032] weightrgb =fusion_weights[:,0].view(-1,1,1,1)

[0033] weight depth =fusion_weights[:,1].view(-1,1,1,1)

[0034] fusion_weights[:,0] means taking the first element of each row in the fusion_weights tensor, that is, the weight of RGB, to achieve element-by-element multiplication with the feature map to achieve weighted RGB features. Similarly, weight depth It is a deformation of the depth feature weight, used to weight the depth feature;

[0035] At the same time, the most representative features are selected through the attention mechanism for the RGB features and depth features after channel expansion and alignment. The enhanced features are expressed as follows:

[0036]

[0037] Among them, SA(·) represents the Shuffle Attention (SA) module, and the feature fusion process is expressed as follows:

[0038] F_out=Fr_out·weight rgb +Fd_out·weight depth

[0039] Step S223: In the channel projection layer, the features are projected using a combination of a 3×3 convolution with a ReLU activation function and two 1×1 convolution operations. Therefore, the final output of the ACM module is expressed as:

[0040]

[0041] Where conv_out represents a combination of a 3×3 convolution and two 1×1 convolutions containing a ReLU activation function. Represents the final output of the i-th layer ACM module.

[0042] As a preference of the present invention, step S2 further includes the following steps:

[0043] Step S23: Multi-scale hole attention module MSA,

[0044] The lightweight Multi-Scale Atrous Attention (MSA) module of the LSMNet model is used to extract cross-level and cross-scale features. The lightweight Multi-Scale Atrous Attention module integrates multi-scale atrous convolution, residual feature enhancement, and attention feature enhancement based on GAP and GMP.

[0045] Step S231: In the multi-scale atrous convolution part, a set of atrous convolutions with different dilation rates are used to extract features. The dilation rate set is defined as: {r1=3, r2=6, r3=9, r4=12}. In the fifth stage of the LSMNet model, only r1 is used for atrous convolution. Similarly, in the second stage of the LSMNet model, r1, r2, r3 and r4 are used to form parallel atrous convolutions. The calculation formula of each atrous convolution branch is expressed as follows:

[0046] F i =σ.BN(conv r (X)) /

[0047] Where X∈R C×H×W Represents the input feature map, conv r Denotes a dilated convolution operation with a corresponding dilation rate of r, BN(·) represents a batch normalization operation, and σ(·) represents a ReLU activation function. Using multi-scale features, after obtaining the outputs of all dilated convolutions, the outputs of all branches are concatenated along the channel dimension, which can be expressed as follows:

[0048]

[0049] Represents the result after splicing in the i-th stage. Since the number of channels increases after splicing, in order to reduce the amount of calculation and improve the fusion effect, 1×1 convolution is used for feature compression, which is expressed as follows:

[0050]

[0051] Step S232: In the residual feature fusion part, the underlying spatial information of the original input is retained, and the fused feature On the basis of , residual connection is introduced and feature expression ability is enhanced by element-by-element addition. The enhanced features are expressed as follows:

[0052]

[0053] The global attention mechanism is introduced in the MSA module to dynamically adjust the weight of features. The global attention mechanism uses a combination of global average pooling (GAP) and global maximum pooling (GMP) to enhance feature expression capabilities. The two methods are expressed as follows:

[0054]

[0055] GMP(X)=max i,j X i,j

[0056] Among them, X represents the input feature, that is, the input feature map. Subsequently, the outputs of GAP and GMP are spliced ​​along the channel dimension to form a global feature representation. This process is shown in the formula:

[0057]

[0058] Next, we use 1×1 convolution to extract channel information by first reducing the dimension and then increasing the dimension. By increasing the dimension, we can restore the information, enhance the feature expression ability, and ensure the complete transmission of information. The formula is as follows:

[0059]

[0060] Where W1 and W2 are the convolution kernels for dimensionality reduction and dimensionality increase respectively, δ(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function. Finally, the attention weights are calculated. Acting on residual features Above, as shown in the formula:

[0061]

[0062] where ⊙ represents element-wise multiplication.

[0063] As a preference of the present invention, step S2 further includes the following steps:

[0064] Step S24: Decoder operation,

[0065] In the decoder, the LSMNet model uses four consecutive lightweight multi-scale void attention modules MSA to transmit multi-scale information layer by layer, gradually recovering the high-resolution saliency prediction results to improve the fine-grained expression ability of saliency targets. The decoder starts from the high-stage MSA module, transmits information from the deepest layer to the shallow layer, upsamples the output and combines it with the result of the ACM module of the previous stage to input it into the MSA module of the next stage to maintain the same resolution. Let the output of the ACM module be The input of the MSA module is The output is The information transmission between MSA modules is expressed as follows:

[0066]

[0067] in, It represents bilinear interpolation upsampling of the feature information of the i-th layer to match the spatial size of the i-1 layer.

[0068] As a preference of the present invention, step S2 further includes the following steps:

[0069] Step S25: Loss function calculation,

[0070] The loss function consists of three parts: structural similarity loss function SSIMLoss, binary cross entropy loss function BCELoss and intersection-over-union loss function IOULoss. The calculation formula of the hybrid loss function is as follows:

[0071]

[0072] represents the mixed loss function of the i-th layer.

[0073] As a preference of the present invention, step S2 further includes the following steps:

[0074] Step S26: Evaluation index,

[0075] Four evaluation indicators are used to evaluate the effectiveness of the improved LSMNet model: F-measure, Mean Absolute Error, S-measure and E-measure.

[0076] The beneficial effects of the present invention are as follows: To address the problem that the existing RGB-D salient object detection methods have high computational complexity, which makes them difficult to deploy on mobile terminals, a lightweight two-stream cross-modal interactive RGB-D salient object detection method LSMNet is proposed. The proposed method effectively balances the relationship between model complexity and model efficiency, and can still maintain high computational efficiency with low model complexity. At the same time, it has strong generalization and accuracy when processing various types of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] By referring to the following description in conjunction with the accompanying drawings, and with a more complete understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings:

[0078] Figure 1 This is the overall network structure diagram of the present invention;

[0079] Figure 2This is the ACM module structure diagram of the present invention;

[0080] Figure 3 This is a structural diagram of the MSA module in the present invention; DETAILED DESCRIPTION

[0081] See Figure 1-3 This embodiment provides a lightweight dual-stream cross-modal interactive RGB-D salient object detection method, including the following steps:

[0082] Step S1: Data preparation,

[0083] Obtain an RGB-D dataset for this task for training and testing. A portion of the NJU2K dataset and a portion of the NLPR dataset are used as training sets, and the remaining portions of the NJU2K dataset, the NLPR dataset, the SIP dataset, the STERE dataset, and the SSD dataset are used as test sets. All datasets used in this embodiment are publicly available datasets.

[0084] Step S11: The RGB-D dataset includes RGB images, depth images, and manually annotated ground truth maps. For each sample in the dataset, it includes an RGB image, a corresponding depth image, and a corresponding manually annotated ground truth map.

[0085] Step S12: First adjust the resolution of the RGB image and depth image to 256×256 respectively, and then use them as the input of the backbone network.

[0086] Step S2: Build a network model,

[0087] Step S21: Decoder feature extraction backbone network construction,

[0088] The MobileNetV3 network pre-trained on the ImageNet dataset is used to improve the MobileNetV3-large network and the MobileNetV3-small network. These networks are used as the backbone networks of the RGB stream and the depth stream, respectively, to extract image features of the RGB and depth modalities.

[0089] In the RGB stream branch, to adapt the feature extraction for the RGB-D salient object detection task, the last pooling layer and the last three convolutional layers of the original MobileNetV3-Large network were removed, while some bottleneck blocks from the original MobileNetV3-Large network were retained. Specifically, the MobileNetV3-Large network begins with a single convolutional block followed by 15 bottleneck blocks. These bottleneck blocks employ a unique structure: 1×1 convolutional connections form an expansion layer and a projection layer on either side, with depthwise separable convolutions in between. This structure significantly reduces the number of parameters and computational overhead while maintaining feature representation. The input and output of the bottleneck blocks are connected via residual connections, which not only helps alleviate the vanishing gradient problem but also enables the learning of more complex feature representations and accelerates convergence. Separation (SE) modules are added to the residual connections to extract more representative features. ReLU activation functions are used in the first half of the network layer and h-swish activation functions in the second half. This improves the model's computational efficiency, generalization, and feature representation.

[0090] In the depth stream branch, because depth images contain less information than RGB images, in order to adapt the network to the RGB-D salient object detection task, on the basis of the original MobileNetV3-Small network, the basic bottleneck block is retained and a bottleneck block in the deep layer is added to the shallow layer to extract more underlying features for edge supervision. The number of extended layers and output channels of the MobileNetV3-Small network is also modified. Specifically, a MobileNetV3-Small network with ten bottleneck blocks is constructed. Similarly, the bottleneck block uses residual connections to speed up convergence while using ReLU functions and h-swish functions as activation functions to maintain the integrity and accuracy of information, improve the calculation speed of inference, and help the model learn more advanced semantic information.

[0091] Step S22: Adaptive cross-modal fusion module ACM,

[0092] RGB images and depth images belong to different modal data. RGB images can help the model better extract the detailed features of the target. Depth images are suitable for models to understand more complex scenes and the relative positions of objects in the image. In most cases, RGB images and depth images can enhance each other and improve the expressiveness of features. The ACM (Adaptive Cross-Model Fusion) module is used to fuse RGB image features and depth image features, aiming to effectively integrate feature information from different modalities while considering the influence of the previous layer to improve the performance of the final detection.

[0093] Because lightweight backbone networks contain relatively few convolutional blocks and fewer filters in their layers, their feature extraction capabilities are inevitably weaker. The ACM module is designed with an inverted structure, consisting of a channel expansion layer, a feature fusion layer, and a channel projection layer. In the channel expansion layer, features are expanded to higher dimensions using 1×1 convolutions, ensuring that the RGB and depth streams have the same number of channels. In the feature fusion layer, features are concatenated using a lightweight method for adaptive fusion and filtering. Finally, in the channel projection layer, features are projected to lower dimensions using 1×1 convolutions. This approach increases the number of channels in feature representation at shallow layers of the network, further improving the model's expressive power. This avoids information loss caused by a significant reduction in the number of channels during channel projection in deeper layers of the network. This maintains model accuracy and significantly improves model speed in resource-constrained scenarios, such as mobile devices.

[0094] Step S221: In the channel expansion layer, the features of the RGB image and the depth image are extracted respectively through the lightweight backbone network and Afterwards, because the network uses an asymmetric backbone network, different branches may output inconsistent channels. Therefore, the input RGB image features and depth image features are subjected to 1×1 convolution, batch normalization, and ReLU activation function operations respectively to make the outputs of the RGB stream branch and the depth stream branch have the same dimension for subsequent feature fusion and processing. The specific process is as follows:

[0095]

[0096] Among them, BN represents the batch normalization operation, ReLU represents the activation function, and the RGB stream branch and the depth stream branch features after channel expansion are used and It should be noted that if there are features of the previous layer, the features of the previous layer also need to be subjected to 1×1 convolution, batch normalization, and ReLU operations accordingly, and the number of its channels is adjusted to be the same as the number of channels after the RGB stream branch and the depth stream branch channel are expanded. This process is expressed as:

[0097]

[0098] Where pre_conv and bil represent 1×1 convolution and bilinear interpolation operations, pre_fea represents the result of the previous layer feature operation, conact represents feature fusion, f_conv represents the dimensionality reduction operation through convolution, Fr and Fd represent the results of the feature fusion operation of RGB features and depth features with the processed previous layer features respectively;

[0099] Step S222: In the feature fusion layer, the fused features Fr and Fd are subjected to global average pooling (GAP) and flattening operations respectively to obtain a vector F containing global information of RGB features and depth features. r_global and F d_global , which can be expressed as:

[0100] F r_global =global_avg_pool(Fr).flatten(start_dim=1)

[0101] F d_global =global_avg_pool(Fd).flatten(start_dim=1)

[0102] The global eigenvector F r_global and F d_global Splicing, input to the fully connected layer to calculate the fusion weight fusion_weights. Assuming that the weight matrix of the fully connected layer is W, the bias is b, and the activation function is σ, the calculation of the fusion weight can be expressed as follows:

[0103] fusion_weights=σ([F r_global ; F d_global ]W+b)

[0104] Among them [F r_global ; F d_global ] indicates splicing on the feature dimension, σ indicates the Softmax function, which is used to normalize the output to between 0 and 1, and obtain two weight values, corresponding to the weights of RGB features and depth features respectively. Because the calculated fusion_weights is a two-dimensional tensor, in order to enable it to be element-by-element multiplied with the feature map, it is necessary to use the view operation to adjust its shape to obtain the weights of RGB features and depth features respectively, as shown in the formula:

[0105] weight rgb =fusion_weights[:,0].view(-1,1,1,1)

[0106] weight depth =fusion_weights[:,1].view(-1,1,1,1)

[0107] fusion_weights[:,0] means taking the first element of each row in the fusion_weights tensor, that is, the weight of RGB, to achieve element-by-element multiplication with the feature map to achieve weighted RGB features. Similarly, weight depth It is a deformation of the depth feature weight, used to weight the depth feature;

[0108] At the same time, the most representative features are selected through the attention mechanism for the RGB features and depth features after channel expansion and alignment. The enhanced features are expressed as follows:

[0109]

[0110] Among them, SA(·) represents the Shuffle Attention (SA) module, and the feature fusion process is expressed as follows:

[0111] F_out=Fr_out·weight rgb +Fd_out·weight depth

[0112] Step S223: In the channel projection layer, the features are projected using a combination of a 3×3 convolution with a ReLU activation function and two 1×1 convolution operations. Therefore, the final output of the ACM module is expressed as:

[0113]

[0114] Where conv_out represents a combination of a 3×3 convolution and two 1×1 convolutions containing a ReLU activation function. Represents the final output of the i-th layer ACM module.

[0115] Step S23: Multi-scale hole attention module MSA,

[0116] Efficient feature extraction and multi-scale information fusion are crucial for RGB-D salient object detection. To this end, the lightweight Multi-Scale Atrous Attention (MSA) module of the LSMNet model is utilized to extract cross-layer and cross-scale features. This lightweight MSA module integrates multi-scale atrous convolution, residual feature enhancement, and attention feature enhancement based on GAP and GMP. This effectively captures contextual information while preserving spatial details, improving saliency detection performance.

[0117] Step S231: In the multi-scale atrous convolution part, a set of atrous convolutions with different dilation rates are used to extract features. The dilation rate set is defined as: {r1=3, r2=6, r3=9, r4=12}. In the fifth stage of the LSMNet model, only r1 is used for atrous convolution. Similarly, in the second stage of the LSMNet model, r1, r2, r3 and r4 are used to form parallel atrous convolutions. The calculation formula of each atrous convolution branch is expressed as follows:

[0118] F i =σ.BN(conv r (X)) /

[0119] Where X∈R C×H×W Represents the input feature map, conv r Denotes a dilated convolution operation with a corresponding dilation rate of r, BN(·) represents a batch normalization operation, and σ(·) represents a ReLU activation function. Using multi-scale features, after obtaining the outputs of all dilated convolutions, the outputs of all branches are concatenated along the channel dimension, which can be expressed as follows:

[0120]

[0121] Represents the result after splicing in the i-th stage. Since the number of channels increases after splicing, in order to reduce the amount of calculation and improve the fusion effect, 1×1 convolution is used for feature compression, which is expressed as follows:

[0122]

[0123] Step S232: In the residual feature fusion part, the underlying spatial information of the original input is retained, and the fused feature On the basis of , residual connection is introduced and feature expression ability is enhanced by element-by-element addition. The enhanced features are expressed as follows:

[0124]

[0125] This operation preserves the underlying feature details and helps improve the performance of salient object detection.

[0126] In order to further improve the adaptability of features, a global attention mechanism is introduced in the MSA module to dynamically adjust the weight of features. The global attention mechanism uses a combination of global average pooling (GAP) and global maximum pooling (GMP) to enhance feature expression capabilities. The two methods are expressed as follows:

[0127]

[0128] GMP(X)=max i,j X i,j

[0129] Among them, X represents the input feature, that is, the input feature map. Subsequently, the outputs of GAP and GMP are spliced ​​along the channel dimension to form a global feature representation. This process is shown in the formula:

[0130]

[0131] Next, we use a 1×1 convolution to first reduce the dimension and then increase the dimension to extract channel information. The advantage of this approach is that it can reduce the amount of computation through dimensionality reduction and improve the efficiency of the model. By increasing the dimension, we can restore information, enhance the feature expression ability, and ensure the complete transmission of information. The formula is as follows:

[0132]

[0133] Where W1 and W2 are the convolution kernels for dimensionality reduction and dimensionality increase respectively, δ(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function. Finally, the attention weights are calculated. Acting on residual features Above, as shown in the formula:

[0134]

[0135] where ⊙ represents element-wise multiplication.

[0136] Step S24: Decoder operation,

[0137] In the decoder, the LSMNet model uses four consecutive lightweight multi-scale void attention modules MSA to transmit multi-scale information layer by layer, gradually recovering the high-resolution saliency prediction results to improve the fine-grained expression ability of saliency targets. The decoder starts from the high-stage MSA module, transmits information from the deepest layer to the shallow layer, upsamples the output and combines it with the result of the ACM module of the previous stage to input it into the MSA module of the next stage to maintain the same resolution. Let the output of the ACM module be The input of the MSA module is The output is The information transmission between MSA modules is expressed as follows:

[0138]

[0139] in, This represents bilinear interpolation upsampling of the feature information of layer i to match the spatial size of layer i-1. This fusion method effectively utilizes the global information of high-level features and combines it with the spatial details of low-level features to achieve more refined salient object detection.

[0140] Step S25: Loss function calculation,

[0141] The loss function consists of three parts: structural similarity loss function SSIMLoss, binary cross entropy loss function BCELoss and intersection-over-union loss function IOULoss. The calculation formula of the hybrid loss function is as follows:

[0142]

[0143] represents the mixed loss function of the i-th layer.

[0144] Step S26: Evaluation index,

[0145] Four evaluation indicators are used to evaluate the effectiveness of the improved LSMNet model: F-measure, Mean Absolute Error, S-measure, and E-measure;

[0146] Among them, in the model training stage of this embodiment,

[0147] The model inputs RGB images and three-channel depth images replicated three times, with a resolution of 960×1080. Training images were resized to 256×256 before input. Various data augmentation techniques were used, including random flipping, cropping, and rotation. The initial learning rate was set to 1e-4, and the learning rate was reduced to 1% of its original value every 60 cycles. The batch size was 8, and the Adam algorithm was used as the optimizer for the network. The entire model was run for 200 epochs.

[0148] Model testing phase,

[0149] First, the resolution of the RGB image and depth image is adjusted to 256×256, consistent with the training phase, to ensure the consistency and stability of the model input data. The images with adjusted resolution are input into the trained network. After a series of feature extraction, fusion and processing, the saliency prediction results are finally obtained.

[0150] Four commonly used evaluation indicators are used to evaluate the effectiveness of the network, namely F-measure, MeanAbsolute Error, S-measure and E-measure. The detailed definitions of these four indicators are as follows:

[0151] F-measure(F β ) is a widely used comprehensive evaluation metric that considers both precision and recall scores and is defined as follows:

[0152]

[0153] Where Precision and Recall represent the precision score and recall score respectively, β 2 Set to 0.3 to emphasize accuracy;

[0154] Mean Absolute Error (MAE) represents the average error between the saliency map S and the true value Y, which is defined as follows:

[0155]

[0156] Where W and H represent the width and height of the saliency map respectively;

[0157] S-measure(S λ ) Evaluate the saliency map S r and the saliency label map S o The spatial structural similarity between , which takes into account the saliency of regions and boundaries, is also called structured measurement and is defined as follows:

[0158] S λ =α*S o +(1-α)*S r ;

[0159] Where α∈[0,1] is the balance parameter, which is set to 0.5 by default;

[0160] E-measure (E γ ) is an evaluation metric based on the enhanced alignment mapping between the saliency map and the saliency label map, which is defined as follows:

[0161]

[0162] where W and H represent the width and height of the saliency map, respectively, and φFM(*) represents the enhanced alignment matrix.

[0163] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A lightweight two-stream cross-modal interactive RGB-D salient object detection method, characterized by: The following steps are involved: Step S1: Data preparation, Obtain the RGB-D dataset for this task for training and testing. Part of the NJU2K dataset and part of the NLPR dataset are used as training sets, and the remaining NJU2K dataset, NLPR dataset, SIP dataset, STERE dataset, and SSD dataset are used as test sets. Step S2: Constructing a network model, including: Step S21: Constructing a feature extraction backbone network of the decoder, Step S22: Adaptive cross-modal fusion module ACM, Step S23: Multi-scale void attention module MSA; Step S24: Decoder, Step S25: Loss function calculation, Step S26: Evaluation indicators.

2. The lightweight dual-stream cross-modal interactive RGB-D salient object detection method according to claim 1 is characterized in that: Step S1 also includes the following steps: Step S11: The RGB-D dataset includes RGB images, depth images, and manually annotated true value maps; Step S12: First adjust the resolution of the RGB image and depth image to 256×256 respectively, and then use them as the input of the backbone network.

3. The lightweight dual-stream cross-modal interactive RGB-D salient object detection method according to claim 1 is characterized in that: Step S2 also includes the following steps: Step S21: Decoder feature extraction backbone network construction, The MobileNetV3 network pre-trained on the ImageNet dataset is used to improve the MobileNetV3-large network and the MobileNetV3-small network. These networks are used as the backbone networks of the RGB stream and the depth stream, respectively, to extract image features of the RGB and depth modalities. In the RGB stream branch, the MobileNetV3-large network starts with a convolutional block, followed by 15 bottleneck blocks. The 15 bottleneck blocks are connected on both sides by 1×1 convolutional connections to form an expansion layer and a projection layer, respectively. A depthwise separable convolution operation is used in the middle. The input and output of the bottleneck block are connected by a residual connection. The SE module is added to the residual connection to extract more representative features. The ReLU function is used as the activation function in the first half of the network layer, and the h-swish function is used as the activation function in the second half of the network. In the depth stream branch, based on the original MobileNetV3-Small network, the basic bottleneck block is retained and a bottleneck block from the deep layer is added to the shallow layer to extract more underlying features for edge supervision. The number of extended layers and output channels of the MobileNetV3-Small network is also modified. Specifically, a MobileNetV3-Small network with ten bottleneck blocks is created. Similarly, the bottleneck block uses residual connections to speed up convergence while using ReLU functions and h-swish functions as activation functions to maintain the integrity and accuracy of information.

4. The lightweight dual-stream cross-modal interactive RGB-D salient object detection method according to claim 3 is characterized in that: Step S2 also includes the following steps: Step S22: Adaptive cross-modal fusion module ACM, The ACM module is used to fuse RGB image features and depth image features. The ACM module is divided into a channel expansion layer, a feature fusion layer, and a channel projection layer. In the channel expansion layer, the features are expanded to a high dimension through 1×1 convolution so that the RGB stream branch and the depth stream branch have the same number of channels. In the feature fusion layer, a lightweight method is used to connect the features for adaptive fusion and filtering. Finally, in the channel projection layer, 1×1 convolution is used to project the features to a low dimension.

5. The lightweight dual-stream cross-modal interactive RGB-D salient object detection method according to claim 4 is characterized in that: Step S22 also includes the following steps: Step S221: In the channel expansion layer, the features of the RGB image and the depth image are extracted respectively through the lightweight backbone network and Afterwards, 1×1 convolution, batch normalization, and ReLU activation function operations are performed on the input RGB image features and depth image features respectively, so that the outputs of the RGB stream branch and the depth stream branch have the same dimension for subsequent feature fusion and processing. The specific process is as follows: Among them, BN represents the batch normalization operation, ReLU represents the activation function, and the RGB stream branch and the depth stream branch features after channel expansion are used and Indicates that if there are features of the previous layer, then the features of the previous layer also need to be subjected to 1×1 convolution, batch normalization, and ReLU operations accordingly, and the number of its channels is adjusted to be the same as the number of channels after the RGB stream branch and the depth stream branch channel are expanded. This process is expressed as: Where pre_conv and bil represent 1×1 convolution and bilinear interpolation operations, pre_fea represents the result of the previous layer feature operation, conact represents feature fusion, f_conv represents the dimensionality reduction operation through convolution, Fr and Fd represent the results of the feature fusion operation of RGB features and depth features with the processed previous layer features respectively; Step S222: In the feature fusion layer, the fused features Fr and Fd are subjected to global average pooling and flattening operations respectively to obtain a vector F containing global information of RGB features and depth features. r_global and F d_global , which can be expressed as: F r_global =global_avg_pool(Fr).flatten(start_dim=1) F d_global =global_avg_pool(Fd).flatten(start_dim=1) The global eigenvector F r_global and F d_global Splicing, input to the fully connected layer to calculate the fusion weight fusion_weights. Assuming that the weight matrix of the fully connected layer is W, the bias is b, and the activation function is σ, the calculation of the fusion weight can be expressed by the formula: fusion_weights=σ([F r_global ;F d_global ]W+b) Among them [F r_global ; F d_global ] indicates splicing on the feature dimension, σ indicates the Softmax function, which is used to normalize the output to between 0 and 1, and obtain two weight values, which correspond to the weights of RGB features and depth features respectively. The view operation is used to adjust its shape to obtain the weights of RGB features and depth features respectively, as shown in the formula: weight rgb =fusion_weights[:,0].view(-1,1,1,1) weight depth =fusion_weights[:,1].view(-1,1,1,1) fusion_weights[:,0] means taking the first element of each row in the fusion_weights tensor, that is, the weight of RGB, to achieve element-by-element multiplication with the feature map to achieve weighted RGB features. Similarly, weight depth It is a deformation of the depth feature weight, used to weight the depth feature; At the same time, the most representative features are selected through the attention mechanism for the RGB features and depth features after channel expansion and alignment. The enhanced features are expressed as follows: Among them, SA(·) represents the Shuffle Attention (SA) module, and the feature fusion process is expressed as follows: F_out=Fr_out·weight rgb +Fd_out·weight depth Step S223: In the channel projection layer, the features are projected using a combination of a 3×3 convolution with a ReLU activation function and two 1×1 convolution operations. Therefore, the final output of the ACM module is expressed as: Where conv_out represents a combination of a 3×3 convolution and two 1×1 convolutions containing a ReLU activation function. Represents the final output of the i-th layer ACM module.

6. The lightweight dual-stream cross-modal interactive RGB-D salient object detection method according to claim 5 is characterized in that: Step S2 also includes the following steps: Step S23: Multi-scale hole attention module MSA, The lightweight multi-scale dilated attention module of the LSMNet model is used to extract cross-level and cross-scale features. The lightweight multi-scale dilated attention module integrates multi-scale dilated convolution, residual feature enhancement, and attention feature enhancement based on GAP and GMP. Step S231: In the multi-scale dilated convolution part, a set of dilated convolutions with different dilation rates are used to extract features. The dilation rate set is defined as: {r1=3, r2=6, r3=9, r4=12}. In the fifth stage of the LSMNet model, only r1 is used for dilated convolution. Similarly, in the second stage of the LSMNet model, r1, r2, r3 and r4 are used to form parallel dilated convolutions. The calculation formula of each dilated convolution branch is expressed as follows: F i =σ.BN(conv r (X)) / Where X∈R C×H×W Represents the input feature map, conv r Denotes the dilated convolution operation with the corresponding dilation rate r, BN(·) represents the batch normalization operation, and σ(·) represents the ReLU activation function. Utilizing multi-scale features, after obtaining the outputs of all dilated convolutions, the outputs of all branches are concatenated along the channel dimension, which can be expressed as follows: Represents the result after splicing in the i-th stage. Since the number of channels increases after splicing, in order to reduce the amount of calculation and improve the fusion effect, 1×1 convolution is used for feature compression, which is expressed as follows: Step S232: In the residual feature fusion part, the underlying spatial information of the original input is retained, and the fused feature On the basis of , residual connection is introduced and feature expression ability is enhanced by element-by-element addition. The enhanced features are expressed as follows: The global attention mechanism is introduced in the MSA module to dynamically adjust the weight of features. The global attention mechanism uses a combination of global average pooling and global maximum pooling to enhance feature expression capabilities. The two methods are expressed as follows: GMP(X)=max i,j X i,j Among them, X represents the input feature, that is, the input feature map. Subsequently, the outputs of GAP and GMP are spliced ​​along the channel dimension to form a global feature representation. This process is shown in the formula: Next, we use 1×1 convolution to extract channel information by first reducing the dimension and then increasing the dimension. By increasing the dimension, we can restore the information, enhance the feature expression ability, and ensure the complete transmission of information. The formula is as follows: Where W1 and W2 are the convolution kernels for dimensionality reduction and dimensionality increase respectively, δ(·) represents the ReLU activation function, and σ(·) represents the Sigmoid activation function. Finally, the attention weights are calculated. Acting on residual features Above, as shown in the formula: where ⊙ represents element-wise multiplication.

7. The lightweight dual-stream cross-modal interactive RGB-D salient object detection method according to claim 6 is characterized in that: Step S2 also includes the following steps: Step S24: Decoder operation, In the decoder, the LSMNet model uses four consecutive lightweight multi-scale hole attention modules MSA to pass the The decoder starts from the high-level MSA module and passes information from the deepest layer to the shallowest layer. The output is upsampled and combined with the result of the ACM module of the previous stage to be input into the MSA module of the next stage to maintain the same resolution. The output of the ACM module is set to The input of the MSA module is The output is The information transmission between MSA modules is expressed as follows: in, It represents bilinear interpolation upsampling of the feature information of the i-th layer to match the spatial size of the i-1 layer.

8. The lightweight dual-stream cross-modal interactive RGB-D salient object detection method according to claim 7 is characterized in that: Step S2 also includes the following steps: Step S25: Loss function calculation, The loss function consists of three parts: structural similarity loss function SSIMLoss, binary cross entropy loss function BCELoss and intersection-over-union loss function IOULoss. The calculation formula of the hybrid loss function is as follows: represents the mixed loss function of the i-th layer.

9. The lightweight dual-stream cross-modal interactive RGB-D salient object detection method according to claim 8, characterized in that: Step S2 also includes the following steps: Step S26: Evaluation index, Four evaluation indicators are used to evaluate the effectiveness of the improved LSMNet model: F-measure, Mean Absolute Error, S-measure and E-measure.

Images