A lightweight dual-stream cross-modal interaction RGB-D salient object detection method

Abstract

The present application relates to a kind of lightweight double-flow cross-modal interaction's RGB-D salient target detection method, comprising the following steps: step S1: data preparation, obtain the RGB-D dataset of this task, for training and testing, wherein part of NJU2K dataset and part of NLPR dataset are used as training set, the remaining part of NJU2K dataset and NLPR dataset, SIP dataset, STERE dataset and SSD dataset are collectively used as test set;Step S2: network model is built, including: step S21: the feature extraction main network construction of decoder, step S22: adaptive cross-modal fusion module ACM, step S23: multi-scale hollow attention module MSA;Step S24: decoder, step S25: loss function calculation, step S26: evaluation index;The present application has the advantages that: in the case where having lower model complexity, still can maintain higher computational efficiency, while processing multiple types of scene has stronger generalization and accuracy.

Description

Technical Field

[0001] This invention relates to computer vision and image processing technology, specifically to a lightweight two-stream cross-modal interactive RGB-D salient object detection method. Background Technology

[0002] In recent years, the emergence of intelligent cars, intelligent robots, and various intelligent devices has led to increasing attention being paid to the application of computer vision in mobile devices. Although existing RGB-D salient object detection (SOD) methods have strived to improve model performance and achieved many excellent results, they inevitably result in a large number of parameters and high computational costs, making their deployment on mobile terminals extremely difficult. In recent years, many researchers have devoted themselves to the application research of RGB-DSOD in the real world, actively attempting to optimize and innovate traditional model architectures to meet diverse needs in different scenarios. Traditional backbone networks, such as VGG and ResNet, have demonstrated good performance in tasks such as image feature extraction due to their complex and deep structures. However, with the continuous development of deep learning technology, especially in resource-constrained real-world applications, these traditional networks suffer from problems such as massive computational costs and limitations on mobile devices. Therefore, lightweight backbone networks such as ShuffleNet, MobileNet, and EfficientNet have gradually become research hotspots. They feature high-precision computation, fewer parameters, and fewer convolutional layers and filters, significantly reducing computational costs compared to traditional convolutional neural networks, making them very suitable for handling limited computing resources. However, when lightweight backbone networks perform feature extraction, due to the simplification of the network structure, there will inevitably be a certain degree of information loss, which will lead to a decrease in feature representation ability.

[0003] Therefore, it is necessary to design a lightweight model to achieve an efficient and accurate RGB-D salient object detection method while minimizing information loss. Summary of the Invention

[0004] In view of the above problems, the purpose of this invention is to provide a lightweight two-stream cross-modal interaction network (LSMNet) for RGB-D salient object detection. This method uses a lightweight network for feature extraction and effective cross-modal interaction and integration. This can compensate for the information loss caused by the lightweight network and effectively aggregate the salient features contained in RGB and depth images, thereby improving the performance of the RGB-D salient object detection model.

[0005] This invention provides a lightweight, two-stream, cross-modal interactive RGB-D saliency target detection method, comprising the following steps:

[0006] Step S1: Data preparation

[0007] Obtain the RGB-D dataset for this task for training and testing. A portion of the NJU2K dataset and a portion of the NLPR dataset are used as the training set, and the remaining portion of the NJU2K dataset, NLPR dataset, SIP dataset, STERE dataset, and SSD dataset are used as the test set.

[0008] Step S2: Construct the network model, including: Step S21: Construction of the feature extraction backbone network of the decoder; Step S22: Adaptive cross-modal fusion module (ACM); Step S23: Multi-scale dilated attention module (MSA); Step S24: Decoder; Step S25: Loss function calculation; Step S26: Evaluation metrics.

[0009] As a preferred embodiment of the present invention, step S1 further includes the following step:

[0010] Step S11: The RGB-D dataset includes RGB images, depth images, and manually labeled ground truth images;

[0011] Step S12: First, adjust the resolution of the RGB image and the depth image to 256×256 respectively, and then use them as input to the backbone network.

[0012] As a preferred embodiment of the present invention, step S2 further includes the following step:

[0013] Step S21: Construction of the feature extraction backbone network for the decoder.

[0014] We improved the MobileNetV3-large and MobileNetV3-small networks by using the MobileNetV3 network pre-trained on the ImageNet dataset, and used them as the backbone networks for RGB stream and depth stream, respectively, to extract image features of RGB modality and depth modality;

[0015] In the RGB stream branch, the MobileNetV3-large network starts with a convolutional block, followed by 15 bottleneck blocks. The two sides of each bottleneck block are formed by 1×1 convolutional connections to form an extension layer and a projection layer, respectively. A depthwise separable convolution operation is used in the middle. The input and output of the bottleneck block are connected through residual connections. An SE module is added to the residual connections to extract more representative features. The SE module uses the ReLU function as the activation function in the first half of the network layer and the h-swish function as the activation function in the second half of the network.

[0016] In the deep flow branch, based on the original MobileNetV3-Small network, the basic bottleneck block is retained and a deeper bottleneck block is added to the shallow layer to extract more low-level features for edge supervision. The number of extended layers and output channels of the MobileNetV3-Small network is modified, specifically to include a MobileNetV3-Small network with ten bottleneck blocks. Similarly, the bottleneck blocks use residual connections to speed up convergence while using ReLU and h-swish functions as activation functions to maintain the integrity and accuracy of information.

[0017] As a preferred embodiment of the present invention, step S2 further includes the following step:

[0018] Step S22: Adaptive Cross-Modal Fusion Module (ACM)

[0019] The ACM (Adaptive Cross-Model Fusion) module is used to fuse RGB image features and depth image features. The ACM module consists of a channel expansion layer, a feature fusion layer, and a channel projection layer. In the channel expansion layer, features are expanded to a higher dimension using 1×1 convolutions, ensuring that the RGB and depth flow branches have the same number of channels. In the feature fusion layer, features are concatenated in a lightweight manner for adaptive fusion and filtering. Finally, the channel projection layer uses 1×1 convolutions to project the features to a lower dimension.

[0020] As a preferred embodiment of the present invention, step S22 further includes the following step:

[0021] Step S221: In the channel expansion layer, features of the RGB image and depth image are extracted respectively through a lightweight backbone network. and Next, 1×1 convolutions, batch normalization, and ReLU activation functions are performed on the input RGB image features and depth image features respectively to ensure that the outputs of the RGB stream branch and the depth stream branch have the same dimension, so as to facilitate subsequent feature fusion and processing. The specific process is as follows:

[0022]

[0023] Where BN represents batch normalization, ReLU represents the activation function, and the RGB stream branch and depth stream branch features after channel expansion are used... and This means that if features from the previous layer exist, then the features from the previous layer also need to undergo 1×1 convolution, batch normalization, and ReLU operations to adjust their number of channels to be the same as the number of channels after channel expansion in the RGB flow branch and the depth flow branch. This process is represented as:

[0024]

[0025] Where pre_conv and bil represent 1×1 convolution and bilinear interpolation operations, pre_fea represents the result of the previous layer features after the operation, conact represents feature fusion, f_conv represents the dimensionality reduction operation through convolution, and Fr and Fd represent the results of feature fusion operations between RGB features and depth features and the processed previous layer features, respectively.

[0026] Step S222: In the feature fusion layer, global average pooling and flattening operations are performed on the fused features Fr and Fd respectively to obtain a vector F containing global information of RGB features and depth features. r_global and F d_global This can be expressed as a formula:

[0027] F r_global =global_avg_pool(Fr).flatten(start_dim=1)

[0028] F d_global =global_avg_pool(Fd).flatten(start_dim=1)

[0029] The global feature vector F r_global and F d_global The concatenated values ​​are input to the fully connected layer to calculate the fusion weights. Assuming the weight matrix of the fully connected layer is W, the bias is b, and the activation function is σ, the calculation of the fusion weights can be expressed by the formula:

[0030] fusion_weights=σ([F r_global ;F d_global ]W+b)

[0031] Where [F] r_global ;F d_global The symbol ] indicates concatenation along the feature dimension. σ represents the Softmax function, used to normalize the output to between 0 and 1, resulting in two weight values ​​corresponding to the weights of the RGB and depth features, respectively. The shape is adjusted using the view operation to obtain the weights of the RGB and depth features, as shown in the formula:

[0032] weightrgb =fusion_weights[:,0].view(-1,1,1,1)

[0033] weight depth =fusion_weights[:,1].view(-1,1,1,1)

[0034] `fusion_weights[:,0]` represents taking the first element of each row in the `fusion_weights` tensor, i.e., the RGB weights, to perform element-wise multiplication with the feature map, thus weighting the RGB features. Similarly, `weight`... depth It is a variation of the depth feature weights, used to weight depth features;

[0035] Meanwhile, the most representative features are selected from the RGB and depth features after channel expansion and alignment using an attention mechanism. The enhanced features are expressed by the following formula:

[0036]

[0037] Where SA(·) represents the Shuffle Attention (SA) module, and the feature fusion process is expressed by the formula:

[0038] F_out = Fr_out·weight rgb +Fd_out·weight depth

[0039] Step S223: In the channel projection layer, the features are projected using a combination of a 3×3 convolution with ReLU activation and two 1×1 convolution operations. Therefore, the final output of the ACM module is expressed by the formula:

[0040]

[0041] Where conv_out represents a combination operation of a 3×3 convolution and two 1×1 convolutions containing the ReLU activation function. This represents the final output of the i-th layer ACM module.

[0042] As a preferred embodiment of the present invention, step S2 further includes the following step:

[0043] Step S23: Multi-scale Hollow Attention Module (MSA)

[0044] We utilize the lightweight multi-scale atrous attention (MSA) module of the LSMNet model to extract features across layers and scales. The lightweight multi-scale atrous attention module integrates multi-scale atrous convolution, residual feature enhancement, and attention feature enhancement based on GAP and GMP.

[0045] Step S231: In the multi-scale dilated convolution part, a set of dilated convolutions with different dilation rates are used to extract features. The dilation rate set is defined as: {r1=3, r2=6, r3=9, r4=12}. In the fifth stage of the LSMNet model, only r1 is used for dilated convolution. Similarly, in the second stage of the LSMNet model, r1, r2, r3, and r4 are used to form parallel dilated convolutions. The calculation formula for each dilated convolution branch is as follows:

[0046] F i =σ.BN(conv r (X)) /

[0047] Where, X∈R C×H×W The feature map representing the input, conv r Let represent a dilated convolution operation with a dilation rate of r, BN(·) represent batch normalization, and σ(·) represent the ReLU activation function. Utilizing multi-scale features, after obtaining the outputs of all dilated convolutions, the outputs of all branches are concatenated along the channel dimension, as expressed by the following formula:

[0048]

[0049] This represents the result after concatenation in the i-th stage. Since the number of channels increases after concatenation, 1×1 convolution is used for feature compression to reduce computation and improve fusion performance, as expressed by the following formula:

[0050]

[0051] Step S232: In the residual feature fusion part, the low-level spatial information of the original input is preserved, and the fused features are... Based on this, residual connections are introduced, and the feature representation ability is enhanced by element-wise addition. The enhanced features are expressed by the following formula:

[0052]

[0053] A global attention mechanism is introduced in the MSA module to dynamically adjust feature weights. The global attention mechanism uses a combination of global average pooling (GAP) and global max pooling (GMP) to enhance feature expressiveness. The two methods are expressed by the following formulas:

[0054]

[0055] GMP(X) = max i,j X i,j

[0056] Where X represents the input feature, i.e., the input feature map. Subsequently, the outputs of GAP and GMP are concatenated along the channel dimension to form a global feature representation, as shown in the formula:

[0057]

[0058] Next, 1×1 convolution is used to first reduce the dimensionality and then increase it to extract channel information. The dimensionality increase operation restores the information, enhances the feature representation ability, and ensures complete information transmission. This can be expressed by the following formula:

[0059]

[0060] Where W1 and W2 are the convolution kernels for dimensionality reduction and dimensionality increase operations, respectively; δ(·) represents the ReLU activation function; and σ(·) represents the Sigmoid activation function. Finally, the attention weights are calculated. Acting on residual features As shown in the formula above:

[0061]

[0062] Where ⊙ represents element-wise multiplication.

[0063] As a preferred embodiment of the present invention, step S2 further includes the following step:

[0064] Step S24: Decoder operation,

[0065] In the decoder, the LSMNet model uses four consecutive lightweight multi-scale dilated attention modules (MSA) to progressively pass multi-scale information, gradually recovering high-resolution saliency prediction results to improve the fine-grained representation of salient targets. The decoder starts with the higher-level MSA modules, passing information sequentially from the deepest layer to the shallowest. The output is upsampled and combined with the result of the previous stage's ACM module before being input into the next stage's MSA module to maintain the same resolution. Let the output of the ACM module be... The input to the MSA module is The output is The information transfer between MSA modules can be represented by the following formula:

[0066]

[0067] in, This means that the feature information of the i-th layer is upsampled by bilinear interpolation to match the spatial size of the (i-1)-th layer.

[0068] As a preferred embodiment of the present invention, step S2 further includes the following step:

[0069] Step S25: Loss function calculation,

[0070] The loss function consists of three parts: the structural similarity loss function SSIMLoss, the binary cross-entropy loss function BCELoss, and the intersection-union ratio loss function IOULoss. The formula for calculating the mixed loss function is as follows:

[0071]

[0072] Let represent the hybrid loss function of the i-th layer.

[0073] As a preferred embodiment of the present invention, step S2 further includes the following step:

[0074] Step S26: Evaluation indicators

[0075] The effectiveness of the improved LSMNet model was evaluated using four metrics: F-measure, Mean Absolute Error, S-measure, and E-measure.

[0076] The beneficial effects of this invention are as follows: In response to the problem that the high computational complexity of existing RGB-D salient object detection methods makes mobile deployment difficult, a lightweight dual-stream cross-modal interactive RGB-D salient object detection method, LSMNet, is proposed. The proposed method effectively balances the relationship between model complexity and model efficiency, maintaining high computational efficiency with low model complexity, and exhibiting strong generalization and accuracy when handling various types of scenarios. Attached Figure Description

[0077] Other objects and results of the invention will become more apparent and readily understood with reference to the following description taken in conjunction with the accompanying drawings. In the drawings:

[0078] Figure 1 This is a diagram of the overall network structure in this invention;

[0079] Figure 2This is a structural diagram of the ACM module in this invention;

[0080] Figure 3 This is a structural diagram of the MSA module in this invention; Detailed implementation method:

[0081] See Figure 1-3 This embodiment provides a lightweight two-stream cross-modal interactive RGB-D saliency target detection method, including the following steps:

[0082] Step S1: Data preparation

[0083] The RGB-D dataset for this task is obtained for training and testing. A portion of the NJU2K dataset and a portion of the NLPR dataset are used as the training set, and the remaining portion of the NJU2K dataset, NLPR dataset, SIP dataset, STERE dataset, and SSD dataset are used as the test set. All datasets used in this embodiment are publicly available datasets.

[0084] Step S11: The RGB-D dataset includes RGB images, depth images, and manually labeled ground truth maps; for each sample in the dataset, there is an RGB image, a corresponding depth image, and a corresponding manually labeled ground truth map.

[0085] Step S12: First, adjust the resolution of the RGB image and the depth image to 256×256 respectively, and then use them as input to the backbone network.

[0086] Step S2: Construct the network model.

[0087] Step S21: Construction of the feature extraction backbone network for the decoder.

[0088] We improved the MobileNetV3-large and MobileNetV3-small networks by using the MobileNetV3 network pre-trained on the ImageNet dataset, and used them as the backbone networks for RGB stream and depth stream, respectively, to extract image features of RGB modality and depth modality;

[0089] In the RGB stream branch, to adapt to the RGB-D salient object detection task for feature extraction, the last pooling layer and the last three convolutional layers of the original MobileNetV3-Large network were removed, while some bottleneck blocks from the original MobileNetV3-Large network were retained. That is, the MobileNetV3-Large network starts with one convolutional block followed by 15 bottleneck blocks. These bottleneck blocks employ a special structure: the two sides of each of the 15 bottleneck blocks are formed by 1×1 convolutional connections to create an extension layer and a projection layer, respectively, with depthwise separable convolution operations in the middle. This structure significantly reduces the number of parameters and computational cost while maintaining feature representation capabilities. The input and output of the bottleneck blocks are connected via residual connections, which not only helps alleviate the gradient vanishing problem but also allows for the learning of more complex feature representations and accelerates convergence. SE modules are added to the residual connections to extract more representative features. The SE module uses ReLU as the activation function in the first half of the network layer and h-swish as the activation function in the second half. This improves the model's computational efficiency and generalization ability while enhancing feature representation.

[0090] In the deep flow branch, because depth images contain less information compared to RGB images, to adapt the network to the RGB-D salient object detection task, the basic bottleneck block is retained on the original MobileNetV3-Small network, and a deeper bottleneck block is added to the shallow layer to extract more low-level features for edge supervision. The number of extended layers and output channels of the MobileNetV3-Small network are also modified, resulting in a MobileNetV3-Small network with ten bottleneck blocks. Similarly, the bottleneck blocks use residual connections to speed up convergence while using ReLU and h-swish functions as activation functions to maintain the integrity and accuracy of information, improve the computational speed of inference, and help the model learn higher-level semantic information.

[0091] Step S22: Adaptive Cross-Modal Fusion Module (ACM)

[0092] RGB images and depth images belong to different modalities of data. RGB images can help models extract more detailed features of objects. Depth maps are suitable for models to understand more complex scenes and the relative positions of objects in images. In most cases, RGB images and depth images can enhance each other, improving feature representation. The ACM (Adaptive Cross-Model Fusion) module is used to fuse RGB image features and depth image features, aiming to effectively integrate feature information from different modalities while considering the influence of the previous layer to improve the final detection performance.

[0093] Because lightweight backbone networks contain fewer convolutional blocks and fewer filters in their layers, their feature extraction capabilities are inevitably weaker. To address this, the ACM module is designed with an inverted structure, consisting of a channel expansion layer, a feature fusion layer, and a channel projection layer. In the channel expansion layer, features are expanded to a higher dimension using 1×1 convolutions, ensuring the RGB and depth flow branches have the same number of channels. In the feature fusion layer, features are concatenated using a lightweight approach for adaptive fusion and filtering. Finally, the channel projection layer uses 1×1 convolutions to project features to a lower dimension. This approach increases the number of channels for feature representation in the shallower layers of the network, further enhancing the model's expressive power. In the deeper layers, it avoids a significant reduction in the number of channels during channel projection, preventing information loss. This also maintains model accuracy and significantly improves running speed in resource-constrained scenarios such as mobile devices.

[0094] Step S221: In the channel expansion layer, features of the RGB image and depth image are extracted respectively through a lightweight backbone network. and Subsequently, because the network uses an asymmetric backbone, different branches may output inconsistent channels. Therefore, 1×1 convolutions, batch normalization, and ReLU activation functions are performed on the input RGB image features and depth image features respectively to ensure that the outputs of the RGB stream branch and the depth stream branch have the same dimension, facilitating subsequent feature fusion and processing. The specific process is as follows:

[0095]

[0096] Where BN represents batch normalization, ReLU represents the activation function, and the RGB stream branch and depth stream branch features after channel expansion are used... and It should be noted that if features from the previous layer exist, then those features also need to undergo 1×1 convolution, batch normalization, and ReLU operations to adjust their channel count to be the same as the channel count after channel expansion in the RGB and depth flow branches. This process is represented as follows:

[0097]

[0098] Where pre_conv and bil represent 1×1 convolution and bilinear interpolation operations, pre_fea represents the result of the previous layer features after the operation, conact represents feature fusion, f_conv represents the dimensionality reduction operation through convolution, and Fr and Fd represent the results of feature fusion operations between RGB features and depth features and the processed previous layer features, respectively.

[0099] Step S222: In the feature fusion layer, perform global average pooling (GAP) and flattening operations on the fused features Fr and Fd respectively to obtain a vector F containing global information of RGB features and depth features. r_global and F d_global This can be expressed as a formula:

[0100] F r_global =global_avg_pool(Fr).flatten(start_dim=1)

[0101] F d_global =global_avg_pool(Fd).flatten(start_dim=1)

[0102] The global feature vector F r_global and F d_global The concatenated values ​​are input to the fully connected layer to calculate the fusion weights. Assuming the weight matrix of the fully connected layer is W, the bias is b, and the activation function is σ, the calculation of the fusion weights can be expressed by the formula:

[0103] fusion_weights=σ([F r_global ;F d_global ]W+b)

[0104] Where [F] r_global ;F d_global The symbol ] indicates concatenation along the feature dimension. σ represents the Softmax function, used to normalize the output to between 0 and 1, resulting in two weight values ​​corresponding to the weights of the RGB and depth features, respectively. Because the calculated fusion_weights is a two-dimensional tensor, its shape needs to be adjusted using the view operation to obtain the weights of the RGB and depth features respectively, as shown in the formula:

[0105] weight rgb =fusion_weights[:,0].view(-1,1,1,1)

[0106] weight depth =fusion_weights[:,1].view(-1,1,1,1)

[0107] `fusion_weights[:,0]` represents taking the first element of each row in the `fusion_weights` tensor, i.e., the RGB weights, to perform element-wise multiplication with the feature map, thus weighting the RGB features. Similarly, `weight`... depth It is a variation of the depth feature weights, used to weight depth features;

[0108] Meanwhile, the most representative features are selected from the RGB and depth features after channel expansion and alignment using an attention mechanism. The enhanced features are expressed by the following formula:

[0109]

[0110] Where SA(·) represents the Shuffle Attention (SA) module, and the feature fusion process is expressed by the formula:

[0111] F_out = Fr_out·weight rgb +Fd_out·weight depth

[0112] Step S223: In the channel projection layer, the features are projected using a combination of a 3×3 convolution with ReLU activation and two 1×1 convolution operations. Therefore, the final output of the ACM module is expressed by the formula:

[0113]

[0114] Where conv_out represents a combination operation of a 3×3 convolution and two 1×1 convolutions containing the ReLU activation function. This represents the final output of the i-th layer ACM module.

[0115] Step S23: Multi-scale Hollow Attention Module (MSA)

[0116] In RGB-D salient object detection tasks, efficient feature extraction and multi-scale information fusion are crucial. To this end, we utilize the lightweight Multi-Scale Atrous Attention (MSA) module of the LSMNet model to extract features across layers and scales. This module integrates multi-scale dilated convolution, residual feature enhancement, and attention feature enhancement based on GAP and GMP, effectively capturing contextual information while preserving spatial details, thus improving the performance of salient object detection.

[0117] Step S231: In the multi-scale dilated convolution part, a set of dilated convolutions with different dilation rates are used to extract features. The dilation rate set is defined as: {r1=3, r2=6, r3=9, r4=12}. In the fifth stage of the LSMNet model, only r1 is used for dilated convolution. Similarly, in the second stage of the LSMNet model, r1, r2, r3, and r4 are used to form parallel dilated convolutions. The calculation formula for each dilated convolution branch is as follows:

[0118] F i =σ.BN(conv r (X)) /

[0119] Where, X∈R C×H×W The feature map representing the input, conv r Let represent a dilated convolution operation with a dilation rate of r, BN(·) represent batch normalization, and σ(·) represent the ReLU activation function. Utilizing multi-scale features, after obtaining the outputs of all dilated convolutions, the outputs of all branches are concatenated along the channel dimension, as expressed by the following formula:

[0120]

[0121] This represents the result after concatenation in the i-th stage. Since the number of channels increases after concatenation, 1×1 convolution is used for feature compression to reduce computation and improve fusion performance, as expressed by the following formula:

[0122]

[0123] Step S232: In the residual feature fusion part, the low-level spatial information of the original input is preserved, and the fused features are... Based on this, residual connections are introduced, and the feature representation ability is enhanced by element-wise addition. The enhanced features are expressed by the following formula:

[0124]

[0125] This operation preserves the details of the underlying features, which helps improve the performance of salient object detection.

[0126] To further enhance the adaptability of features, a global attention mechanism is introduced into the MSA module to dynamically adjust feature weights. This global attention mechanism combines global average pooling (GAP) and global max pooling (GMP) to enhance feature expressiveness; the two methods are expressed by the following formulas:

[0127]

[0128] GMP(X) = max i,j X i,j

[0129] Where X represents the input feature, i.e., the input feature map. Subsequently, the outputs of GAP and GMP are concatenated along the channel dimension to form a global feature representation, as shown in the formula:

[0130]

[0131] Next, channel information is extracted using a 1×1 convolution method that first reduces dimensionality and then increases it. The advantage of this approach is that dimensionality reduction reduces computation and improves model efficiency. Dimensionality increase restores information, enhances feature representation, and ensures complete information transfer, as expressed by the following formula:

[0132]

[0133] Where W1 and W2 are the convolution kernels for dimensionality reduction and dimensionality increase operations, respectively; δ(·) represents the ReLU activation function; and σ(·) represents the Sigmoid activation function. Finally, the attention weights are calculated. Acting on residual features As shown in the formula above:

[0134]

[0135] Where ⊙ represents element-wise multiplication.

[0136] Step S24: Decoder operation,

[0137] In the decoder, the LSMNet model uses four consecutive lightweight multi-scale dilated attention modules (MSA) to progressively pass multi-scale information, gradually recovering high-resolution saliency prediction results to improve the fine-grained representation of salient targets. The decoder starts with the higher-level MSA modules, passing information sequentially from the deepest layer to the shallowest. The output is upsampled and combined with the result of the previous stage's ACM module before being input into the next stage's MSA module to maintain the same resolution. Let the output of the ACM module be... The input to the MSA module is The output is The information transfer between MSA modules can be represented by the following formula:

[0138]

[0139] in, This means that the feature information of the i-th layer is upsampled using bilinear interpolation to match the spatial size of the (i-1)-th layer. This fusion method can effectively utilize the global information of high-level features and combine it with the spatial details of low-level features to achieve more refined salient object detection.

[0140] Step S25: Loss function calculation,

[0141] The loss function consists of three parts: the structural similarity loss function SSIMLoss, the binary cross-entropy loss function BCELoss, and the intersection-union ratio loss function IOULoss. The formula for calculating the mixed loss function is as follows:

[0142]

[0143] Let represent the hybrid loss function of the i-th layer.

[0144] Step S26: Evaluation indicators

[0145] The effectiveness of the improved LSMNet model was evaluated using four evaluation metrics: F-measure, Mean Absolute Error, S-measure, and E-measure.

[0146] In this embodiment, the model training phase...

[0147] The model input consisted of an RGB image and a three-channel depth image copied three times, with an image resolution of 960×1080. Before inputting the training images into the model, the image size was adjusted to 256×256. Various data augmentation techniques were employed, including random flipping, cropping, and rotation. The initial learning rate was set to 1e-4, decreasing to 1% of its original value every 60 epochs. The batch size was 8. The Adam algorithm was used as the optimizer to optimize the network, and the entire model was run for 200 epochs.

[0148] Model testing phase

[0149] First, the resolution of the RGB and depth images is adjusted to 256×256 to maintain consistency with the training phase, ensuring the consistency and stability of the model input data. The images with adjusted resolution are then input into the trained network. After a series of feature extraction, fusion, and processing steps, the final saliency prediction results are obtained.

[0150] Four commonly used evaluation metrics are used to assess the effectiveness of a network: F-measure, Mean Absolute Error, S-measure, and E-measure. The detailed definitions of these four metrics are as follows:

[0151] F-measure(F β Precision and recall are widely used comprehensive evaluation metrics that consider both precision and recall scores, and are defined as follows:

[0152]

[0153] Where Precision and Recall represent the precision score and recall score, respectively, β 2 Setting it to 0.3 emphasizes precision;

[0154] Mean Absolute Error (MAE) represents the average error between the significance plot S and the true value Y, and it is defined as follows:

[0155]

[0156] Where W and H represent the width and height of the saliency plot, respectively;

[0157] S-measure(S λ )Assess the significance of the plot S r With salience label plot S o The spatial structural similarity between regions, taking into account the salience of regions and boundaries, is also known as a structured measurement, and is defined as follows:

[0158] S λ =α*S o +(1-α)*S r ;

[0159] Where α∈[0,1] is the balance parameter, which is set to 0.5 by default;

[0160] E-measure(E γ The saliency map is an evaluation metric based on the enhanced alignment mapping between the saliency map and the saliency label map, and it is defined as follows:

[0161]

[0162] Where W and H represent the width and height of the saliency map, respectively, and φFM(*) represents the enhancement alignment matrix.

[0163] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A lightweight dual-stream cross-modal interaction RGB-D salient object detection method, characterized in that, Comprising the following steps: Step S1: data preparation, Obtain the RGB-D dataset of the task for training and testing, wherein part of the NJU2K dataset and part of the NLPR dataset are taken as the training set, and the remaining part of the NJU2K dataset and the NLPR dataset, the SIP dataset, the STERE dataset and the SSD dataset are taken together as the test set; Step S2: constructing a network model, comprising: step S21: constructing a feature extraction backbone network of a decoder, step S22: an adaptive cross-modal fusion module ACM, step S23: a multi-scale hollow attention module MSA, step S24: a decoder, step S25: loss function calculation, and step S26: evaluation index; Step S22: an adaptive cross-modal fusion module ACM, The ACM module is used to fuse RGB image features and depth image features. The ACM module is divided into a channel expansion layer, a feature fusion layer and a channel projection layer. In the channel expansion layer, the features are expanded to a high dimension through 1x1 convolution to make the RGB stream branch and the depth stream branch have the same number of channels. In the feature fusion layer, the features are connected in a lightweight manner for adaptive fusion and filtering. Finally, in the channel projection layer, the features are projected to a low dimension using 1x1 convolution. Step S221: In the channel expansion layer, the features of the RGB image and the depth image are extracted respectively through the lightweight backbone network and Then, 1x1 convolution, batch normalization and ReLU activation function operations are performed on the input RGB image features and depth image features respectively, so that the outputs of the RGB stream branch and the depth stream branch have the same dimension, so as to perform feature fusion and processing subsequently. The specific process is as follows: ； ； where BN denotes the batch normalization operation, ReLU denotes the activation function, and the RGB stream branch and the depth stream branch features after channel expansion are denoted as and denotes that if the feature of the previous layer exists, the corresponding 1x1 convolution, batch normalization and ReLU operation on the feature of the previous layer are also needed to adjust the channel number to the same as the channel number of the RGB stream branch and the depth stream branch after channel expansion, and this process is denoted as ； ； ； wherein and denote 1x1 convolution and bilinear interpolation operation, denote the result of the previous layer feature after operation, denote feature fusion, denote the operation of dimension reduction by convolution, and denote the result of the RGB feature and the depth feature respectively after the feature fusion operation with the processed previous layer feature respectively; Step S222: In the feature fusion layer, the fused features and respectively perform global average pooling and flattening operation to obtain a vector containing global information of RGB features and depth features and , which is expressed by the formula: ； ； concatenate the global feature vector and , input into a fully connected layer to calculate fusion weights, assuming the weight matrix of the fully connected layer is , the bias is , and the activation function is The calculation of the fusion weight is represented by the formula: ； wherein denotes concatenation over the feature dimension, denotes a Softmax function to normalize the output between 0-1 to get two weight values, corresponding to the weights of the RGB feature and the depth feature, respectively, and the shape is adjusted by the view operation to get the weights of the RGB feature and the depth feature, respectively, as shown in the formula: ； ； Indicates taking The first element of each row in the tensor is the RGB weight, used to perform element-wise multiplication with the feature map, thus weighting the RGB features. Similarly, It is a variation of the depth feature weights, used to weight depth features; At the same time, the RGB features and the depth features aligned by the channel expansion are selected through an attention mechanism to select the most representative features. The enhanced features are represented by the following formula: ； ； wherein, denotes a Shuffle Attention module, and the feature fusion process is represented by the formula ； In the channel projection layer, a 3x3 convolution with a ReLU activation function and two 1x1 convolution operations are combined to project the features. Therefore, the final output of the ACM module is represented by the following formula: ； wherein denotes a combined operation of one 3x3 convolution and two 1x1 convolutions including a ReLU activation function, denotes the output of the first layer ACM module.

2. The lightweight dual-stream cross-modal interaction based RGB-D salient object detection method according to claim 1, characterized in that, In step S1, the following steps are further included: Step S11: The RGB-D dataset includes RGB images, depth images and manually annotated label truth maps; Step S12: First, adjust the resolutions of the RGB images and the depth images to 256x256, and then input them into the backbone network. 3.The lightweight dual-stream cross-modal interaction based RGB-D salient object detection method of claim 1, wherein, In step S2, the following steps are further included: Step S21: constructing a feature extraction backbone network of a decoder, The MobileNetV3-large network and the MobileNetV3-small network are improved using the MobileNetV3 network pre-trained on the ImageNet dataset, which are used as the backbone networks of the RGB stream and the depth stream, respectively, to extract image features of the RGB modality and the depth modality; In the RGB stream branch, the MobileNetV3-large network starts from a convolution block, followed by 15 bottleneck blocks. The two sides of the 15 bottleneck blocks are connected by 1x1 convolution to form an expansion layer and a projection layer. The middle part uses a depth separable convolution operation. The input and output of the bottleneck block are connected through a residual connection. An SE module is added in the residual connection to extract more representative features. The ReLU function is used as the activation function in the first half of the network, and the h-swish function is used as the activation function in the second half of the network; In the deep stream branch, on the basis of the original MobileNetV3-Small network, the basic bottleneck block is retained and one deep bottleneck block is added to the shallow level to extract more bottom features for edge supervision. The expansion layer number and output channel number of the MobileNetV3-Small network are modified. Specifically, the MobileNetV3-Small network contains ten bottleneck blocks. Similarly, the bottleneck block uses residual connection to speed up the convergence speed, and uses ReLU function and h-swish function as activation function to keep the integrity and accuracy of information.

4. The lightweight dual-stream cross-modal interaction based RGB-D salient object detection method according to claim 1, characterized in that, In step S2, the following steps are further included: Step S24: decoder operation, In the decoder, the LSMNet model uses four consecutive lightweight multi-scale hollow attention modules MSA to pass multi-scale information layer by layer, gradually restores the high-resolution saliency prediction result to improve the fine-grained expression ability of the saliency target, and the decoder starts from the MSA module of the high stage, and sequentially passes information from the deepest layer to the shallow layer, performs upsampling operation on the output, and inputs the result of the ACM module of the previous stage into the MSA module of the next stage to maintain the same resolution. Assuming that the output of the ACM module is , the input of the MSA module is , and the output is , the information transmission between the MSA modules is represented by the following formula: ； wherein, representing upsampling the feature information of the first layer by bilinear interpolation to match the spatial size of the second layer. layer by bilinear interpolation to match the spatial size of the second layer. layer by bilinear interpolation to match the spatial size of the second layer.

5. The lightweight dual-stream cross-modal interaction based RGB-D salient object detection method according to claim 4, characterized in that, In step S2, the following steps are further included: Step S25: loss function calculation, The loss function is composed of three parts, namely structural similarity loss function SSIMLoss, binary cross entropy loss function BCELoss and intersection over union loss function IOULoss. The calculation formula of the hybrid loss function is as follows: ； represents the first layer's mixed loss function.

6. The lightweight dual-stream cross-modal interaction RGB-D salient object detection method according to claim 5, characterized in that, In step S2, the following steps are further included: Step S26: evaluation index, Four evaluation indexes are used to evaluate the effectiveness of the improved LSMNet model, and the four evaluation indexes are F-measure, Mean Absolute Error, S-measure and E-measure.

7. The lightweight dual-stream cross-modal interaction based RGB-D salient object detection method according to claim 1, wherein, In step S2, the following steps are further included: Step S23: multi-scale hollow attention module MSA, The lightweight multi-scale hollow attention module of LSMNet model is used to extract cross-layer and cross-scale features. The lightweight multi-scale hollow attention module integrates multi-scale hollow convolution, residual feature enhancement and attention feature enhancement based on GAP and GMP. Step S231: In the multi-scale dilated convolution part, a set of dilated rates different from each other are used to extract features, and the set of dilated rates is defined as: In the fifth stage of the LSMNet model, only is used for dilated convolution. Similarly, in the second stage of the LSMNet model, the following , , and are used for parallel dilated convolution, and the calculation formula of each dilated convolution branch is represented as follows: ； in, ∈ The feature map representing the input. The corresponding expansion rate is indicated by The dilated convolution operation, Representative batch normalization operation, Representing the ReLU activation function, it utilizes multi-scale features and, after obtaining the outputs of all dilated convolutions, concatenates the outputs of all branches along the channel dimension, as expressed by the following formula: ； ∈ represents the result of the first stage splicing. Due to the increase in the number of channels after splicing, in order to reduce the amount of calculation and improve the fusion effect, a 1x1 convolution is used for feature compression, which is expressed by the formula: ； Step S232: In the residual feature fusion part, the original input bottom layer spatial information is retained, and the enhanced feature expression ability is introduced on the basis of the fused feature The enhanced feature is expressed by the following formula: ； Global attention mechanism is introduced in the MSA module to dynamically adjust the weight of the feature. The global attention mechanism uses the combination of global average pooling and global maximum pooling to enhance the feature expression ability. The two methods are represented by the following formulas respectively: ； ； wherein, represents the input feature, i.e., the input feature map, and then the outputs of GAP and GMP are concatenated along the channel dimension to form the global feature representation, which is shown in the following equation: ； Then, 1×1 convolution is used to extract channel information by reducing dimension and then restoring dimension. The dimension restoration operation enhances the feature expression ability and ensures the complete transmission of information, which is represented by the following formula: ； wherein , are the convolution kernels for dimension reduction and dimension increase operations, respectively, represents the ReLU activation function, represents the Sigmoid activation function, and finally, the calculated attention weight acts on the residual features , as shown in the formula: ； wherein represents an element-wise multiplication.

Images