Multimodal salient object detection method and system based on lightweight three-branch encoder-decoder network

Abstract

This paper discloses a multimodal salient object detection method and system based on a lightweight three-branch codec network. The method includes the following steps: Step 1: Obtain and preprocess RGB-D-T datasets for training and testing tasks; Step 2: Build and train a lightweight deep learning-based salient object detection model for RGB-D-T images using the training dataset; Step 3: Use the trained network model to receive the test dataset for salient object detection. After training, the proposed lightweight network model achieves high accuracy in salient object detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and multimodal image processing, and in particular relates to a lightweight RGB-DT salient object detection method and system based on deep learning technology. Background Art

[0002] The salient object detection (SOD) task aims to mimic the human visual attention guidance mechanism in computer systems to detect the most salient objects in various scenes. RGB images have been the primary processing target for SOD tasks in the past. However, in special situations such as complex lighting and shadow scenes, multiple salient objects, and cluttered backgrounds, methods based solely on RGB images struggle to achieve good results. Currently, with the increasing popularity of depth sensors and thermal infrared sensors, an increasing number of SOD tasks are supplementing RGB images with depth and thermal images. Leveraging the spatial information of depth images and the temperature information of thermal infrared images can help improve the detection and localization accuracy of RGB images in various complex scenes. This multimodal approach integrates more comprehensive and rich scene information, effectively compensating for the shortcomings of single-modality methods. This has led to the emergence of RGB-D and RGB-TSOD methods, which have become a hot topic in the field of salient object detection and have made significant progress in recent years. However, multimodal input requires the model to process a wider variety of image features. To better complement the features of different modalities and improve object detection, the network must become wider and longer, resulting in an exponential increase in the number of model parameters and computational complexity. This will make the model unfeasible for deployment on mobile platforms such as robots and smartphones with limited computing resources. Therefore, for the multimodal SOD task, it is urgent to design a lightweight and efficient salient object detection method. Summary of the Invention

[0003] To address the technical challenges of existing multimodal salient object detection models, which suffer from high parameter counts and computational complexity, this paper proposes a lightweight three-branch encoder-decoder network. This network uses a lightweight backbone as an encoder to extract trimodal features at different levels. A parameter-free modal fusion method is then employed for bimodal fusion. The bimodal fusion features are then fed into a bimodal decoder and a trimodal decoder for full modal and hierarchical interaction. This results in an accurate salient object prediction map with a reduced number of parameters and simple computation. After training, the proposed network model achieves lightweight performance while maintaining high accuracy.

[0004] The present invention adopts the following technical solutions:

[0005] A multimodal salient object detection method based on a lightweight three-branch encoder-decoder network has the following basic steps:

[0006] Step 1: Obtain the RGB-DT dataset for training and testing the task and preprocess the image dataset;

[0007] Step 2: Build and use the training dataset to train a lightweight RGB-DT image salient object detection model based on deep learning;

[0008] Step 3: Use the trained network model to receive the test dataset for salient object detection.

[0009] Furthermore, step 1 is as follows:

[0010] Download the publicly available VDT2048 dataset. The original VDT2048 dataset contains 2048 sets of images from different scenes. The training set and test set include 1024 and 1000 sets of images, respectively. Each set contains four images: RGB, depth, thermal, and ground truth images. All images in the training and test sets are used for training and testing, respectively.

[0011] Furthermore, preprocessing includes random image flipping, cropping, and rotation.

[0012] Furthermore, step 2 is as follows:

[0013] Step 2.1: Three-branch encoder extracts features

[0014] The MobileNet-v2 network is used as the backbone network for extracting RGB image features, depth image features, and thermal infrared image features, and serves as the RGB encoder, depth map encoder, and thermal infrared map encoder.

[0015] The RGB image, depth image and thermal infrared image are input into the corresponding RGB encoder, depth image encoder and thermal infrared image encoder respectively, and each generates a 5-level feature map, represented as f i v 、f i d and f i t , where i∈{1,2,3,4,5}.

[0016] Step 2.2: Fusion of two-modal features

[0017] The modal feature fusion network consists of 10 modal information guided fusion modules (MIGF), of which 5 are grouped together. Each group fuses the RGB images at different levels with the corresponding depth images and thermal infrared images to perform two-modal feature fusion. MIGF first uses parameter-free spatial attention to extract spatial features from the RGB images to obtain the spatial features f i sThe parameter-free spatial attention here uses a fixed-weight convolution kernel that does not change during the model training process and contains negligible learning parameters. By using spatial features to optimize and fuse the other modality, the final two-modal fusion feature f is obtained. i vt and f i vd The specific method is as follows:

[0018]

[0019] f i s =NSA(f i v ),i∈{1,2,3,4,5} (2)

[0020]

[0021] Among them, NSA(I) represents the spatial feature extraction of feature map I, f avg () and f max () represents average channel pooling and maximum channel pooling, f 3,1 () represents a 3×3 convolution module with a fixed convolution kernel weight of 1 and a filling rate of 1, f 1,0 () represents a 1×1 convolution module with a fixed convolution kernel weight of 1 and a filling rate of 0. represents element-by-element addition, ⊙ represents element-by-element multiplication, Cat(,) represents channel concatenation, and Sigmoid() represents the use of the Sigmoid activation function.

[0022] Step 2.3: Bimodal Decoder

[0023] After extracting the two-modal features at different levels, the two-modal features are decoded separately so that each layer of features f i vt and f i vd After passing through a 3×3 convolutional layer and upsampling, the features are then channel-wise concatenated with the previous layer's features. Before concatenation, the top three layers of features pass through a context enhancement unit (CEU). This unit consists of three different branches, each of which applies convolution operations with kernels of different sizes to the input feature map to capture high-level semantic features at different scales. During model testing, the three branches of the CEU can be reparameterized into a single branch, reducing the model's computational complexity and parameter count. This is detailed as follows:

[0024]

[0025]

[0026] Among them, Conv 1×1 () and Conv 3×3 () represents the convolution operation with the convolution kernel size of 1×1 and 3×3, UP() represents the upsampling operation, and Cat(,) represents channel splicing. Represents element-by-element addition, and CE() represents a reparameterizable convolution operation of different sizes on the input features.

[0027] Step 2.4: Trimodal Decoder

[0028] The two-modal decoding feature F i vt and F i vd The input is sent to the trimodal decoder for modal and hierarchical interaction. The trimodal decoder specifically includes a semantic feature interaction module and a context enhancement module.

[0029] 1) Semantic feature interaction module

[0030] The high-level decoder features of the two modalities RGB-T and RGB-D and Perform channel attention cross fusion and obtain trimodal features of the same size through convolution and upsampling After multiplying these features, the multi-level semantic weight S is obtained through global average pooling. vdt .

[0031] Then, the trimodal features F at each level vdt Through the dilated convolution with different expansion rates, different receptive field semantic information is captured from multi-level features to obtain semantic features at multiple levels. Semantic features at different levels can be fused using channel splicing, and the fused features are combined with the weight S vdt Multiply the three modal high-level semantic features H vdt The details are as follows:

[0032]

[0033] Among them, Conv 3×3 () represents a convolution operation with a convolution kernel size of 3×3, UP() represents an upsampling operation, Cat(,) represents channel splicing, C3(), C5(), and C7() represent dilated convolution operations with dilation rates of 3, 5, and 7, respectively. Represents dimension-broadcasted multiplication.

[0034] 2) Semantic feature enhancement module

[0035] The bottom two-modal features are fused into trimodal bottom-level detail features by element-by-element addition. Where i = {1, 2}. The trimodal high-level semantic features H vdtInput, use parameter-free spatial attention to obtain H vdt The spatial weight of the underlying features and Perform weighted optimization to obtain and The details are as follows:

[0036]

[0037] Among them, UP() represents upsampling operation, Cat(,) represents channel splicing, represents element-wise addition, ⊙ represents element-wise multiplication, and NSA() represents parameter-free spatial attention. Finally, the outputs of the two-modal decoder and the three-modal decoder are convolved to generate the saliency map P. vd 、P vt and P vdt , as follows:

[0038]

[0039] Among them, Conv 1×1 () indicates a convolution operation with a convolution kernel size of 1×1, UP() indicates an upsampling operation, Represents element-wise addition.

[0040] Step 2.5: Train the three-stream encoder-decoder network using the dataset obtained in step 1

[0041] The saliency map P predicted by the present invention vd 、P vt and P vdt The mixed loss function is calculated with the ground truth graph G, optimized using the Adam algorithm, and the learning rate is adjusted using the cosine annealing strategy. A total of 200 rounds of training are performed. The mixed loss function calculation formula is as follows:

[0042]

[0043] L(x,y)=L BCE (x,y)+L IoU (x,y)+L SSIM (x,y) (11)

[0044] Among them, x and y represent the salient object and the true value respectively, μ x and μ y are the mean values ​​of image x and y respectively, and are the variances of images x and y, σ xy is the covariance of images x and y, c1 and c2 are small constants. The final loss function is defined as:

[0045] L all =L(P vd ,G)+L(P vt ,G)+L(P vdt ,G) (12)

[0046] Furthermore, step 3 is as follows:

[0047] First, load the network model parameter weights trained in step 2 and update the parameters in the model. Second, pass the test set obtained in step 1 as input data into the network model to obtain the final saliency prediction map P for detection. vdt .

[0048] Furthermore, after step 3, the significance prediction map is evaluated by MAE, E-measure, F-measure and S-measure evaluation indicators.

[0049] The present invention also discloses a multimodal salient object detection system based on a lightweight three-branch encoding and decoding network, which includes the following modules based on the above method:

[0050] Dataset acquisition module: obtains RGB-DT datasets for training and testing tasks and preprocesses the datasets;

[0051] Object detection model training module: Build and use the training dataset to train a lightweight RGB-DT image salient object detection model based on deep learning;

[0052] Salient object detection module: Use the trained network model to receive the test dataset for salient object detection.

[0053] The present invention proposes a lightweight three-branch encoding and decoding network. Specifically, the network fuses different modalities with lower computational complexity and fewer parameters. The proposed dual-modal decoder fully interacts with the multi-level features of the two modalities. In order to better fuse the three modalities, the tri-modal decoder proposed in the present invention explores the semantic features of the three modalities at different scales and absorbs the specific features of the two modalities through the context enhancement module to further integrate the information between the different modalities. After training, the lightweight network model proposed in the present invention has a high accuracy in salient target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a network model structure of a preferred embodiment of the present invention;

[0055] Figure 2 This is a structural diagram of a modal information guided fusion module according to a preferred embodiment of the present invention;

[0056] Figure 3 Provides a modular structure diagram for the context of a preferred embodiment of the present invention;

[0057] Figure 4 This is a structural diagram of a semantic interaction module according to a preferred embodiment of the present invention;

[0058] Figure 5 This is a structural diagram of a semantic enhancement module according to a preferred embodiment of the present invention;

[0059] Figure 6 This is a result diagram of a preferred embodiment of the present invention.

[0060] Figure 7 This is a flow chart of a multimodal salient target detection method based on a lightweight three-branch encoding and decoding network in a preferred embodiment of the present invention.

[0061] Figure 8 This is a block diagram of a multimodal salient object detection system based on a lightweight three-branch encoding and decoding network in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0062] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and preferred embodiments.

[0063] like Figure 1-7 As shown in FIG, this embodiment is a multimodal salient object detection method based on a lightweight three-branch encoding and decoding network, and its basic steps are:

[0064] Step 1: Obtain the RGB-DT dataset for training and testing tasks and preprocess the dataset.

[0065] Dataset acquisition means obtaining the image dataset required for network training in step 2.

[0066] Download the publicly available VDT2048 dataset. The original VDT2048 dataset contains 2048 sets of images from different scenes. The training set and test set include 1024 and 1000 sets of images, respectively. Each set contains four images: RGB, depth, thermal, and ground truth images. All images in the training and test sets are used for training and testing, respectively.

[0067] Step 2: Build and use the training dataset to train a lightweight RGB-DT image salient object detection model based on deep learning.

[0068] Construct a lightweight three-branch encoding and decoding network model:

[0069] Step 2.1: Three-branch encoder extracts features

[0070] The MobileNet-v2 network is used as the backbone network for salient object detection. The global average pooling and the last fully connected layer in the backbone are deleted, and a three-branch encoder is constructed to extract RGB image features, depth image features, and thermal infrared image features respectively.

[0071] The depth map is spliced ​​into three channels through the channel and input into the depth map encoder to generate 5 levels of feature maps, where the 5 levels of depth map representation are and Similarly, the RGB image and depth image also generate 5 feature maps respectively. The output scales of the 5 feature maps are: 16×176×176, 24×88×88, 32×44×44, 96×22×22, 320×11×11 (number of channels × height × width).

[0072] Step 2.2: Fusion of two-modal features.

[0073] Reference Attachment Figure 2 The modal feature fusion network consists of 10 modal information guided fusion modules (MIGFs), of which 5 are grouped together. Each group fuses the RGB images at different levels with the corresponding depth images and thermal infrared images. The modal information guided fusion module first extracts spatial features from the RGB image using parameter-free spatial attention to obtain the spatial features f i s The parameter-free spatial attention here uses a fixed-weight convolution kernel that does not change during the model training process and contains negligible learning parameters. By using spatial features to optimize and fuse the other modality, the final two-modal fusion feature f is obtained. i vt and f i vd The specific method is as follows:

[0074]

[0075] f i s =NSA(f i v ),i∈{1,2,3,4,5} (2)

[0076]

[0077] Among them, f avg () and f max () represents average channel pooling and maximum channel pooling, f 3,0 () represents a 3×3 convolution module with a fixed convolution kernel weight of 1 and a filling rate of 0, f 1,1() represents a 1×1 convolution module with a fixed convolution kernel weight of 1 and a filling rate of 1. represents element-by-element addition, ⊙ represents element-by-element multiplication, Cat(,) represents channel concatenation, and Sigmoid() represents the use of the Sigmoid activation function.

[0078] Step 2.3: Bimodal Decoder

[0079] Reference Attachment Figure 3 After extracting the two-modal features at different levels, the two-modal features are decoded respectively so that each layer of features f i vt and f i vd After passing through a 3×3 convolutional layer and upsampling, the features are then channel-wise concatenated with the previous layer's features. Before concatenation, the top three layers of features pass through a semantic feature enhancement unit. This unit consists of three different branches, each of which uses convolution operations with kernels of different sizes on the input feature map to capture high-level semantic features at different scales. During the model testing phase, the three branches of the semantic feature enhancement unit can be reparameterized into a single branch, reducing the model's computational complexity and parameter count. The details are as follows:

[0080]

[0081]

[0082] Among them, Conv 1×1 () and Conv 3×3 () represents the convolution operation with the convolution kernel size of 1×1 and 3×3, UP() represents the upsampling operation, and Cat(,) represents channel splicing. Represents element-by-element addition, and CE() represents a reparameterizable convolution operation of different sizes on the input features.

[0083] Step 2.4: Trimodal Decoder

[0084] Decoding features of the two modalities and The input is sent to the trimodal decoder for modal and hierarchical interaction. The trimodal decoder specifically includes a semantic feature interaction module and a context enhancement module.

[0085] 1) Semantic feature interaction module

[0086] Reference Attachment Figure 4 , the two-modal high-level decoder features of RGB-T and RGB-D and Perform channel attention cross fusion and obtain trimodal features of the same size through convolution and upsampling After multiplying these features, we use global average pooling to obtain multi-level semantic weights. The details are as follows:

[0087]

[0088] Trimodal features at each level Through the dilated convolution with different expansion rates, different receptive field semantic information is captured from multi-level features to obtain semantic features at different levels. Semantic features at different levels are fused using channel splicing, and the fused features are combined with the weight S vdt Multiply the three modal high-level semantic features H vdt The details are as follows:

[0089]

[0090] Among them, Conv 3×3 () represents a convolution operation with a convolution kernel size of 3×3, UP() represents an upsampling operation, Cat(,) represents channel splicing, C3(), C5(), and C7() represent dilated convolution operations with dilation rates of 3, 5, and 7, respectively. Represents dimension-broadcasted multiplication.

[0091] 2) Semantic Enhancement Module

[0092] Reference Attachment Figure 5 , the bottom two modal features are fused into trimodal bottom detail features by element-by-element addition Where i = {1, 2}. The trimodal high-level semantic features H vdt Input, use parameter-free spatial attention to obtain H vdt The spatial weight of the underlying features and Perform weighted optimization to obtain and The details are as follows:

[0093]

[0094] Among them, UP() represents upsampling operation, Cat(,) represents channel splicing, represents element-wise addition, ⊙ represents element-wise multiplication, and NSA() represents parameter-free spatial attention. Finally, the outputs of the two-modal decoder and the three-modal decoder are convolved to generate the saliency map P. vd 、P vt and P vdt , as follows:

[0095]

[0096] Among them, Conv 1×1() indicates a convolution operation with a convolution kernel size of 1×1, UP() indicates an upsampling operation, Represents element-wise addition.

[0097] Step 2.5: Train the three-stream encoder-decoder network using the dataset obtained in step 1

[0098] The saliency map P predicted by the present invention vd 、P vt and P vdt The mixed loss function is calculated with the ground truth graph G, optimized using the Adam algorithm, and the learning rate is adjusted using the cosine annealing strategy. A total of 200 rounds of training are performed. The mixed loss function calculation formula is as follows:

[0099]

[0100] L(x,y)=L BCE (x,y)+L IoU (x,y)+L SSIM (x,y) (11)

[0101] Among them, x and y represent the salient object and the true value respectively, μ x and μ y are the mean values ​​of image x and y respectively, and are the variances of images x and y, σ xy is the covariance of images x and y, c1 and c2 are small constants. The final loss function is defined as:

[0102] L all =L(P vd ,G)+L(P vt ,G)+L(P vdt ,G) (12)

[0103] Step 3: Use the trained network model to receive the test dataset for salient object detection.

[0104] First, load the network model parameter weights trained in step 2. Pass the test set obtained in step 1 as input data into the network model to obtain the final saliency prediction map P for detection. vdt , Figure 6 The effect diagram of the present invention is shown.

[0105] Finally, the significance prediction map is evaluated by MAE, E-measure, F-measure and S-measure evaluation indicators.

[0106] Table 1 shows the quantitative comparison of the salient object detection algorithm proposed in this paper with the current mainstream algorithms. The algorithm proposed in this paper is significantly better than the existing algorithms in terms of MAE, E-measure, F-measure and S-measure evaluation indicators.

[0107] Table 1

[0108] method MAE E-measure F-measure S-measure CIR 0.0043 0.7899 0.5978 0.9053 DCFM 0.0038 0.9219 0.7473 0.9054 HAI 0.0033 0.9689 0.8287 0.9124 MGAI 0.0029 0.9527 0.8158 0.9190 Ours 0.0024 0.9845 0.8769 0.9310

[0109] like Figure 8 As shown, this embodiment discloses a multimodal salient object detection system based on a lightweight three-branch encoding and decoding network, based on the above method embodiment, including the following modules:

[0110] Dataset acquisition module: obtains RGB-DT datasets for training and testing tasks and preprocesses the datasets;

[0111] Object detection model training module: Build and use the training dataset to train a lightweight RGB-DT image salient object detection model based on deep learning;

[0112] Salient object detection module: Use the trained network model to receive the test dataset for salient object detection.

[0113] For other contents of this embodiment, please refer to the above method embodiment.

[0114] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

[0115] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

Claims

1. A multimodal salient object detection method based on a lightweight three-branch encoder-decoder network, characterized by Follow these steps: Step 1: Obtain the RGB-DT dataset for training and testing tasks and preprocess the dataset; Step 2: Build and use the training dataset to train a lightweight RGB-DT image salient object detection model based on deep learning; Step 2 is as follows: Step 2.1: Three-branch encoder extracts features The MobileNet-v2 network is used as the backbone network for extracting RGB image features, depth image features, and thermal infrared image features, and serves as the RGB encoder, depth map encoder, and thermal infrared image encoder. The RGB image, depth image and thermal infrared image are input into the corresponding RGB encoder, depth map encoder and thermal infrared map encoder respectively, and each generates a 5-level feature map, which is expressed as and where i∈{1,2,3,4,5}; Step 2.2: Fusion of two-modal features The modal feature fusion network consists of 10 modal information guided fusion modules MIGF, of which 5 are grouped together. Each group fuses the RGB images at different levels with the corresponding depth images and thermal infrared images for two-modal feature fusion. MIGF first uses parameter-free spatial attention to extract spatial features from the RGB images to obtain spatial features. By using spatial features to optimize and fuse the other modality, the final two-modal fusion features are obtained. and The details are as follows: Among them, NSA(I) represents the spatial feature extraction of feature map I, f avg () and f max () represents average channel pooling and maximum channel pooling, f 3,1 () represents a 3×3 convolution module with a fixed convolution kernel weight of 1 and a filling rate of 1, f 1,0 () represents a 1×1 convolution module with a fixed convolution kernel weight of 1 and a filling rate of 0. represents element-by-element addition, ⊙ represents element-by-element multiplication, Cat(,) represents channel concatenation, and Sigmoid() represents the use of the Sigmoid activation function; Step 2.3: Bimodal Decoder After extracting the two-modal features at different levels, the two-modal features are decoded separately so that each layer of features f i vt and f i vd After passing through a 3×3 convolutional layer and upsampling, the features are then channel-wise concatenated with the features of the previous layer. The features of the top three layers pass through a context enhancement unit before concatenation. This unit consists of three different branches, each of which uses convolution operations with convolution kernels of different sizes on the input feature map to capture high-level semantic features of different scales. During the model testing phase, the three branches of the context enhancement unit are reparameterized into a single branch. The details are as follows: Among them, Conv 1×1 () and Conv 3×3 () represents the convolution operation with the convolution kernel size of 1×1 and 3×3, UP() represents the upsampling operation, and Cat(,) represents channel splicing. Represents element-by-element addition, and CE() represents a reparameterizable convolution operation of different sizes on the input features; Step 2.4: Trimodal Decoder Decoding features of the two modalities and The input is fed into the trimodal decoder for modal and hierarchical interaction; the trimodal decoder specifically includes a semantic feature interaction module and a context enhancement module; Step 2.5: Train the three-stream encoder-decoder network using the dataset obtained in step 1 The predicted saliency map P vd 、P vt and P vdt The mixed loss function is calculated with the true value graph G respectively. The mixed loss function calculation formula is as follows: L(x,y)=L BCE (x,y)+L IoU (x,y)+L SSIM (x,y) (11) Among them, x and y represent the salient object and the true value respectively, μ x and μ y are the mean values ​​of image x and y respectively, and are the variances of images x and y, σ xy is the covariance of images x and y, c1 and c2 are constants; the final loss function is defined as: L all =L(P vd ,G)+L(P vt ,G)+L(P vdt ,G)(12)Step 3: Use the trained network model to receive the test dataset for salient object detection.

2. The multimodal salient object detection method based on a lightweight three-branch encoding / decoding network according to claim 1, characterized in that: Step 1 is as follows: The VDT2048 dataset is used. The VDT2048 dataset contains 2048 groups of pictures from different scenes. The training set and test set include 1048 and 1000 groups of pictures respectively. Each group of pictures contains 4 images, namely RGB image, depth image, thermal infrared image and saliency ground truth image. All the pictures in the training set and test set are used for training and testing respectively.

3. The multimodal salient object detection method based on a lightweight three-branch encoding / decoding network as claimed in claim 2, characterized in that: In step 2.4: 1) Semantic feature interaction module The high-level decoder features of the two modalities RGB-T and RGB-D and Perform channel attention cross fusion and obtain trimodal features of the same size through convolution and upsampling After multiplying these features, the multi-level semantic weight S is obtained through global average pooling. vdt ; The trimodal features F at each level vdt Through the dilated convolution with different expansion rates, the semantic information of different receptive fields is captured from the multi-level features to obtain semantic features of multiple levels; the semantic features of different levels are fused using channel splicing, and the fused features are combined with the weight S vdt Multiply the three modal high-level semantic features H vdt ; The details are as follows: Among them, Conv 3×3 () represents a convolution operation with a convolution kernel size of 3×3, UP() represents an upsampling operation, Cat(,) represents channel splicing, C3(), C5(), and C7() represent dilated convolution operations with dilation rates of 3, 5, and 7, respectively. Represents dimension-broadcast multiplication; 2) Context Enhancement Module The bottom two-modal features are fused into trimodal bottom-level detail features by element-by-element addition. Where i = {1, 2}; the trimodal high-level semantic features H vdt Input, use parameter-free spatial attention to obtain H vdt The spatial weight of the underlying features and Perform weighted optimization to obtain and The details are as follows: Among them, UP() represents upsampling operation, Cat(,) represents channel splicing, Represents element-by-element addition, ⊙ represents element-by-element multiplication, and NSA() represents parameter-free spatial attention; the outputs of the two-modal decoder and the three-modal decoder are respectively convolved to generate the saliency map P vd 、P vt and P vdt , as follows: Among them, Conv 1×1 () indicates a convolution operation with a convolution kernel size of 1×1, UP() indicates an upsampling operation, Represents element-wise addition.

4. The multimodal salient object detection method based on a lightweight three-branch encoding / decoding network as claimed in claim 3, characterized in that: Step 3 is as follows: Load the network model parameter weights generated by step 2 training, pass the test set obtained in step 1 as input data into the network model, and obtain the significance prediction map P for detection vdt .

5. The multimodal salient object detection method based on a lightweight three-branch encoding / decoding network according to any one of claims 1 to 4, characterized in that: After step 3, the MAE, E-measure, F-measure, and S-measure evaluation indicators are used to evaluate the significance prediction map.

6. A multimodal salient object detection system based on a lightweight three-branch codec network, based on the method according to any one of claims 1 to 5, characterized in that Includes the following modules: Dataset acquisition module: obtains RGB-DT datasets for training and testing tasks and preprocesses the datasets; Object detection model training module: Build and use the training dataset to train a lightweight RGB-DT image salient object detection model based on deep learning; Salient object detection module: Use the trained network model to receive the test dataset for salient object detection.

Images