An RGB-D underwater salient object detection method based on semantic guidance fusion

Abstract

The application discloses an RGB-D underwater salient object detection method based on semantic guidance fusion. First, an RGB-D underwater salient object detection dataset for training and testing the task is acquired, and image dataset is preprocessed; then a lightweight RGB-D underwater salient object detection model based on deep learning is constructed and trained using the training dataset; finally, the trained network model is used to receive the test dataset for salient object detection. The network model acquires features of different modalities and fuses them with lower computational complexity and fewer parameters, and the proposed semantic perception module guides the interaction of high-level two-modal features. After training, the lightweight network model has high precision in salient object detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and multi-modal image processing, and particularly relates to a lightweight RGB-D underwater salient object detection method based on deep learning technology. BACKGROUND

[0002] The underwater salient object detection (USOD) task is an important branch of the salient object detection (SOD) task, and its core purpose is to imitate the human visual attention mechanism to identify and highlight the most eye-catching objects in the underwater scene. The development history of USOD is relatively short, and the early underwater detection methods mainly rely on traditional hand-crafted feature-based techniques, which identify salient objects by analyzing basic visual features such as color and texture of the picture. With the advent of convolutional neural networks (CNN) and their remarkable ability in feature extraction, many researchers began to apply CNN to salient object detection. In the traditional SOD task, the main processing is RGB images. However, relying solely on RGB images in dealing with low brightness, multiple salient objects and complex background challenging scenes often does not meet expectations. At present, with the rapid development and wide popularization of depth sensors, the USOD field begins to introduce depth maps, and the spatial information of depth maps can help improve the accuracy of RGB image detection and positioning in complex underwater scenes. However, with the increase in the number of input images, the parameters and computational complexity of the model also increase dramatically. In this particular environment of underwater, the deployment of the model on mobile platforms becomes particularly urgent. Therefore, it is particularly important to develop a lightweight and efficient multi-modal salient object detection technology for the USOD task. SUMMARY

[0003] The technical problem solved: In view of the problem of large model parameter quantity and computational complexity of the above underwater salient object detection model, the present application provides an RGB-D underwater salient object detection method based on semantic guidance fusion.

[0004] The present application proposes a lightweight RGB-D underwater salient object detection model based on deep learning. Specifically, first, a lightweight backbone is used as an encoder to extract 5 levels of modal features, then a semantic perception (SP) module is used to obtain the semantic information of high-level features, in order to use the semantic information to guide the high-level modal fusion, we propose a multi-scale modal fusion (MMF) module for two-modal fusion, which extracts scale information of different modalities and performs modal interaction, this module aims to fully fuse the information between different modalities and reduce the parameter quantity as much as possible. Finally, in the decoding process, we combine the fusion features of different levels by absorbing shallow features by deep features, and generate the final saliency prediction map.

[0005] An RGB-D underwater salient object detection method based on semantic guidance fusion, the basic steps of which are:

[0006] Step 1: Obtain the RGB-D underwater salient object detection dataset for training and testing the task, and pre-process the image dataset;

[0007] Step 2: Construct and train a lightweight RGB-D underwater salient object detection model based on deep learning using the training dataset;

[0008] Step 3: Use the trained network model to receive the test dataset for salient object detection;

[0009] Further, the specific method of step 1 is as follows:

[0010] Download the public dataset USOD10K dataset, the original USOD10k dataset contains 10255 sets of underwater pictures. Among them, the training set, the validation set and the test set respectively include 7178, 2051 and 1026 groups of pictures, each group of pictures contains 3 images, namely RGB image, depth image and salient real image, the whole training set is used to train the model, and the validation set is used to test the trained model. In order to improve the generalization ability of the model, random cropping, horizontal inversion and random rotation operations are performed on the image dataset.

[0011] Further, the specific method of step 2 is as follows:

[0012] The lightweight RGB-D underwater salient object detection model based on deep learning includes an encoder, a semantic perception module, a multi-scale modal fusion module and a decoder.

[0013] Step 2.1: Dual-flow encoder feature extraction;

[0014] A lightweight backbone MobileNet-v2 network is used as an encoder to extract RGB features and depth image features respectively. The RGB image and the depth image are input into the encoder to generate 5 level feature maps respectively, represented as and Where i∈{1,2,3,4,5}.

[0015] Step 2.2: Semantic perception module;

[0016] The semantic perception (SP) module uses the RGB image features and the depth image features output by the 5th layer of the encoder to obtain semantic perception information through an attention mechanism, and then uses the semantic perception information as guidance to optimize the modal fusion of the 3rd, 4th and 5th layer outputs of the encoder.

[0017] The semantic perception module firstly concatenates the RGB feature and the depth feature in the channel to obtain the cross-modal complementary information:

[0018]

[0019] wherein Cat() represents channel concatenation, then average pooling and maximum pooling along the channel dimension are used to obtain the depth semantic weight, and the RGB semantic weight is obtained through the spatial attention mechanism, and the two weights are added to the cross-modal complementary information through element-wise multiplication to obtain the semantic perception information (SPW). Specifically as follows:

[0020]

[0021] wherein Conv 3×3 () represents a convolution operation with a kernel size of 3x3, UP() represents an up-sampling operation, CP avg () and CP max () represent average channel pooling and maximum channel pooling respectively, SA() represents spatial attention, Sigmoid() represents using a Sigmoid activation function, element-wise addition, and represents element-wise multiplication.

[0022] Step 2.3: multi-scale modal fusion module;

[0023] The multi-scale modal fusion module fuses the outputs of each level of the encoder. The inputs of the multi-scale modal fusion module at the 3rd, 4th and 5th levels of the encoder include the RGB feature, the depth feature and the semantic perception information SPW of the corresponding level, while the inputs of the remaining levels do not include the semantic perception information SPW. The input RGB feature is subjected to two convolution kernel 3x3 operations, and the filling rates are set to 1 and 3 respectively. The depth feature is also subjected to two convolution kernel 3x3 operations, and the filling rates are set to 5 and 7 respectively. The outputs of the four branches are concatenated to obtain the multi-scale cross-modal information MSF:

[0024]

[0025] wherein C1(), C3(), C5() and C7() represent the dilated convolution operations with the dilation rates of 1, 3, 5 and 7 respectively, Conv 3×3 () represents a convolution operation with a kernel size of 3x3, and Cat() represents channel concatenation. In order to retain the unique information of the two modalities, the MSF is integrated with the two modal feature and of the corresponding level, and finally the two modal fusion features

[0026]

[0027] Among them: SPW represents semantic perception information, represents element-by-element addition, and ⊙ represents element-by-element multiplication.

[0028] Step 2.4: Decoder;

[0029] The two-modal features at different stages of the encoder Input to the decoder, which is the feature of the 5th layer At the beginning, the features are upsampled and then connected in series with the features of the previous layer to interact hierarchically and restore the resolution. The results are then interacted with the features of the next layer until they are connected with the bimodal features of the first layer of the encoder. The interaction is completed, and the predicted saliency map P is finally obtained. vd , the formula is as follows:

[0030]

[0031] Among them: CBG includes 3×3 convolution operation, batch normalization operation and GELU activation function.

[0032] Step 2.5: Use the dataset obtained in step 1 to train a lightweight RGB-D underwater salient object detection model based on deep learning;

[0033] The training set data obtained in step 1 is input into the constructed lightweight RGB-D underwater salient target detection model based on deep learning, and the predicted salient map P is used to detect underwater salient targets. vd The mixed loss function is calculated with the ground-truth graph G, optimized using the Adam algorithm, and the learning rate is adjusted using the cosine annealing strategy. A total of 200 rounds of training are performed. The mixed loss function includes the binary cross entropy (BCE) loss function, the IoU loss function, and the SSIM loss function. The calculation formula is as follows:

[0034]

[0035] Among them, x and y represent the salient object and the true value respectively, μ x and μ y are the mean values ​​of image x and y respectively, and are the variances of images x and y, σ xy is the covariance of images x and y, c1 and c2 are small constants. The final loss function is defined as:

[0036] L all =L BCE (P vd ,G)+L IoU (P vd ,G)+LSSIM (P vd ,G) (7)

[0037] Further, the specific method of step 3 is as follows:

[0038] Firstly, the network model weight trained by step 2.5 is loaded to update the parameters in the lightweight RGB-D underwater saliency target detection model. Secondly, the verification set obtained in step 1 is input into the network model as input data, so as to obtain the final saliency prediction map P used for detection. vd The saliency prediction map is evaluated by MAE, E-measure, F-measure and S-measure evaluation indexes.

[0039] The present application has the following advantages:

[0040] The present application proposes a lightweight RGB-D underwater saliency target detection model based on deep learning. Specifically, the network model obtains and fuses features of different modalities with lower computational complexity and fewer parameters, and the proposed semantic perception module guides the interaction of high-level two-modal features. After training, the lightweight network model proposed by the present application has high precision for salient target detection. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 Fig. 1 is a network model structure diagram of the present application;

[0042] Figure 2 Fig. 3 is a structure diagram of the semantic perception module of the present application;

[0043] Figure 3 Fig. 4 is a structure diagram of the multi-scale modal fusion module of the present application;

[0044] Figure 4 Fig. 5 is a result comparison diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions of the present application are further described below in combination with the drawings and embodiments.

[0046] Reference is made to the accompanying drawings Figure 1 A RGB-D underwater saliency target detection method based on semantic guidance fusion, and the basic steps are as follows:

[0047] Step 1: data set acquisition;

[0048] The image data set required for network training in step 2 is obtained.

[0049] Download the public data set USOD10K data set, the original USOD10k data set contains 10255 sets of underwater pictures. Among them, the training set, the validation set and the test set include 7178, 2051 and 1026 groups of pictures respectively, each group of pictures contains 3 images, namely RGB image, depth image and saliency real image, we use all the training set to train the model, and use the image number more validation set to test the trained model. In order to improve the generalization ability of the model, we carry out random cropping, horizontal inversion and random rotation and other preprocessing operations on the image data set.

[0050] Step 2: Construct and train a lightweight RGB-D underwater saliency target detection model based on deep learning using the training data set;

[0051] The lightweight RGB-D underwater saliency target detection model based on deep learning comprises an encoder, a semantic perception module, a multi-scale modal fusion module and a decoder.

[0052] Step 2.1: dual-flow encoder extracts features;

[0053] In order to reduce the parameter amount of the model, the application adopts a lightweight backbone MobileNet-v2 network as an encoder to extract RGB features and depth image features respectively. The RGB image and the depth image are input into the encoder to generate 5 level feature maps respectively, represented as and Wherein i belongs to {1, 2, 3, 4, 5}.

[0054] Step 2.2: semantic perception module;

[0055] The high-level features have good semantic information, and the semantic perception (SP) module uses the RGB image features and the depth image features obtained by the encoder 5th layer to obtain semantic perception information through attention mechanism, and then the semantic perception information is used as guidance for modal fusion of the encoder 3rd, 4th and 5th layer output, and since the bottom noise contains detail and edge information, we do not guide it.

[0056] As shown in Figure 2 , the semantic perception module first concatenates the channels of the RGB features and the depth features to obtain cross-modal complementary information:

[0057]

[0058] wherein: Cat(,) represents channel concatenation, then the average pooling and the maximum pooling along the channel dimension are used to obtain the depth semantic weight, and the RGB semantic weight is obtained through the spatial attention mechanism, and the two weights are added in the cross-modal complementary information through element-wise multiplication to obtain the semantic perception information (SPW). Specifically as follows:

[0059]

[0060] wherein: Conv 3×3 () represents a convolution operation with a kernel size of 3x3, UP() represents an up-sampling operation, CP avg () and CP max () represent average channel pooling and maximum channel pooling respectively, SA() represents spatial attention, Sigmoid() represents using a Sigmoid activation function, represents element-wise addition, and represents element-wise multiplication.

[0061] Step 2.3: multi-scale modal fusion module;

[0062] The multi-scale modal fusion module fuses the outputs of each level of the encoder respectively. The inputs of the multi-scale modal fusion module at the 3rd, 4th and 5th levels of the encoder include the RGB features, the depth features and the semantic perception information SPW of the corresponding levels, and the inputs of the remaining levels do not include the semantic perception information SPW. The input RGB features respectively pass through 2 convolution kernels with a size of 3x3, and the filling rates are set to 1 and 3 respectively, and the depth features also respectively pass through 2 convolution kernels with a size of 3x3, and the filling rates are set to 5 and 7 respectively. The outputs of the four branches are concatenated to obtain multi-scale cross-modal information MSF:

[0063]

[0064] wherein: C1(), C3(), C5() and C7() respectively represent the dilated convolution operations with dilations of 1, 3, 5 and 7, Conv 3×3 () represents a convolution operation with a kernel size of 3x3, and Cat(,) represents channel concatenation. In order to retain the unique information of the two modalities, the MSF is integrated with the two modal features and of the corresponding level, and finally the two modal fusion features

[0065]

[0066] wherein: SPW represents semantic perception information, represents element-wise addition, and represents element-wise multiplication. Figure 3Structure diagram of multi-scale modal fusion module of the application

[0067] Step 2.4: decoder

[0068] Two modal features at different stages in the encoder are input into the decoder, which starts from the features at the 5th layer , performs hierarchical interaction after upsampling the features and concatenating them with the previous layer features to restore the resolution, and obtains the result, which continues to perform hierarchical interaction with the features at the next layer until the two modal features at the first layer of the encoder are interacted, and finally the predicted saliency map P vd is obtained .

[0069]

[0070] Wherein CBG includes 3x3 convolution operation, batch normalization operation and GELU activation function.

[0071] Step 2.5: training the lightweight RGB-D underwater saliency object detection model based on deep learning through the data set obtained in step 1;

[0072] The training set data obtained in step 1 is input into the lightweight RGB-D underwater saliency object detection model based on deep learning constructed, the predicted saliency map P vd and the ground truth map G are calculated by a hybrid loss function, optimized by an Adam algorithm, and the learning rate is adjusted by a cosine annealing strategy, and a total of 200 rounds of training are performed. The hybrid loss function includes a binary cross entropy (BCE) loss function, an IoU loss function and an SSIM loss function, and the calculation formula is as follows:

[0073]

[0074] Wherein x and y represent the salient object and the true value respectively, and mu x and mu y are the average values of images x and y respectively, and are the variances of images x and y respectively, and sigma xy is the covariance of images x and y, and c1 and c2 are small constants. Finally, the loss function is defined as:

[0075] L all = L BCE (P vd ,G) + L IoU (P vd ,G) + L SSIM (P vd ,G) (7)

[0076] Step 3: using the trained network model to receive a test data set for salient object detection;

[0077] Firstly, the network model weight trained by step 2.5 is loaded to update the parameters in the lightweight RGB-D underwater salient object detection model. Secondly, the verification set obtained in step 1 is input into the network model as input data, so as to obtain the final saliency prediction map P vd The saliency prediction map is evaluated by MAE, E-measure, F-measure and S-measure evaluation indexes.

[0078] Figure 4 The effect diagram of the present application is shown. The saliency prediction map is evaluated by MAE, E-measure, F-measure and S-measure evaluation indexes. Table 1 shows the quantitative results of the salient object detection algorithm proposed in the present application compared with the current mainstream algorithm. The algorithm proposed in the present application is obviously superior to the existing algorithm in the MAE, E-measure, F-measure and S-measure evaluation indexes.

[0079] Table 1

[0080] Method MAE E-measure F-measure S-measure CIRNet 0.0347 0.9213 0.8621 0.9044 BBSNet 0.0272 0.9382 0.8860 0.9155 HAINet 0.0302 0.9401 0.8852 0.9091 Ours 0.0255 0.9509 0.8966 0.9119

[0081] The above is a further detailed description of the present application in combination with specific / preferred embodiments, and cannot be regarded as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, some alternatives or modifications can be made to the described embodiments without departing from the concept of the present application, and these alternatives or modifications shall be regarded as belonging to the protection scope of the present application.

[0082] The part not described in detail in the present application belongs to the known technology of those skilled in the art.

Claims

1. A method for underwater salient object detection based on RGB-D semantic guidance fusion, characterized by: The basic steps are: Step 1: Obtain the RGB-D underwater salient object detection dataset for training and testing tasks and preprocess the image dataset; Step 2: Build and use the training dataset to train a lightweight RGB-D underwater salient object detection model based on deep learning; The lightweight RGB-D underwater salient object detection model based on deep learning includes an encoder, a semantic perception module, a multi-scale modality fusion module and a decoder; Step 3: Use the trained network model to receive the test dataset for salient object detection; Step 2: Step 2.1: The dual-stream encoder extracts features; The lightweight backbone MobileNet-v2 network is used as the encoder to extract RGB features and depth image features respectively; the RGB image and depth image are input into the encoder, and each generates a 5-level feature map, which is expressed as and where i∈{1,2,3,4,5}; Step 2.2: Semantic perception module; The semantic perception module uses the RGB image features output by the 5th layer of the encoder and deep image features The semantic perception information is obtained through the attention mechanism, and then the semantic perception information is used as a guide to optimize the modal fusion of the encoder's 3rd, 4th and 5th layer outputs; The semantic perception module first concatenates the RGB features and the depth features to obtain cross-modal complementary information: Where: Cat(,) represents channel concatenation, and then average pooling and maximum pooling along the channel dimension are used to obtain the deep semantic weight. At the same time, the RGB semantic weight is obtained through the spatial attention mechanism. The two weights are added to the cross-modal complementary information through element-by-element multiplication to obtain the semantic perception information (SPW); the details are as follows: Among them: Conv 3×3 () indicates the convolution operation with a convolution kernel size of 3×3, UP() indicates the upsampling operation, CP avg () and CP max () represents average channel pooling and maximum channel pooling, SA() represents spatial attention, Sigmoid() represents the use of Sigmoid activation function, represents element-by-element addition, and ⊙ represents element-by-element multiplication; Step 2.3: Multi-scale modality fusion module; The multi-scale modal fusion module fuses the output of each level of the encoder respectively. The input of the multi-scale modal fusion module at the 3rd, 4th and 5th levels of the encoder includes the RGB features, depth features and semantic perception information SPW of the corresponding levels, while the input of the remaining levels does not have semantic perception information SPW; the input RGB features are respectively subjected to two 3×3 dilated convolution operations with filling rates set to 1 and 3, and the depth features are also respectively subjected to two 3×3 dilated convolution operations with filling rates of 5 and 7; the outputs of the four branches are concatenated to obtain the multi-scale cross-modal information MSE: Among them: C1(), C3(), C5() and C7() represent the dilation rates of 1, 3, 5 and 7 respectively. 3×3 () represents the convolution operation with a convolution kernel size of 3×3, and Cat(,) represents channel splicing; in order to retain the unique information of the two modalities, the MSF is combined with the two modal features of the corresponding level. and Integrate and finally obtain the two-modal fusion features Among them: SPW represents semantic perception information, represents element-by-element addition, and ⊙ represents element-by-element multiplication; Step 2.4: Decoder; The two-modal features at different stages of the encoder Input to the decoder, which is the feature of the 5th layer At the beginning, the features are upsampled and then connected in series with the features of the previous layer to interact hierarchically and restore the resolution. The results are then interacted with the features of the next layer until they are connected with the bimodal features of the first layer of the encoder. The interaction is completed, and the predicted saliency map P is finally obtained. vd , the formula is as follows: Among them: CBG includes 3×3 convolution operation, batch normalization operation and GELU activation function; Step 2.5: Use the dataset obtained in step 1 to train a lightweight RGB-D underwater salient object detection model based on deep learning; The training set data obtained in step 1 is input into the constructed lightweight RGB-D underwater salient target detection model based on deep learning, and the predicted salient map P is used to detect underwater salient targets. vd The mixed loss function is calculated with the ground-truth graph G, optimized by the Adam algorithm, and the learning rate is adjusted using the cosine annealing strategy. A total of 200 rounds of training are performed; the mixed loss function includes the binary cross entropy BCE loss function, the IoU loss function, and the SSIM loss function. The calculation formula is as follows: Among them, x and y represent the salient object and the true value respectively, μ x and μ y are the mean values ​​of image x and y respectively, and are the variances of images x and y, σ xy is the covariance of images x and y, c1 and c2 are small constants; the final loss function is defined as: L all =L BCE (P vd ,G)+L IoU (P vd ,G)+L SSIM (P vd ,G) (7)。 2. The RGB-D underwater salient object detection method based on semantic guidance fusion according to claim 1 is characterized in that: Step 1: Download the public dataset USOD10K dataset. The original USOD10k dataset contains 10,255 sets of underwater images; the training set, validation set, and test set include 7,178, 2,051, and 1,026 sets of images, respectively. Each set of images contains three images: an RGB image, a depth image, and a saliency ground truth image. The model is trained using the entire training set, and the validation set is used to test the trained model. To improve the generalization ability of the model, the image dataset is randomly cropped, horizontally flipped, and randomly rotated.

3. The method for underwater salient object detection based on RGB-D semantic guidance fusion according to claim 2 is characterized in that: Step 3: First, load the network model weights trained in step 2.5 and update the parameters in the lightweight RGB-D underwater salient object detection model; secondly, pass the validation set obtained in step 1 as input data into the network model to obtain the final saliency prediction map P for detection. vd ; The significance prediction map is evaluated using MAE, E-measure, F-measure and S-measure evaluation indicators.

Images