RGBD underwater conspicuous object detection method with multiple attention guided by edge

Abstract

To provide an RGBD underwater conspicuous object detection method with multiple attentions guided by edges that effectively reduces redundant information of an underwater image and enhances detection precision and recognition capability for an underwater object.SOLUTION: An underwater conspicuous object detection method includes: a step S1 of acquiring an RMD image for an RGB image and a depth map; a step S2 of inputting the RMD image to a conspicuous object detection model which has been trained and includes an encoder structure and a decoder structure; and a step S3 of acquiring a conspicuous image output from the conspicuous object detection model and an edge image to obtain an object detection result. The encoder generates feature maps sequentially decreasing in resolution, layer by layer, from the RMD image, and the decoder structure includes an edge detection branch used to merge the feature maps, layer by layer, and a conspicuous detection branch used to merge the feature maps and edge feature maps, layer by layer.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to the technical field of image processing, and in particular to an edge-guided RGBD underwater salient object detection method with multi-attention. [Background technology]

[0002] The salient object detection (SOD) task aims to detect the most attention-grabbing objects in a scene and then extract pixel-accurate interest region masks for them. The advantage of salient object detection is that it has many applications, including foreground image cropping, foreground image evaluation, visual tracking, image retrieval, etc.

[0003] In recent years, the salient object detection task has made rapid progress thanks to the ability of convolutional neural networks (CNNs) in multi-scale extraction of deep semantics and shallow details. Based on CNNs, the development of new architectures has also promoted the progress of the salient object detection task, among which the U-Net architecture has attracted the most attention due to its bottom-up simple architecture that can create rich feature maps. The attention mechanism is a mechanism that achieves the role of filtering by highlighting important information and suppressing noise, and in the salient object detection task, the most attention-grabbing objects in the scene are detected. Therefore, since the attention mechanism was proposed, it has been widely applied to the salient object detection task, bringing significant progress to the salient detection task.

[0004] Although the task of salient object detection has made great progress, there are very few studies on underwater scenes. However, research shows that all previously disclosed salient object detection techniques are for land scenes. Currently, methods are proposed to process RGB images and depth maps using two-branch encoders, such as the self-adaptive guided fusion network for RGB-D salient object detection (Patent Document 1), which can effectively improve the effect, but the introduction of the depth map will be redundant unless the improvement in effect compensates for the computational cost. In addition, the channel attention mechanism is introduced to highlight the semantic information of the channel dimension, which makes the model more favorable, but there is still a blurring phenomenon in the edge region.

[0005] Due to the complexity of underwater scenes and the strong absorption and scattering of light by the water medium and suspended particles, the salient object detection task faces challenging factors such as complex backgrounds and color shifts. If traditional SOD methods are directly used on underwater scenes, the effect is poor. However, the ocean contains abundant resources, and the marine ecosystem has high research value. Salient object detection can effectively reduce redundant information in underwater images and improve the detection accuracy and recognition ability of underwater objects, which can effectively assist in the mining of marine resources and the carrying out of marine ecosystem research, so it is necessary to carry out the underwater salient object detection task. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] China Patent Application Publication No. 116452914 Summary of the Invention [Problem to be solved by the invention]

[0007] In view of the shortcomings of the prior art, the present invention provides an edge-guided RGBD underwater salient object detection method with multi-attention. The present invention preprocesses the input image with underwater imaging features, and uses the edge prediction image to guide deep layer salient object detection. The method accurately predicts salient objects and enhances the edge details of the salient image. [Means for solving the problem]

[0008] The technical means of the present invention are as follows.

[0009] The edge-guided RGBD underwater salient object detection method according to one embodiment of the present invention includes the following steps: Step S1: obtaining an RGB image and a depth map, performing channel dimension preprocessing on the RGB image and the depth map, and serially connecting them to obtain an RMD image; Step S2 of inputting the RMD image into a trained salient object detection model including an encoder structure and a decoder structure; and step S3 of obtaining a salient image and an edge image output from the salient object detection model as an object detection result; The encoder generates feature maps with successively reduced resolution layers based on the residual neural network and the RMD image; The decoder structure includes an edge detection branch that is used to fuse feature maps layer by layer and output an edge image, and a saliency detection branch that is used to fuse feature maps and edge feature maps layer by layer and output a saliency image with enhanced edge details.

[0010] Furthermore, the step of performing channel dimension preprocessing on the RGB image and the depth map and serially connecting them to obtain an RMD image includes: taking the maximum value of the G channel and the B channel in the RGB image as the M channel; and serially concatenating the R channel, the M channel and the depth map in the channel dimension to form a new input RMD image.

[0011] Furthermore, the encoder structure comprises: A preprocessing layer that includes one 7×7 convolutional layer and one 2×2 max pooling layer to obtain feature maps whose resolution is reduced to 1 / 4 of the original size; A basic convolution block that includes three convolution layers, each of which uses batch normalization and ReLU activation functions. a skip connection is added between the input and output of each elementary convolution block to avoid vanishing or exploding gradients; Basic convolutional blocks and skip connections are combined into one residual neural module, and the output feature map has multiple scales from shallow to deep layers.

[0012] Further, the edge detection branch includes an edge pyramid convolution module and an edge detection branch spatial attention module; The edge pyramid convolution module inputs the feature map of the deepest layer into a multi-scale convolution module, and further inputs the outputs of the convolutions at different scales into a 1×1 convolution module to output an edge feature map; the edge detection branch spatial attention module receives an edge feature map and a feature map, upsamples the small-scale edge feature map to have the same scale as the large-scale feature map, serially concatenates the edge feature map and the feature map and both pixel-level dot product matrices in a channel dimension to obtain a serially concatenated feature map; The serially connected feature map is converted into spatial attention weights, which are used to correct the spatial features of the serially connected feature map to obtain an edge attention image; First, the obtained edge attention image is residually connected to the original feature map and the upsampled edge feature map, respectively. Then, the two branches are serially connected in the channel dimension, and then a convolution operation is performed. Finally, the edge feature map is output.

[0013] Further, the saliency detection branch includes a scale pyramid convolution module and a saliency detection branch spatial attention module; The scale pyramid convolution module inputs the feature map of the deepest layer into a different scale convolution, and further inputs the output of the multiple different scale convolutions into a 1×1 convolution to output an edge feature map; The saliency detection branch spatial attention module receives the edge feature map, the saliency feature map and the feature map, normalizes the edge feature map by a function to generate an edge attention image for correcting the spatial features of the saliency feature map, and serially connects the corrected saliency feature map and the edge feature map in the channel dimension to obtain an edge-enhanced saliency feature map, generates three tensors, i.e., query, key and value, by convolution and transformation, and inputs them into a self-attention module, and outputs an attention feature map by the self-attention module; The attention feature map, the feature map, and the dot product matrix of both of them are serially connected in the channel dimension to obtain a serially connected feature map; the serially connected feature map is normalized by a function to generate a semantic attention image for correcting the channel semantic feature; and a semantic-augmented feature map is obtained by performing pixel dot product operation between the feature map and the semantic attention image; The semantics-enhanced feature map is residually connected to the original feature map and the attention feature map respectively, and a convolution operation is performed to obtain the final salient feature map.

[0014] Further, the step of training the salient object detection model further comprises: obtaining training set data including land and underwater images and GroundTruth salient object images; training a network using the above land images and underwater images as input data for the network; The method includes a step of calculating a loss value between the salient object image output from the network and the GroundTruth salient object image in the training set, and performing error backpropagation using the loss value to update the weights of the network.

[0015] Further, the step of calculating a loss value between the salient object image output from the network and the salient object image of GroundTruth in the training set, and performing backpropagation using the loss value to update the weights of the network includes: The method includes a step of calculating loss values ​​of the actual saliency image and edge image for the output of each layer of the edge detection branch and the saliency detection branch, respectively, and minimizing the loss values ​​using a stochastic gradient descent method to obtain an optimal model. Effect of the Invention

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] 1. The present invention uses the method of first connecting in series, and introduces a pre-processing module when testing underwater datasets, so that the input is more consistent with the underwater scene, and thus adapts to the characteristics of underwater lighting.

[0018] 2. The present invention proposes to sequentially guide saliency detection by multi-layer edges, solving the problem that information may be lost in the method guided by different resolutions.

[0019] 3. Considering that the spatial features of edge images are more salient, the present invention designs a spatial attention mechanism, which suppresses noise edges, highlights real edges, and makes coarse edge images more refined through weighting operations on edge image pixels, thereby contributing to accurately guiding salient object detection. [Brief description of the drawings]

[0020] In order to more clearly explain the technical means in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings necessary for the description of the embodiments or the prior art. It goes without saying that the following accompanying drawings are some embodiments of the present invention, and those skilled in the art can further obtain other accompanying drawings from these accompanying drawings without creative effort.

[0021] [Figure 1] 2 is a schematic diagram of the structure of a salient object detection model in an embodiment of the present invention; [Diagram 2] FIG. 2 is a structural diagram of an edge pyramid convolution module in an embodiment of the present invention; [Diagram 3] FIG. 13 is a structural diagram of an edge prediction branch spatial attention module in an embodiment of the present invention; [Figure 4] FIG. 2 is a structural diagram of a multi-scale pyramid convolution module in an embodiment of the present invention; [Diagram 5] FIG. 2 is a structural diagram of a saliency detection branch spatial channel attention module in an embodiment of the present invention; [Figure 6] FIG. 2 is a schematic diagram of a training procedure for a salient object detection model in an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] In order to make clearer the purpose, technical means and advantages of the embodiments of the present invention, the technical means of the embodiments of the present invention will be described clearly and completely below with reference to the drawings in the embodiments of the present invention, but it goes without saying that the described embodiments are not all the embodiments but only some of the embodiments of the present invention. Any other embodiments that a person skilled in the art can obtain based on the embodiments of the present invention without any creative effort shall be included in the protection scope of the present invention.

[0023] The present invention provides an edge-guided RGBD underwater salient object detection method with multi-attention, which includes the following steps S1 to S3.

[0024] S1: Obtain an RGB image and a depth map, and serially connect the RGB image and the depth map in the channel dimension to obtain an RMD image.

[0025] Specifically, we use the underwater image pre-processing module. Taking into account the characteristics of underwater light, we take the maximum value of the G channel and B channel in the RGB image to be the M channel, and then serially connect the R channel, M channel and the depth map Depth in the channel dimension to form a new input RMD image, which can be expressed as follows:

number

number

[0026] S2: Input the RMD image into a trained salient object detection model, which includes an encoder structure and a decoder structure, and the encoder generates a feature map with a layer-by-layer sequentially reduced resolution according to the RMD image based on a residual neural network, and the decoder structure includes an edge detection branch and a saliency detection branch, where the edge detection branch is used to fuse the feature map layer-by-layer and output an edge image, and the saliency detection branch is used to fuse the feature map and the edge feature map layer-by-layer and output a salient image with enhanced edge details.

[0027] Furthermore, the encoder uses a residual neural network as the backbone network to generate feature maps with successively reduced resolution layer by layer, which mainly includes a preprocessing layer, basic convolutional blocks, and skip connections, and finally combines multiple basic convolutional blocks and skip connections into one residual neural module.

[0028] Preferably, the preprocessing layer consists of one 7x7 convolution, one batch normalization layer, one ReLU activation function and one 2x2 max pooling layer to obtain a feature map Res1 whose resolution is reduced to 1 / 4 of the original size.

[0029] Preferably, the basic convolution block comprises three convolution layers, the first convolution layer comprising a 1×1 convolution, a batch normalization layer and a ReLU activation function, the second convolution layer comprising a 3×3 convolution with a step size of 2, a batch normalization layer and a ReLU activation function, where the step size is set to 2 to have a downsampling effect, and the third convolution layer comprising a 1×1 convolution and a batch normalization layer.

[0030] Preferably, a skip connection is added between the input and output of each basic convolution block to avoid gradient vanishing or gradient explosion. The input of the basic convolution block is skip-connected to the output of the basic convolution block through a downsampling layer, and then the output is obtained through a ReLU activation function by the skip connection, where the downsampling layer is composed of a 1×1 convolution with a step size of 2 and a batch normalization layer.

[0031] Preferably, a plurality of basic convolution blocks and skip connections are combined into one residual neural module. Take Resnet-50 as an example, the first residual neural module is composed of three basic convolution blocks and skip connections, and outputs a feature map Res2, the second residual neural module is composed of four basic convolution blocks and skip connections, and outputs a feature map Res3, the third residual neural module is composed of six basic convolution blocks and skip connections, and outputs a feature map Res4, and the fourth residual neural module is composed of three basic convolution blocks and skip connections, and outputs a feature map Res5. In this way, a feature map Res with layers from shallow layers (large scale) to deep layers (small scale) is generated. i where i={1,2,3,4,5}.

[0032] Furthermore, in the decoding stage, two branches, edge detection and saliency detection, are used, and the edge detection branch is fused into the saliency detection branch at each layer.

[0033] Preferably, from the fourth layer, the edge feature map output from the deep layer is upsampled and then input into the edge prediction branch spatial attention module of the shallow layer together with the residual neural network feature map of the present layer to output the edge feature map of the present layer; the edge feature map output from the deep layer and the saliency feature map output from the deep layer are input into the saliency detection branch spatial attention module to output the spatial attention feature map; and the spatial attention feature map and the residual neural network feature map of the present layer are input into the saliency detection branch dot product attention module to output the saliency feature map of the present layer.

[0034] Preferably, in order to learn the representation of multi-scale edge features in deep layers, an edge pyramid convolution module is designed in the present invention, which extracts edge features by convolutions of different sizes, as shown in Figure 2. First, the deepest layer feature map Res5 is input, and then it is output as different outputs y through convolutions of different sizes. i where i={1,2,3,4,5}, then serially connect them in the channel dimension, and finally output the edge feature map edge4 through a 1×1 convolution, batch normalization layer and ReLU activation function, which can be expressed as follows:

number

[0035] Preferably, in order to extract a more accurate edge feature map, a novel edge prediction branch spatial attention module is designed in this invention, as shown in FIG.

[0036] First, a small-scale edge feature map edge i+1 The large-scale feature map Res i Then, we upsample it to the same scale as the edge feature map edge i+1 , feature map Res i And then we serially connect both pixel-level dot product matrices in the channel dimension to obtain the serially connected feature map edge sa and express it as follows:

number

number

[0037] b. Next, the feature map edge after serial connection sa Then, we take the average value in the channel dimension for , and convert it into spatial attention weights by convolution and sigmoid function normalization. Then, we use the feature map edge after serial connection. sa The spatial features of the edge are corrected by the spatial attention weights and the feature map edge sa Then, we perform the matrix dot product operation to obtain the edge attention image edge spa and express it as follows:

number

[0038] c. Finally, the obtained edge attention image edge spa respectively, the original feature map Res i, the edge feature map after upsampling i+1 Then, the two branches are serially connected in the channel dimension and a convolution operation is performed to obtain the final edge feature map edge i and express it as follows:

number

number

[0039] Preferably, in order to learn the representation of multi-scale semantic features in deep layers, a multi-scale pyramid convolution module is designed in the present invention, which extracts semantic features by convolutions of different sizes, as shown in Figure 4. First, the feature map Res5 of the deepest layer is input, and then it is output as different outputs y through convolutions of different sizes. i Then, i = {1, 2, 3, 4, 5}, and then serially connected in the channel dimension. Finally, conv 1×1 After convolution, the feature map f4 is output and can be expressed as follows:

number

[0040] Preferably, in order to realize effective fusion of the salient feature map and the edge feature map, a novel salient feature-edge feature fusion module is designed in the present invention. In order to extract more accurate salient region location information, the present invention uses a spatial channel attention module. The above modules are shown in Figure 5.

[0041] First, the edge feature map edge is generated using a sigmoid function. i+1 Normalize the edge attention image to correct the spatial features of the saliency map, and i+1 Then, the residuals are connected by performing a matrix dot product operation with the saliency map after correction, and the saliency map after correction is combined with the edge feature map edge i+1 We serially connect the vectors in the channel dimension to obtain an edge-enhanced saliency map f e , which can be expressed as follows:

number

number

[0042] b. Then, we apply convolution and transformation to the saliency map f e to obtain the query f e q , Key fe k and Value fe v These three tensors can be expressed as follows:

number

[0043] Input the above three tensors to the dot product attention module. Query fe q , Key fe k and Value fe v Enter the three tensors and run the query fe q and key fe k The two tensors are multiplied by a matrix and the softmax function is used to obtain the attention matrix, and the value fe v The attention matrix is ​​multiplied by the matrix and then connected to the salient feature map f e , and the attention matrix f att Then, the dot product attention module can be expressed as follows:

number

[0044] c. Next, the attention feature map f att and feature map Res i And then connect both dot product matrices in series in the channel dimension to get the feature map f ca Then, the feature map f ca We normalize,f,to generate a semantic attention image for the correction of channel,semantic features, and then use the feature map,f, ca Then, we perform pixel-level dot product operation on the semantic attention image to obtain a semantic augmented feature map f gap and express it as follows:

number

number

[0045] d. Finally, the semantics-enhanced feature map f gaprespectively, the original feature map Res i , attention feature map f att Then, the two branches are serially connected in the channel dimension and then a convolution operation is performed to obtain the final feature map f i and express it as follows:

number

number

[0046] Through the above means, a salient object detection model is constructed. In order to obtain an optimal detection model, the model needs to be trained. The present invention uses a land dataset and an underwater dataset to train the proposed underwater salient object detection network model. At present, there is little research on underwater salient object detection, so a land dataset is used to verify the model performance and compare it with other land methods. In order to improve the model generalization ability, for example, to improve the accuracy of diver detection by learning clear human images on land, a combination of a land training set and an underwater test set is used. As shown in FIG. 6, the specific training steps are as follows:

[0047] The labels operator is used to extract target edge images from the actual salient images in the dataset to obtain the actual edge images. In the training phase, two training policies are used: training using the land dataset and training using the mixed dataset, where the mixed dataset includes the land dataset and the underwater dataset. For the land dataset, 700 samples, 1500 samples and 785 samples are selected from the NLPR dataset, NJUD dataset and DUT-D dataset, respectively, for a total of 2985 samples, and the other samples are used for testing. For the underwater dataset, 2051 samples are selected from the USOD10k dataset to obtain the training set, and the other samples are used for testing.

[0048] During the training process, the learning rate was adjusted using a warm-up and linear decay policy, the learning rate of the backbone was set to a maximum of 0.05, and the momentum and decay rates were set to 0.9 and 5e-4, respectively. The loss between the output of each layer of the two branches and the actual image was calculated, and the loss value was minimized using stochastic gradient descent to obtain the optimal model. Finally, the predicted saliency images and edge images were output in the testing phase.

[0049] For supervised training, we use the classical binary cross entropy function (BCE) as the loss function in the salient object detection branch. s ,

number

number

[0050] In addition, to achieve a more suitable edge effect, we use Intersection-over-Union (IoU) as another loss function, which can be expressed as follows:

number

[0051] In the edge detection branch, we use the binary cross entropy function (BCE) as the loss function, y e ,

number

number

[0052] S3: Obtain the salient images and edge images output from the trained salient object detection model as the object detection results.

[0053] Finally, the following should be noted: The above embodiments are merely for explaining the technical means of the present invention, and are not intended to limit it, and the present invention has been described in detail with reference to the above embodiments, but it is possible to modify the technical means described in the above embodiments, or to implement equivalent replacements for part or all of the technical features, and it is obvious to those skilled in the art that such modifications and replacements do not depart from the essence of the corresponding technical means of the embodiments of the present invention.

[0054] (Additional Note) (Appendix 1) Step S1: obtaining an RGB image and a depth map, performing channel dimension preprocessing on the RGB image and the depth map, and serially connecting them to obtain an RMD image; Step S2 of inputting the RMD image into a trained salient object detection model including an encoder structure and a decoder structure; and step S3 of obtaining a salient image and an edge image output from the salient object detection model as an object detection result; The encoder generates feature maps with successively reduced resolution layers based on the residual neural network and the RMD image; The decoder structure includes an edge detection branch used to fuse feature maps layer by layer and output an edge image, and a saliency detection branch used to fuse feature maps and edge feature maps layer by layer and output a saliency image with enhanced edge details.

[0055] (Appendix 2) The step of performing channel dimension preprocessing on the RGB image and the depth map and serially connecting them to obtain an RMD image includes: taking the maximum value of the G channel and the B channel in the RGB image as the M channel; The edge-guided RGBD underwater salient object detection method with multi-attention described in Appendix 1 is characterized in that it includes a step of serially connecting the R channel, M channel and depth map in the channel dimension to obtain a new input RMD image.

[0056] (Appendix 3) The encoder structure comprises: A preprocessing layer that includes one 7×7 convolutional layer and one 2×2 max pooling layer to obtain feature maps whose resolution is reduced to 1 / 4 of the original size; A basic convolution block that includes three convolution layers, each of which uses batch normalization and ReLU activation functions. a skip connection is added between the input and output of each elementary convolution block to avoid vanishing or exploding gradients; The edge-guided RGBD underwater salient object detection method with multi-attention described in Appendix 1 is characterized in that basic convolutional blocks and skip connections are combined into one residual neural module, and the output feature map has multiple scales from shallow layers to deep layers.

[0057] (Appendix 4) The edge detection branch includes an edge pyramid convolution module and an edge detection branch spatial attention module; The edge pyramid convolution module inputs the feature map of the deepest layer into a multi-scale convolution module, and further inputs the outputs of the convolutions at different scales into a 1×1 convolution module to output an edge feature map; the edge detection branch spatial attention module receives an edge feature map and a feature map, upsamples the small-scale edge feature map to have the same scale as the large-scale feature map, serially concatenates the edge feature map and the feature map and both pixel-level dot product matrices in a channel dimension to obtain a serially concatenated feature map; The serially connected feature map is converted into spatial attention weights, which are used to correct the spatial features of the serially connected feature map to obtain an edge attention image; First, the obtained edge attention image is residually connected to the original feature map and the upsampled edge feature map, respectively; then, the two branches are serially connected in the channel dimension, and then a convolution operation is performed; and finally, the edge feature map is output.

[0058] (Appendix 5) The saliency detection branch includes a scale pyramid convolution module and a saliency detection branch spatial attention module; The scale pyramid convolution module inputs the feature map of the deepest layer into a different scale convolution, and further inputs the output of the multiple different scale convolutions into a 1×1 convolution to output an edge feature map; The saliency detection branch spatial attention module receives the edge feature map, the saliency feature map and the feature map, normalizes the edge feature map by a function to generate an edge attention image for correcting the spatial features of the saliency feature map, and serially connects the corrected saliency feature map and the edge feature map in the channel dimension to obtain an edge-enhanced saliency feature map, generates three tensors, i.e., query, key and value, by convolution and transformation, and inputs them into a self-attention module, and outputs an attention feature map by the self-attention module; The attention feature map, the feature map, and the dot product matrix of both of them are serially connected in the channel dimension to obtain a serially connected feature map; the serially connected feature map is normalized by a function to generate a semantic attention image for correcting the channel semantic feature; and a semantic-augmented feature map is obtained by performing pixel dot product operation between the feature map and the semantic attention image; The edge-guided RGBD underwater salient object detection method with multi-attention described in Appendix 1 is characterized in that the semantic-enhanced feature map is residually connected to the original feature map and the attention feature map, respectively, and a convolution operation is performed to obtain the final salient feature map.

[0059] (Appendix 6) Training the salient object detection model comprises: obtaining training set data including land and underwater images and GroundTruth salient object images; training a network using the above land images and underwater images as input data for the network; calculating a loss value between the salient object images output from the network and the GroundTruth salient object images in the training set, and performing backpropagation using the loss value to update the weights of the network.

[0060] (Appendix 7) Calculating a loss value between the salient object image output from the network and the GroundTruth salient object image in the training set, and performing backpropagation using the loss value to update the weights of the network; The method for detecting underwater salient objects with multi-attention guided by edges described in Appendix 6 includes a step of calculating loss values ​​between an actual salient image and an edge image for each output of the edge detection branch and the saliency detection branch, respectively, and minimizing the loss values ​​using a stochastic gradient descent method to obtain an optimal model.

Claims

1. S1: obtaining an RGB image and a depth map, performing channel-dimensional preprocessing on the RGB image and the depth map, and serially connecting the RGB image and the depth map to obtain an RMD image; Step S2 of inputting the RMD image into a trained salient object detection model including an encoder structure and a decoder structure; and step S3 of obtaining a salient image and an edge image output from the salient object detection model as an object detection result; The encoder generates feature maps with successively reduced resolution layers based on the RMD images based on a residual neural network; The decoder structure includes an edge detection branch used to fuse feature maps layer by layer and output an edge image, and a saliency detection branch used to fuse feature maps and edge feature maps layer by layer and output a saliency image with enhanced edge details.

2. The step of performing channel dimension pre-processing on the RGB image and the depth map and serially connecting them to obtain an RMD image includes: taking the maximum value of the G channel and the B channel in the RGB image as the M channel; The method for edge-guided RGBD underwater salient object detection with multi-attention according to claim 1, further comprising the step of serially connecting the R channel, the M channel and the depth map in the channel dimension to obtain a new input RMD image.

3. The encoder structure comprises: A preprocessing layer that includes one 7x7 convolutional layer and one 2x2 max pooling layer to obtain feature maps whose resolution is reduced to ¼ of the original size; A basic convolution block including three convolution layers, each of which uses batch normalization and a ReLU activation function; a skip connection is added between the input and output of each elementary convolution block to avoid vanishing or exploding gradients; The edge-guided RGBD underwater salient object detection method with multi-attention as claimed in claim 1, characterized in that basic convolutional blocks and skip connections are combined into one residual neural module, and the output feature map has multiple scales from shallow layers to deep layers.

4. The edge detection branch includes an edge pyramid convolution module and an edge detection branch spatial attention module; The edge pyramid convolution module inputs the feature map of the deepest layer into a multi-scale convolution module, and further inputs the outputs of the convolutions at different scales into a 1×1 convolution module to output an edge feature map; the edge detection branch spatial attention module receives an edge feature map and a feature map, upsamples the small-scale edge feature map to have the same scale as the large-scale feature map, serially concatenates the edge feature map and the feature map and both pixel-level dot product matrices in a channel dimension to obtain a serially concatenated feature map; The serially connected feature map is converted into spatial attention weights, which are used to correct the spatial features of the serially connected feature map to obtain an edge attention image; The edge-guided RGBD underwater salient object detection method with multi-attention as described in claim 1, characterized in that first, the obtained edge attention image is residually connected to the original feature map and the upsampled edge feature map respectively, then the two branches are serially connected in the channel dimension and then a convolution operation is performed, and finally the edge feature map is output.

5. The saliency detection branch includes a scale pyramid convolution module and a saliency detection branch spatial attention module; The scale pyramid convolution module inputs the deepest layer feature map to a different scale convolution, and further inputs the output of the different scale convolutions to a 1×1 convolution, and outputs an edge feature map; The saliency detection branch spatial attention module receives the edge feature map, the saliency feature map and the feature map, normalizes the edge feature map by a function to generate an edge attention image for correcting the spatial features of the saliency feature map, and serially connects the corrected saliency feature map and the edge feature map in the channel dimension to obtain an edge-enhanced saliency feature map, generates three tensors, i.e., query, key and value, by convolution and transformation, and inputs them into a self-attention module, and outputs an attention feature map by the self-attention module; The attention feature map, the feature map, and the dot product matrix of both of them are serially connected in the channel dimension to obtain a serially connected feature map; the serially connected feature map is normalized by a function to generate a semantic attention image for correcting the channel semantic feature; and a semantic-augmented feature map is obtained by performing pixel dot product operation between the feature map and the semantic attention image; The edge-guided RGBD underwater salient object detection method with multi-attention as claimed in claim 1, characterized in that the semantics-enhanced feature map is residually connected to the original feature map and the attention feature map respectively, and a convolution operation is performed to obtain a final salient feature map.

6. Training the salient object detection model comprises: Obtaining training set data including land and underwater images and GroundTruth salient object images; training a network using the above land images and underwater images as input data for the network; The method for detecting underwater salient objects with multi-attention guided by edges according to claim 1, further comprising a step of: calculating a loss value between the salient object image output from the network and the GroundTruth salient object image in the training set; and performing backpropagation using the loss value to update the weights of the network.

7. A step of calculating a loss value between the salient object image output from the network and the salient object image of the GroundTruth in the training set, and performing backpropagation using the loss value to update the weights of the network, The method for detecting underwater salient objects with multi-attention guided by edges, as described in claim 6, further comprising a step of calculating loss values ​​of actual salient images and edge images for the output of each layer of the edge detection branch and the saliency detection branch, respectively, and minimizing the loss values ​​using a stochastic gradient descent method to obtain an optimal model.

Images