Construction and detection method of lightweight salient object detection model based on multi-scale learning

Abstract

The application discloses a kind of construction and detection method of lightweight salient object detection model based on multi-scale learning, first, the features of color three channels RGB image are extracted using multi-scale learning mechanism and depth separable convolution;Second, the features of depth image are extracted using reverse residual block;Third, the features of depth image are enhanced using channel attention mechanism and spatial attention mechanism;Then, the features of color three channels RGB image and enhanced depth image are fused;Finally, the fused features pass through decoding network and output layer to generate saliency prediction map.The model reduces the number of parameters through depth separable convolution, performs multi-scale learning through dilated convolution, and improves the effect of salient object detection by mining the effective information of depth image through attention mechanism.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for constructing and detecting a lightweight salient object detection model based on multi-scale learning. Background Technology

[0002] Saliency detection, also known as saliency assessment, aims to detect the objects or regions in natural images that are of most interest to the human eye. Saliency detection technology has wide applications in computer vision, including image retrieval, image segmentation, object detection, visual tracking, and scene classification. In recent years, due to the rapid development of convolutional neural networks, deep learning-based saliency detection methods have achieved great success. However, deep learning-based methods require powerful computing capabilities, making saliency detection technology difficult to apply in practice. Its performance is particularly poor in scenarios requiring real-time processing and limited computing resources, such as mobile devices, autonomous driving, and robotics. Designing a lightweight saliency detection model is undoubtedly key to solving these problems. Therefore, this paper studies saliency detection of RGBD images based on lightweight convolutional neural networks and proposes a lightweight saliency detection model based on multi-scale learning to improve the performance of saliency detection under limited resources. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of current RGBD image saliency detection methods by proposing a lightweight saliency target detection model based on multi-scale learning for construction and detection, thereby improving the effectiveness of saliency target detection.

[0004] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a lightweight salient object detection model based on multi-scale learning, comprising the following steps:

[0005] Step S1: Establish the training sample set. Select M color three-channel RGB images (i.e., RGB three-channel color images), the corresponding depth images of the color images, and the corresponding ground truth images of the color images to form the training sample set;

[0006] Step S2: Build a convolutional neural network. This network includes an input layer, hidden layers, and an output layer;

[0007] Step S3: Input each color image and its corresponding depth image from the training sample set into the convolutional neural network for training, to obtain five saliency prediction images corresponding to each color image in the training set, denoted as {Q}. i}, where 1≤i≤5;

[0008] Step S4: Calculate the loss function value between the saliency prediction map and the corresponding true saliency map for each color image in the training set, denoted as {lossi}, where 1≤i≤5, loss i Let be the loss function value between the i-th saliency prediction map and the true saliency map, using the binary cross-entropy loss function. The comprehensive loss function value is... Where 0≤α≤1 are the weighting coefficients;

[0009] Step S5: Repeat steps S3 and S4. In each iteration, input the training samples into the convolutional neural network, calculate the comprehensive loss function value, perform backpropagation, and use the gradient descent method to optimize the network parameters to obtain the optimal weight vector and the optimal bias term.

[0010] Step S6: Input the color image to be predicted and its corresponding depth image into the trained convolutional neural network model to obtain the saliency prediction map Q1 corresponding to the color image, thus completing the saliency target detection task.

[0011] In this invention, firstly, multi-scale learning mechanisms and depthwise separable convolutions are used to extract color three-channel RGB image features; secondly, inverse residual blocks are used to extract depth image features; thirdly, channel attention and spatial attention mechanisms are used to enhance the depth image features; next, the color three-channel RGB image features and the enhanced depth image features are fused; finally, the fused features are passed through a decoding network and an output layer to generate a saliency prediction map. The model reduces the number of parameters through depthwise separable convolutions, performs multi-scale learning through dilated convolutions, and mines effective information from the depth image through attention mechanisms, thereby improving the performance of salient object detection.

[0012] In step S2, the input layer includes a color image input layer and a depth image input layer, and the hidden layer includes a color image lightweight feature extraction network, a depth image lightweight feature extraction network, a depth image information enhancement network, a global feature extraction network, and a decoding network.

[0013] The color image input layer converts the three-channel RGB image into a training three-channel RGB image, which is then input into the color image lightweight feature extraction network. The width and height of the training three-channel RGB image must be integer multiples of 32.

[0014] The depth map input layer transforms the depth image into a training depth image, which is then fed into the depth map lightweight feature extraction network. The width and height of the training depth image must both be integer multiples of 32.

[0015] The color image lightweight feature extraction network contains five sequentially connected color lightweight network blocks, the depth image lightweight feature extraction network contains five sequentially connected depth lightweight network blocks, and the depth image information enhancement network contains five depth enhancement network blocks. The first depth lightweight network block receives the training depth image from the depth image input layer, and its output is connected to the second depth lightweight network block and the first depth enhancement network block. The output of the second depth lightweight network block is connected to the third depth lightweight network block and the second depth enhancement network block. The output of the third depth lightweight network block is connected to the fourth depth lightweight network block and the third depth enhancement network block. The output of the fourth depth lightweight network block is connected to the fifth depth lightweight network block and the fourth depth enhancement network block. The output of the fifth depth lightweight network block is connected to the fifth depth enhancement network block.

[0016] The decoding network consists of five decoding neural network blocks. The outputs of the fifth lightweight color network block and the fifth deep augmentation network block undergo element-wise addition to obtain feature map F5, which is then input to the global feature extraction network. The output of the global feature extraction network serves as one input to the fifth decoding neural network block, and feature map F5 is the other input to the fifth decoding neural network block. The output of the fifth decoding neural network block serves as one input to the fourth decoding neural network block. The outputs of the fourth lightweight color network block and the fourth deep augmentation network block undergo element-wise addition to obtain feature map F4, which is then the other input to the fourth decoding neural network block. The output of the fourth decoding neural network block serves as one input to the third decoding neural network block. The outputs of the third lightweight color network block and the third deep augmentation network block undergo element-wise addition to obtain feature map F3, which is then the other input to the third decoding neural network block. The output of the third decoding neural network block serves as one input to the second decoding neural network block. The outputs of the second lightweight color network block and the second deep augmentation network block undergo element-wise addition to obtain feature map F3, which is then the other input to the third decoding neural network block. After the addition operation, the resulting feature map F2 serves as another input to the second decoding neural network block; the output of the second decoding neural network block serves as one of the inputs to the first decoding neural network block. The outputs of the first color lightweight network block and the first depth augmentation network block undergo element-wise addition, resulting in feature map F1, which serves as another input to the first decoding neural network block. The output layer contains five prediction neural network blocks, each receiving the outputs of the five decoding neural network blocks as input, and outputting five saliency prediction maps.

[0017] The input to the first lightweight color network block is a training color three-channel RGB image. The first lightweight color network block comprises a first downsampling neural network block, a first batch of normalized layers, a first activation layer, a first multi-scale neural network block, a second batch of normalized layers, and a second activation layer, connected in sequence. The first downsampling neural network block contains two branches: the first branch contains a first convolutional layer, and the second branch contains a first pooling layer. The outputs of the first convolutional layer and the first pooling layer are stacked, and the result is used as the output of the first downsampling neural network block. The second lightweight color network block comprises a second downsampling neural network block, a third batch of normalized layers, a third activation layer, a second multi-scale neural network block, a fourth batch of normalized layers, and a fourth activation layer, connected in sequence. The second downsampling neural network block contains two branches: the first branch contains a second convolutional layer and a first depthwise separable convolutional layer, connected in sequence; the second branch contains a second pooling layer. The outputs of the first depthwise separable convolutional layer and the second pooling layer are stacked, and the result is used as the output of the second downsampling neural network block. The third lightweight color network block contains a third downsampling neural network block, a fifth batch normalization layer, a fifth activation layer, a third multi-scale neural network block, a sixth batch normalization layer, and a sixth activation layer, all connected in sequence. The third downsampling neural network block has two branches: the first branch contains a third convolutional layer and a second depthwise separable convolutional layer, connected in sequence; the second branch contains a third pooling layer. The outputs of the second depthwise separable convolutional layer and the third pooling layer are stacked, and the result is used as the output of the third downsampling neural network block. The fourth lightweight color network block contains a fourth downsampling neural network block, a seventh batch normalization layer, a seventh activation layer, a fourth multi-scale neural network block, an eighth batch normalization layer, and an eighth activation layer, all connected in sequence. The fourth downsampling neural network block has two branches: the first branch contains a fourth convolutional layer and a third depthwise separable convolutional layer, connected in sequence; the second branch contains a fourth pooling layer. The outputs of the third depthwise separable convolutional layer and the fourth pooling layer are stacked, and the result is used as the output of the fourth downsampling neural network block. The fifth lightweight colored network block contains a fifth downsampling neural network block, a ninth batch normalization layer, a ninth activation layer, a fifth multi-scale neural network block, a tenth batch normalization layer, and a tenth activation layer, all connected in sequence. The fifth downsampling neural network block has two branches: the first branch contains a fifth convolutional layer and a fourth depthwise separable convolutional layer connected in sequence; the second branch contains a fifth pooling layer. The outputs of the fourth depthwise separable convolutional layer and the fifth pooling layer are stacked, and the result is used as the output of the fifth downsampling neural network block. The activation mode for the first to tenth activation layers is ReLU. The first convolutional layer has a 3×3 kernel size, a stride of 2, and padding of 1. The second to fifth convolutional layers all have a 1×1 kernel size, a stride of 1, and padding of 0. The number of kernels in the first to fifth convolutional layers are 13, 8, 8, 64, and 224, respectively.The kernel size of the first to fourth depthwise separable convolutional layers is 5×5, the stride is 2, and the padding is 2. The number of kernels in the first to fourth depthwise separable convolutional layers are 8, 8, 64, and 224, respectively. The first to fifth pooling layers all use max pooling, with a pooling window size of 2×2 and a stride of 2.

[0018] The first multi-scale neural network block comprises two branches: the first branch contains a first multi-scale attention fusion network block, and the second branch contains a twelfth convolutional layer. The outputs of the first multi-scale attention fusion network block and the twelfth convolutional layer undergo element-wise addition, and the result is used as the output of the first multi-scale neural network block. The second multi-scale neural network block comprises two branches: the first branch contains a second multi-scale attention fusion network block, and the second branch contains a thirteenth convolutional layer. The outputs of the second multi-scale attention fusion network block and the thirteenth convolutional layer undergo element-wise addition, and the result is used as the output of the second multi-scale neural network block. The third multi-scale neural network block comprises two branches: the first branch contains the third to fifth multi-scale attention fusion network blocks connected sequentially, and the second branch contains a fourteenth convolutional layer. The outputs of the fifth multi-scale attention fusion network block and the fourteenth convolutional layer undergo element-wise addition, and the result is used as the output of the third multi-scale neural network block. The fourth multi-scale neural network block contains two branches. The first branch contains sequentially connected multi-scale attention fusion network blocks from the sixth to the tenth. The second branch contains the fifteenth convolutional layer. The outputs of the tenth and fifteenth multi-scale attention fusion network blocks undergo element-wise addition, and the result is used as the output of the fourth multi-scale neural network block. The fifth multi-scale neural network block also contains two branches. The first branch contains sequentially connected multi-scale attention fusion network blocks from the eleventh to the thirteenth. The second branch contains the sixteenth convolutional layer. The outputs of the thirteenth and sixteenth multi-scale attention fusion network blocks undergo element-wise addition, and the result is used as the output of the fifth multi-scale neural network block. The kernel size of the twelfth to sixteenth convolutional layers is 1×1, the stride is 1, and the padding is 0. The number of kernels is 16, 24, 32, 96, and 320, respectively.

[0019] The first to thirteenth multi-scale attention fusion network blocks have the same structure. Each multi-scale attention fusion network block contains five processing branches. The input to the first processing branch is the input to the multi-scale attention fusion network block itself. The first processing branch contains a sixth pooling layer, a sixth convolutional layer, an eleventh activation layer, a seventh convolutional layer, and a twelfth activation layer connected in sequence. The input to the first processing branch and the output of the twelfth activation layer are multiplied by a dimension expansion operation, and the result is used as the output of the first processing branch. The input to the second processing branch is the output of the first processing branch. The second processing branch contains a seventh pooling layer, an eighth convolutional layer, and a thirteenth activation layer connected in sequence. The input to the second processing branch and the output of the thirteenth activation layer are multiplied by a channel expansion operation, and the result is used as the output of the second processing branch. The input to the third processing branch is the input to the multi-scale attention fusion network block. This branch contains a ninth convolutional layer and first to fourth depthwise separable convolutional layers. The input to the ninth convolutional layer is the input to the third processing branch, and its output is fed into the first to fourth depthwise separable convolutional layers. The outputs of these layers undergo element-wise addition, and the result is the output of the third processing branch. The input to the fourth processing branch is the output of the third processing branch. This branch contains an eleventh batch normalized layer, a fourteenth activation layer, a tenth convolutional layer, and a twelfth batch normalized layer, connected sequentially. The input to the multi-scale attention fusion network block and the output of the twelfth batch normalized layer undergo element-wise addition, and the result is the output of the fourth processing branch. The input to the fifth processing branch is the output of the fourth processing branch. This branch contains an eleventh convolutional layer, a thirteenth batch normalized layer, and a fifteenth activation layer, connected sequentially. The output of the fifteenth activation layer is the output of the fifth processing branch. The outputs of the second and fifth processing branches are subjected to element-wise multiplication, resulting in the output of the multi-scale attention fusion network block. The number of input channels for this block is denoted as ch. The kernel size of the sixth, seventh, ninth, and tenth convolutional layers is 1×1, with a stride of 1 and padding of 0. The number of kernels in these layers is ch / 8, ch, ch*4, and ch, respectively. The kernel size of the eighth convolutional layer is 7×7, with a stride of 1 and padding of 3. The number of kernels in the eleventh convolutional layer is 3×3, with a stride of 1 and padding of 1. The number of kernels is ch. The sixth pooling layer uses global max pooling. The seventh pooling layer uses channel-wise global max pooling. The activation methods for the eleventh, fourteenth, and fifteenth activation layers are all ReLU. The activation methods for the twelfth and thirteenth activation layers are both sigmoid.The kernel size of the first to fourth dilated depth-separable convolutional layers is 3×3, the stride is 1, the padding is 1, 2, 4, and 8 respectively, the dilation coefficient is 1, 2, 4, and 8 respectively, and the number of kernels is ch*4 for each layer. The first multi-scale attention fusion network block has 16 input channels, the second has 24, the third to fifth have 32, the sixth to tenth have 96, and the eleventh to thirteenth have 320.

[0020] The first to fifth deep lightweight network blocks have the same structure. Each deep lightweight network block contains a sequentially connected seventeenth convolutional layer, fourteenth batch normalized layer, sixteenth activation layer, first depthwise separable convolutional layer, fifteenth batch normalized layer, seventeenth activation layer, eighteenth convolutional layer, sixteenth batch normalized layer, nineteenth convolutional layer, seventeenth batch normalized layer, eighteenth activation layer, second depthwise separable convolutional layer, eighteenth batch normalized layer, nineteenth activation layer, twentieth convolutional layer, and nineteenth batch normalized layer. The outputs of the sixteenth and nineteenth batch normalized layers undergo element-wise addition, and the result is used as the output of the deep lightweight network block. The number of input channels of the deep lightweight network block is denoted as inc, and the number of output channels is denoted as ouc. The kernel size of the seventeenth to twentieth convolutional layers is 1×1, the stride is 1, the padding is 0, and the number of kernels is inc*4, ouc, ouc*4, and ouc, respectively. The activation method for layers 16 through 19 is ReLU. The kernel size for the first and second depthwise separable convolutional layers is 3×3, with strides of 2 and 1 respectively, padding of 1, and the number of kernels is inc*4 and ouc*4 respectively. The number of input channels for the first through fifth depthwise lightweight network blocks are 1, 16, 24, 32, and 96 respectively. The number of output channels for the first through fifth depthwise lightweight network blocks are 16, 24, 32, 96, and 320 respectively.

[0021] The first to fifth deep augmentation network blocks have the same structure, each containing two sequentially connected processing branches. The first processing branch consists of a sequentially connected eighth pooling layer, a twenty-first convolutional layer, a twentieth activation layer, a twenty-second convolutional layer, and a twenty-first activation layer. The input of the deep augmentation network block and the output of the twenty-first activation layer are multiplied by a dimension-expanding operation, and the result is used as the output of the first processing branch. The second processing branch consists of a sequentially connected ninth pooling layer, a twenty-third convolutional layer, and a twenty-second activation layer. The output of the first processing branch and the output of the twenty-second activation layer are multiplied by a channel-expanding operation, and the result is used as the output of the second processing branch, which is the output of the deep augmentation network block. The eighth pooling layer uses global max pooling, and the ninth pooling layer uses channel-wise global max pooling. The number of input channels for the deep augmentation network block is denoted as inc. The kernel size of the twenty-first and twenty-second convolutional layers is 1×1, the stride is 1, the padding is 0, and the number of kernels is inc / 8 and inc, respectively. The 23rd convolutional layer has a 7×7 kernel size, a stride of 1, padding of 3, and 1 kernel. The 20th activation layer uses ReLU activation, while the 21st and 22nd activation layers use Sigmoid activation. The number of input channels for the first to fifth deep augmentation network blocks are 16, 24, 32, 96, and 320, respectively.

[0022] The global feature extraction network comprises five processing branches. The first branch consists of a first adaptive average pooling layer, a twenty-fourth convolutional layer, a twenty-second batch normalization layer, a twenty-third activation layer, and a first upsampling layer, connected in sequence. The second branch consists of a second adaptive average pooling layer, a twenty-fifth convolutional layer, a twenty-first batch normalization layer, a twenty-fourth activation layer, and a second upsampling layer, connected in sequence. The third branch consists of a third adaptive average pooling layer, a twenty-sixth convolutional layer, a twenty-second batch normalization layer, a twenty-fifth activation layer, and a third upsampling layer, connected in sequence. The fourth branch consists of a fourth adaptive average pooling layer, a twenty-seventh convolutional layer, a twenty-third batch normalization layer, a twenty-sixth activation layer, and a fourth upsampling layer, connected in sequence. The outputs of the first to fourth branches are concatenated with the input of the global feature extraction network, and the result is used as the input of the fifth branch. The fifth branch consists of a twenty-eighth convolutional layer, a twenty-fourth batch normalization layer, and a twenty-seventh activation layer, connected in sequence. The output of the fifth branch is the output of the global feature extraction network. The output feature map sizes of the first to fourth adaptive average pooling layers are 2×2, 4×4, 6×6, and 8×8, respectively. The convolutional kernel size of the twenty-fourth to twenty-eighth convolutional layers is 1×1, the stride is 1, the padding is 0, and the number of kernels is 80, 80, 80, 80, and 320, respectively. The activation method of the twenty-third to twenty-seventh activation layers is ReLU. The sampling method of the first to fourth upsampling layers is bilinear interpolation, and the output feature map size is the same as the input feature map size of the global feature extraction network.

[0023] The fifth decoding neural network block comprises two sequentially connected processing branches. The input to the first processing branch is the feature map F5. This first processing branch contains a 29th convolutional layer and a 25th batch normalization layer, connected sequentially. The output of the global feature extraction network and the output of the 25th batch normalization layer undergo an element-wise addition operation, the result of which is the output of the first processing branch. The second processing branch contains a 5th dilated depthwise separable convolutional layer, a 26th batch normalization layer, a 28th activation layer, a 30th convolutional layer, a 27th batch normalization layer, and a 29th activation layer, connected sequentially. The output of the 29th activation layer is the output of the fifth decoding neural network block. The 29th convolutional layer has a 3×3 kernel size, a stride of 1, padding of 1, and 320 kernels. The 30th convolutional layer has a 1×1 kernel size, a stride of 1, padding of 0, and 96 kernels. The 5th dilated depthwise separable convolutional layer has a 3×3 kernel size, a stride of 1, padding of 1, a dilation factor of 1, and 320 kernels. The activation method for the 28th and 29th activation layers is "ReLU".

[0024] The fourth to first decoding neural network blocks have the same structure. Each decoding neural network block contains three processing branches. The first processing branch takes the corresponding feature map as input and contains a 31st convolutional layer and a 28th batch normalization layer connected sequentially. The second processing branch takes the corresponding decoding neural network block as input and contains a 5th upsampling layer. The outputs of the first and second processing branches undergo an element-wise addition operation, the result of which is input to the third processing branch. The third processing branch contains a 6th dilated depthwise separable convolutional layer, a 29th batch normalization layer, a 30th activation layer, a 32nd convolutional layer, a 30th batch normalization layer, and a 31st activation layer connected sequentially. The output of the 31st activation layer is the output of the decoding neural network block. The number of input and output channels of the decoding neural network block is denoted as inc and ouc, respectively. The 31st convolutional layer has a 3×3 kernel size, a stride of 1, padding of 1, and an inc number of kernels. The 32nd convolutional layer has a 1×1 kernel size, a stride of 1, padding of 0, and an ouc number of kernels. The sixth dilated depthwise separable convolutional layer has a kernel size of 3×3, a stride of 1, padding of 2, a dilation factor of 2, and an inc number of kernels. The thirtieth and thirty-first activation layers use ReLU activation. The number of input channels for the fourth to first decoding neural network blocks are 96, 32, 24, and 16, respectively. The number of output channels for the fourth to first decoding neural network blocks are 32, 24, 16, and 16, respectively. The fifth upsampling layer has coefficients of 2, and the sampling method is bilinear interpolation.

[0025] The first to fifth prediction neural network blocks have the same structure. Each prediction neural network block contains a 33rd convolutional layer, a 32nd activation layer, and a 6th upsampling layer connected sequentially. The output of the 6th upsampling layer is the output of the prediction neural network block. The 33rd convolutional layer has a 1×1 kernel size, a stride of 1, zero padding, and one kernel. The 32nd activation layer uses the "Sigmoid" activation method. The 6th upsampling layer uses bilinear interpolation, and the output feature map size is the same as the training color image size.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] 1) This invention proposes a novel lightweight multi-scale color image feature extraction module, which uses dilated convolution for multi-scale learning and depthwise separable convolution to reduce the number of model parameters, thereby achieving lightweight multi-scale learning.

[0028] 2) This invention uses inverse residual blocks to extract lightweight depth map features and uses channel attention and spatial attention mechanisms to enhance the extracted depth map features.

[0029] 3) This invention designs a global feature extraction network and uses a pyramid pooling mechanism to obtain global context information to guide low-level features and obtain an accurate and clear saliency prediction map. Attached Figure Description

[0030] Figure 1 A general block diagram of the convolutional neural network constructed by the method of the present invention;

[0031] Figure 2a A structural block diagram of the second colored lightweight network block in the convolutional neural network constructed by the method of the present invention;

[0032] Figure 2b A structural block diagram of the second downsampling neural network block in the convolutional neural network constructed by the method of the present invention;

[0033] Figure 2c A structural block diagram of the second deep lightweight network block in the convolutional neural network constructed by the method of the present invention;

[0034] Figure 3 This is a graph showing the accuracy and recall performance of the significance detection method of the present invention.

[0035] Figure 4a The maximum F-measure value represents the significance detection performance of the method of this invention.

[0036] Figure 4b This represents the mean absolute error value of the significance detection performance of the method of the present invention;

[0037] Figure 5a This is the first original color three-channel RGB image;

[0038] Figure 5b for Figure 5a The corresponding depth image;

[0039] Figure 5c To utilize the method of the present invention for Figure 5a The saliency prediction image obtained by making predictions;

[0040] Figure 6a This is the first original color three-channel RGB image;

[0041] Figure 6b for Figure 6a The corresponding depth image;

[0042] Figure 6c To utilize the method of the present invention for Figure 6a The saliency prediction image obtained by making predictions;

[0043] Figure 7a This is the first original color three-channel RGB image;

[0044] Figure 7b for Figure 7a The corresponding depth image;

[0045] Figure 7c To utilize the method of the present invention for Figure 7a The saliency prediction image obtained from the prediction. Detailed Implementation

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0047] like Figure 1 As shown, the present invention proposes a lightweight salient object detection model based on multi-scale learning, which includes the following steps:

[0048] Step S1: Establish the training sample set. Select M color three-channel RGB images, the depth images corresponding to the color images, and the real salient images corresponding to the color images to form the training sample set;

[0049] Step S2: Build a convolutional neural network. This network includes an input layer, hidden layers, and an output layer;

[0050] Step S3: Input each color image and its corresponding depth image from the training sample set into the convolutional neural network for training, to obtain five saliency prediction images corresponding to each color image in the training set, denoted as {Q}. i}, where 1≤i≤5;

[0051] Step S4: Calculate the loss function value between the saliency prediction map and the corresponding true saliency map for each color image in the training set, denoted as {loss i}, where 1≤i≤5, loss i Let be the loss function value between the i-th saliency prediction map and the true saliency map, using the binary cross-entropy loss function. The comprehensive loss function value is... Where 0≤α≤1 are the weighting coefficients;

[0052] Step S5: Repeat steps S3 and S4. In each iteration, input the training samples into the convolutional neural network, calculate the comprehensive loss function value, perform backpropagation, and use the gradient descent method to optimize the network parameters to obtain the optimal weight vector and the optimal bias term.

[0053] Step S6: Input the color image to be predicted and its corresponding depth image into the trained convolutional neural network model to obtain the saliency prediction map Q1 corresponding to the color image, thus completing the saliency target detection task.

[0054] In step S2, the input layer includes a color image input layer and a depth image input layer, and the hidden layer includes a color image lightweight feature extraction network, a depth image lightweight feature extraction network, a depth image information enhancement network, a global feature extraction network, and a decoding network.

[0055] The color image input layer converts the three-channel RGB image into a training three-channel RGB image, which is then input into the color image lightweight feature extraction network. The width and height of the training three-channel RGB image must be integer multiples of 32.

[0056] The depth map input layer transforms the depth image into a training depth image, which is then fed into the depth map lightweight feature extraction network. The width and height of the training depth image must both be integer multiples of 32.

[0057] The color image lightweight feature extraction network contains five sequentially connected color lightweight network blocks, the depth image lightweight feature extraction network contains five sequentially connected depth lightweight network blocks, and the depth image information enhancement network contains five depth enhancement network blocks. The first depth lightweight network block receives the training depth image from the depth image input layer, and its output is connected to the second depth lightweight network block and the first depth enhancement network block. The output of the second depth lightweight network block is connected to the third depth lightweight network block and the second depth enhancement network block. The output of the third depth lightweight network block is connected to the fourth depth lightweight network block and the third depth enhancement network block. The output of the fourth depth lightweight network block is connected to the fifth depth lightweight network block and the fourth depth enhancement network block. The output of the fifth depth lightweight network block is connected to the fifth depth enhancement network block.

[0058] The decoding network consists of five decoding neural network blocks. The outputs of the fifth lightweight color network block and the fifth deep augmentation network block undergo element-wise addition to obtain feature map F5, which is then input to the global feature extraction network. The output of the global feature extraction network serves as one input to the fifth decoding neural network block, and feature map F5 is the other input to the fifth decoding neural network block. The output of the fifth decoding neural network block serves as one input to the fourth decoding neural network block. The outputs of the fourth lightweight color network block and the fourth deep augmentation network block undergo element-wise addition to obtain feature map F4, which is then the other input to the fourth decoding neural network block. The output of the fourth decoding neural network block serves as one input to the third decoding neural network block. The outputs of the third lightweight color network block and the third deep augmentation network block undergo element-wise addition to obtain feature map F3, which is then the other input to the third decoding neural network block. The output of the third decoding neural network block serves as one input to the second decoding neural network block. The outputs of the second lightweight color network block and the second deep augmentation network block undergo element-wise addition to obtain feature map F3, which is then the other input to the third decoding neural network block. After the addition operation, the resulting feature map F2 serves as another input to the second decoding neural network block; the output of the second decoding neural network block serves as one of the inputs to the first decoding neural network block. The outputs of the first color lightweight network block and the first depth augmentation network block undergo element-wise addition, resulting in feature map F1, which serves as another input to the first decoding neural network block. The output layer contains five prediction neural network blocks, each receiving the outputs of the five decoding neural network blocks as input, and outputting five saliency prediction maps.

[0059] In this specific embodiment, the input to the first lightweight color network block is a training color three-channel RGB image. The first lightweight color network block comprises a first downsampling neural network block, a first batch of normalized layers, a first activation layer, a first multi-scale neural network block, a second batch of normalized layers, and a second activation layer, connected in sequence. The first downsampling neural network block contains two branches: the first branch contains a first convolutional layer, and the second branch contains a first pooling layer. The outputs of the first convolutional layer and the first pooling layer are stacked, and the result is used as the output of the first downsampling neural network block. The second lightweight color network block comprises a second downsampling neural network block, a third batch of normalized layers, a third activation layer, a second multi-scale neural network block, a fourth batch of normalized layers, and a fourth activation layer, connected in sequence. The second downsampling neural network block contains two branches: the first branch contains a second convolutional layer and a first depthwise separable convolutional layer, connected in sequence; the second branch contains a second pooling layer. The outputs of the first depthwise separable convolutional layer and the second pooling layer are stacked, and the result is used as the output of the second downsampling neural network block. The third lightweight color network block contains a third downsampling neural network block, a fifth batch normalization layer, a fifth activation layer, a third multi-scale neural network block, a sixth batch normalization layer, and a sixth activation layer, all connected in sequence. The third downsampling neural network block has two branches: the first branch contains a third convolutional layer and a second depthwise separable convolutional layer, connected in sequence; the second branch contains a third pooling layer. The outputs of the second depthwise separable convolutional layer and the third pooling layer are stacked, and the result is used as the output of the third downsampling neural network block. The fourth lightweight color network block contains a fourth downsampling neural network block, a seventh batch normalization layer, a seventh activation layer, a fourth multi-scale neural network block, an eighth batch normalization layer, and an eighth activation layer, all connected in sequence. The fourth downsampling neural network block has two branches: the first branch contains a fourth convolutional layer and a third depthwise separable convolutional layer, connected in sequence; the second branch contains a fourth pooling layer. The outputs of the third depthwise separable convolutional layer and the fourth pooling layer are stacked, and the result is used as the output of the fourth downsampling neural network block. The fifth lightweight colored network block contains a fifth downsampling neural network block, a ninth batch normalization layer, a ninth activation layer, a fifth multi-scale neural network block, a tenth batch normalization layer, and a tenth activation layer, all connected in sequence. The fifth downsampling neural network block has two branches: the first branch contains a fifth convolutional layer and a fourth depthwise separable convolutional layer connected in sequence; the second branch contains a fifth pooling layer. The outputs of the fourth depthwise separable convolutional layer and the fifth pooling layer are stacked, and the result is used as the output of the fifth downsampling neural network block. The activation mode for the first to tenth activation layers is ReLU. The first convolutional layer has a 3×3 kernel size, a stride of 2, and padding of 1; the second to fifth convolutional layers all have a 1×1 kernel size, a stride of 1, and padding of 0.The number of convolutional kernels in the first to fifth convolutional layers are 13, 8, 8, 64, and 224, respectively. The kernel size of the first to fourth depthwise separable convolutional layers is 5×5, the stride is 2, and the padding is 2. The number of convolutional kernels in the first to fourth depthwise separable convolutional layers are 8, 8, 64, and 224, respectively. The first to fifth pooling layers all use max pooling, with a pooling window size of 2×2 and a stride of 2.

[0060] In this specific embodiment, the first multi-scale neural network block comprises two branches: the first branch contains a first multi-scale attention fusion network block, and the second branch contains a twelfth convolutional layer. The outputs of the first multi-scale attention fusion network block and the twelfth convolutional layer undergo element-wise addition, and the result is used as the output of the first multi-scale neural network block. The second multi-scale neural network block comprises two branches: the first branch contains a second multi-scale attention fusion network block, and the second branch contains a thirteenth convolutional layer. The outputs of the second multi-scale attention fusion network block and the thirteenth convolutional layer undergo element-wise addition, and the result is used as the output of the second multi-scale neural network block. The third multi-scale neural network block comprises two branches: the first branch contains the third to fifth multi-scale attention fusion network blocks connected sequentially, and the second branch contains a fourteenth convolutional layer. The outputs of the fifth multi-scale attention fusion network block and the fourteenth convolutional layer undergo element-wise addition, and the result is used as the output of the third multi-scale neural network block. The fourth multi-scale neural network block contains two branches. The first branch contains sequentially connected multi-scale attention fusion network blocks from the sixth to the tenth. The second branch contains the fifteenth convolutional layer. The outputs of the tenth and fifteenth multi-scale attention fusion network blocks undergo element-wise addition, and the result is used as the output of the fourth multi-scale neural network block. The fifth multi-scale neural network block also contains two branches. The first branch contains sequentially connected multi-scale attention fusion network blocks from the eleventh to the thirteenth. The second branch contains the sixteenth convolutional layer. The outputs of the thirteenth and sixteenth multi-scale attention fusion network blocks undergo element-wise addition, and the result is used as the output of the fifth multi-scale neural network block. The kernel size of the twelfth to sixteenth convolutional layers is 1×1, the stride is 1, and the padding is 0. The number of kernels is 16, 24, 32, 96, and 320, respectively.

[0061] In this specific embodiment, the first to thirteenth multi-scale attention fusion network blocks have the same structure. Each multi-scale attention fusion network block contains five processing branches. The input to the first processing branch is the input to the multi-scale attention fusion network block. The first processing branch contains a sixth pooling layer, a sixth convolutional layer, an eleventh activation layer, a seventh convolutional layer, and a twelfth activation layer connected in sequence. The input to the first processing branch and the output of the twelfth activation layer are multiplied by dimensionality expansion, and the result is used as the output of the first processing branch. The input to the second processing branch is the output of the first processing branch. The second processing branch contains a seventh pooling layer, an eighth convolutional layer, and a thirteenth activation layer connected in sequence. The input to the second processing branch and the output of the thirteenth activation layer are multiplied by channel expansion, and the result is used as the output of the second processing branch. The input to the third processing branch is the input to the multi-scale attention fusion network block. This branch contains a ninth convolutional layer and first to fourth depthwise separable convolutional layers. The input to the ninth convolutional layer is the input to the third processing branch, and its output is fed into the first to fourth depthwise separable convolutional layers. The outputs of these layers undergo element-wise addition, and the result is the output of the third processing branch. The input to the fourth processing branch is the output of the third processing branch. This branch contains an eleventh batch normalized layer, a fourteenth activation layer, a tenth convolutional layer, and a twelfth batch normalized layer, connected sequentially. The input to the multi-scale attention fusion network block and the output of the twelfth batch normalized layer undergo element-wise addition, and the result is the output of the fourth processing branch. The input to the fifth processing branch is the output of the fourth processing branch. This branch contains an eleventh convolutional layer, a thirteenth batch normalized layer, and a fifteenth activation layer, connected sequentially. The output of the fifteenth activation layer is the output of the fifth processing branch. The outputs of the second and fifth processing branches undergo element-wise multiplication, and the result is the output of the multi-scale attention fusion network block. The number of input channels in the multi-scale attention fusion network block is denoted as ch. The kernel size of convolutional layers 6, 7, 9, and 10 is 1×1, the stride is 1, and the padding is 0. The number of kernels in convolutional layers 6, 7, 9, and 10 are ch / 8, ch, ch*4, and ch, respectively. The kernel size of convolutional layer 8 is 7×7, the stride is 1, the padding is 3, and the number of kernels is 1. The kernel size of convolutional layer 11 is 3×3, the stride is 1, the padding is 1, and the number of kernels is ch. The sixth pooling layer uses global max pooling. The seventh pooling layer uses channel-wise global max pooling. The activation methods of activation layers 11, 14, and 15 are all "ReLU". The activation methods of activation layers 12 and 13 are all "sigmoid".The kernel size of the first to fourth dilated depth-separable convolutional layers is 3×3, the stride is 1, the padding is 1, 2, 4, and 8 respectively, the dilation coefficient is 1, 2, 4, and 8 respectively, and the number of kernels is ch*4 for each layer. The first multi-scale attention fusion network block has 16 input channels, the second has 24, the third to fifth have 32, the sixth to tenth have 96, and the eleventh to thirteenth have 320.

[0062] In this specific embodiment, the first to fifth deep lightweight network blocks have the same structure. Each deep lightweight network block contains a seventeenth convolutional layer, a fourteenth batch normalized layer, a sixteenth activation layer, a first depthwise separable convolutional layer, a fifteenth batch normalized layer, a seventeenth activation layer, an eighteenth convolutional layer, a sixteenth batch normalized layer, a nineteenth convolutional layer, a seventeenth batch normalized layer, an eighteenth activation layer, a second depthwise separable convolutional layer, an eighteenth batch normalized layer, a nineteenth activation layer, a twentieth convolutional layer, and a nineteenth batch normalized layer, all connected sequentially. The outputs of the sixteenth and nineteenth batch normalized layers undergo element-wise addition, and the result is used as the output of the deep lightweight network block. The number of input channels of the deep lightweight network block is denoted as inc, and the number of output channels is denoted as ouc. The kernel size of the seventeenth to twentieth convolutional layers is 1×1, the stride is 1, the padding is 0, and the number of kernels is inc*4, ouc, ouc*4, and ouc, respectively. The activation method for layers 16 through 19 is ReLU. The kernel size for the first and second depthwise separable convolutional layers is 3×3, with strides of 2 and 1 respectively, padding of 1, and the number of kernels is inc*4 and ouc*4 respectively. The number of input channels for the first through fifth depthwise lightweight network blocks are 1, 16, 24, 32, and 96 respectively. The number of output channels for the first through fifth depthwise lightweight network blocks are 16, 24, 32, 96, and 320 respectively.

[0063] In this specific embodiment, the first to fifth deep augmentation network blocks have the same structure, and each deep augmentation network block contains two sequentially connected processing branches. The first processing branch contains a sequentially connected eighth pooling layer, a twenty-first convolutional layer, a twentieth activation layer, a twenty-second convolutional layer, and a twenty-first activation layer. The input of the deep augmentation network block and the output of the twenty-first activation layer are multiplied by dimensionality expansion, and the result is used as the output of the first processing branch. The second processing branch contains a sequentially connected ninth pooling layer, a twenty-third convolutional layer, and a twenty-second activation layer. The output of the first processing branch and the output of the twenty-second activation layer are multiplied by channel expansion, and the result is used as the output of the second processing branch, which is the output of the deep augmentation network block. The eighth pooling layer uses global max pooling, and the ninth pooling layer uses channel-wise global max pooling. The number of input channels of the deep augmentation network block is denoted as inc. The kernel size of the twenty-first and twenty-second convolutional layers is 1×1, the stride is 1, the padding is 0, and the number of kernels is inc / 8 and inc, respectively. The 23rd convolutional layer has a 7×7 kernel size, a stride of 1, padding of 3, and 1 kernel. The 20th activation layer uses ReLU activation, while the 21st and 22nd activation layers use Sigmoid activation. The number of input channels for the first to fifth deep augmentation network blocks are 16, 24, 32, 96, and 320, respectively.

[0064] In this specific embodiment, the global feature extraction network comprises five processing branches. The first processing branch comprises a first adaptive average pooling layer, a twenty-fourth convolutional layer, a twenty-second batch normalization layer, a twenty-third activation layer, and a first upsampling layer, connected in sequence. The second processing branch comprises a second adaptive average pooling layer, a twenty-fifth convolutional layer, a twenty-first batch normalization layer, a twenty-fourth activation layer, and a second upsampling layer, connected in sequence. The third processing branch comprises a third adaptive average pooling layer, a twenty-sixth convolutional layer, a twenty-second batch normalization layer, a twenty-fifth activation layer, and a third upsampling layer, connected in sequence. The fourth processing branch comprises a fourth adaptive average pooling layer, a twenty-seventh convolutional layer, a twenty-third batch normalization layer, a twenty-sixth activation layer, and a fourth upsampling layer, connected in sequence. The outputs of the first to fourth processing branches and the input of the global feature extraction network are concatenated, and the result is used as the input of the fifth processing branch. The fifth processing branch comprises a twenty-eighth convolutional layer, a twenty-fourth batch normalization layer, and a twenty-seventh activation layer, connected in sequence, and the output of the fifth processing branch is the output of the global feature extraction network. The output feature map sizes of the first to fourth adaptive average pooling layers are 2×2, 4×4, 6×6, and 8×8, respectively. The convolutional kernel size of the twenty-fourth to twenty-eighth convolutional layers is 1×1, the stride is 1, the padding is 0, and the number of kernels is 80, 80, 80, 80, and 320, respectively. The activation method of the twenty-third to twenty-seventh activation layers is ReLU. The sampling method of the first to fourth upsampling layers is bilinear interpolation, and the output feature map size is the same as the input feature map size of the global feature extraction network.

[0065] In this specific embodiment, the fifth decoding neural network block comprises two sequentially connected processing branches. The input to the first processing branch is the feature map F5. The first processing branch contains a 29th convolutional layer and a 25th batch normalization layer, which are connected sequentially. The output of the global feature extraction network and the output of the 25th batch normalization layer undergo an element-wise addition operation, the result of which is the output of the first processing branch. The second processing branch contains a fifth dilated depthwise separable convolutional layer, a 26th batch normalization layer, a 28th activation layer, a 30th convolutional layer, a 27th batch normalization layer, and a 29th activation layer, which are connected sequentially. The output of the 29th activation layer is the output of the fifth decoding neural network block. The 29th convolutional layer has a 3×3 kernel size, a stride of 1, padding of 1, and 320 kernels. The 30th convolutional layer has a 1×1 kernel size, a stride of 1, padding of 0, and 96 kernels. The fifth dilated depthwise separable convolutional layer has a 3×3 kernel size, a stride of 1, padding of 1, a dilation factor of 1, and 320 kernels. The activation method for the 28th and 29th activation layers is "ReLU".

[0066] In this specific embodiment, the fourth to first decoding neural network blocks have the same structure. The decoding neural network block contains three processing branches. The input to the first processing branch is the corresponding feature map. The first processing branch contains a 31st convolutional layer and a 28th batch normalization layer connected sequentially. The input to the second processing branch is the corresponding decoding neural network block. The second processing branch contains a 5th upsampling layer. The outputs of the first and second processing branches undergo an element-wise addition operation, and the result is input to the third processing branch. The third processing branch contains a 6th dilated depthwise separable convolutional layer, a 29th batch normalization layer, a 30th activation layer, a 32nd convolutional layer, a 30th batch normalization layer, and a 31st activation layer connected sequentially. The output of the 31st activation layer is the output of the decoding neural network block. The number of input and output channels of the decoding neural network block is denoted as inc and ouc, respectively. The kernel size of the 31st convolutional layer is 3×3, the stride is 1, the padding is 1, and the number of kernels is inc. The kernel size of the 32nd convolutional layer is 1×1, the stride is 1, the padding is 0, and the number of kernels is ouc. The sixth dilated depthwise separable convolutional layer has a kernel size of 3×3, a stride of 1, padding of 2, a dilation factor of 2, and an inc number of kernels. The thirtieth and thirty-first activation layers use ReLU activation. The number of input channels for the fourth to first decoding neural network blocks are 96, 32, 24, and 16, respectively. The number of output channels for the fourth to first decoding neural network blocks are 32, 24, 16, and 16, respectively. The fifth upsampling layer has coefficients of 2, and the sampling method is bilinear interpolation.

[0067] In this specific embodiment, the first to fifth prediction neural network blocks have the same structure. Each prediction neural network block contains a 33rd convolutional layer, a 32nd activation layer, and a 6th upsampling layer connected sequentially. The output of the 6th upsampling layer is the output of the prediction neural network block. The 33rd convolutional layer has a 1×1 kernel size, a stride of 1, zero padding, and one kernel. The 32nd activation layer uses the "Sigmoid" activation method. The 6th upsampling layer uses bilinear interpolation, and the output feature map size is the same as the training color image size.

[0068] To further verify the feasibility and effectiveness of this invention, experiments were conducted. A lightweight, multi-scale salient object detection model was developed using the PyTorch framework and written in Python. The experimental setup consisted of an Intel i5-7500 processor and an NVIDIA RTX 2080Ti graphics card. The NJU2K dataset for salient object detection was used to verify the detection performance of the method (500 color three-channel RGB images and their corresponding depth images). To test the performance of the method, three commonly used objective parameters for evaluating salient object detection methods were used as evaluation metrics: Precision Recall Curve (PR), F-Measure, and Mean Absolute Error (MAE). The overall block diagram of the convolutional neural network is shown below. Figure 1 As shown, the accuracy and recall curves reflecting the performance of the method of the present invention are as follows: Figure 3 As shown, the maximum F metric (Max F) reflecting the performance of the method of the present invention is as follows: Figure 4a As shown, the maximum F-metric value is 0.8926, and the mean absolute error reflecting the performance of the method of the present invention is as follows: Figure 4b As shown, the mean absolute error is 0.0643.

[0069] To test the processing speed of the method of this invention, the trained model was used to process images in the dataset. Different batch sizes were set, and the total processing time was obtained to calculate the processing speed. The processing speed of the method of this invention is shown in Table 1.

[0070] Table 1. Processing speed index values ​​of the method of the present invention.

[0071]

[0072] As can be seen from Table 1, the lightweight network designed using the method of the present invention has a faster processing speed. This is because the method of the present invention adopts a lightweight design, which reduces the number of model parameters (6.2M) without sacrificing detection performance, thus resulting in faster computation speed.

[0073] Figure 5a The first original three-channel color RGB image is given; Figure 5b Given Figure 5a The corresponding depth image, Figure 5c The method of the present invention is given for... Figure 5a The saliency prediction image obtained by making predictions; Figure 6a The second original three-channel color RGB image is given; Figure 6b Given Figure 6a The corresponding depth image, Figure 6c The method of the present invention is given for... Figure 6a The saliency prediction image obtained by making predictions; Figure 7a The third original color three-channel RGB image is given; Figure 7b Given Figure 7a The corresponding depth image, Figure 7c The method of the present invention is given for... Figure 7a The saliency prediction image obtained from the prediction process. Comparison. Figure 5a and Figure 5c , Figure 6a and Figure 6c , Figure 7a and Figure 7c It can be seen that the saliency prediction map obtained by the method of the present invention has high detection accuracy.

Claims

1. A method for constructing and detecting a lightweight salient object detection model based on multi-scale learning, characterized in that, The method comprises the following steps: Step S1: establishing a training sample set, specifically comprising: selecting RGB three-channel color images, depth images corresponding to the color images, and real saliency images corresponding to the color images to form a training sample set; Step S2: establishing a convolutional neural network, the convolutional neural network comprising an input layer, a hidden layer, and an output layer; the input layer comprises a color image input layer and a depth image input layer, the hidden layer comprises a color image lightweight feature extraction network, a depth image lightweight feature extraction network, a depth image information enhancement network, a global feature extraction network, and a decoding network; the color image lightweight feature extraction network comprises five color lightweight network blocks connected in sequence; the depth image lightweight feature extraction network comprises five depth lightweight network blocks connected in sequence; the depth image information enhancement network comprises five depth enhancement network blocks; the first depth lightweight network block receives a training depth image transmitted by the depth image input layer, and outputs are connected to the second depth lightweight network block and the first depth enhancement network block respectively, the output of the second depth lightweight network block is connected to the third depth lightweight network block and the second depth enhancement network block respectively, the output of the third depth lightweight network block is connected to the fourth depth lightweight network block and the third depth enhancement network block respectively, the output of the fourth depth lightweight network block is connected to the fifth depth lightweight network block and the fourth depth enhancement network block respectively, and the output of the fifth depth lightweight network block is connected to the fifth depth enhancement network block; Step S3: input each color image and the corresponding depth image in the training sample set into the convolutional neural network for training, to obtain five saliency prediction images corresponding to each color image in the training sample set, denoted as {Q i} where 1≤i≤5; Step S4: Calculate the loss function value between the saliency prediction map corresponding to each color image in the training sample set and the corresponding real saliency map, denoted as {loss i}, wherein 1≤i≤5, loss i is the loss function value between the i-th saliency prediction map and the real saliency map, and the binary cross-entropy loss function is adopted, and the comprehensive loss function value is wherein 0≤α≤1 is a weight coefficient; Step S5: steps S3 and S4 are executed in cycles, in each cycle, a training sample is input into the convolutional neural network, a comprehensive loss function value is calculated, and back propagation is performed, a gradient descent method is used to optimize network parameters, an optimal weight vector and an optimal bias term are obtained, a trained convolutional neural network model is obtained, which is a lightweight saliency target detection model based on multi-scale learning; Step S6: a color image to be predicted and a depth image corresponding to the color image are input into the lightweight saliency target detection model based on multi-scale learning, a saliency prediction image Q1 corresponding to the color image is obtained, and a saliency target detection task is completed.

2. The method of claim 1, wherein the method further comprises: the decoding network comprises five decoding neural network blocks, the outputs of the fifth color lightweight network block and the fifth depth enhancement network block are subjected to Element-wise addion operation, and a feature map F5 is obtained and input into the global feature extraction network; the output of the global feature extraction network is used as one of the inputs of the fifth decoding neural network block, and the feature map F5 is used as another input of the fifth decoding neural network block; the output of the fifth decoding neural network block is used as one of the inputs of the fourth decoding neural network block, the outputs of the fourth color lightweight network block and the fourth depth enhancement network block are subjected to Element-wise addion operation, and a feature map F4 is obtained and used as another input of the fourth decoding neural network block; The output of the fourth decoding neural network block is taken as one of the inputs of the third decoding neural network block, and the output of the third color lightweight network block and the third depth enhancement network block is subjected to Element-wise addion operation to obtain a feature map F3 as another input of the third decoding neural network block; The output of the third decoding neural network block is taken as one of the inputs of the second decoding neural network block, and the output of the second color lightweight network block and the second depth enhancement network block is subjected to Element-wise addion operation to obtain a feature map F2 as another input of the second decoding neural network block; The output of the second decoding neural network block is taken as one of the inputs of the first decoding neural network block, and the output of the first color lightweight network block and the first depth enhancement network block is subjected to Element-wise addion operation to obtain a feature map F1 as another input of the first decoding neural network block; The output layer comprises five prediction neural network blocks, which respectively receive the outputs of the five decoding neural network blocks as inputs, and output five saliency prediction maps.

3. The method of claim 2, wherein the method further comprises: The input of the first color lightweight network block is a training RGB three-channel color image, and the first color lightweight network block comprises a first downsampling neural network block, a first batch normalization layer, a first activation layer, a first multi-scale neural network block, a second batch normalization layer and a second activation layer connected in sequence; the first downsampling neural network block comprises two branches, the first branch comprises a first convolutional layer, and the second branch comprises a first pooling layer, and the outputs of the first convolutional layer and the first pooling layer are subjected to stacking operation, and the result is taken as the output of the first downsampling neural network block; The second color lightweight network block comprises a second downsampling neural network block, a third batch normalization layer, a third activation layer, a second multi-scale neural network block, a fourth batch normalization layer and a fourth activation layer connected in sequence; the second downsampling neural network block comprises two branches, the first branch comprises a second convolutional layer and a first depth separable convolutional layer connected in sequence, and the second branch comprises a second pooling layer, and the outputs of the first depth separable convolutional layer and the second pooling layer are subjected to stacking operation, and the result is taken as the output of the second downsampling neural network block; The third color lightweight network block comprises a third downsampling neural network block, a fifth batch normalization layer, a fifth activation layer, a third multi-scale neural network block, a sixth batch normalization layer and a sixth activation layer connected in sequence; the third downsampling neural network block comprises two branches, the first branch comprises a third convolutional layer and a second depth separable convolutional layer connected in sequence, and the second branch comprises a third pooling layer, and the outputs of the second depth separable convolutional layer and the third pooling layer are subjected to stacking operation, and the result is taken as the output of the third downsampling neural network block; The fourth color lightweight network block comprises a fourth downsampling neural network block, a seventh batch normalization layer, a seventh activation layer, a fourth multi-scale neural network block, an eighth batch normalization layer and an eighth activation layer connected in sequence; the fourth downsampling neural network block comprises two branches, the first branch comprises a fourth convolutional layer and a third depth separable convolutional layer connected in sequence, and the second branch comprises a fourth pooling layer; the outputs of the third depth separable convolutional layer and the fourth pooling layer are subjected to a stacking operation, and the result is taken as the output of the fourth downsampling neural network block; The fifth color lightweight network block comprises a fifth downsampling neural network block, a ninth batch normalization layer, a ninth activation layer, a fifth multi-scale neural network block, a tenth batch normalization layer and a tenth activation layer connected in sequence; the fifth downsampling neural network block comprises two branches, the first branch comprises a fifth convolutional layer and a fourth depth separable convolutional layer connected in sequence, and the second branch comprises a fifth pooling layer; the outputs of the fourth depth separable convolutional layer and the fifth pooling layer are subjected to a stacking operation, and the result is taken as the output of the fifth downsampling neural network block.

4. The method of claim 3, wherein the method further comprises: The first multi-scale neural network block comprises two branches, the first branch comprises a first multi-scale attention fusion network block, and the second branch comprises a twelfth convolutional layer; the outputs of the first multi-scale attention fusion network block and the twelfth convolutional layer are subjected to an Element-wise addition operation, and the result is taken as the output of the first multi-scale neural network block; The second multi-scale neural network block comprises two branches, the first branch comprises a second multi-scale attention fusion network block, and the second branch comprises a thirteenth convolutional layer; the outputs of the second multi-scale attention fusion network block and the thirteenth convolutional layer are subjected to an Element-wise addition operation, and the result is taken as the output of the second multi-scale neural network block; The third multi-scale neural network block comprises two branches, the first branch comprises a third multi-scale attention fusion network block, a fourth multi-scale attention fusion network block and a fifth multi-scale attention fusion network block connected in sequence, and the second branch comprises a fourteenth convolutional layer; the outputs of the fifth multi-scale attention fusion network block and the fourteenth convolutional layer are subjected to an Element-wise addition operation, and the result is taken as the output of the third multi-scale neural network block; The fourth multi-scale neural network block comprises two branches, the first branch comprises a sixth multi-scale attention fusion network block, a seventh multi-scale attention fusion network block, an eighth multi-scale attention fusion network block and a ninth multi-scale attention fusion network block connected in sequence, and the second branch comprises a fifteenth convolutional layer; the outputs of the tenth multi-scale attention fusion network block and the fifteenth convolutional layer are subjected to an Element-wise addition operation, and the result is taken as the output of the fourth multi-scale neural network block; The fifth multi-scale neural network block comprises two branches, the first branch comprises an eleventh multi-scale attention fusion network block, a twelfth multi-scale attention fusion network block and a thirteenth multi-scale attention fusion network block connected in sequence, and the second branch comprises a sixteenth convolutional layer; the outputs of the thirteenth multi-scale attention fusion network block and the sixteenth convolutional layer are subjected to an Element-wise addition operation, and the result is taken as the output of the fifth multi-scale neural network block. The first to the thirteenth multi-scale attention fusion network block structures are same, the multi-scale attention fusion network block includes five processing branches, the input of the first processing branch is the input of the multi-scale attention fusion network block, the first processing branch includes the sixth pooling layer, the sixth convolutional layer, the eleventh activation layer, the seventh convolutional layer and the twelfth activation layer connected in sequence, the input of the first processing branch and the output of the twelfth activation layer are subjected to dimension expansion multiplication operation, and the result is taken as the output of the first processing branch; The input of the second processing branch is the output of the first processing branch, the second processing branch includes the seventh pooling layer, the eighth convolutional layer and the thirteenth activation layer connected in sequence, the input of the second processing branch and the output of the thirteenth activation layer are subjected to channel expansion multiplication operation, and the result is taken as the output of the second processing branch; The input of the third processing branch is the input of the multi-scale attention fusion network block, the third processing branch includes the ninth convolutional layer, the first to the fourth dilated depth separable convolutional layers, the input of the ninth convolutional layer is the input of the third processing branch, the outputs of the ninth convolutional layer are input into the first to the fourth dilated depth separable convolutional layers respectively, and the outputs of the first to the fourth dilated depth separable convolutional layers are subjected to Element-wise addition operation, and the result is taken as the output of the third processing branch; The input of the fourth processing branch is the output of the third processing branch, the fourth processing branch includes the eleventh batch normalization layer, the fourteenth activation layer, the tenth convolutional layer and the twelfth batch normalization layer connected in sequence, the input of the multi-scale attention fusion network block and the output of the twelfth batch normalization layer are subjected to Element-wise addition operation, and the result is taken as the output of the fourth processing branch; The input of the fifth processing branch is the output of the fourth processing branch, the fifth processing branch includes the eleventh convolutional layer, the thirteenth batch normalization layer and the fifteenth activation layer connected in sequence, and the output of the fifteenth activation layer is the output of the fifth processing branch; The output of the second processing branch and the output of the fifth processing branch are subjected to Element-wise multiplication operation, and the result is the output of the multi-scale attention fusion network block. 5.The method of claim 1, wherein, The first to the fifth depth lightweight network block structures are same, the depth lightweight network block includes the seventeenth convolutional layer, the fourteenth batch normalization layer, the sixteenth activation layer, the first depth separable convolutional layer, the fifteenth batch normalization layer, the seventeenth activation layer, the eighteenth convolutional layer, the sixteenth batch normalization layer, the nineteenth convolutional layer, the seventeenth batch normalization layer, the eighteenth activation layer, the second depth separable convolutional layer, the eighteenth batch normalization layer, the nineteenth activation layer, the twentieth convolutional layer and the nineteenth batch normalization layer connected in sequence; The output of the sixteenth batch normalization layer and the output of the nineteenth batch normalization layer are subjected to Element-wise addition operation, and the result is taken as the output of the depth lightweight network block; The first to the fifth depth enhancement network block structures are same, the depth enhancement network block includes two processing branches connected in sequence. The first processing branch comprises a twenty-first convolutional layer, a twenty-second activation layer, a twenty-third convolutional layer, and a twenty-second activation layer connected in sequence; the input of the deep reinforcement network block and the output of the twenty-first activation layer are multiplied in a dimension expansion operation, and the result is taken as the output of the first processing branch; The second processing branch comprises a twenty-fourth convolutional layer, a twenty-third activation layer, and a twenty-second activation layer connected in sequence; the output of the first processing branch and the output of the twenty-second activation layer are multiplied in a channel expansion operation, and the result is taken as the output of the second processing branch, which is the output of the deep reinforcement network block. 6.The method of claim 1, wherein, The global feature extraction network comprises five processing branches, the first processing branch comprises a twenty-fourth convolutional layer, a twenty-third activation layer, and a first up-sampling layer connected in sequence; The second processing branch comprises a twenty-fifth convolutional layer, a twenty-first batch normalization layer, a twenty-fourth activation layer, and a second up-sampling layer connected in sequence; The third processing branch comprises a twenty-sixth convolutional layer, a twenty-second batch normalization layer, a twenty-fifth activation layer, and a third up-sampling layer connected in sequence; The fourth processing branch comprises a twenty-seventh convolutional layer, a twenty-third batch normalization layer, a twenty-sixth activation layer, and a fourth up-sampling layer connected in sequence; The outputs of the first to fourth processing branches and the input of the global feature extraction network are subjected to a Concatenation operation, and the result is taken as the input of the fifth processing branch; The fifth processing branch comprises a twenty-eighth convolutional layer, a twenty-fourth batch normalization layer, and a twenty-seventh activation layer connected in sequence, and the output of the fifth processing branch is the output of the global feature extraction network.

7. The method of claim 2, wherein the method further comprises: The fifth decoding neural network block comprises two processing branches connected in sequence, the input of the first processing branch is the feature map F5, the first processing branch comprises a twenty-ninth convolutional layer and a twenty-fifth batch normalization layer connected in sequence, the output of the global feature extraction network and the output of the twenty-fifth batch normalization layer are subjected to an Element-wise addition operation, and the result is the output of the first processing branch, the second processing branch comprises a fifth dilated depth separable convolutional layer, a twenty-sixth batch normalization layer, a twenty-eighth activation layer, a thirtieth convolutional layer, a twenty-seventh batch normalization layer, and a twenty-ninth activation layer connected in sequence, and the output of the twenty-ninth activation layer is the output of the fifth decoding neural network block; The fourth to the first decoding neural network block has the same structure, the decoding neural network block comprises three processing branches, the input of the first processing branch is a corresponding feature map, the first processing branch comprises a thirty-first convolutional layer and a twenty-eighth batch normalization layer connected in sequence, the input of the second processing branch is a corresponding decoding neural network block, the second processing branch comprises a fifth upsampling layer, the outputs of the first processing branch and the second processing branch are subjected to an Element-wise addition operation, and the result is input into the third processing branch, the third processing branch comprises a sixth dilated depth separable convolutional layer, a twenty-ninth batch normalization layer, a thirtieth activation layer, a thirty-second convolutional layer connected in sequence, the output of the thirtieth batch normalization layer and the thirty-first activation layer is the output of the decoding neural network block; The first to the fifth prediction neural network block has the same structure, the prediction neural network block comprises a thirty-third convolutional layer, a thirty-second activation layer and a sixth upsampling layer connected in sequence, and the output of the sixth upsampling layer is the output of the prediction neural network block.

Images