A salient object detection method based on depth estimation

Abstract

The application provides a salient object detection method based on depth estimation, and belongs to the technical field of object detection; solves the problem of inaccurate, unclear and wrong detection results of the detected object in the prior art; and comprises the following steps: image data set acquisition; image preprocessing; a neural network model is built based on an encoder-decoder neural network, wherein the encoder part comprises a depth estimation module and a multi-level feature extraction module; the depth estimation module is used for extracting an initial depth prediction map; the multi-level feature extraction module is used for obtaining deep semantic feature information while extracting color, texture and direction shallow feature information, so that an initial saliency prediction map is obtained; the initial depth prediction map and the initial saliency prediction map are simultaneously input into the decoder for fusion to obtain a saliency prediction map; and the synthesized saliency prediction map is used as a loss function to supervise the training of the encoder-decoder network; and the application is applied to object detection.

Description

TECHNICAL FIELD

[0001] The application provides a salient object detection method based on depth estimation, and belongs to the technical field of salient object detection. BACKGROUND

[0002] With the rapid development of deep learning, high-performance graphics processors and the ability to process data, computer vision technology can simulate human vision to solve real-world problems in smart video surveillance, robots, industrial detection and other application scenarios. The core function of salient object detection technology is to identify important information from images according to human visual attention mechanisms. Salient object detection technology is one of the most important computer vision data preprocessing technologies and is a hot technology currently studied by the academic and industrial communities.

[0003] Classification and advantages and disadvantages of prior art: salient object detection is divided into traditional methods and deep learning-based methods as of 2014. In the traditional method, prior information is extracted by hand, which consumes a lot of time and effort, and the effect of the saliency prediction map obtained is not satisfactory. There are several problems, (1) with the explosive growth of image data, manual processing methods are difficult to achieve the goal; (2) it is difficult to extract the depth features of the image, and some salient object detection only extracts shallow features, with poor detection performance. The salient object detection method based on deep learning extracts various feature information of the image using a convolutional neural network, providing rich features for the final detection of salient object detection. Among them, the feature information of the image is widely extracted, including not only shallow feature information but also deep global context feature information, which is applied in different salient object detection algorithms, so that the neural network is better trained and the salient detection performance is effectively improved.

[0004] At present, the salient object detection method based on deep learning is divided into two categories, one is bottom-up salient object detection, and the other is top-down salient object detection research. Among them, the bottom-up salient object detection is data-driven, while the top-down salient object detection is task-driven.

[0005] With the rapid development of many applications such as automatic driving, 3D vision, VR and human-computer interaction, the depth information of the image has become the focus of attention, and how to recover the distance information of different targets from a two-dimensional image will directly affect the final image result. Compared with the use of professional equipment such as laser radar, there are several problems: (1) high cost, using related physical equipment to obtain image information will consume a lot of manpower and material resources; (2) less applicable scene, for the current large amount of image data, professional acquisition of the corresponding depth image is only suitable for a small part of the scene.

[0006] Therefore, monocular depth estimation method has attracted widespread attention, and monocular depth estimation method is divided into two kinds, one is unsupervised learning method, which does not need to use real depth image for supervision in the process, uses the encoder of neural network to generate predicted depth image during training, and constructs a group of corresponding image pairs with the original image, and then uses the network structure of the decoder to generate the final depth, and combines the applicable loss function to supervise the whole neural network. The other is supervised learning method, which usually uses external devices such as depth camera or laser radar. The method obtains the depth image as the predicted label, and converts the monocular depth estimation research into a regression task or a classification task for processing. The monocular depth estimation method generally adopts the neural network structure of encoding-decoding, but the extracted feature information is insufficient and part of the feature information is easily lost in the feature extraction process. Moreover, considering the real complex scene, the neural network is difficult to extract the context semantic information, resulting in that the obtained depth image is fuzzy and part of the result is missing. Therefore, most scholars propose to introduce a neural network structure of aggregating multiple features into monocular depth estimation, which not only considers the low-level feature information, but also pays attention to the deep feature information of the image, so as to improve the quality of the depth image. In the neural network model used in the monocular image depth estimation method, the main use is still the design concept of the traditional convolution structure, and the detection of the target in the scene of the image is lack of attention, only the depth of view field in the image is considered, and the depth distribution of the depth image is only applied in the processing research process of the two-dimensional image, and there are still many deficiencies, therefore, the application proposes a method combining salient target detection and depth estimation, which increases the depth information of the image in the fusion process of the multi-feature feature information, and constrains the scene detection ability of the salient target. SUMMARY

[0007] The application proposes a salient target detection method based on depth estimation to solve the problems in the prior art, such as that the boundary of the detected target is fuzzy and unclear at the joint with other backgrounds, it is difficult to detect the most salient target in the detected image with multiple targets, and the detection result is wrong when detecting images with similar foreground and background.

[0008] To solve the above technical problems, the technical scheme adopted by the present application is as follows: a salient object detection method based on depth estimation, comprising the following steps:

[0009] S1: image data set acquisition: the image data set includes two parts of content: one part is a data set for depth estimation, including an RGB image and its corresponding depth image; the other part is a data set including an RGB image and a ground truth image;

[0010] S2: image preprocessing;

[0011] S3: constructing a neural network: a neural network model is built based on an encoder-decoder neural network framework, wherein the encoder part includes a depth estimation module and a multi-level feature extraction module, the depth estimation module is used to extract an initial depth prediction map, and the multi-level feature extraction module is used to obtain deep semantic feature information while extracting color, texture and direction shallow feature information, so as to obtain an initial saliency prediction map;

[0012] The decoder includes an upsampling and fusion module, the initial depth prediction map and the initial saliency prediction map are input into the decoder for fusion to obtain a saliency prediction map, and the synthesized saliency prediction map is used as a loss function to supervise the training of the encoder-decoder network;

[0013] S4: outputting the corresponding initial saliency detection image after supervision, and outputting the processed final saliency detection image from the last layer of the encoder.

[0014] The depth estimation module includes a superpixel segmentation module, a depth estimation network, a depth mask network and a context global information extraction module, and a conditional random field is added for final fitting, and the depth estimation module is trained by a depth image;

[0015] The preprocessed RGB image is input into the depth estimation module for depth estimation to obtain a predicted depth image;

[0016] The context semantic information extraction module shares the neural network structure with the encoder.

[0017] The superpixel segmentation module adopts simple linear iterative clustering.

[0018] The structure of the depth estimation network includes five convolution layers of ConvP1, ConvP2, ConvP3, ConvP4 and ConvP5, ConvP1 is a 7*7 convolution kernel with a step of 2, an input channel number of 3 and an output channel number of 16;

[0019] ConvP2 is a 5*5 convolution kernel with a step of 2, an input channel number of 16 and an output channel number of 32;

[0020] ConvP3 is a 3*3 convolution kernel, the step is 2, the input channel number is 32, and the output channel number is 64;

[0021] ConvP4 is a 3*3 convolution kernel, the step is 2, the input channel number is 64, and the output channel number is 128;

[0022] ConvP5 is a 3*3 convolution kernel, the step is 2, the input channel number is 128, and the output channel number is 256.

[0023] The composition of the deep mask network is composed of 5 convolutional layers, ConvP1, ConvP2, ConvP3, ConvP4 and ConvP5 are used at the same time with the pose estimation network, and the mask mask processing predicted depth image is increased, and the supervision is realized by using the Depth image.

[0024] The multi-level feature extraction module is used for extracting shallow feature information and context semantic information, and the network model is divided into 5 parts, which are Conv1, Conv2_1, Conv2_2, Conv3_1 and Conv3_2, wherein Conv1 is input 7*7*64 convolution, Conv2_1, Conv2_2, Conv3_1 and Conv3_2 are respectively composed of different residual blocks;

[0025] Conv2_1 is composed of two residual blocks, the connection mode of the residual block is modified based on the residual module of ResNet, which is connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation function are added, and the input saliency image feature is down-sampled to 112*112 resolution;

[0026] 5.3Conv2_2 is composed of two residual blocks, the connection mode of the residual block is modified based on the residual module of ResNet, which is connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation function are added, and the input saliency image feature is down-sampled to 56*56 resolution;

[0027] 5.4Conv3_1 is composed of three residual blocks, the connection mode of the residual block is modified based on the residual module of ResNet, which is connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation function are added, and the input saliency image feature is down-sampled to 28*28 resolution;

[0028] 5.5Conv3_2 consists of three residual blocks, the connection mode of residual block is modified based on the residual module of ResNet, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation function are added, which down-samples the input saliency image features to the resolution of 14*14.

[0029] The decoder mainly includes five parts, one-to-one corresponding to the encoder, which are Deconv1, Deconv2_1, Deconv2_2, Deconv3_1 and Deconv3_2, wherein Deconv3_2 is responsible for fusing the predicted depth image and global context feature information, consisting of two residual blocks, the connection mode of residual block is modified based on the residual module of ResNet, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation function are added, which up-samples the predicted saliency image to the resolution of 14*14;

[0030] Deconv3_1 consists of two residual blocks, two-layer residual block and three-layer residual block, wherein the connection mode of residual block is modified based on the residual module of ResNet, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation function are added, which up-samples the predicted saliency image to the resolution of 28*28;

[0031] Deconv2_2 consists of two residual blocks, which up-samples the predicted saliency image to the resolution of 56*56;

[0032] Deconv2_1 is responsible for fusing the predicted depth image and low-level feature information, consisting of two residual blocks, the connection mode of residual block is modified based on the residual module of ResNet, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation function are added, which up-samples the predicted saliency image to the resolution of 112*112;

[0033] Deconv1 consists of two residual blocks, the connection mode of residual block is modified based on the residual module of ResNet, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation function are added, which up-samples the predicted saliency image to the resolution of 224*224.

[0034] The loss function is composed of the loss weighted sum of each layer network, that is,

[0035]

[0036] In the formula: L represents the loss function, k represents a specific layer in the neural network, ConvP1, ConvP2, ConvP3, ConvP4 and ConvP5 five convolutional layers in the depth estimation module, k=d1, d2, d3, d4, d5; Conv1, Conv2_1, Conv2_2, Conv3_1 and Conv3_2 five convolutional layers in the multi-level feature extraction module, k=c1, c2, c3, c4, c5; K represents the total number of layers of the neural network, represents different parameters added according to different neural networks, l (k) represents the loss of each layer in the neural network;

[0037] The loss of each layer is composed of three parts:

[0038] In the formula: represents the depth estimation loss function, represents the binary classification cross-entropy loss function, represents the structural similarity loss function;

[0039] The three parts of loss are:

[0040]

[0041] In the formula, when k=d1, d2, d3, d4, d5, it represents that the loss function supervises ConvP1, ConvP2, ConvP3, ConvP4 and ConvP5 parts of the depth estimation module; when k=c1, c2, c3, c4, c5, it represents that the loss function supervises five parts of the decoder: Deconv1, Deconv2_1, Deconv2_2, Deconv3_1 and Deconv3_2.

[0042] The depth estimation loss function introduces a depth estimation mask Y, and combines the calculation method of the backpropagation Huber loss algorithm and is modified to calculate the supervision loss, which is represented as:

[0043]

[0044]

[0045]

[0046] In the formula: Y represents an estimation mask, which is constructed according to a predicted depth map generated by the depth estimation module, l Dep represents the loss function of the depth estimation module, represents the output result of the depth estimation network, d pPseudo label of the depth estimation network; x represents the coordinate position of the pixel point, n represents the total number of the output results of the depth estimation network, c represents the threshold value of the parameter, which is one fifth of the maximum value of the elements, is respectively calculated for each pixel point in the depth image, and the calculation result is regarded as a matrix, is the maximum value of all elements in the above matrix.

[0047] A large number of experiments prove that the significant target detection method based on depth estimation proposed by the application can achieve a higher level in the field of significant target detection, and has a leading position in detection ability compared with the popular algorithm, and has the beneficial effects that:

[0048] 1. By building a new type of neural network structure, a new type of depth estimation network is constructed, which solves the problem that the significant target detection network is difficult to detect the relative depth information of the image. At the same time, the input of the depth estimation network is the features extracted by the encoder network and the image processed by the superpixel segmentation, which provides a relatively complete input for the operation of the depth estimation module, speeds up the operation of the network, and is more beneficial to the generation of the predicted image.

[0049] 2. The encoder-decoder structure is built and the network component modules are changed. After the image is input into the encoder, not only the shallow color, structure, edge and other features can be extracted, but also the context semantic feature information of the image can be fully considered. Compared with the existing U-Net method, the extracted feature information is more and the hierarchy is richer, so that the image features input into the decoder for upsampling contain relatively sufficient information. The integration of the predicted depth estimation image in the decoder can effectively avoid the loss of part of the important feature information caused by the upsampling of the decoder, and ensure the quality of the significant target detection result.

[0050] 3. The new loss function is combined, and the depth estimation loss function and the structure loss function are improved to ensure that the network is supervised from two directions at the same time, which can not only ensure the high quality of the input predicted depth image, but also supervise the network structure of the encoder-decoder, and optimize the neural network. BRIEF DESCRIPTION OF DRAWINGS

[0051] The application will be further described below in combination with the drawings:

[0052] Fig. 1 is the flowchart of the target detection method of the application;

[0053] Fig. 2 is the structure schematic diagram of the neural network of the application;

[0054] Fig. 3 is the embodiment schematic diagram of the target detection by using the detection method of the application. DETAILED DESCRIPTION

[0055] As Figs. 1 to 3 shown, the present application provides a salient object detection method based on depth estimation, comprising the following steps:

[0056] Step 1: Image dataset acquisition: the image dataset includes two parts of content: one part is a dataset for depth estimation, which has RGB images and corresponding depth images; the other part is a dataset with RGB images and ground truth images.

[0057] Step 2: Image preprocessing: pre-process the RGB image to be searched and the depth image and ground truth image in the dataset, correct the image name, and obtain the RGB image to be searched and the corresponding depth image and ground truth image.

[0058] Step 3: Building neural network: building a neural network model based on an encoder-decoder neural network framework;

[0059] 3.1 Build the encoder part of the neural network, which mainly consists of two parts: one is a depth estimation module for extracting initial depth images; the other is a multi-level feature extraction module that extracts color, texture and direction, etc. shallow feature information while obtaining deep semantic feature information. The overall encoder-decoder neural network is used to estimate the depth of the image and the change from the initial saliency map to the ground truth map, and the synthesized saliency map is used as a loss to supervise the training of the encoder-decoder network.

[0060] 3.2 Build the decoder part of the neural network, the role of the decoder is upsampling and fusion module.

[0061] Step 4: Image depth estimation module: mainly includes superpixel segmentation module, depth estimation network, depth mask network and context global information extraction module, and increases conditional random field for the last fitting, the depth estimation module is trained by depth image. The pre-processed RGB image is input into the neural network for depth estimation to obtain the predicted depth image; the context semantic information extraction module shares the neural network structure with the encoder.

[0062] 4.1 Building a neural network depth estimation module part, wherein the superpixel segmentation module uses simple linear iterative clustering (SLIC), the specific implementation steps of SLIC are: setting initial seed points and uniformly distributing according to the set number, selecting seed points in the field according to the initial seed points and assigning cluster labels, calculating the distance of each pixel point from the seed and recording, and iterating and optimizing until the error converges. SLIC uses Euclidean distance to measure the similarity of pixel points, only considers the relationship between pixel points and cluster centers, and cannot reflect the variance of colors, so the detail processing effect of colorful image regions is relatively poor.

[0063] 4.2 The composition of the depth estimation network is ConvP1, ConvP2, ConvP3, ConvP4 and ConvP5, ConvP1 is a 7*7 convolution kernel with a step of 2, an input channel number of 3 and an output channel number of 16; ConvP2 is a 5*5 convolution kernel with a step of 2, an input channel number of 16 and an output channel number of 32; ConvP3 is a 3*3 convolution kernel with a step of 2, an input channel number of 32 and an output channel number of 64; ConvP4 is a 3*3 convolution kernel with a step of 2, an input channel number of 64 and an output channel number of 128; ConvP5 is a 3*3 convolution kernel with a step of 2, an input channel number of 128 and an output channel number of 256.

[0064] 4.3 The composition of the depth mask network is composed of 5 convolutional layers, which uses ConvP1, ConvP2, ConvP3, ConvP4 and ConvP5 at the same time as the depth estimation network, and increases the mask mask processing of the predicted depth image according to the particularity of the depth estimation network, and realizes supervision using the Depth image.

[0065] Step 5: Multi-level feature extraction module: input the preprocessed RGB image into the neural network for multi-feature extraction to obtain multi-level feature information of the image;

[0066] 5.1 Building a neural network multi-level feature extraction module part, the function of network feature extraction is to extract shallow feature information and context semantic information, and the network model is divided into 5 parts, which are Conv1, Conv2_1, Conv2_2, Conv3_1 and Conv3_2. Conv1 is a 7*7*64 convolution, and Conv2_1, Conv2_2, Conv3_1 and Conv3_2 are respectively composed of different residual block combinations.

[0067] 5.2Conv2_1 consists of two residual blocks, the connection of residual blocks is based on the modification of ResNet residual module, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation functions are added. The input saliency image features are down-sampled to a resolution of 112*112.

[0068] 5.3Conv2_2 consists of two residual blocks, the connection of residual blocks is based on the modification of ResNet residual module, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation functions are added. The input saliency image features are down-sampled to a resolution of 56*56.

[0069] 5.4Conv3_1 consists of three residual blocks, the connection of residual blocks is based on the modification of ResNet residual module, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation functions are added. The input saliency image features are down-sampled to a resolution of 28*28.

[0070] 5.5Conv3_2 consists of three residual blocks, the connection of residual blocks is based on the modification of ResNet residual module, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation functions are added. The input saliency image features are down-sampled to a resolution of 14*14.

[0071] Step 6: Image feature fusion: input the predicted depth image into the decoder, and fuse with the multi-level feature information of the image, and obtain the final saliency image through the operation of the decoder.

[0072] 6.1 input the predicted depth image into the decoder, the decoder mainly includes 5 parts, one-to-one corresponding to the encoder, which are Deconv1, Deconv2_1, Deconv2_2, Deconv3_1 and Deconv3_2.

[0073] 6.2Deconv3_2 is responsible for fusing the predicted depth image and global context feature information, which consists of two residual blocks, the connection of residual blocks is based on the modification of ResNet residual module, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation functions are added. The predicted saliency image is up-sampled to a resolution of 14*14.

[0074] 6.3 Deconv3_1 consists of two residual blocks, two-layer residual block and three-layer residual block, respectively, wherein the connection mode of the residual blocks is modified based on the residual module of ResNet, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation functions are added. The predicted saliency image is up-sampled to a resolution of 28*28.

[0075] 6.4 Deconv2_2 consists of two residual blocks, and the predicted saliency image is up-sampled to a resolution of 56*56.

[0076] 6.5 Deconv2_1 is responsible for fusing the predicted depth image and low-level feature information, and consists of two residual blocks, wherein the connection mode of the residual blocks is modified based on the residual module of ResNet, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation functions are added. The predicted saliency image is up-sampled to a resolution of 112*112.

[0077] 6.6 Deconv1 consists of two residual blocks, wherein the connection mode of the residual blocks is modified based on the residual module of ResNet, connected by three convolutions of 1*1, 3*3 and 1*1, and Batch Normal and Relu activation functions are added. The predicted saliency image is up-sampled to a resolution of 224*224.

[0078] Step 7: Construct the loss function loss, wherein the loss is equal to the weighted sum of the loss of each layer: And the loss of each layer is composed of three parts loss:

[0079] The three parts loss are:

[0080]

[0081] When k = d1, d2, d3, d4, d5, it represents that the loss function supervises the ConvP1, ConvP2, ConvP3, ConvP4 and ConvP5 parts of the depth estimation module; when k = c1, c2, c3, c4, c5, it represents that the loss function supervises the five parts of the decoder: Deconv1, Deconv2_1, Deconv2_2, Deconv3_1 and Deconv3_2.

[0082] 7.1 Depth estimation loss function:

[0083] A depth estimation mask Y is introduced, and the calculation method of the reverse Huber loss algorithm is combined and modified to calculate the supervision loss, which can be represented as:

[0084]

[0085]

[0086]

[0087] In the formula, Y represents the estimated mask, constructed based on the predicted depth map generated by the depth estimation module, l Dep This represents the loss function of the depth estimation module. This represents the output of the depth prediction network, d. p The symbols represent pseudo-labels, p stands for Pseudo-Labelling, x represents the coordinates of a pixel, n represents the total number of outputs from the depth estimation network, and c represents the threshold of the parameters, which is one-fifth of the maximum value of the elements. The calculation is performed on each pixel in the depth image, and the result is viewed as a matrix. This refers to selecting the maximum value among all elements of the matrix mentioned above.

[0088] 7.2pixel-level Binary Cross Entropy (BCE) loss:

[0089]

[0090] l bce =-∑ (r，c) [G(r,c)log(S(r,c))+(1-G(r,c))log(1-S(r,c))];

[0091] Where G(r, c) ∈ {0, 1} represents the true value at the r-th row and c-th column, and S(r, c) represents the predicted value. As can be seen from the formula, the loss is related to each pixel, therefore it is a pixel-level loss.

[0092] 7.3patch-level Structural SIMilarity (SSIM) loss:

[0093]

[0094] Where x = {x j j = 1, ..., N 2} and y = {y j j = 1, ..., N 2} represents the N*N region extracted from the predicted saliency map and ground truth. μ x μy and sigma x , sigma y respectively represent the mean and variance of x and y, and sigma xy represents their covariance, let C1 = 0.01 2 , C2 = 0.03 2 . Analysis loss can find that the loss generated by each pixel point is related to the local patch (here is N*N patch) nearby, so in the process of training, the loss value of the edge part of the object is strengthened, and the non-edge part is inhibited. Because of the existence of this loss, the algorithm can pay attention to more edge details of the salient target information.

[0095] Step 8: using the loss function to supervise the network decoder, and output the corresponding initial saliency detection image after supervision.

[0096] Step 9: the processed final saliency detection image is output by the last layer of the encoder.

[0097] The present application, aiming at the problem of salient target detection in real complex scenes, proposes a deep learning network structure fusing depth information and feature information. The depth information extraction module includes a superpixel segmentation module, a depth estimation network, a depth mask network and a context global information extraction module, and the last fitting is carried out in the decoder, and the depth estimation module is trained by the Depth image. The residual network is used to build the encoder-decoder structure, the feature information (including global and local feature extraction modules) is extracted in the encoder, and the decoder includes a decoding module. The encoder and the decoder are connected correspondingly, and the depth information is input. The RGB true value image is used to train the decoder. A new loss function is established, which is a loss fusion function, and is used to supervise the depth estimation module and the decoder module respectively. The depth estimation function is used to supervise the depth estimation module, and the mixed loss function is used to supervise the decoder module.

[0098] It should be noted that the connection relationship between the components and modules of the present application is determined and can be realized. Except for the special description in the embodiments, the specific connection relationship can bring about the corresponding technical effects, and based on the premise of not relying on the corresponding software program execution, the technical problems proposed by the present application are solved. The model of the components, modules, specific elements, the connection mode between them, and the conventional use method and the expected technical effects brought by the above technical features, except for the specific description, all belong to the public content that can be obtained by the prior art before the application date, such as patents, journal papers, technical manuals, technical dictionaries, textbooks, or belong to the existing technology of the conventional technology and common knowledge in the art, and do not need to be described in detail. The technical solutions provided in the present application are clear, complete, and can be realized, and the corresponding entity products can be reproduced or obtained according to the technical means.

[0099] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A salient target detection method based on depth estimation, characterized in that: Includes the following steps: S1: Image Dataset Acquisition: The image dataset consists of two parts: one part is a dataset for depth estimation, containing RGB images and their corresponding depth images; the other part is a dataset containing RGB images and ground truth images. S2: Image preprocessing; S3: Constructing a Neural Network: Based on the encoder-decoder neural network framework, a neural network model is built. The encoder part includes a depth estimation module and a multi-level feature extraction module. The depth estimation module is used to extract the initial depth prediction map, and the multi-level feature extraction module is used to obtain deep semantic feature information while extracting shallow feature information such as color, texture, and orientation, thereby obtaining the initial saliency prediction map. The multi-level feature extraction module is used to extract shallow feature information and contextual semantic information. The network model is divided into 5 parts, namely Conv1, Conv2_1, Conv2_2, Conv3_1 and Conv3_2. Conv1 is a 7*7*64 input convolution, and Conv2_1, Conv2_2, Conv3_1 and Conv3_2 are composed of different combinations of residual blocks. Conv2_1 consists of two residual blocks. The connection method of the residual blocks is based on the residual module of ResNet and is modified by three convolutions of 1*1, 3*3 and 1*1. Batch Normal and ReLU activation functions are added to downsample the input saliency image features to a resolution of 112*112. Conv2_2 consists of two residual blocks. The connection method of the residual blocks is based on the residual module of ResNet and is modified by connecting three convolutions of 1*1, 3*3 and 1*1. Batch Normal and ReLU activation functions are added to downsample the input saliency image features to a resolution of 56*56. Conv3_1 consists of three residual blocks. The connection method of the residual blocks is based on the residual module of ResNet and is modified by three convolutions of 1*1, 3*3 and 1*1. Batch Normal and ReLU activation functions are added to downsample the input saliency image features to a resolution of 28*28. Conv3_2 consists of three residual blocks. The connection method of the residual blocks is based on the residual module of ResNet and is modified by connecting three convolutions of 1*1, 3*3 and 1*1. Batch Normal and ReLU activation functions are added to downsample the input saliency image features to a resolution of 14*14. The decoder includes an upsampling and fusion module, which simultaneously inputs the initial depth prediction map and the initial saliency prediction map into the decoder for fusion to obtain a saliency prediction map. The synthesized saliency prediction map is used as a loss function to supervise the training of the encoder-decoder network. The decoder consists of five parts, corresponding one-to-one with the encoder: Deconv1, Deconv2_1, Deconv2_2, Deconv3_1, and Deconv3_2. Deconv3_2 is responsible for fusing the predicted depth image and global contextual feature information. It consists of two residual blocks. The connection method of the residual blocks is based on the residual module of ResNet, which is modified by connecting three convolutions of 1*1, 3*3, and 1*1. Batch Normal and ReLU activation functions are added to upsample the predicted saliency image to a resolution of 14*14. Deconv3_1 consists of two residual blocks, namely a two-layer residual block and a three-layer residual block. The connection method of the residual block is based on the residual module of ResNet, which is modified by connecting three convolutions of 1*1, 3*3 and 1*1, and adding Batch Normal and ReLU activation functions to upsample the predicted saliency image to a resolution of 28*28. Deconv2_2 consists of two residual blocks that upsample the predicted saliency image to a resolution of 56*56. Deconv2_1 is responsible for fusing the predicted depth image and low-level feature information. It consists of two residual blocks. The connection method of the residual blocks is based on the residual module of ResNet. It is connected by three convolutions of 1*1, 3*3 and 1*1, and adds BatchNormal and ReLU activation functions to upsample the predicted saliency image to a resolution of 112*112. Deconv1 consists of two residual blocks. The connection method of the residual blocks is based on the residual module of ResNet. It is connected by three convolutions of 1*1, 3*3 and 1*1, and adds Batch Normal and ReLU activation functions to upsample the predicted saliency image to a resolution of 224*224. S4: The initial saliency detection image is output after supervision, and the final saliency detection image is output by the last layer of the encoder.

2. The salient target detection method based on depth estimation according to claim 1, characterized in that: The depth estimation module includes a superpixel segmentation module, a depth prediction network, a depth mask network, and a context global information extraction module. A conditional random field is added for the final fitting. The depth estimation module is trained using a depth image. The preprocessed RGB image is input into the depth estimation module for depth estimation to obtain the predicted depth image; The context semantic information extraction module shares the same neural network structure as the encoder.

3. The salient target detection method based on depth estimation according to claim 2, characterized in that: The superpixel segmentation module employs simple linear iterative clustering.

4. The salient target detection method based on depth estimation according to claim 2, characterized in that: The structure of the depth prediction network includes five convolutional layers: ConvP1, ConvP2, ConvP3, ConvP4, and ConvP5. ConvP1 is a 7*7 convolutional kernel with a stride of 2, 3 input channels, and 16 output channels. ConvP2 is a 5x5 convolution kernel with a stride of 2, 16 input channels, and 32 output channels. ConvP3 is a 3x3 convolution kernel with a stride of 2, 32 input channels, and 64 output channels. ConvP4 is a 3x3 convolution kernel with a stride of 2, 64 input channels, and 128 output channels. ConvP5 is a 3x3 convolution kernel with a stride of 2, 128 input channels, and 256 output channels.

5. The salient target detection method based on depth estimation according to claim 4, characterized in that: The depth mask network consists of five convolutional layers, which are used in conjunction with the pose estimation network (ConvP1, ConvP2, ConvP3, ConvP4, and ConvP5). The predicted depth image is further processed by masking, and supervision is achieved using the depth image.

6. The salient target detection method based on depth estimation according to claim 1, characterized in that: The loss function is composed of a weighted sum of the losses of each layer of the network, i.e. ； In the above formula: L represents the loss function, k represents a specific layer in the neural network, and K represents the total number of layers in the neural network. This indicates the different parameters added depending on the type of neural network. This represents the loss of each specific layer in the neural network; The loss of each layer consists of three parts: ; In the above formula: This represents the depth estimation loss function. This represents the cross-entropy loss function for binary classification. Represents the structural similarity loss function; The three loss components are as follows: ； In the above formula, when k=d1, d2, d3, d4, d5, it represents the ConvP1, ConvP2, ConvP3, ConvP4, and ConvP5 parts of the loss function supervised deep estimation module; when k=c1, c2, c3, c4, c5, it represents the 5 parts of the loss function supervised decoder: Deconv1, Deconv2_1, Deconv2_2, Deconv3_1, and Deconv3_2.

7. The salient target detection method based on depth estimation according to claim 6, characterized in that: The depth estimation loss function incorporates a depth estimation mask Y, and after modification using the calculation method of the inverse HuBer loss algorithm, the supervised loss is calculated. The supervised loss is expressed as: ； ； ； In the above formula: Y represents the estimated mask, constructed based on the predicted depth map generated by the depth estimation module. This represents the loss function of the depth estimation module. This represents the output of the depth prediction network. This represents the pseudo-label of the depth estimation network, where x represents the pixel coordinates, n represents the total number of outputs from the depth estimation network, and c represents the threshold value of the parameters. The calculation is performed on each pixel in the depth image, and the result is viewed as a matrix. This refers to selecting the maximum value among all elements of the matrix mentioned above.

Images