A method for detecting a salient object based on edge feature guidance

Abstract

The embodiment of the application relates to the technical field of saliency object detection, in particular to a saliency object detection method based on edge feature guidance, comprising the following steps: extracting edge data of a saliency object from a saliency object ground truth map, and taking the edge data as an additional preliminary training set; extracting image features by using a basic network, and respectively obtaining edge features of the saliency object and multi-scale position features based on the image features; dynamically learning the relationship between the edge features and the multi-scale position features based on a multilayer perception machine to obtain edge-enhanced fusion features; and sending the edge-enhanced fusion features into a convolution network for refinement processing, and obtaining a final segmentation result based on the refined edge-enhanced fusion features. The method provided by the application effectively utilizes saliency object edge information and position information, and overcomes the defects that the existing saliency object detection method does not sufficiently utilize edge information and lacks consideration of the connection between edge position features and global features.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of salient object detection, and particularly relates to a salient object detection method based on edge feature guidance. BACKGROUND

[0002] In deep learning, salient object detection (SOD) aims to find the most eye-catching object from the input image, and is one of the important preprocessing steps in computer vision tasks. Its research scope covers images, three-dimensional data, light fields, videos and many other aspects, and the related technology is widely used in information retrieval, target tracking, image editing and other fields. In recent years, thanks to the rapid development of related technologies, especially the substantial progress of semantic segmentation technology combined with deep learning, the research goals and problems in the field of salient object detection have been further refined and in-depth. In the field of salient object detection for static images, the research goal has gradually changed from single-target detection task to complex tasks including multi-target, camouflage target and other problems.

[0003] Consistent with most computer vision topics, salient object detection has experienced a transition from traditional methods to deep learning methods, and currently deep learning methods firmly occupy the top position of the accuracy ranking list. The main difference between the two methods is that traditional methods need to design features manually, while deep learning methods can automatically extract features using convolutional neural networks. In traditional methods, salient object features are generally designed according to the characteristics of the image, and then combined with some machine learning methods. In deep learning methods, with the help of convolutional neural networks, the position and boundary of the salient object can be converted into features with stronger expression ability, so the research focus shifts to how to find and use the information part related to the salient object in the image.

[0004] There are still some problems that have not been solved in the existing salient object detection methods. First, the use of boundary information is not sufficient. Most methods do not explicitly use the edge information of the salient object, which makes the method unable to well predict the complete object contour. Especially in the case where the contrast of the neighborhood of the boundary of the salient object is low, the surrounding pixels are relatively close, resulting in that the features obtained after convolution are very similar to the pixels near the edge, and it is difficult to distinguish the boundary of the salient object in the subsequent processing process, which is also the direct reason for the blurred boundary or even the inability to predict the edge in the prediction result. Second, the existing salient object detection methods are very sensitive to the size of the input image, and the prediction results of the same image under different sizes are not consistent. Third, the connection between the edge position feature and the global feature is not considered. The commonly used processing method is to use independent branches to process the global feature and the edge position feature, and then simply fuse them without considering the difference and connection between the two features, which introduces noise in the fusion process. SUMMARY

[0005] The embodiment of the present application provides a salient object detection method based on edge feature guidance, which effectively utilizes edge information and position information of a salient object and considers the relationship with global information to overcome the defects of insufficient utilization of edge information and lack of consideration of the relationship between edge position features and global features in the prior salient object detection method.

[0006] To solve the above technical problems, the embodiment of the present application provides a salient object detection method based on edge feature guidance, which comprises the following steps: first, extracting edge data of a salient object from a salient object ground truth map and taking the edge data as an additional preliminary training set; then, extracting image features by using a basic network and respectively obtaining edge features and multi-scale position features of the salient object based on the image features; next, dynamically learning the relationship between the edge features and the multi-scale position features based on a multi-layer perception to obtain edge-enhanced fusion features; finally, sending the edge-enhanced fusion features into a convolution network for refinement processing and obtaining a final segmentation result based on the refined edge-enhanced fusion features.

[0007] In some example embodiments, the edge data of the salient object is extracted from the salient object ground truth map and taken as an additional preliminary training set, which comprises: calculating the amplitude and direction of image pixel gradient based on the salient object ground truth map to obtain calculated image pixel gradient; performing non-maximum suppression on the calculated image pixel gradient to obtain an edge binary image of a preliminary salient object; screening the edge binary image using a double threshold to obtain an edge image close to the real edge of the salient object; extracting the edge data of the salient object based on the edge image and converting the edge data into png format, and taking the png format edge data as an additional preliminary training set.

[0008] In some example embodiments, the image features are extracted by using a basic network and the edge features and multi-scale position features of the salient object are respectively obtained based on the image features, which comprises: performing regression learning on the features obtained by the convolution block of the basic network and the edge image to obtain the edge features of the salient object; performing regression learning on the features obtained by the convolution block of the basic network and the salient object ground truth map to obtain the position features of the salient object; wherein the regression process of the edge features of the salient object comprises: performing binary cross-entropy judgment on the pixel points corresponding to the prediction boundary map and the pixel points corresponding to the contour ground truth map to obtain a judgment result; after accumulating and averaging the judgment result, obtaining the loss between the image edge prediction result and the edge contour ground truth map; and the expression of the regression process of the edge features of the salient object is:

[0009] lbce (x, y) = y log(x) + (1 - y) log(1 - x)

[0010]

[0011] wherein, l bce (x, y) function is a binary cross-entropy function, indicating the difference between two probability distributions; G e is an edge contour ground truth map; n represents the number of pixel points, k represents the index of the pixel point, e k and represent the pixel points of the predicted boundary map and the contour ground truth map, respectively.

[0012] In some example embodiments, the process of regressing the features obtained by the convolutional block of the base network with the salient object ground truth map further includes: multi-scale rough salient object regression; wherein the rough salient object regression process includes: performing binary classification cross-entropy judgment on the pixel points corresponding to the predicted multi-scale salient object result and the pixel points corresponding to the salient object ground truth map, to obtain a judgment result; and accumulating and averaging the judgment result to obtain the loss between the rough salient object prediction result and the salient object ground truth map; the expression of the rough salient object regression process is:

[0013] l bce (x, y) = y log(x) + (1 - y) log(1 - x)

[0014]

[0015] wherein, l bce (x, y) function is a binary cross-entropy function, indicating the difference between two probability distributions; G Si is a salient object ground truth map; n represents the number of pixel points; k represents the index of the pixel point; and represent the pixel points of the predicted salient object result and the salient object ground truth map, respectively; i represents the scale corresponding to the salient object feature, i = 3, 4, 5, 6.

[0016] In some example embodiments, based on the multi-layer perception, the relationship between the edge feature and the multi-scale position feature is dynamically learned to obtain an edge-enhanced fusion feature, including: performing spatial position coding on the edge feature, and refining the edge feature based on an attention mechanism; resampling the significant object features of different scales until the resolution of the significant object features is consistent with that of the edge feature; concatenating the significant object features of different scales and the edge feature, and then inputting the same into a convolution layer for learning to respectively obtain an edge coding weight of the significant object and a position coding weight of the significant object; and based on the edge coding weight of the significant object and the position coding weight of the significant object, giving specific weights to the edge feature and the position feature, and fusing the edge feature and the position feature to obtain an edge-enhanced fusion feature.

[0017] In some example embodiments, the expression of the calculation process of refining the edge feature is:

[0018]

[0019] wherein, the delta function is a ReLU activation function; the symbol "⊙" represents pixel-level multiplication; E is an edge feature; W and b are parameters generated by a series of fully connected layers for the edge feature; wherein, the dimension of the edge feature of the significant object is consistent with that of the basic feature of the position of the significant object.

[0020] In some example embodiments, the expression of the calculation process of the edge coding weight of the significant object and the position coding weight of the significant object is:

[0021] f i =Concat(E,upsample(S i ))

[0022] (W E ,W Si )=σ(GAP(δ(conv(f i ,W)))

[0023] wherein, S i represents the significant object feature of the i-th layer; E represents an edge feature; the Concat function is a feature channel concatenation operation; upsample represents up-sampling, wherein, after up-sampling, the resolution of S i is the same as that of the edge feature E; W represents the parameters of the convolution layer; δ represents a ReLU activation function; GAP represents a global average pooling; σ is a sigmoid function used for normalization processing; W E and W Si are respectively the edge coding weight of the significant object and the position coding weight of the significant object; and the expression of the calculation process of obtaining the edge-enhanced fusion feature is:

[0024]

[0025] wherein, is an edge-enhanced fusion feature obtained by fusing the i-th layer salient object feature and the edge feature.

[0026] In some example embodiments, the edge-enhanced fusion feature is sent into a convolution network for refinement processing, and based on the refined edge-enhanced fusion feature, a final segmentation result is obtained, including: sending the edge-enhanced fusion feature into a convolution block for decoding; performing regression on the decoded features at multiple scales and the salient objects corresponding thereto to obtain regression results; converting the regression results into saliency scores to obtain a robust saliency segmentation result; fusing the multi-scale saliency features after edge enhancement to obtain a final fine saliency prediction map, and simultaneously calculating a loss using an edge-enhanced cross-entropy function and an intersection over union loss function for back propagation.

[0027] In some example embodiments, the calculation process of the intersection over union loss function is: dividing the intersection of two segmentation maps by the union to obtain the value of the intersection over union; and the calculation formula of the intersection over union loss function is as follows:

[0028]

[0029]

[0030] wherein, IOU is the intersection over union; A and B represent two images for which the intersection over union is to be calculated; C is the number of image pixels; y gt represents a pixel point of a salient object ground truth map; y pred represents a final fine saliency prediction map obtained by fusing the multi-scale saliency features after edge enhancement.

[0031] In some example embodiments, when the loss is calculated using the edge-enhanced cross-entropy function and the intersection over union loss function, the calculation rule of the loss function is defined as follows: the weight of the region near the salient object boundary is enhanced using an edge map, a two-dimensional normal distribution is introduced to improve the generalization ability, and the Gaussian kernel function G(x, y) is as follows:

[0032]

[0033] Let E gt be a boundary picture, and the boundary picture is subjected to Gaussian kernel convolution to obtain a weight w g :

[0034] w g = Conv(G(x, t), E)

[0035] wherein, wg a weight assigned to the boundary adjacent region;

[0036] The final edge enhancement loss function is shown as follows:

[0037]

[0038] wherein, l ee represents the final edge enhancement loss function, l bce represents a binary cross-entropy function;

[0039] In the calculation of the loss by using the edge enhancement cross-entropy function and the intersection over union loss function, the calculation rule for calculating the loss is defined as follows: let l coarse and l fine correspond to the rough saliency segmentation result of the first half part and the fine saliency object segmentation result of the second half part respectively, and the calculation process is as follows:

[0040] L coarse = l edge (E, G e ) + l sal (S i , G si )

[0041]

[0042] wherein, E, S i respectively represent the edge feature of the first half part and the coarse-grained feature of the salient object, and F pred is the final multi-scale fused saliency object prediction result;

[0043] The final loss function is the sum of the rough prediction loss and the fine prediction loss of the saliency object detection, and the expression is shown as follows:

[0044] l total = l coarse + l fine

[0045] wherein, l total represents the final loss function.

[0046] The technical scheme provided by the embodiments of the present application has at least the following advantages:

[0047] The embodiment of the present application provides a salient object detection method based on edge feature guidance, which comprises the following steps: first, edge data of a salient object is extracted from a salient object true value graph, and the edge data is taken as an additional preliminary training set; then, image features are extracted by using a basic network, and based on the image features, edge features of the salient object and multi-scale position features are obtained respectively; next, based on a multi-layer perception, the relationship between the edge features and the multi-scale position features is dynamically learned to obtain edge-enhanced fusion features; finally, the edge-enhanced fusion features are sent into a convolution network for refinement processing, and based on the refined edge-enhanced fusion features, a final segmentation result is obtained.

[0048] Firstly, the present application designs a conversion rule for applying image edge features to salient object positions, each multi-scale fusion branch enhances the information of the edge features, and adaptively fuses with the salient object information, solving the problem of introducing noise when fusing edge features and object features with large span.

[0049] Secondly, the present application simultaneously uses salient object edge and position information to assist salient object detection, and simultaneously considers the relationship between the two kinds of information and global information, the difference between multi-scale features, and the misleading situation possibly produced in the up-sampling fusion process, makes up for the shortcomings of possibly introducing noise in the fusion process of auxiliary information and salient object information, and realizes more accurate salient object detection.

[0050] Thirdly, the present application considers the universality and portability of the scheme, more fully utilizes the edge information, gives higher weight to the edge neighborhood pixel points in the loss function, and further strengthens the role of the edge features. Due to the introduction of multiple sets of edge and position features, the present application adds a relationship feature weighting step, so that the user can obtain the best weight of the corresponding relationship feature without manual allocation, and the scheme is more universal. BRIEF DESCRIPTION OF DRAWINGS

[0051] One or more embodiments are exemplified by the pictures in the corresponding drawings, which do not constitute a limitation on the embodiments, and the drawings in the drawings do not constitute a proportional limitation, unless specifically stated.

[0052] Figure 1 A flowchart of a salient object detection method based on edge feature guidance provided by an embodiment of the present application is shown in the figure;

[0053] Figure 2 A framework diagram of a salient object detection method based on edge feature guidance provided by an embodiment of the present application is shown in the figure;

[0054] Figure 3is a flowchart of an edge position relationship feature calculation process provided by an embodiment of the present application;

[0055] Figure 4 is a weight diagram of edge enhancement loss provided by an embodiment of the present application;

[0056] Figure 5 is a visualization diagram of a comparison of multiple salient object detection methods provided by an embodiment of the present application. DETAILED DESCRIPTION

[0057] As can be known from the background, the existing salient object detection methods have the technical problems of insufficient utilization of edge information and lack of consideration of the connection between edge position features and global features.

[0058] As mentioned earlier, salient object detection has undergone a process of transition from traditional methods to deep learning methods. The main difference between traditional methods and deep learning methods is that traditional methods require manual design of features, while deep learning methods can automatically extract features using convolutional neural networks. In traditional methods, salient object features are generally designed according to the characteristics of the image, and then combined with some machine learning methods. A related technology proposes a generalized principal component analysis, which uses a linear subspace method to replace the actual segmented image, and selects the salient region by measuring the feature contrast value and the region geometric attribute. Another related technology proposes to introduce similar images to make the judgment result more accurate, which points out that similar images will provide a good approximation of the background region, even if the salient region of each image may not be close, but distinguishing the background region is equivalent to distinguishing the salient region.

[0059] In deep learning methods, with the help of convolutional neural networks, the position and boundary of salient objects and other information can be converted into features with stronger expression ability, so the research focus shifts to how to find and use the information part related to salient objects in the image. A related technology proposes a model composed of two parts, namely a recurrent localization network and a boundary refinement network. This model can help better locate salient objects by repeatedly focusing on the spatial distribution of various scenes, and also help refine the salient map by the relationship between each pixel and its neighbors. Another related technology proposes to expand the influence of pooling to solve this problem, proposes pyramid pooling, and adds an edge detection branch to the pooling network to sharpen the details of salient objects. A related technology proposes to use BCE, SSIM, and IOU joint loss to solve the problem of blurred boundaries of segmented images, so that the network pays more attention to boundary quality, rather than just focusing on region accuracy as before.

[0060] Since the existing salient object detection method does not sufficiently utilize the boundary information, and the existing salient object detection method lacks consideration of the connection between the edge position feature and the global feature. The commonly used processing method is to use independent branch processing for the global feature and the edge position feature, and then simply fuse, without considering the difference and connection between the two features, which introduces noise in the fusion process. Therefore, it is of great practical significance to develop a salient object detection method which effectively utilizes the edge and position information of the salient object and considers the relationship between the global information.

[0061] To solve the above technical problems, the embodiment of the present application provides a salient object detection method based on edge feature guidance, which comprises the following steps: first, extracting edge data of a salient object from a salient object ground truth map, and taking the edge data as an additional preliminary training set; then, extracting image features using a basic network, and based on the image features, respectively obtaining edge features of the salient object and multi-scale position features; next, based on a multi-layer perception, dynamically learning the relationship between the edge features and the multi-scale position features to obtain edge-enhanced fusion features; finally, sending the edge-enhanced fusion features into a convolution network for refinement processing, and based on the refined edge-enhanced fusion features, obtaining a final segmentation result. The embodiment of the present application provides a salient object detection method based on edge feature guidance, which effectively utilizes the edge information and position information of the salient object and considers the relationship between the global information, to overcome the problem that the existing salient object detection method does not sufficiently utilize the edge information and lacks consideration of the connection between the edge position feature and the global feature.

[0062] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, the technical solutions claimed by the present application can be realized even without these technical details and various changes and modifications based on the following embodiments.

[0063] Referring to Figure 1 The embodiment of the present application provides a salient object detection method based on edge feature guidance, which comprises the following steps:

[0064] Step S1, extracting edge data of a salient object from a salient object ground truth map, and taking the edge data as an additional preliminary training set.

[0065] Step S2, extracting image features using a basic network, and based on the image features, respectively obtaining edge features of the salient object and multi-scale position features.

[0066] Step S3, based on the multi-layer perceptron, the relationship between the edge feature and the multi-scale position feature is dynamically learned, and the edge enhanced fusion feature is obtained.

[0067] Step S4, the edge enhanced fusion feature is sent into the convolution network for refinement processing, and based on the refined edge enhanced fusion feature, the final segmentation result is obtained.

[0068] Specifically, the edge feature guided salient object detection method provided in the application first extracts the edge data of the salient object from the salient object ground truth image, and uses the edge data as an additional preliminary training set, thereby preparing for subsequent learning of robust edge features. Next, using a mature and reliable base network to extract image features, and further obtaining multi-scale salient object edge features and multi-scale position features through the FPN structure. Then, the relationship between the edge feature and the salient object position feature is dynamically learned by using the multi-layer perceptron, and the edge enhanced fusion feature is obtained by complementarily using the salient object position feature and the edge feature. Here, the fused multi-scale feature is also called edge and object enhanced feature; finally, the fused multi-scale feature is further refined in the convolution network, and the refined multi-scale feature is added, thereby obtaining the final segmentation result. In order to enhance the edge learning effect of the network, the edge auxiliary information is used in the cross entropy loss function to give higher weight to the area near the boundary.

[0069] Figure 2 Based on Figure 1 The workflow shown in the figure shows the basic framework of the entire scheme. Next, the present specification will be described in detail according to the above four steps.

[0070] In some embodiments, the edge data of the salient object is extracted from the salient object ground truth image in step S1, and the edge data is used as an additional preliminary training set, including:

[0071] Step S101, based on the salient object ground truth image, the amplitude and direction of the image pixel gradient are calculated, and the calculated image pixel gradient is obtained.

[0072] Step S102, non-maximum suppression is performed on the calculated image pixel gradient, and an edge binary image of the preliminary salient object is obtained.

[0073] Step S103, using double threshold, the edge binary image is screened, and an edge image close to the real edge of the salient object is obtained.

[0074] Step S104, based on the edge image, the edge data of the salient object is extracted, and the edge data is converted into png format, and the png format edge data is used as an additional preliminary training set.

[0075] Specifically, the specific method of step S1 is: selecting an edge detection operator to process the saliency segmentation ground truth that needs to extract edges, obtaining saliency object edge data, and the main selection standard is accuracy. In this embodiment, the edge detection operator is Canny operator. When calculating the gradient and direction of a pixel point, for a picture, the gradient can well reflect the change of its pixels, and the greater the gradient change, the greater the difference between adjacent pixels. In terms of the whole picture, there is an edge in a certain region, which is visually black to white. The calculation of the gradient is divided into size and direction. First, the gradient in each direction is calculated, and then the square root and tangent line are calculated. The following is the calculation method of the gradient in x and y directions:

[0076]

[0077]

[0078] Then, the process of double threshold and lag boundary tracking is performed. Specifically, a threshold upper limit and a threshold lower limit are set. If a pixel point in the image is greater than the threshold upper limit, it is considered to be a boundary (strong boundary), and if it is less than the threshold lower limit, it is considered not to be a boundary. The two are considered to be candidates (weak boundaries), which need to be judged again by lag boundary tracking.

[0079] In some embodiments, the step of extracting the saliency object edge and the multi-scale position feature is: taking VGG16 backbone network as an example, since the low-level features contain more detailed information and the high-level features contain more semantic information, the features obtained by the last convolution layer of the second convolution block are used to learn the edge features by regression with the edge map, and the features of the subsequent layers are used to learn the position features by regression with the saliency object ground truth.

[0080] In some embodiments, the feature extraction method is: using a feature pyramid network to extract multi-scale features of the original image. Each scale of feature is predicted separately, and the prediction loss of each scale is added to the final loss function.

[0081] In some embodiments, in step S2, the image features are extracted by using a basic network, and based on the image features, the edge features of the saliency object and the multi-scale position features are obtained, including:

[0082] Step S201, the features obtained by the convolution block of the basic network are used to learn the edge features of the saliency object by regression with the edge image.

[0083] Step S202, the features obtained by the convolution block of the basic network are used to learn the multi-scale position features of the saliency object by regression with the saliency object ground truth.

[0084] Specifically, the specific method of step S2 is: first, the image is sent into the basic network to extract features. The selection of the basic network takes into account the accuracy and efficiency, such as Figure 2 As shown in the figure, the embodiment selects the VGG16 network pre-trained on the ImageNet dataset, uses the convolution block 2 (Conv2-2) to extract edge features, and uses the convolution block 3 to the convolution block 6 (the convolution block 6 uses the convolution block 5) to extract significant object features. Then, using the FPN structure, the significant object features and the edge features are respectively regressed with the significant object ground truth map (provided by the dataset) and the significant object edge map to further obtain the corresponding multi-scale significant object basic features S i (i = 3, 4, 5, 6) and boundary basic features E.

[0085] Wherein, the regression process of the edge features of the significant object includes: performing binary cross-entropy judgment on the pixel points corresponding to the predicted boundary map and the pixel points corresponding to the contour ground truth map to obtain a judgment result; and after accumulating and averaging the judgment result, obtaining the loss between the image edge prediction result and the edge contour ground truth map.

[0086] The expression of the regression process of the edge features of the significant object is:

[0087] l bce (x, y) = ylog(x) + (1-y)log(1-x)

[0088]

[0089] Wherein, l bce (x, y) function is a binary cross-entropy function, indicating the difference between two probability distributions; G e is the edge contour ground truth map; n represents the number of pixel points, k represents the index of the pixel point, e k and respectively represent the pixel points of the predicted boundary map and the contour ground truth map.

[0090] In some embodiments, the process of regressing the features obtained by the convolution block of the basic network with the significant object ground truth map in step S202 also includes: regression of a multi-scale rough significant object.

[0091] Wherein, the regression process of the multi-scale rough significant object includes: performing binary cross-entropy judgment on the pixel points corresponding to the predicted multi-scale significant object result and the pixel points corresponding to the significant object ground truth map to obtain a judgment result; and after accumulating and averaging the judgment result, obtaining the loss between the rough significant object prediction result and the significant object ground truth map; the expression of the regression process of the rough significant object is:

[0092] l bce(x, y)=ylog(x)+(1-y)log(1-x)

[0093]

[0094] Among them, l bce The (x, y) function is a binary cross-entropy function, representing the difference between two probability distributions; G Si This is a truth map of salient objects; n represents the number of pixels; k represents the index of the corresponding pixel. and represents the pixel points of the predicted salient object result and the salient object ground truth map, respectively; i represents the scale corresponding to the salient object feature, i = 3, 4, 5, 6.

[0095] In some embodiments, step S3 involves dynamically learning the relationship between the edge features and the multi-scale positional features based on a multilayer perceptron to obtain edge enhancement fusion features, including:

[0096] Step S301: Encode the edge features spatially and refine them based on an attention mechanism.

[0097] Step S302: Resample the salient object features at different scales until the resolution of the salient object features is consistent with that of the edge features.

[0098] Step S303: After concatenating the salient object features and edge features at different scales, the convolutional layer is fed into the learning layer to obtain the edge encoding weights and position encoding weights of the salient objects.

[0099] Step S304: Based on the edge encoding weight and the position encoding weight of the salient object, assign specific weights to the edge features and the position features, and fuse the edge features and the position features to obtain edge enhancement fusion features.

[0100] like Figure 3 As shown, the specific method of step S3 is as follows: First, spatial location encoding is performed on the edge feature E, the channel dimension is compressed, and the edge feature is refined using an attention mechanism.

[0101] In some embodiments, the expression for the calculation process of refining the edge features is:

[0102]

[0103] Wherein, the delta function is the ReLU activation function; the symbol "⊙" represents pixel-level multiplication; E is the edge feature; W and b are parameters generated by a series of fully connected layers for the edge feature; and, for the convenience of processing, the dimension of the edge feature of the salient object is consistent with the dimension of the basic feature of the salient object position.

[0104] The salient object feature S i is resampled to be consistent with the resolution of the edge feature E, and the two are spliced and then input into a subsequent convolutional layer and a pooling layer to learn the corresponding basic feature weights W E of the salient object edge and position. Si .

[0105] In some embodiments, the expression of the calculation process of the edge encoding weight of the salient object and the position encoding weight of the salient object is as follows:

[0106] f i = Concat(E, upsample(S i ))

[0107] (W E , W Si ) = σ(GAP(δ(conv(f i , W))

[0108] Wherein, S i represents the salient object feature of the i-th layer; E represents the edge feature; the Concat function is a feature channel splicing operation; upsample represents up-sampling, wherein, after up-sampling, S i is the same in resolution as the edge feature E; W represents the parameters of the convolutional layer; δ represents the ReLU activation function; GAP represents the global average pooling; σ is the sigmoid function used for normalization processing; W E and W Si are the edge encoding weight of the salient object and the position encoding weight of the salient object, respectively.

[0109] After obtaining the weights W E and W Si , specific weights are given to the edge and position features based on the weights W E and W Si , and the two relationship features are fused to obtain the preliminary edge-enhanced fusion feature.

[0110] The expression of the calculation process of the edge-enhanced fusion feature is as follows:

[0111]

[0112] Wherein, This is the result of fusing the salient object features and edge features of the i-th layer, i.e., edge enhancement fusion features.

[0113] In some embodiments, step S4 feeds the edge enhancement fusion features into a convolutional network for refinement, and based on the refined edge enhancement fusion features, obtains the final segmentation result, including:

[0114] Step S401: Feed the edge enhancement fusion feature into the convolutional block for decoding.

[0115] Step S402: Regress the decoded features at multiple scales with their corresponding salient objects to obtain the regression results.

[0116] Step S403: Convert the regression results into significance scores to obtain robust significance segmentation results.

[0117] Step S404: Fuse the multi-scale saliency features after edge enhancement to obtain the final refined saliency prediction map. At the same time, use the edge enhancement cross-entropy function and cross-union ratio loss function to calculate the loss for backpropagation.

[0118] Specifically, the method for step S4 is as follows: First, the multi-scale edge enhancement fusion features are... The data is fed into the decoder for decoding. Then, the decoded features at multiple scales are regressed against the corresponding salient objects. The regression results are converted into saliency scores using the sigmoid function to obtain robust saliency segmentation results, i.e., saliency result maps at each scale. Finally, all multi-scale saliency features are fused to obtain the final refined saliency prediction map. At the same time, the loss is calculated using the edge enhancement cross-entropy function and the IOU function for backpropagation.

[0119] In some embodiments, the crossover ratio (CRR) loss function is calculated as follows: the intersection of the two segmentation images is divided by their union to obtain the CRR value; the formula for calculating the CRR loss function is as follows:

[0120]

[0121]

[0122] Where IOU stands for Intersection over Union (IoU); A and B represent the two images for which IoU calculation is performed; C is the number of pixels in the image; y gt y represents the pixels in the truth map of salient objects; pred This represents the final, refined saliency prediction map obtained by fusing multi-scale saliency features after edge enhancement.

[0123] See Figure 4, shows the calculation process of edge enhancement loss. In some embodiments, when calculating the loss by using the edge enhancement cross-entropy function and the intersection over union loss function, the calculation rule of the loss function is defined as follows: the edge map is used to enhance the weight of the neighborhood near the boundary of the salient object, a two-dimensional normal distribution is introduced to improve the generalization ability, and the Gaussian kernel function G(x, y) is as follows:

[0124]

[0125] Let E gt be the edge map, and the edge map is convolved with the Gaussian kernel to obtain the weight w g :

[0126] w g = Conv(G(x, y), E)

[0127] where w g represents the weight assigned to the boundary neighborhood region;

[0128] The final edge enhancement loss function is as follows:

[0129]

[0130] where l ee represents the final edge enhancement loss function, and l bce represents the binary cross-entropy function.

[0131] When calculating the loss by using the edge enhancement cross-entropy function and the intersection over union loss function, the loss function calculation rule of the framework is defined as follows: let l coarse and l fine correspond to the rough saliency segmentation result of the first half and the fine salient object segmentation result of the second half, respectively, and the calculation process is as follows:

[0132] l coarse = l edge (E, G e )+l sal (S i , G Si )

[0133]

[0134] where E, S i represent the edge feature of the first half and the coarse-grained feature of the salient object, respectively, and F pred is the final multi-scale fused salient object prediction result;

[0135] The final loss function is the sum of the rough prediction loss and the fine prediction loss of the salient object detection, and the expression is as follows:

[0136] l total = l coarse + l fine

[0137] wherein, l total denotes the final loss function.

[0138] Experimental results of the salient object detection method based on edge feature guidance:

[0139] To verify the actual performance of the present application on the salient object detection task, experiments were conducted on four well-known public data sets DUTS, ECSSD, HKU-IS and DUT-OMRON.

[0140] DUTS is the largest salient object detection data set at present, containing 10553 training images and 5019 test images. The training images are selected from the training set of ImageNet, and the test images are selected from the test set of ImageNet and the SUN data set, and the pixel-level annotation is completed by 50 participants manually.

[0141] The ECSSD data set is composed of 1000 complex scene images selected from MSRA, which contains many semantically meaningful but complex structured images, and is commonly used for model evaluation. The PASCAL-S data set contains 4447 images with pixel-level annotation of salient objects, which at least meet one of the following conditions: at least one salient object touches the boundary of the image; the color contrast is less than 0.7; there are multiple unconnected salient objects. DUT-OMRON contains 5168 high-quality images manually selected from 14000 images, all of which contain one or more salient objects and relatively complex backgrounds. In line with most methods, the present application uses MAE and F-measure as evaluation criteria, wherein the smaller the value of MAE (mean absolute error) indicates better performance, and the F-measure considers the precision and recall comprehensively, and the larger the value indicates better performance, denoted by the symbol F β .

[0142] Table 1 gives the comparison of the present application and other methods in MAE and F β measure on the DUTS validation set.

[0143] Table 1 MAE and F β

[0144]

[0145] According to Table 1, it can be seen that the present application achieves excellent accuracy on this data set, and still achieves certain improvement in the case of MAE reaching below 0.05 on this data set, and F βClose to the optimal value. It is worth noting that our improvement is not only reflected in the data, but also in the visual effect. We have done some visualizations in Figure 5 which can be seen that the application is superior to other methods, especially in the effect of distinguishing the boundary of the salient object. The application balances and strengthens the boundary information, which can better identify the interest points near the boundary of the salient object.

[0146] Table 2 gives the comparison of MAE and F β measurements of the application and other methods on the ECSSD test set.

[0147] Table 2 MAE and F β

[0148]

[0149] According to Table 2, it can be seen that the application achieves the optimal accuracy on this data set, and also achieves a greater improvement in the MAE and F β measurements of 0.038 and 0.935 respectively on this data set.

[0150] Table 3 gives the comparison of MAE and F β measurements of the application and other methods on the HKU-IS test set.

[0151] Table 3 MAE and F β

[0152]

[0153] According to Table 3, it can be seen that the application achieves the optimal accuracy on this data set, and also achieves a greater improvement in the MAE and F β measurements of 0.031 and 0.931 respectively on this data set.

[0154] Table 4 gives the comparison of MAE and F β measurements of the application and other methods on the DUT-OMRON test set.

[0155] Table 4 MAE and F β

[0156]

[0157] According to Table 4, it can be seen that the application achieves the optimal accuracy on this data set, and also achieves a smaller improvement in the MAE accuracy of about 0.05, while the F β measurements are flat with the optimal accuracy.

[0158] To make sure the edge-enhanced relational features play a role on this dataset, Figure 5 Visualization is performed, and direct comparison is made between the proposed method and other excellent methods. The results show that the proposed method can make the relational weights concentrate on the adjacent areas most relevant to the edges of the salient object, and the areas irrelevant to the edges obtain lower weights. This result shows that the salient edge features learned from the edge image generation contain boundary information, and the learning of the edge position relational weights complements these boundary information and salient object position information. This process is similar to the human visual recognition of the salient object in the picture, which shows that the proposed method can help the computer to further understand the picture.

[0159] According to the technical scheme, the embodiment of the application provides a salient object detection method based on edge feature guidance, which comprises the following steps: first, edge data of a salient object is extracted from a salient object ground truth map, and the edge data is taken as an additional preliminary training set; then, image features are extracted by using a basic network, and based on the image features, edge features of a salient object and multi-scale position features are obtained respectively; next, based on a multi-layer perception, the relationship between the edge features and the multi-scale position features is dynamically learned to obtain edge-enhanced fusion features; finally, the edge-enhanced fusion features are sent to a convolution network for refinement processing, and based on the refined edge-enhanced fusion features, a final segmentation result is obtained.

[0160] Firstly, the application designs a conversion rule for applying image edge features to salient object positions, each multi-scale fusion branch enhances the information of the edge features, and adaptively fuses with the salient object information, solving the problem of introducing noise when fusing edge features and salient object features with large span.

[0161] Secondly, the application uses salient object edges and position information to assist salient object detection, and considers the relationship between the two kinds of information and global information. The differences between multi-scale features and the misleading situation that may be caused in the up-sampling fusion process complement the shortcomings of the auxiliary information and the salient object information that may introduce noise in the fusion process, and realize more accurate salient object detection.

[0162] Thirdly, the application considers the universality and portability of the scheme, and more fully utilizes the edge information to give higher weights to the edge neighborhood pixels in the loss function, and further strengthens the role of edge features. Due to the introduction of multiple sets of edge and position features, the application adds a relational feature weighting step, and the user can obtain the best weight of the corresponding relational feature without manual allocation, so that the scheme is more universal.

[0163] Those skilled in the art can understand that the above-mentioned embodiments are specific examples for realizing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application. Any person skilled in the art can make respective changes and modifications without departing from the spirit and scope of the present application, and therefore the protection scope of the present application should be limited by the scope defined by the claims.

Claims

1. A method for salient object detection based on edge feature guidance, characterized in that, The method comprises the following steps: extracting edge data of the salient object from the salient object ground truth image, and taking the edge data as an additional preliminary training set; extracting image features by using a basic network, and obtaining edge features of the salient object and multi-scale position features based on the image features; dynamically learning the relationship between the edge features and the multi-scale position features based on a multi-layer perception to obtain edge-enhanced fusion features; sending the edge-enhanced fusion features into a convolution network for refinement processing, and obtaining a final segmentation result based on the refined edge-enhanced fusion features; dynamically learning the relationship between the edge features and the multi-scale position features based on a multi-layer perception to obtain edge-enhanced fusion features, comprising: performing spatial position coding on the edge features, and refining the edge features based on an attention mechanism; resampling salient object features of different scales until the resolution of the salient object features is consistent with that of the edge features; concatenating the salient object features of different scales and the edge features, and sending the concatenated features into a convolution layer for learning to obtain edge coding weights of the salient object and position coding weights of the salient object; assigning specific weights to the edge features and the position features based on the edge coding weights of the salient object and the position coding weights of the salient object, and fusing the edge features and the position features to obtain edge-enhanced fusion features; sending the edge-enhanced fusion features into a convolution network for refinement processing, and obtaining a final segmentation result based on the refined edge-enhanced fusion features, comprising: sending the edge-enhanced fusion features into a convolution block for decoding; regressing decoded features under multiple scales and corresponding salient objects to obtain regression results; converting the regression results into saliency scores to obtain robust saliency segmentation results; fusing multi-scale saliency features after edge enhancement to obtain a final fine saliency prediction map, and simultaneously calculating losses by using an edge-enhanced cross-entropy function and an intersection-over-union loss function for back propagation.

2. The edge feature guided salient object detection method of claim 1, wherein, extracting edge data of the salient object from the salient object ground truth image, and taking the edge data as an additional preliminary training set, comprising: calculating the amplitude and direction of the image pixel gradient based on the salient object ground truth image to obtain the calculated image pixel gradient; performing non-maximum suppression on the calculated image pixel gradient to obtain an edge binary image of the preliminary salient object; screening the edge binary image by using a double threshold to obtain an edge image close to the real edge of the salient object; extracting edge data of the salient object based on the edge image, and converting the edge data into a png format, and taking the png format edge data as an additional preliminary training set.

3. The edge feature based salient object detection method of claim 2, wherein, extracting image features by using a basic network, and obtaining edge features of the salient object and multi-scale position features based on the image features, comprising: regressing the features obtained by the convolution block of the basic network and the edge image to obtain the edge features of the salient object; The feature obtained by the convolution block of the base network is subjected to regression learning with the salient object ground truth, to obtain a position feature of the salient object; The regression process of the edge feature of the salient object includes: The pixel points corresponding to the predicted boundary map and the pixel points corresponding to the contour ground truth are subjected to binary cross-entropy judgment, to obtain a judgment result; The judgment result is subjected to cumulative averaging, to obtain a loss between the image edge prediction result and the edge contour ground truth; The expression of the regression process of the edge feature of the salient object is: wherein, is a binary cross-entropy function, representing the difference between two probability distributions; is an edge contour ground truth map; n represents the number of pixel points, and k represents the index of a pixel point, and respectively represent the pixel points of the predicted boundary map and the contour ground truth map.

4. The edge feature guided salient object detection method of claim 2, wherein, The process of regression learning of the feature obtained by the convolution block of the base network with the salient object ground truth further includes: Regression of a multi-scale rough salient object The regression process of the multi-scale rough salient object includes: The pixel points corresponding to the predicted multi-scale salient object result and the pixel points corresponding to the salient object ground truth are subjected to binary cross-entropy judgment, to obtain a judgment result; The judgment result is subjected to cumulative averaging, to obtain a loss between the rough salient object prediction result and the salient object ground truth; The expression of the regression process of the rough salient object is: wherein, is a binary cross-entropy function, representing the difference between two probability distributions; is a ground truth saliency map; n represents the number of pixel points; k represents the index of a pixel point; and respectively represent the pixel points of the predicted saliency result and the saliency ground truth map; i represents the scale corresponding to the saliency feature, i = 3, 4, 5, 6.

5. The edge feature guided salient object detection method of claim 1, wherein, The expression of the calculation process of the edge feature refinement is: wherein, The function is a ReLU activation function. represents pixel-level multiplication; E is an edge feature; W and b are parameters generated by a series of fully connected layers for the edge feature; wherein the dimension of the edge feature of the salient object is consistent with the dimension of the base feature of the salient object position.

6. The edge feature guided salient object detection method of claim 1, wherein, The expression of the calculation process of the edge encoding weight of the salient object and the position encoding weight of the salient object is: wherein, represents the salient object feature of the i-th layer; E represents the edge feature; the Concat function is a feature channel concatenation operation; upsample represents up-sampling, wherein the resolution of the up-sampled is the same as that of the edge feature E; W represents the parameters of the convolutional layer; represents a ReLU activation function; GAP represents global average pooling; is a sigmoid function used for normalization processing; respectively represent the edge encoding weight of the salient object and the position encoding weight of the salient object. The expression of the calculation process of the edge-enhanced fusion feature is: wherein, is the edge enhanced fusion feature obtained after the i-th layer salient object feature and edge feature are fused.

7. The edge feature guided salient object detection method of claim 1, wherein, The calculation process of the intersection-over-union loss function is: the intersection-over-union value is obtained by dividing the intersection of two segmentation maps by the union; The calculation formula of the intersection-over-union loss function is as follows: Wherein, IOU is the intersection over union; A and B represent two images which need to be calculated for the intersection over union; C is the number of image pixels; represents a pixel point of a salient object ground truth image; represents a final fine saliency prediction map obtained after fusing the multi-scale saliency features after edge enhancement.

8. The edge feature guided salient object detection method of claim 1, wherein, When the edge-enhanced cross-entropy function and the intersection-over-union loss function are used to calculate the loss, the calculation rule of the loss function is defined as follows: The weight of the field near the boundary of the salient object is enhanced by using the edge map, and a two-dimensional normal distribution is introduced to improve the generalization ability. The Gaussian kernel function G(x, y) is as follows: Set For the boundary picture, the boundary picture is subjected to Gaussian kernel convolution to obtain weights : wherein, represents the weight assigned to the border adjacent region; The final edge-enhanced loss function is as follows: wherein, denotes the final edge enhancement loss function, denotes the binary cross-entropy function; When the edge-enhanced cross-entropy function and the intersection-over-union loss function are used to calculate the loss, the calculation rule of the loss calculation is defined as follows: Set and correspond to the coarse saliency segmentation result of the former half and the fine saliency object segmentation result of the latter half, respectively, and the calculation process is as follows: wherein E, respectively represent the first half edge feature, the coarse-grained feature of the salient object, is the final multi-scale fused saliency object prediction result. The final loss function is the sum of the rough prediction loss and the fine prediction loss of the salient object detection, and the expression is as follows: wherein, represents the final loss function.

Images