A method for detecting a significant object, and a computer program product

Abstract

The present invention relates to a salient target detection method, comprising: S1, obtaining an image to be detected; S2, inputting the image to be detected into a trained salient detection model to obtain a salient map; the salient detection model comprises: an encoding component and a decoding component; the first n decoding units of the decoding component correspond to the encoding unit, and the encoding component outputs encoding features and a low-dimensional feature vector based on the image input into the salient detection model; the m-th decoding unit of the decoding component decodes the low-dimensional feature vector to obtain a decoding feature; the a-th decoding unit is connected to the a-1-th decoding unit through a Bayesian pooling module, and the decoding features obtained by the a-th decoding unit are subjected to Bayesian pooling to obtain a processed feature map; the x-th decoding unit receives the encoding feature and the feature map after the Bayesian pooling process, processes them using an edge tightening and foreground exposure attention mechanism, and outputs a decoding feature.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a salient object detection method and a computer program product. Background Art

[0002] Salient object detection (SOD) is to identify the most visually attractive objects from a given image by simulating the human visual system (HVS). The study of SOD not only contributes to our understanding of complex visual differences, but also promotes innovations in practical applications in various fields. The technology has been successfully applied to many computer vision tasks, ranging from action recognition in video surveillance to eye lesion detection in medical imaging. It also helps in hidden defect detection in manufacturing, thus emphasizing its wide applicability.

[0003] In the prior art, there is a SOD method that uses joint supervised prediction at different resolutions in multiple stages, that is, joint supervised prediction at different resolutions in multiple stages, adjusting the input image to different resolutions, obtaining images of multiple scales, and using a salient object detection model for prediction, and then fusing the prediction results of multiple scales to obtain the final salient object detection result. However, this method has slow inference speed and low efficiency.

[0004] In the prior art, there is also a SOD method that uses a transformer model based on U-net. The Transformer model learns the correlation between features and introduces nonlinear mapping to model complex feature relationships. It is then integrated into the U-Net architecture, and upsampling and skip connections are used in the U-Net decoder to restore the detailed information of the feature map, so that features at different levels are fused in the decoder, effectively utilizing the information of the underlying and high-level features. However, this type of method does not adequately simulate HVS, the edge positioning is not accurate enough, and the significant object detection effect is not good.

[0005] In the prior art, there is also a SOD method based on prior background. Prior background knowledge is obtained in advance through offline preprocessing and used to guide the target detection process in the algorithm. It helps understand the position, shape and size of the target in the image. However, this type of method relies too much on prior knowledge in a specific field, which may limit the generalization ability of the model in other fields. At the same time, if the prior knowledge used is outdated, the model will perform poorly. Summary of the invention

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a salient target detection method and a computer program product, which solve the technical problems existing in the prior art.

[0007] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0008] In a first aspect, an embodiment of the present invention provides a salient object detection method, comprising:

[0009] S1, obtaining the image to be detected;

[0010] S2, inputting the image to be detected into a trained salient detection model to obtain a salient map corresponding to the image to be detected;

[0011] The salient detection model comprises: an encoding component having a feature enhancement network and n sequentially connected encoding units, and a decoding component having m decoding units; the first n decoding units of the decoding component correspond to the n encoding units of the encoding component one by one, and the encoding component outputs encoding features through each encoding unit and outputs a low-dimensional feature vector through the feature enhancement network based on an image input to the salient detection model; wherein m=n+1;

[0012] The mth decoding unit of the decoding component is used to decode the low-dimensional feature vector to obtain a decoding feature;

[0013] The ath decoding unit and the a-1th decoding unit in the decoding component are connected through a Bayesian pooling module, and the Bayesian pooling module is used to perform Bayesian pooling processing on the decoding features obtained by the ath decoding unit, obtain the feature map after Bayesian pooling processing and input it into the a-1th decoding unit; 2≤a≤m;

[0014] The x-th decoding unit in the decoding component is used to receive the encoding features output by the corresponding encoding unit, the feature map after Bayesian pooling obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit, and process them using the edge tightening and foreground exposure attention mechanism to output the decoding features; when x is 1, the decoding features output by the first decoding unit are used as the salient map corresponding to the image input to the salient detection model; 1≤x≤n.

[0015] Preferably, before S1, it also includes:

[0016] S0. Using training data, training the salient detection model to obtain a trained salient detection model;

[0017] The training data includes multiple sets of image data, each set of image data includes: an original image with a target and a true value image of the original image;

[0018] The trained salient detection model is obtained by adjusting weight parameters in the salient detection model based on the first loss, the second loss, and the third loss obtained in the process of training the salient detection model;

[0019] The first loss is obtained based on the true value image of the original image and the saliency map output by the saliency detection model;

[0020] The second loss is obtained based on the ratio of the intersection and union of the bounding box of the object in the saliency map output by the saliency detection model and the bounding box of the object in the true value image of the original image;

[0021] The third loss is obtained based on the true value image of the original image and the decoded features of each decoding unit.

[0022] Preferably,

[0023] The low-dimensional feature vector is obtained by a feature enhancement network in the encoding component, and the feature enhancement network obtains the low-dimensional feature vector using formula (1) based on the encoding features output by each encoding unit in the encoding component;

[0024] The formula (1) is:

[0025] f=Rec(CBR(f l )+f l+1 ), l = 1, ..., n;

[0026] Among them, f l represents the output characteristics of the lth coding unit;

[0027] CBR(f l ) indicates f l Output after entering the CBR module;

[0028] Rec() represents a recursive operation;

[0029] f is a low-dimensional feature vector.

[0030] Preferably,

[0031] The Bayesian pooling module between the a-th decoding unit and the a-1-th decoding unit in the decoding component uses formula (2) to perform Bayesian pooling processing on the decoding features obtained by the a-th decoding unit to obtain a feature map after Bayesian pooling processing;

[0032] Wherein, the formula (2) is:

[0033] R = BayesPool(F);

[0034] F is the decoding feature obtained by the decoding unit;

[0035] R is the feature map of the decoded features obtained by the decoding unit after Bayesian pooling processing;

[0036] BayesPool is the Bayesian pooling process;

[0037] The Bayesian pooling process is to convert each pixel in the decoding feature obtained by the decoding unit through formula (3) to obtain the corresponding probability value;

[0038]

[0039] in, is the pixel value of the pixel with pixel number i in the decoding feature obtained by the a-th decoding unit;

[0040] θ j is the probability value of the pixel with pixel number j in the feature map after Bayesian pooling processing; where pixel number j corresponds to pixel number i;

[0041] D B is the coverage domain of the Bayesian pooling module.

[0042] Preferably,

[0043] The mth decoding unit of the decoding component is the first convolution module including: a transposed CBR module, a convolution layer Conv, and a sigmoid function;

[0044] The xth decoding unit of the decoding component includes: a CBPter conversion module and a setting convolution module;

[0045] The setting convolution module in the nth decoding unit is a second convolution module including: a transposed CBR module, a transposed convolution layer transposed Conv, and a sigmoid function;

[0046] The set convolution module in the y-th decoding unit is the third convolution module including: CBR module, transposed CBR module, transposed convolution layer transposed Conv, sigmoid function; 1≤y≤n-1;

[0047] Each CBPter conversion module includes a multi-layer perceptron MLP.

[0048] Preferably, in S0, in the process of training the salient detection model using training data, the x-th decoding unit in the decoding component receives the encoding features output by the encoding unit corresponding thereto, the feature map after Bayesian pooling processing obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit, processes them using edge tightening and foreground exposure attention mechanisms, and outputs decoding features, specifically including:

[0049] S01, the x-th decoding unit in the decoding component obtains the first dot product result based on formula (4); the formula (4) is: first dot product result = A*C; A is the feature map after Bayesian pooling processing obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit; C is the attention map of the foreground exposure attention mechanism; * represents the dot product;

[0050] S02, the x-th decoding unit in the decoding component obtains a second dot product result based on formula (5); the formula (5) is: second dot product result = A*B; B is the encoding feature output by the encoding unit corresponding to the x-th decoding unit;

[0051] S03, adding the first dot product result to the second dot product result to obtain a first intermediate result;

[0052] S04, adding the first intermediate result to the second intermediate result to obtain a first output result; the second intermediate result is obtained after the first intermediate result is processed by the LN unit and the multilayer perceptron MLP in the x-th decoding unit in sequence;

[0053] S05. The x-th decoding unit in the decoding component obtains a fourth dot product result based on formula (6); the formula (6) is: fourth dot product result = first output result * C;

[0054] S06, adding the first dot product result to the fourth dot product result to obtain a third intermediate result;

[0055] S07. Add the third intermediate result to the fourth intermediate result to obtain the decoding feature of the x-th decoding unit; the fourth intermediate result is obtained after the third intermediate result is processed by the LN unit and the multilayer perceptron MLP in the x-th decoding unit in sequence.

[0056] Preferably,

[0057] The attention map C of the foreground exposure attention mechanism is obtained by formula (7);

[0058] The formula (7) is:

[0059]

[0060] Wherein, Q, K, and V are matrices obtained by the dot product of the coding feature B output by the coding unit corresponding to the x-th decoding unit and the feature map A after Bayesian pooling obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit; R is the matrix obtained by linear transformation of the feature map after Bayesian pooling; d k is the dimension of the K matrix; Att(Q, K, V, R) is the attention map C of the foreground exposure attention mechanism.

[0061] Preferably,

[0062] In S0, each set of image data in the training data is used to train the salient detection model to obtain the trained salient detection model; the training data includes Z sets of image data;

[0063] During the training process of each set of image data, the weight parameters of the salient detection model are adjusted based on the total loss;

[0064] The total loss is obtained based on the first loss, the second loss and the third loss using formula (8);

[0065] The formula (8) is:

[0066] L total =L BCE +L IoU +λL esCE ;

[0067] Among them, L BCE For the first loss;

[0068] L IoU For the second loss;

[0069] L esCE For the third loss;

[0070] λ is a preset coefficient;

[0071] L total is the total loss.

[0072] Preferably,

[0073] Among them, the third loss L esCE Based on the true value image of the original image and the decoding features of each decoding unit, it is obtained by formula (9);

[0074] The formula (9) is:

[0075]

[0076] In the formula, S a is the decoding feature obtained by the a-th decoding unit;

[0077] AvgPool() represents the average pooling operation;

[0078] is the decoding feature obtained by the a-th decoding unit and the final truth graph G corresponding to the a-th decoding unit a Binary cross entropy loss between

[0079] Among them, G a =AvePool(G a )×(1-G a )+G a ;

[0080] AvePool() is the average pooling operation;

[0081] G a =maxPool(G a-1 );

[0082] maxPool() is the maximum pooling operation;

[0083] G a is the initial truth map corresponding to the a-th decoding unit;

[0084] G1 is the true value image of the original image.

[0085] On the other hand, this embodiment further provides a computer program product, including a computer program, which, when executed by a processor, implements the salient object detection method of the first aspect.

[0086] The beneficial effects of the present invention are as follows: a salient target detection method of the present invention uses a salient detection model to perform salient target detection on an image to be detected to obtain a corresponding salient map, and the salient detection model in the present invention uses a recursive U-net structure, obtains coding features and low-dimensional feature vectors in the image to be detected through an encoding component, and then uses a decoding component to process, a decoding unit in the decoding component obtains decoding features, and the a-th decoding unit and the a-1-th decoding unit in the decoding component are connected through a Bayesian pooling module, and the Bayesian pooling module is used to perform Bayesian pooling processing on the decoding features obtained by the a-th decoding unit to obtain a feature map after Bayesian pooling processing and input it to the a-1-th decoding unit. Compared with the prior art, the a-1-th decoding unit can seamlessly integrate the obtained feature map after Bayesian pooling processing and the corresponding coding features, and then perform edge tightening and foreground exposure attention mechanism processing, and output decoding features. The salient map finally obtained through the processing process of the decoding unit in the decoding component in the salient detection model of the present invention has high accuracy in salient target detection, is conducive to protecting edge details and refining the precise edge of the target, and has high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 is a flow chart of a salient target detection method of the present invention;

[0088] Figure 2 Schematic diagram of the structure of the salient object detection model in the present invention;

[0089] Figure 3 is a flow chart of a salient object detection method in an embodiment of the present invention;

[0090] Figure 4 A framework diagram of Bayesian pooling in an embodiment of the present invention;

[0091] Figure 5 It is a structural diagram of the CBPTe conversion module and the foreground exposure attention mechanism FEA in an embodiment of the present invention;

[0092] Figure 6 is the initial truth graph G corresponding to the a-th decoding unit in the embodiment of the present invention a The outline diagram has been preprocessed by edge extension;

[0093] Figure 7 This is a qualitative comparison effect diagram of the method of this embodiment and other existing methods on 5 data sets in this embodiment;

[0094] Figure 8 It is a heat map comparison diagram of the foreground exposure attention mechanism FEA embedded in the Swin conversion module in this embodiment and the traditional self-attention mechanism;

[0095] Fig. 9 This is a visualization diagram of the feature map after Bayesian pooling processing, the heat map generated by the foreground exposure attention mechanism FEA, and the decoding features generated by the CBPTe conversion module at each stage in this embodiment. DETAILED DESCRIPTION

[0096] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0097] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0098] Embodiment 1

[0099] See also Figure 1 , this embodiment provides a salient target detection method, including:

[0100] S1, obtaining the image to be detected;

[0101] S2, inputting the image to be detected into a trained salient detection model to obtain a salient map corresponding to the image to be detected;

[0102] See also Figure 2 , the salient detection model comprises: an encoding component having a feature enhancement network and n sequentially connected encoding units, and a decoding component having m decoding units; the first n decoding units of the decoding component correspond to the n encoding units of the encoding component one by one, and the encoding component outputs encoding features through each encoding unit and outputs a low-dimensional feature vector through the feature enhancement network based on the image input to the salient detection model; wherein m=n+1;

[0103] The mth decoding unit of the decoding component is used to decode the low-dimensional feature vector to obtain a decoding feature;

[0104] The ath decoding unit and the a-1th decoding unit in the decoding component are connected through a Bayesian pooling module, and the Bayesian pooling module is used to perform Bayesian pooling processing on the decoding features obtained by the ath decoding unit, obtain the feature map after Bayesian pooling processing and input it into the a-1th decoding unit; 2≤a≤m;

[0105] The x-th decoding unit in the decoding component is used to receive the encoding features output by the corresponding encoding unit, the feature map after Bayesian pooling obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit, and process them using the edge tightening and foreground exposure attention mechanism to output the decoding features; when x is 1, the decoding features output by the first decoding unit are used as the salient map corresponding to the image input to the salient detection model; 1≤x≤n.

[0106] In the present embodiment, a salient target detection method is used to perform salient target detection on an image to be detected by using a salient detection model to obtain a corresponding salient map. The salient detection model in the present invention uses a recursive U-net structure. The encoding features and low-dimensional feature vectors in the image to be detected are obtained by the encoding component, and then the decoding component is used to process. The decoding unit in the decoding component obtains the decoding features. The a-th decoding unit and the a-1-th decoding unit in the decoding component are connected through a Bayesian pooling module. The Bayesian pooling module is used to perform Bayesian pooling on the decoding features obtained by the a-th decoding unit to obtain the feature map after the Bayesian pooling process and input it to the a-1-th decoding unit. Compared with the prior art, the a-1-th decoding unit can seamlessly integrate the obtained feature map after the Bayesian pooling process and the corresponding encoding features, and then perform edge tightening and foreground exposure attention mechanism for processing, and output the decoding features. The salient map finally obtained by the processing process of the decoding unit in the decoding component in the salient detection model of the present invention has high accuracy in salient target detection, is conducive to protecting edge details and refining the precise edge of the target, and has high efficiency.

[0107] Embodiment 2

[0108] See also Figure 3 , this embodiment provides a salient target detection method, including:

[0109] S0. Using training data, training the salient detection model to obtain a trained salient detection model;

[0110] The training data includes multiple sets of image data, each set of image data includes: an original image with a target and a true value image of the original image;

[0111] S1, obtaining the image to be detected;

[0112] S2, inputting the image to be detected into a trained salient detection model to obtain a salient map corresponding to the image to be detected;

[0113] See also Figure 2 , the salient detection model includes: an encoding component having a feature enhancement network and n sequentially connected encoding units, and a decoding component having m decoding units; the first n decoding units of the decoding component correspond to the n encoding units of the encoding component one by one, and the encoding component outputs encoding features through each encoding unit based on the image input to the salient detection model and outputs a low-dimensional feature vector through the feature enhancement network; wherein, m=n+1; in the actual application of this embodiment, n=4, m=5.

[0114] The encoding unit in this embodiment is a Swin conversion module.

[0115] The mth decoding unit of the decoding component is used to decode the low-dimensional feature vector to obtain a decoding feature;

[0116] The ath decoding unit and the a-1th decoding unit in the decoding component are connected through a Bayesian pooling module, and the Bayesian pooling module is used to perform Bayesian pooling processing on the decoding features obtained by the ath decoding unit, obtain the feature map after Bayesian pooling processing and input it into the a-1th decoding unit; 2≤a≤m;

[0117] See also Figure 2 , the x-th decoding unit in the decoding component is used to receive the encoding features output by the corresponding encoding unit, the feature map after Bayesian pooling obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit, and use the edge tightening and foreground exposure attention mechanism to process and output the decoding features; when x is 1, the decoding features output by the first decoding unit are used as the salient map corresponding to the image input to the salient detection model; 1≤x≤n.

[0118] The trained salient detection model is obtained by adjusting weight parameters in the salient detection model based on the first loss, the second loss, and the third loss obtained in the process of training the salient detection model;

[0119] The first loss is obtained based on the true value image of the original image and the saliency map output by the saliency detection model;

[0120] The second loss is obtained based on the ratio of the intersection and union of the bounding box of the object in the saliency map output by the saliency detection model and the bounding box of the object in the true value image of the original image;

[0121] The third loss is obtained based on the true value image of the original image and the decoded features of each decoding unit.

[0122] In the practical application of this embodiment, the low-dimensional feature vector is obtained by a feature enhancement network in the encoding component, and the feature enhancement network obtains the low-dimensional feature vector using formula (1) based on the encoding features output by each encoding unit in the encoding component;

[0123] The formula (1) is:

[0124] f=Rec(CBR(f l )+f l+1 ), l = 1, ..., n;

[0125] Among them, f l represents the output characteristics of the lth coding unit;

[0126] CBR(f l ) indicates f l Output after entering the CBR module;

[0127] Rec() represents a recursive operation;

[0128] f is a low-dimensional feature vector.

[0129] In the practical application of this embodiment, the Bayesian pooling module between the a-th decoding unit and the a-1-th decoding unit in the decoding component uses formula (2) to perform Bayesian pooling processing on the decoding features obtained by the a-th decoding unit to obtain a feature map after Bayesian pooling processing;

[0130] Wherein, the formula (2) is:

[0131] R = BayesPool(F);

[0132] F is the decoding feature obtained by the decoding unit;

[0133] R is the feature map of the decoded features obtained by the decoding unit after Bayesian pooling processing;

[0134] BayesPool is the Bayesian pooling process;

[0135] In the present embodiment, a local safety area (i.e., supervision range) is obtained from the decoding features obtained by each decoding unit through Bayesian pooling. The pixel values ​​of the feature map after Bayesian pooling are redefined as the probability values ​​belonging to the target object. Therefore, the feature map after Bayesian pooling of the output of the Bayesian pooling module is also referred to as the foreground probability map. Among them, the pixel value that absolutely belongs to the target in the feature map after Bayesian pooling is 1, the pixel value that belongs to the background is 0, and the pixel value range that belongs to the fuzzy safety area (compact background) is from 0 to 1. Among them, the fuzzy safety area is also called the compact background to prevent edge damage. At the same time, since the contribution of the pixel in the compact background is uncertain, it needs to be further evaluated in the next decoding unit. If the pixel in the feature map after Bayesian pooling is not completely 0 or 1, it may contain an edge because only the pixels close to the edge show fuzzy attribution.

[0136] See also Figure 4 In this embodiment, the Bayesian pooling module performs 2×2 pooling with a step size of 2. Therefore, Bayesian pooling visually expands the decoded features (also called coarse saliency maps), that is, restricts the target object to a wider area in the prediction map. This method not only prevents damage to the edges of objects, but also enhances features within compact backgrounds.

[0137] The Bayesian pooling process is to convert each pixel in the decoding feature obtained by the decoding unit through formula (3) to obtain the corresponding probability value;

[0138]

[0139] in, is the pixel value of the pixel with pixel number i in the decoding feature obtained by the a-th decoding unit;

[0140] θ j is the probability value of the pixel with pixel number j in the feature map after Bayesian pooling processing; where pixel number j corresponds to pixel number i;

[0141] D B is the coverage domain of the Bayesian pooling module.

[0142] Specifically, the mth decoding unit of the decoding component is a first convolution module including: a transposed CBR module, a convolution layer Conv, and a sigmoid function; the mth decoding unit (that is, the 5th decoding unit) in this embodiment only includes one first convolution module.

[0143] The xth decoding unit of the decoding component includes: a CBPter conversion module and a setting convolution module;

[0144] The setting convolution module in the nth decoding unit (that is, the fourth decoding unit) is a second convolution module including: a transposed CBR module, a transposed convolution layer transposed Conv, and a sigmoid function;

[0145] The set convolution module in the y-th decoding unit is the third convolution module including: CBR module, transposed CBR module, transposed convolution layer transposed Conv, sigmoid function; 1≤y≤n-1;

[0146] Each CBPter conversion module includes a multi-layer perceptron MLP.

[0147] In this embodiment, CBR (Convolution-BatchNorm-ReLU) is a common structure in the convolution module, which consists of three steps: convolution, batch normalization, and activation function (ReLU). It is mainly used in the forward propagation process to extract feature information from the input data. It can effectively extract the spatial information of the input feature map and introduce nonlinear transformation to enhance the expression ability of the model.

[0148] Transposed CBR is also called deconvolution module or transposed CBR module, which is mainly used for deconvolution operation, that is, upsampling the feature map to restore it to the original input size. Similar to the CBR structure, it consists of three steps: transpose convolution, batch normalization, and activation function (ReLU). It can effectively upsample low-resolution feature maps to improve the spatial resolution of the model, and can introduce nonlinear transformations to enhance the expressiveness of the model.

[0149] In actual application, in S0, in the process of training the salient detection model using training data, the x-th decoding unit in the decoding component receives the encoding features output by the corresponding encoding unit, the feature map after Bayesian pooling processing obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit, and processes it using the edge tightening and foreground exposure attention mechanism to output the decoding features, which specifically includes:

[0150] S01, the x-th decoding unit in the decoding component obtains the first dot product result based on formula (4); the formula (4) is: first dot product result = A*C; A is the feature map after Bayesian pooling processing obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit; C is the attention map of the foreground exposure attention mechanism; * represents the dot product;

[0151] S02, the x-th decoding unit in the decoding component obtains a second dot product result based on formula (5); the formula (5) is: second dot product result = A*B; B is the encoding feature output by the encoding unit corresponding to the x-th decoding unit;

[0152] S03, adding the first dot product result to the second dot product result to obtain a first intermediate result;

[0153] S04, adding the first intermediate result to the second intermediate result to obtain a first output result; the second intermediate result is obtained after the first intermediate result is processed by the LN unit and the multilayer perceptron MLP in the x-th decoding unit in sequence;

[0154] S05. The x-th decoding unit in the decoding component obtains a fourth dot product result based on formula (6); the formula (6) is: fourth dot product result = first output result * C;

[0155] S06, adding the first dot product result to the fourth dot product result to obtain a third intermediate result;

[0156] S07. Add the third intermediate result to the fourth intermediate result to obtain the decoding feature of the x-th decoding unit; the fourth intermediate result is obtained after the third intermediate result is processed by the LN unit and the multilayer perceptron MLP in the x-th decoding unit in sequence.

[0157] For an example, see Figure 5 The structure diagram of the CBPTe conversion module and foreground exposure attention mechanism in the x-th decoding unit in the decoding component, wherein the CBPTe conversion module includes a multi-layer perceptron MLP. In the decoding unit in this embodiment, before entering the CBPTe module, the encoded feature map needs to be dot-producted with the feature map (that is, the foreground probability map) after Bayesian pooling processing obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit. In addition, the feature map (foreground probability map) after Bayesian pooling processing is dot-producted with the attention map obtained by the foreground exposure attention mechanism FEA. Therefore, it can be seen that the decoding unit in this embodiment has improved the input and jump connections, and all improvements are intended to limit the operating range of the CBPTe conversion module in this embodiment to a compact background.

[0158] like Figure 5 The structure diagram of the CBPter conversion module and the foreground exposure attention mechanism FEA in this embodiment is shown. Figure 5 In this embodiment, the structure on the left side of the Swin conversion module (that is, the corresponding encoding unit) is a block 1 structure. Figure 5 The structure on the right side of the foreground probability map is the block 2 structure. In this embodiment, the block 1 structure and the block 2 structure together constitute the CBPter conversion module in this embodiment.

[0159] for Figure 5 In the middle block 1 structure, the first dot product result is obtained by formula (4); the formula (4) is: the first dot product result = A*C; A is the feature map after Bayesian pooling processing obtained by the Bayesian pooling module between the x+1th decoding unit and the xth decoding unit; C is the attention map of the foreground exposure attention mechanism; * represents the dot product; the second dot product result is obtained based on formula (5); the formula (5) is: the second dot product result = A*B; B is the encoding feature output by the encoding unit corresponding to the xth decoding unit; then, the first dot product result is added to the second dot product result to obtain the first intermediate result; the first intermediate result is added to the second intermediate result to obtain the first output result; the second intermediate result is obtained after the first intermediate result is processed by the LN unit and the multilayer perceptron MLP in the xth decoding unit in sequence.

[0160] for Figure 5 In the middle block 2 structure, the fourth dot product result is obtained based on formula (6); the formula (6) is: the fourth dot product result = the first output result * C; then, the first dot product result is added to the fourth dot product result to obtain the third intermediate result; finally, the third intermediate result is added to the fourth intermediate result to obtain the decoding feature of the x-th decoding unit; the fourth intermediate result is obtained after the third intermediate result is processed by the LN unit and the multilayer perceptron MLP in the x-th decoding unit in sequence.

[0161] in, Figure 5 SW-FEA in [5] represents the foreground exposure attention mechanism FEA with shifted window partition to capture more information on the entire feature map.

[0162] The x-th decoding unit seamlessly integrates the encoded features output by the corresponding encoding unit and the feature map processed by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit, and feeds them into the CBPter block, so that the foreground probability of the previous stage (that is, the feature map processed by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit) distribution has a positive impact on the x-th decoding unit. In addition, the embedded foreground exposure attention mechanism FEA guides the salient detection model to focus on the content in the compact background. It is conducive to accurate edge positioning.

[0163] Specifically, the attention map C of the foreground exposure attention mechanism is obtained by formula (7);

[0164] The formula (7) is:

[0165]

[0166] Wherein, Q, K, and V are matrices obtained by the dot product of the coding feature B output by the coding unit corresponding to the x-th decoding unit and the feature map A after Bayesian pooling obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit; R is the matrix obtained by linear transformation of the feature map after Bayesian pooling; d k is the dimension of the K matrix; Att(Q, K, V, R) is the attention map C of the foreground exposure attention mechanism.

[0167] In a specific implementation, in S0, each set of image data in the training data is used to train the salient detection model to obtain the trained salient detection model; the training data includes Z sets of image data;

[0168] It should be noted that the epoch of the saliency detection model in this embodiment is 110.

[0169] During the training process of each set of image data, the weight parameter of the salient detection model is adjusted based on the total loss; in this embodiment, the weight parameter of the salient detection model is a pre-set coefficient λ.

[0170] The total loss is obtained based on the first loss, the second loss and the third loss using formula (8);

[0171] The formula (8) is:

[0172] L total =L BCE +L IoU +λL esCE ;

[0173] Among them, L BCE is the first loss; L IoU is the second loss; L esCE is the third loss; λ is a preset coefficient; L total is the total loss.

[0174] Specifically, the first loss in this embodiment is a binary cross entropy (BCE) loss; used for final saliency map supervision; the second loss in this embodiment is an intersection over union (IoU) loss, which is obtained based on the ratio of the intersection and union of the bounding box of the target in the saliency map output by the saliency detection model and the bounding box of the target in the true value image of the original image, and the third loss in this embodiment is an edge shrinkage cross entropy loss. The intersection over union (IoU) loss in this embodiment is a common loss function.

[0175] Among them, the third loss L esCE Based on the true value image of the original image and the decoding features of each decoding unit, it is obtained by formula (9);

[0176] The formula (9) is:

[0177]

[0178] In the formula, S a is the decoding feature obtained by the a-th decoding unit;

[0179] AvgPool() represents the average pooling operation;

[0180] is the decoding feature obtained by the a-th decoding unit and the final truth graph G corresponding to the a-th decoding unit a Binary cross entropy loss between

[0181] In this embodiment, when calculating When the a-th decoding unit corresponds to the initial truth graph G a After edge extension preprocessing, the final truth graph G is obtained a ·. See Figure 6 , in this embodiment, is the initial truth graph G corresponding to the a-th decoding unit a Summary image after edge extension preprocessing.

[0182] Among them, G a =AvePool(G a )×(1-G a )+G a ;

[0183] AvePool() is the average pooling operation;

[0184] Ga =maxPool(G a-1 );

[0185] maxPool() is the maximum pooling operation;

[0186] G a is the initial truth map corresponding to the a-th decoding unit;

[0187] It should be noted that when a=1, G1 is the true value image of the original image. a And the final truth graph G a ·, it can be seen that the pixels in the edge area are linearly mapped to the range of 0 to 1. The extended edge pixel distribution more realistically reflects the fuzzy edge state of the rough saliency map. Therefore, the proposed third loss L esCE This helps to shrink more smoothly towards the boundaries of real objects.

[0188] On the other hand, this embodiment further provides a computer program product, including a computer program, which implements the salient object detection method in the first embodiment when executed by a processor.

[0189] Experimental example:

[0190] Figure 7 The qualitative comparison of the salient object detection method of this embodiment with 9 existing methods on 5 datasets is shown. The experiment carefully selected 6 images from the dataset, which present scenes with complex edges. After testing various models on these challenging images, the method of this embodiment consistently achieved excellent performance, surpassing other competing models. Figure 7 From the observation of the predicted edge in the embodiment, it is found that the method in the embodiment can protect the edge more comprehensively. Therefore, these comparison results also highlight the significant effect of the salient detection model in this embodiment in preventing edge damage.

[0191] Figure 8 A heat map comparison of the foreground exposure attention mechanism FEA embedded in the Swin transformation module and the traditional self-attention mechanism is shown. Qualitative analysis is performed through visualization to evaluate the effectiveness of the foreground exposure attention mechanism FEA. It is obvious that the foreground exposure attention mechanism FEA can locate high-attention areas more accurately than the traditional self-attention mechanism, especially the edges of objects. This proves its effectiveness in revealing the foreground, especially in emphasizing the edges of objects. It shows the superiority of the salient object detection method of this embodiment in refining the precise edges of the target.

[0192] like Fig. 9As shown, in this embodiment, the feature map (foreground probability map) after Bayesian pooling, the heat map generated by the foreground exposure attention mechanism FEA, and the decoded features (rough saliency map) generated by the CBPter conversion module are combined at each stage (see Figure 2 ) was visualized. Fig. 9 It can be seen that from stage 1 to stage 4, the edge of the object gradually shrinks, causing the exposure area to gradually concentrate around the actual edge.

[0193] In fact, the above phenomena show an inherent correlation. "Expansion" defines the operating range of "exposure" and "contraction". It is the basic premise of "exposure" and "contraction". "Exposure" and "contraction" promote each other. Specifically, the former provides the latter with precise positioning of high-attention areas, while the latter in turn improves the positioning of high-attention areas and makes it more precise. Figure 8 This can also be seen in the gradually expanding high attention area in the figure.

[0194] In the present embodiment, a salient target detection method is used to perform salient target detection on an image to be detected by using a salient detection model to obtain a corresponding salient map. The salient detection model in the present invention uses a recursive U-net structure. The encoding features and low-dimensional feature vectors in the image to be detected are obtained by the encoding component, and then the decoding component is used to process. The decoding unit in the decoding component obtains the decoding features. The a-th decoding unit and the a-1-th decoding unit in the decoding component are connected through a Bayesian pooling module. The Bayesian pooling module is used to perform Bayesian pooling on the decoding features obtained by the a-th decoding unit to obtain the feature map after the Bayesian pooling process and input it to the a-1-th decoding unit. Compared with the prior art, the a-1-th decoding unit can seamlessly integrate the obtained feature map after the Bayesian pooling process and the corresponding encoding features, and then perform edge tightening and foreground exposure attention mechanism for processing, and output the decoding features. The salient map finally obtained by the processing process of the decoding unit in the decoding component in the salient detection model of the present invention has high accuracy in salient target detection, is conducive to protecting edge details and refining the precise edge of the target, and has high efficiency.

[0195] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0196] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0197] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.

[0198] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0199] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A salient object detection method, characterized in that: include: S1, obtaining the image to be detected; S2, inputting the image to be detected into a trained salient detection model to obtain a salient map corresponding to the image to be detected; The salient detection model comprises: an encoding component having a feature enhancement network and n sequentially connected encoding units, and a decoding component having m decoding units; the first n decoding units of the decoding component correspond to the n encoding units of the encoding component one by one, and the encoding component outputs encoding features through each encoding unit and outputs a low-dimensional feature vector through the feature enhancement network based on an image input to the salient detection model; wherein m=n+1; The mth decoding unit of the decoding component is used to decode the low-dimensional feature vector to obtain a decoding feature; The ath decoding unit and the a-1th decoding unit in the decoding component are connected through a Bayesian pooling module, and the Bayesian pooling module is used to perform Bayesian pooling processing on the decoding features obtained by the ath decoding unit, obtain the feature map after Bayesian pooling processing and input it into the a-1th decoding unit; 2≤a≤m; The x-th decoding unit in the decoding component is used to receive the encoding features output by the corresponding encoding unit and the feature map after Bayesian pooling obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit, and use the edge tightening and foreground exposure attention mechanism to process and output the decoding features; when x is 1, the decoding features output by the first decoding unit are used as the salient map corresponding to the image input to the salient detection model; 1≤x≤n; The Bayesian pooling module between the a-th decoding unit and the a-1-th decoding unit in the decoding component uses formula (2) to perform Bayesian pooling processing on the decoding features obtained by the a-th decoding unit to obtain a feature map after Bayesian pooling processing; Wherein, the formula (2) is: R = BayesPool(F); F is the decoding feature obtained by the decoding unit; R is the feature map of the decoded features obtained by the decoding unit after Bayesian pooling processing; BayesPool is the Bayesian pooling process; The Bayesian pooling process is to convert each pixel in the decoding feature obtained by the decoding unit through formula (3) to obtain the corresponding probability value; in, is the pixel value of the pixel with pixel number i in the decoding feature obtained by the a-th decoding unit; θ j is the probability value of the pixel with pixel number j in the feature map after Bayesian pooling processing; where pixel number j corresponds to pixel number i; D B is the coverage domain of the Bayesian pooling module.

2. The salient object detection method according to claim 1, characterized in that: Prior to S1, it also included: S0. Using training data, training the salient detection model to obtain a trained salient detection model; The training data includes multiple sets of image data, each set of image data includes: an original image with a target and a true value image of the original image; The trained salient detection model is obtained by adjusting weight parameters in the salient detection model based on the first loss, the second loss, and the third loss obtained in the process of training the salient detection model; The first loss is obtained based on the true value image of the original image and the saliency map output by the saliency detection model; The second loss is obtained based on the ratio of the intersection and union of the bounding box of the object in the saliency map output by the saliency detection model and the bounding box of the object in the true value image of the original image; The third loss is obtained based on the true value image of the original image and the decoded features of each decoding unit.

3. The salient object detection method according to claim 2, characterized in that: The low-dimensional feature vector is obtained by a feature enhancement network in the encoding component, and the feature enhancement network obtains the low-dimensional feature vector using formula (1) based on the encoding features output by each encoding unit in the encoding component; The formula (1) is: f=Rec(CBR(f l )+f l+1 );l=1,...,n; Among them, f l represents the output characteristics of the lth coding unit; CBR(f l ) indicates f l Output after entering the CBR module; Rec() represents a recursive operation; f is a low-dimensional feature vector; The CBR module consists of three steps: convolution, batch normalization and activation function.

4. The salient object detection method according to claim 3, characterized in that: The mth decoding unit of the decoding component is the first convolution module including: a transposed CBR module, a convolution layer Conv, and a sigmoid function; The xth decoding unit of the decoding component includes: a CBPter conversion module and a setting convolution module; The setting convolution module in the nth decoding unit is a second convolution module including: a transposed CBR module, a transposed convolution layer transposed Conv, and a sigmoid function; The set convolution module in the y-th decoding unit is the third convolution module including: CBR module, transposed CBR module, transposed convolution layer transposed Conv, sigmoid function; 1≤y≤n-1; Among them, each CBPter conversion module includes a multi-layer perceptron MLP; The transposed CBR module consists of three steps: transposed convolution, batch normalization and activation function.

5. The salient object detection method according to claim 4, characterized in that: In S0, in the process of training the salient detection model using training data, the x-th decoding unit in the decoding component receives the encoding features output by the corresponding encoding unit and the feature map processed by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit, processes them using the edge tightening and foreground exposure attention mechanism, and outputs the decoding features, which specifically includes: S01, the x-th decoding unit in the decoding component obtains the first dot product result based on formula (4); the formula (4) is: first dot product result = A*C; A is the feature map after Bayesian pooling processing obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit; C is the attention map of the foreground exposure attention mechanism; * represents the dot product; S02, the x-th decoding unit in the decoding component obtains a second dot product result based on formula (5); the formula (5) is: second dot product result = A*B; B is the encoding feature output by the encoding unit corresponding to the x-th decoding unit; S03, adding the first dot product result to the second dot product result to obtain a first intermediate result; S04, adding the first intermediate result to the second intermediate result to obtain a first output result; the second intermediate result is obtained after the first intermediate result is processed by the LN unit and the multilayer perceptron MLP in the x-th decoding unit in sequence; S05. The x-th decoding unit in the decoding component obtains a fourth dot product result based on formula (6); the formula (6) is: fourth dot product result = first output result * C; S06, adding the first dot product result to the fourth dot product result to obtain a third intermediate result; S07. Add the third intermediate result to the fourth intermediate result to obtain the decoding feature of the x-th decoding unit; the fourth intermediate result is obtained after the third intermediate result is processed by the LN unit and the multilayer perceptron MLP in the x-th decoding unit in sequence.

6. The salient object detection method according to claim 5, characterized in that: The attention map C of the foreground exposure attention mechanism is obtained by formula (7); The formula (7) is: Wherein, Q, K, and V are matrices obtained by the dot product of the coding feature B output by the coding unit corresponding to the x-th decoding unit and the feature map A after Bayesian pooling obtained by the Bayesian pooling module between the x+1-th decoding unit and the x-th decoding unit; R is the matrix obtained by linear transformation of the feature map after Bayesian pooling; d k is the dimension of the K matrix; Att(Q, K, V, R) is the attention map C of the foreground exposure attention mechanism.

7. The salient object detection method according to claim 6, characterized in that: In S0, each set of image data in the training data is used to train the salient detection model to obtain the trained salient detection model; the training data includes Z sets of image data; During the training process of each set of image data, the weight parameters of the salient detection model are adjusted based on the total loss; The total loss is obtained based on the first loss, the second loss and the third loss using formula (8); The formula (8) is: THE total =L BCE +L IoU +λL esCE ; Among them, L BCE For the first loss; L IoU For the second loss; L esCE For the third loss; λ is a preset coefficient; L total is the total loss.

8. The salient object detection method according to claim 7, characterized in that: Among them, the third loss L esCE Based on the true value image of the original image and the decoding features of each decoding unit, it is obtained by formula (9); The formula (9) is: In the formula, S a is the decoding feature obtained by the a-th decoding unit; AvgPool() represents the average pooling operation; is the decoding feature obtained by the a-th decoding unit and the final truth graph G corresponding to the a-th decoding unit a Binary cross entropy loss between Among them, G a =AvePool(G a )×(1-G a )+G a ; AvePool() is an average pooling operation; G a =maxPool(G a-1 ); maxPool() is the maximum pooling operation; G a is the initial truth map corresponding to the a-th decoding unit; G1 is the true value image of the original image.

9. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the salient object detection method according to any one of claims 1 to 8.

Images