Weakly supervised salient object detection method and system based on mixed label screening

Abstract

The application discloses a weakly supervised salient object detection method and system based on mixed label screening. In the first round of training, the output end of the trained optimization network is connected with the input end of a first discriminator to obtain a first generative adversarial network. The first data set is input into the first generative adversarial network for training to obtain the trained first generative adversarial network. N images in a second data set that do not participate in training are combined with the first data set and then input into the trained first generative adversarial network to optimize the pseudo label of the output image. The first discriminator scores the pseudo label, and the top T images in the ranking are selected as a third data set. The third data set is used as the input value of a prediction network to complete the training of the prediction network and a second discriminator. The training is repeated for several rounds. An image to be subjected to salient object detection is input into the prediction network after the last round of training to obtain a salient object detection result.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a weakly supervised saliency target detection method and system based on hybrid label screening. Background Technology

[0002] Saliency object detection mimics the human eye's visual attention mechanism, aiming to detect the most attractive object in an image or video. It has been applied to varying degrees in multiple tasks, such as semantic segmentation, video compression, and object tracking. In recent years, more and more methods based on deep neural networks have emerged, achieving significant results in detection accuracy. However, most of these methods rely on pixel-level annotations as supervision during training, which is time-consuming, labor-intensive, and costly. Therefore, lower-cost weakly supervised and unsupervised methods have emerged. These methods mostly rely on sparse labels, which offer strong interactivity and are quick to create, and have already found numerous applications in semantic segmentation, object detection, and other fields. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a weakly supervised saliency target detection method and system based on hybrid label screening;

[0004] On the one hand, a weakly supervised saliency target detection method based on hybrid label screening is provided, including:

[0005] In the first round of training, the first dataset is input into the optimization network for training, resulting in the trained optimization network. The output of the trained optimization network is connected to the input of the first discriminator to obtain the first generative adversarial network (GAN). The first dataset is then input into the first GAN for training, resulting in the trained first GAN. N images from the second dataset that were not used in training are merged with the first dataset and input into the trained first GAN to optimize the pseudo-labels of the output images. The first discriminator scores the pseudo-labels and sorts the images in descending order of scores, selecting the top T images as the third dataset. Based on the pseudo-labels of the T images in the third dataset, the original labels of the corresponding images in the first and second datasets are updated to pseudo-labels.

[0006] The third dataset is input into the prediction network to train it, resulting in a trained prediction network. The output of the trained prediction network is connected to the input of the second discriminator to obtain a second generative adversarial network (GAN). The third dataset is then input into the second GAN to train it, resulting in a trained second GAN. The first and second datasets are input into the trained second GAN, and the prediction network outputs pseudo-labels for the images. The second discriminator scores the pseudo-labels and sorts the images in descending order of scores, selecting the top T images as the fourth dataset. Based on the pseudo-labels of the T images in the fourth dataset, the original labels of the corresponding images in the first and second datasets are updated to pseudo-labels. This completes one round of training.

[0007] Repeat the training several times until all images in the second dataset have been trained, then the training ends; obtain the images of the objects to be detected, and input them into the prediction network after the last round of training to obtain the salient object detection results.

[0008] On the other hand, a weakly supervised saliency target detection system based on hybrid label screening is provided, including:

[0009] The first training module is configured as follows: In the first round of training, the first dataset is input into the optimization network and trained to obtain the trained optimization network; the output of the trained optimization network is connected to the input of the first discriminator to obtain the first generative adversarial network (GAN); the first dataset is input into the first GAN and trained to obtain the trained first GAN; N images from the second dataset that were not used in training are merged with the first dataset and input into the trained first GAN to optimize the pseudo-labels of the network output images; the first discriminator scores the pseudo-labels and sorts the images in descending order of scores, selecting the top T images as the third dataset; based on the pseudo-labels of the T images in the third dataset, the original labels of the corresponding images in the first and second datasets are updated to pseudo-labels.

[0010] The second training module is configured as follows: The third dataset is input into the prediction network to train it, resulting in a trained prediction network. The output of the trained prediction network is connected to the input of the second discriminator to obtain a second generative adversarial network (GAN). The third dataset is then input into the second GAN to train it, resulting in a trained second GAN. The first and second datasets are input into the trained second GAN, and the prediction network outputs pseudo-labels for the images. The second discriminator scores the pseudo-labels and sorts the images according to their scores from highest to lowest, selecting the top T images as the fourth dataset. Based on the pseudo-labels of the T images in the fourth dataset, the original labels of the corresponding images in the first and second datasets are updated to the pseudo-labels. This completes one round of training.

[0011] The loop training module is configured to repeat training for several rounds until all images in the second dataset have been trained, at which point training ends.

[0012] The salient object detection module is configured to: acquire an image of the salient object to be detected, input the image of the salient object to be detected into the prediction network after the last round of training, and obtain the salient object detection result.

[0013] Furthermore, an electronic device is also provided, including:

[0014] Memory, used for non-transitory storage of computer-readable instructions; and

[0015] Processor, for executing the computer-readable instructions,

[0016] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0017] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0018] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0019] The above technical solution has the following advantages or beneficial effects:

[0020] This invention proposes a novel recurrent generative adversarial framework that uses an iterative training strategy, starting from simple samples and continuously improving, to filter and optimize coarse labels. It also proposes a global-local label selection strategy that selects labels that are well predicted by traditional methods, as well as labels that perform well on certain pixels, iteratively enhancing the network's prediction capabilities. The proposed method outperforms other weakly supervised methods and even surpasses some strongly supervised methods. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a flowchart of the method in Example 1;

[0023] Figure 2 This is a visualization of the results from Example 1. Detailed Implementation

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0026] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0027] With the continuous development of deep learning, fully supervised salient object detection (SOD) has achieved breakthrough progress, with significant assurance in accuracy. However, this method often relies on pixel-level labels for supervision, which are costly and time-consuming to produce. Therefore, weakly supervised methods based on sparse labels have been continuously developed. This invention proposes a hybrid label system, which consists of a small number of pixel-level labels and a large number of coarse labels generated by traditional mathematical labels. This invention finds that in simple scenarios, noisy labels have an accuracy close to that of manually labeled labels, but in difficult scenarios, they contain a lot of noise. Based on this finding, this invention proposes a label selection scheme that combines local and global selection, employing a training strategy from easy to difficult to gradually improve the network's salient object segmentation ability. In addition, this invention also constructs a two-part recurrent generative adversarial network framework to help the invention better select labels that meet the current stage objectives.

[0028] Example 1

[0029] This embodiment provides a weakly supervised saliency target detection method based on hybrid label screening;

[0030] Weakly supervised saliency target detection methods based on hybrid label screening include:

[0031] S101: In the first round of training, the first dataset is input into the optimization network and trained to obtain the trained optimization network; the output of the trained optimization network is connected to the input of the first discriminator to obtain the first generative adversarial network; the first dataset is input into the first generative adversarial network and trained to obtain the trained first generative adversarial network.

[0032] The N images from the second dataset that were not used in training are merged with the first dataset and then input into the trained first generative adversarial network to optimize the pseudo-labels of the network output images. The first discriminator scores the pseudo-labels and sorts the images in descending order of scores, selecting the top T images as the third dataset. Based on the pseudo-labels of the T images in the third dataset, the original labels of the corresponding images in the first and second datasets are updated to pseudo-labels.

[0033] S102: Input the third dataset into the prediction network and train the prediction network to obtain the trained prediction network; connect the output of the trained prediction network to the input of the second discriminator to obtain the second generative adversarial network; input the third dataset into the second generative adversarial network and train it to obtain the trained second generative adversarial network.

[0034] The first and second datasets are input into the trained second generative adversarial network to predict the pseudo-labels of the network output images. The second discriminator scores the pseudo-labels and sorts the images in descending order of scores, selecting the top T images as the fourth dataset. Based on the pseudo-labels of the T images in the fourth dataset, the original labels of the corresponding images in the first and second datasets are updated to pseudo-labels. At this point, one round of training is complete.

[0035] S103: Repeat S101 to S102 until all images in the second dataset have been used in training, then training ends;

[0036] S104: Obtain the image of the target to be detected, input the image of the target to be detected into the prediction network after the last round of training, and obtain the target detection result.

[0037] Furthermore, the method S101 further includes the following steps before:

[0038] S100: Construct a salient target detection model; the salient target detection model includes: a first generative adversarial network and a second generative adversarial network; wherein, the first generative adversarial network includes: an optimization network and a first discriminator connected in sequence; the second generative adversarial network includes: a prediction network and a second discriminator connected in sequence.

[0039] Furthermore, before S101 and after S100, the method further includes:

[0040] S100-1: Construct a first dataset and a second dataset; the first dataset includes: several images, each image including a true label and a coarse label; the second dataset includes: several images, each image including a coarse label.

[0041] Furthermore, S101 selects the top T images as the third dataset, specifically including:

[0042] S101-1: Images with pseudo-label scores greater than a set threshold are directly used as data in the third dataset;

[0043] S101-2: For images with pseudo-label scores less than a set threshold, perform pixel region segmentation, and use the images after pixel region segmentation as data in the third dataset.

[0044] The pixel region division is achieved by dividing the pixel region into stable pixel regions and unstable pixel regions based on pixel confidence. Pixel regions with a confidence level greater than a set threshold are considered stable pixel regions, while those with a confidence level less than the set threshold are considered unstable pixel regions.

[0045] Optionally, both the optimization network and the prediction network are implemented using CNN networks.

[0046] Furthermore, the coarse labels are either manually labeled or labels output by the salient object detection network after the image is input into the network. Optionally, the salient object detection network includes GCPANet for salient object detection and MINet for salient object detection, etc., but this embodiment does not limit this to a specific network.

[0047] Furthermore, the pixel region is divided into stable pixel regions and unstable pixel regions based on pixel confidence, where pixel confidence refers to:

[0048] In the K stages of training the first generative adversarial network, the first mean absolute error of the predicted saliency map of the network output in stage K-3 and stage K is calculated, the second mean absolute error of the predicted saliency map of the network output in stage K-2 and stage K is calculated, and the third mean absolute error of the predicted saliency map of the network output in stage K-1 and stage K is calculated. The sum of the three mean absolute errors is used as the pixel confidence.

[0049] Furthermore, training the first generative adversarial network further includes:

[0050] The parameters of the first generative adversarial network are optimized using the first total loss function.

[0051] Furthermore, the first total loss function is expressed as follows:

[0052] L R =L bce +α·L lsc +V(D R ,R);

[0053]

[0054]

[0055]

[0056] Among them, y i For image x i The corresponding tag, p i It is image x i The prediction significance plot, K x It is the set of pixels within a k×k range surrounding pixel x, s m and s y These are the significance scores for pixels x and y, respectively. These are the regularization weights, where P(·) and I(·) are the position coordinates and RGB color of the corresponding pixels, respectively. and These are all hyperparameters of the Gaussian kernel, where m and n are pixels.

[0057] Furthermore, training the second generative adversarial network further includes:

[0058] The parameters of the second generative adversarial network are optimized using a second total loss function.

[0059] Furthermore, the second total loss function L S Its expression is:

[0060] L S =Lpre +V(D S ,S);

[0061] L pre =w i ·L bce (Pre i ,L i );

[0062]

[0063] Where MAE represents the mean absolute error, w i Training stability confidence for each image; L bce It is the cross-entropy loss function, Pre i L represents the prediction result of the prediction network. i These are the corresponding training labels.

[0064] This invention employs a novel form of weak supervision for salient object detection tasks: hybrid labels. Hybrid labels consist of a large number of coarse labels with extremely low acquisition costs, and a very small number of accurately labeled pixel-level labels. Based on this, the invention further reduces the proportion of accurately labeled labels to 1 / 20. In practical applications, it was found that not all coarsely labeled pixels have low prediction accuracy. Therefore, this invention aims to utilize accurately labeled images in simple scenes, as well as accurately predicted regions in more complex scenes, to help the network learn the ability to detect salient objects. Thus, a global-local filtering strategy based on generative adversarial networks is proposed. This strategy filters the hybrid labels, starting training with accurate pixels from both simple and complex scenes, gradually improving itself, thereby solving the salient object detection task from a weakly supervised perspective.

[0065] First, this invention filters labels based on the overall difficulty of the image. Although salient object detection is a class-independent task (i.e., it does not focus on the category of the detected object, but only on the salience of the detected object relative to the overall environment), the feature distributions among foreground objects should be similar. In this case, this invention can use images with a small number of accurate labels (pixel-level labels) as the target feature distribution, and use a generative adversarial network to score each saliency prediction result. The higher the score, the closer the detection result is to the true target distribution, thus filtering out the better-predicted labels from the coarse label data. These labels are likely to be detection results in simple scenes. By continuously filtering out labels that are closer to the true distribution, more and more accurate labels are obtained, and these are used to supervise the training of the network, enhancing the network's ability to predict salient objects.

[0066] Secondly, considering that traditional methods do not always perform poorly at the pixel level in complex scenes, the difficulty in predicting complex scenes is due to issues such as dark scenes and low foreground-background contrast. However, there are also localized areas of simplicity within complex scenes. To address this, this invention proposes a local filtering strategy. This strategy scores all pixels in each image, fully utilizing the pixels with higher prediction accuracy in complex scenes for weighted training. Combined with the overall filtering strategy, high-accuracy prediction areas are selected from both the perspective of the entire image and each individual pixel. Iterative training is then performed, gradually progressing from easier to more difficult models to improve detection performance.

[0067] Furthermore, this invention does not directly combine the prediction network and discriminator for generative adversarial training. Instead, it adds an additional optimization network. First, the optimization network is used to correct coarse labels while simultaneously training the generative adversarial network. During training, local scores are used to weight the network, fully utilizing the local accuracy of pixels. Subsequently, based on the scores obtained by the discriminator, more accurate labels are selected and used to train the prediction network. Because this invention uses an iterative approach, the prediction results of the prediction network are further incorporated into the optimization network for further refinement. Therefore, this invention also incorporates a generative adversarial strategy into the selection of the prediction network, forming a cyclical generative adversarial training strategy.

[0068] The hybrid tags used in this invention can be divided into two parts: a small number of pixel-level true tags and a large number of coarse tags, wherein the coarse tags are generated using traditional unsupervised methods. The overall framework is as follows: Figure 1 As shown, the network consists of an optimization network (Op-Net) and a prediction network (Pr-Net). Op-Net is designed as a two-stream encoder-decoder architecture, taking the original RGB image and coarse labels as input and outputting updated pseudo-labels. Pr-Net is a replacement saliency detection network that takes the original RGB image as input and the pseudo-labels generated by Op-Net as supervision signals.

[0069] This invention first employs a generative adversarial network (GAN) training method, filtering coarse labels and pseudo-labels generated during training based on the distribution of real labels. Secondly, it utilizes the fluctuation range of the predicted value of each pixel during training for pixel-level filtering. Combining these two methods continuously improves the prediction quality, training the entire framework from easy to difficult. Finally, this invention designs a cyclic adversarial generative strategy to better ensure that training optimizes in the desired direction, further constraining the quality of the image labels used for training.

[0070] For conventional weakly supervised labels, such as dot labels, doodle labels, and category labels, labels are only present at a subset of pixels. During training, these labels often need to be expanded to ensure stable network training. For example, the traditional flooding algorithm expands the foreground and background pixels of dot labels, and the expanded labels are used for training, achieving good detection performance. However, this method also has limitations. Because the traditional algorithm simply treats adjacent regions of the image as connected graphs, ignoring other image features (such as color and texture), there is a high probability of incorrect foreground and background pixel expansion, thus interfering with network training. Therefore, this invention employs a hybrid labeling approach, further reducing the number of accurate labels to 1 / 20 of the total training data.

[0071] The core problem facing hybrid label training is that noise contained in coarse labels can severely limit network performance. Reducing and limiting this noise is crucial for improving performance. Experimental observations show that in some scenes, the contrast between salient objects and the background is high, and the colors of salient boundaries change significantly, allowing traditional methods to produce good results. Training with these coarse labels still yields good initial results. Secondly, for images where detection is less than ideal, a considerable number of foreground pixels of salient objects are still detected. Selectively utilizing accurate foreground pixels from coarse labels for training is a worthwhile area of ​​research. To address these two situations, this invention designs label selection methods from both global and local perspectives.

[0072] A Global Label Filtering Strategy Based on Generative Adversarial Networks (GANs): Generative Adversarial Networks (GANs) have achieved many breakthroughs in image generation and other fields, such as StyleGAN. Many methods also utilize GANs to distinguish between real and generated distributions, jointly training GANs and prediction networks to make the prediction results closer to the distribution of real detection results. For example, Xiao et al. used GANs to discriminate semantic segmentation networks with only category labels, continuously fitting the segmentation results of different categories to the accurate segmentation result. This invention also employs GANs to filter coarse labels and pseudo-labels generated during training.

[0073] Image generation methods based on GANs typically input noise into a generator to produce a target domain image, and then use a discriminator to distinguish between the real and generated images. Through "adversarial deception," the image generated by the generator network becomes increasingly closer to the real image. Since this invention is for a specific saliency detection task, the original RGB image and coarse labels are directly input into the generator network (i.e., the optimization network Op-Net) to obtain optimized saliency pseudo-labels.

[0074] Given a set of images (N represents the number of images). The RGB images and coarse labels are input into the optimization network Op-Net. The optimization network Op-Net will participate in generative adversarial training as a generator to obtain the prediction results:

[0075] P i =R(x) i ,c i ),

[0076] Among them, P i R represents the prediction map for each image, R is the optimization network Op-Net, and x is the value. i For i RGB images in the dataset, c i Its corresponding rough label.

[0077] Unlike the role of the discriminator in the generation task, this invention uses a discriminator to judge coarse labels that are closer to the real labels (pixel-level labels). The direct approach is to directly input the saliency map predicted by the correction network and the corresponding real labels into the discriminator network for distribution judgment. However, since the saliency map is only a binary map and contains very few image features, it is difficult to achieve good results in feature learning. Therefore, it is combined with a color image and fed into the discriminator network for distribution learning.

[0078] This invention not only filters out the foreground image but also the corresponding background color image, allowing the network to learn the true foreground distribution from both the foreground and background. Thus, when the predicted saliency map contains noise, the color features input to the discriminator will include background features. Due to the difference in foreground and background distribution, the discriminator's score decreases, forcing the network to optimize towards detecting less background. Furthermore, when the predicted saliency map has poor integrity, the detected background features will contain some foreground features. Under the constraint of distribution, the generator will optimize towards more complete object detection. Under these two simultaneous constraints, the optimized network achieves further improvements in both detection accuracy and integrity, resulting in better predictive performance.

[0079] This invention utilizes accurately labeled data as the true data distribution, first constructing a predicted data set and a true data set:

[0080]

[0081]

[0082] Among them, F p and F s These represent the predicted image and the ground truth image, respectively. Cat indicates the cascade operation. S represents element-wise multiplication. i x represents i The corresponding pixel-level accurate labels.

[0083] Training a generative adversarial network (GAN) will enable the prediction network to make its predictions as close as possible to the true and accurate label distribution. In other words, the generative network should try to fool the discriminator network, while the discriminator network should try to distinguish between the true and predicted distributions, thereby further improving the prediction accuracy.

[0084] The network loss function is defined as follows:

[0085]

[0086] Here, D is the discriminator, and the training of the generative adversarial network is completed by constraining the discriminator and the generator respectively.

[0087] Then, the trained discriminator is used to score the labels, and labels that are closer to the true label distribution are selected. This part is the simple scene image, defined as follows:

[0088]

[0089] Among them, M i For the image x to be filtered ii The corresponding mask label, Score i This represents the overall score for that label. Based on the score, the training set for the prediction network is obtained, defined as follows:

[0090] Ts p ={(x i ,y i Score i >thr},

[0091] Where thr is the threshold used for filtering, set to 0.5, x i and y i These are images whose scores exceed the threshold and their labels.

[0092] A pixel-level local filtering strategy based on prediction stability: Observation reveals that most of the noise in coarse labels originates from false detections, i.e., detecting non-salient parts. In other words, most coarse labels still contain relatively complete salient objects. Therefore, how to effectively utilize these detected pixels and suppress non-salient objects is key to further improving performance. Existing technologies determine the confidence and uncertainty pixels in an image from different perspectives. For example, they use a fixed low threshold to filter out low-confidence regions in the prediction results, or they simultaneously train two models with the same network structure but different initializations to predict the same label, evaluating the difference between the prediction results to weight the uncertainty perception loss, thereby suppressing the uncertainty. This invention aims to filter out these uncertain pixels within a single training phase without using a manual threshold.

[0093] The ST method considers the entire image and finds a positive correlation between the network's final prediction accuracy and training stability. Inspired by this, experimental observations revealed the same principle at the pixel level. During training, the network tends to make more confident judgments about foreground and background pixels with higher confidence, and the prediction results for these pixels fluctuate less in each training iteration. Corresponding to the early loops of the network in this invention, these pixels are often relatively simple object parts with prominent textures and color contrasts—the simple pixel samples mentioned earlier. In the later stages of the network loop, the framework of this invention gradually learns how to detect salient objects from simple samples, achieving self-improvement and gradually learning the ability to detect salient objects in complex scenes.

[0094] Specifically, for an RGB image x i For X, this invention saves the network weights at the end of each training set iteration during the training process. For a training process with K iterations, this invention saves the weight parameters of K checkpoints. When the network stabilizes and reaches its best performance in the later stages of training, this invention uses the saved weights from the last few stages to adjust x. i The prediction significance map M is obtained by making predictions. ij The evaluation is performed in the last few stages and between the final stage to obtain stable and unstable pixel regions, and to determine the easy and hard sample pixels in the training process, as shown in the following formula:

[0095]

[0096] Where MAE represents the mean absolute error, w iA training stability confidence map is generated for each image. This invention uses the training results from the last three stages to perform the evaluation. Simultaneously, this invention uses the obtained confidence maps to weight the loss function of the prediction network training: L pre =w i ·L bce (Pre i ,L i ), where L bce It is the cross-entropy loss function, Pre i L represents the prediction results of the prediction network Pr-Net. i These are the corresponding training labels.

[0097] To further ensure the stability of training, this invention also performs generative adversarial training on the prediction network Pr-Net, forming a cyclic generative adversarial strategy.

[0098] Specifically, this invention includes data with both accurate and coarse labels, comprising 1 / 20 of the total data; this portion is referred to as the real dataset. The remaining data lacks accurate pixel-level labels, possessing only coarse labels; this portion is referred to as the coarse dataset.

[0099] In the first round of training, Op-Net is pre-trained using a real dataset, and then a discriminator is used in conjunction with Op-Net for generative adversarial training. Secondly, since the coarse labels contain many images with masks and almost no target information, this invention removes them in the first round. The trained first discriminator scores all coarse labels, sorts the scores of each image, and selects the top T images for training.

[0100]

[0101] Where Rank represents the ranking function, and e is the number of training iterations. Subsequently, this invention filters this data into simple and complex scene images. The dataset obtained from global filtering is used as simple samples and added to the prediction network for training, while the remaining samples are classified as complex scene images and processed using local filtering.

[0102] In the training of the prediction network, for simple samples, this invention directly inputs the image into the prediction network and trains it under the supervision of its corresponding label; for complex samples, a local screening strategy is used to evaluate stability and obtain a confidence map, which is then used to weight the training loss function of Pr-Net.

[0103] After the prediction network is trained, a discriminator with the same structure but different initialization is used in conjunction with Pr-Net for generative adversarial training. The discriminator is used again to score and rank the data, and the top T data are selected as the training data for the next round of Op-Net. This invention will then iteratively train according to this strategy until all data has been added to the training.

[0104] Binary cross-entropy loss is used as the loss function for training Op-Net and Pr-Net, as shown in the following formula:

[0105]

[0106] Among them, y i For image x i The corresponding tag, p i It is image x i The prediction significance plot.

[0107] Furthermore, this invention introduces a Gaussian bandwidth filter to constrain the similarity scores between pixels with the same features. For pixels m and n, a local saliency consistency loss function is defined:

[0108]

[0109] Among them, K x It is the set of pixels within a k×k range surrounding pixel x, s x and s y Let x and y be the saliency scores for pixels x and y, respectively. The function F(x,y) is defined as:

[0110]

[0111] in, These are the regularization weights, where P(·) and I(·) are the position coordinates and RGB color of the corresponding pixels, respectively. and These are all hyperparameters of the Gaussian kernel.

[0112] The total loss function of Op-Net, trained using a joint generative adversarial network, is expressed as:

[0113] L R =L bce +α·L lsc +V(D R ,R),

[0114] Here, α is set to 0.3.

[0115] The total loss function for training Pr-Net is as follows:

[0116] L S =Lpre +V(D S ,S),

[0117] Among them, L pre The cross-entropy loss function is weighted by the stability score.

[0118] Figure 2 The visualization results of the design model of this invention are shown. The method of this invention achieves good results in terms of both accuracy and completeness. For example, in the second image, the method of this invention completely detected the leopard's forelimbs and hindlimbs. At the same time, in other challenging scenarios, such as the first image, where the squirrel's tail and number of colors are very similar, the method of this invention also performs well.

[0119] Example 2

[0120] This embodiment provides a weakly supervised saliency target detection system based on hybrid label screening, including:

[0121] The first training module is configured as follows: In the first round of training, the first dataset is input into the optimization network and trained to obtain the trained optimization network; the output of the trained optimization network is connected to the input of the first discriminator to obtain the first generative adversarial network (GAN); the first dataset is input into the first GAN and trained to obtain the trained first GAN; N images from the second dataset that were not used in training are merged with the first dataset and input into the trained first GAN to optimize the pseudo-labels of the network output images; the first discriminator scores the pseudo-labels and sorts the images in descending order of scores, selecting the top T images as the third dataset; based on the pseudo-labels of the T images in the third dataset, the original labels of the corresponding images in the first and second datasets are updated to pseudo-labels.

[0122] The second training module is configured as follows: The third dataset is input into the prediction network to train it, resulting in a trained prediction network. The output of the trained prediction network is connected to the input of the second discriminator to obtain a second generative adversarial network (GAN). The third dataset is then input into the second GAN to train it, resulting in a trained second GAN. The first and second datasets are input into the trained second GAN, and the prediction network outputs pseudo-labels for the images. The second discriminator scores the pseudo-labels and sorts the images in descending order of scores, selecting the top T images as the fourth dataset. Based on the pseudo-labels of the T images in the fourth dataset, the original labels of the corresponding images in the first and second datasets are updated to pseudo-labels. This completes one round of training.

[0123] The loop training module is configured to repeat training for several rounds until all images in the second dataset have been trained, at which point training ends.

[0124] The salient object detection module is configured to: acquire an image of the salient object to be detected, input the image of the salient object to be detected into the prediction network after the last round of training, and obtain the salient object detection result.

[0125] It should be noted that the first training module, the second training module, the loop training module, and the salient object detection module mentioned above correspond to steps S101 to S104 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should be noted that these modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0126] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0127] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0128] Example 3

[0129] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0130] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0131] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0132] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0133] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0134] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0135] Example 4

[0136] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0137] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A weakly supervised salient object detection method based on mixed label screening, characterized in that, The method comprises the following steps: constructing a first data set and a second data set; the first data set comprises a plurality of images, each image comprising a real label and a coarse label; the second data set comprises a plurality of images, each image comprising a coarse label; in a first round of training, the first data set is input into an optimization network for training to obtain a trained optimization network; the output end of the trained optimization network is connected with the input end of a first discriminator to obtain a first generative adversarial network, the first data set is input into the first generative adversarial network for training to obtain a trained first generative adversarial network; N images in the second data set that do not participate in training are combined with the first data set and then input into the trained first generative adversarial network, and the optimization network outputs pseudo labels of the images; the first discriminator scores the pseudo labels and sorts the images in descending order of scores, and selects the top T images as a third data set; the original labels of the corresponding images in the first data set and the second data set are updated to the pseudo labels according to the pseudo labels of the T images in the third data set; the third data set is input into a prediction network for training to obtain a trained prediction network; the output end of the trained prediction network is connected with the input end of a second discriminator to obtain a second generative adversarial network, the third data set is input into the second generative adversarial network for training to obtain a trained second generative adversarial network; the first data set and the second data set are input into the trained second generative adversarial network, and the prediction network outputs pseudo labels of the images; the second discriminator scores the pseudo labels and sorts the images in descending order of scores, and selects the top T images as a fourth data set; the original labels of the corresponding images in the first data set and the second data set are updated to the pseudo labels according to the pseudo labels of the T images in the fourth data set; thus, one round of training is completed; the training is repeated for several rounds until all the images of the second data set have participated in the training, and then the training is completed; an image to be detected for salient objects is obtained, and the image to be detected for salient objects is input into the prediction network after the last round of training to obtain a salient object detection result.

2. The weakly supervised salient object detection method based on hybrid label filtering according to claim 1, characterized in that, The method further comprises: constructing a salient object detection model; the salient object detection model comprises a first generative adversarial network and a second generative adversarial network; the first generative adversarial network comprises an optimization network and a first discriminator connected in sequence; the second generative adversarial network comprises a prediction network and a second discriminator connected in sequence.

3. The weakly supervised salient object detection method based on hybrid label filtering according to claim 1, characterized in that, selecting the top T images as a third data set, specifically comprising: images with pseudo label scores greater than a set threshold value are directly used as data of the third data set; images with pseudo label scores less than the set threshold value are subjected to pixel region division, and the images after the pixel region division are also used as data of the third data set; The pixel region division is performed by pixel confidence, and the pixel region is divided into a stable pixel region and an unstable pixel region; the pixel region with a pixel confidence greater than a set threshold is the stable pixel region, and the pixel region with a pixel confidence less than the set threshold is the unstable pixel region.

4. The weakly supervised salient object detection method based on hybrid label filtering according to claim 3, characterized in that, The pixel confidence refers to: In a first stage of the first generative adversarial network training , a first mean absolute error of the predicted saliency map output by the first stage and the second stage network is calculated, a second mean absolute error of the predicted saliency map output by the first stage and the second stage network, a third mean absolute error of the predicted saliency map of the first stage and the predicted saliency map output by the second stage network is calculated, a sum of the three mean absolute errors is calculated, and the sum is taken as the pixel confidence.

5. The weakly supervised salient object detection method based on hybrid label filtering of claim 1, wherein, The first total loss function is expressed as: ； ； ； ； wherein, is an image corresponding label, is an image predicted saliency map, is a pixel surrounding a pixel set region within a range of and are saliency scores of pixels and corresponding to the pixel is a regularization weight, and are position coordinates and RGB color of the corresponding pixel, and are hyperparameters of the Gaussian kernel, and are pixels; The inputting the third data set into the second generative adversarial network, training the second generative adversarial network, and obtaining the trained second generative adversarial network further include: adopting a second total loss function to optimize parameters of the second generative adversarial network; the second total loss function The expression is: ； ； For an RGB image ∈X, during training, the network weights at the end of each training set iteration are saved; for a given... Each iteration of the training process is saved. Each checkpoint weight parameter When the network stabilizes and reaches its best performance in the later stages of training, the weights from the last few stages are used to adjust the performance. The prediction significance map is obtained by making predictions. The evaluation is performed in the last few stages and between the final stage to obtain stable and unstable pixel regions, and to determine the easy and hard sample pixels in the training process, as shown in the following expression: ； where MAE represents the mean absolute error, a training stability confidence corresponding to each image; is a cross-entropy loss function, represents a prediction result of the prediction network, is a corresponding training label.

6. A weakly supervised salient object detection system based on mixed label screening, comprising: The first training module is configured to input the first data set into the optimization network in the first round of training, train the optimization network, and obtain a trained optimization network; The output end of the trained optimization network is connected with the input end of the first discriminator to obtain a first generative adversarial network, the first data set is input into the first generative adversarial network, the first generative adversarial network is trained, and a trained first generative adversarial network is obtained; After the N images in the second data set that do not participate in training are combined with the first data set, the combined images are input into the trained first generative adversarial network, and the optimization network outputs pseudo labels of the images; the first discriminator scores the pseudo labels, and sorts the images in descending order of scores, and selects the top T images as a third data set; the original labels of the corresponding images in the first data set and the second data set are updated to the pseudo labels according to the pseudo labels of the T images in the third data set; The second training module is configured to input the third data set into the prediction network, train the prediction network, and obtain a trained prediction network; The output end of the trained prediction network is connected with the input end of the second discriminator to obtain a second generative adversarial network, the third data set is input into the second generative adversarial network, the second generative adversarial network is trained, and a trained second generative adversarial network is obtained; The first data set and the second data set are input into the trained second generative adversarial network, and the prediction network outputs pseudo labels of the images; The second discriminator scores the pseudo labels, and sorts the images in descending order of scores, and selects the top T images as a fourth data set; The original labels of the corresponding images in the first data set and the second data set are updated to the pseudo labels according to the pseudo labels of the T images in the fourth data set; thus, one round of training is completed; The cycle training module is configured to repeat several rounds of training until all the images in the second data set have participated in training, and then the training is completed; The salient object detection module is configured to obtain an image to be salient object detected, input the image to be salient object detected into the prediction network after the last round of training, and obtain a salient object detection result. The first data set and the second data set are constructed; the first data set includes a plurality of images, and each image includes a true label and a coarse label; and the second data set includes a plurality of images, and each image includes a coarse label.

7. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; and a processor for executing the computer readable instructions, wherein the computer readable instructions, when executed by the processor, perform the method of any of claims 1-5. a non-transitory computer readable medium storing computer readable instructions, wherein the instructions, when executed by a computer, perform the method of any of claims 1-5.

8. A storage medium, characterized by a non-transitory a computer program for implementing the method of any of claims 1-5 when executed on one or more processors.

9. A computer program product, characterised in that, ​

Images