Hybrid supervised training method, device and storage medium for salient object detection
By employing a hybrid supervised training method, utilizing finely labeled and unlabeled data, and combining data augmentation and pixel reliability partitioning, the problem of insufficient data in salient object detection models is solved, thereby improving the model's performance and generalization ability while reducing data acquisition costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING SHUNSHI BROTHERS TECH CO LTD
- Filing Date
- 2022-12-19
- Publication Date
- 2026-05-12
AI Technical Summary
Existing salient object detection models suffer from performance bottlenecks due to the limited number of images in the dataset, and the high cost of acquiring large-scale labeled data, coupled with limited data augmentation methods, make it difficult to effectively utilize unlabeled data.
A hybrid supervised training method is adopted, which uses finely labeled data to train multiple network models as discriminators, combines unlabeled data for data augmentation and pixel reliability segmentation, and uses the cross-entropy loss function to train the model, expand the data samples and improve the model's generalization ability.
By using a hybrid supervised training method, the model's generalization ability was improved, the data acquisition cost was reduced, unlabeled data was effectively utilized, and the model's performance was enhanced.
Smart Images
Figure CN115861118B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, specifically to a hybrid supervised training method, device, and storage medium for salient target detection. Background Technology
[0002] The information disclosed in this background section is intended only to enhance understanding of the overall background of the invention and is not necessarily to be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.
[0003] In recent years, research based on deep learning technology has achieved remarkable results in the field of computer vision. Taking salient object detection as an example, deep learning-based methods have higher accuracy and generalization ability compared to traditional methods, and can capture salient regions of images with greater reliability. However, deep learning methods require a large number of data samples for training network models. For example, widely used image classification datasets such as ImageNet contain millions of images, while the largest dataset in the field of salient object detection contains only about ten thousand images. In addition, training salient object detection models requires pixel-level annotated data samples, and obtaining large-scale, finely annotated datasets faces significant time consumption and labor investment.
[0004] Since salient object detection is based on predicting human visual attention, its primary goal is to detect and segment the most prominent object regions in an image. Therefore, data augmentation methods that introduce black edges, such as MixUp, CutMix, and Mosaic, are unsuitable for salient object detection. Currently, commonly used data augmentation methods for salient object detection are limited to affine transformations, random cropping, and random flipping. Furthermore, supervised learning methods rely on large-scale labeled data, which is prohibitively expensive to acquire in practice. To alleviate this problem, assigning pseudo-labels to unlabeled pixels and making reasonable use of unlabeled data samples becomes crucial. Summary of the Invention
[0005] To address this issue, this invention provides a hybrid supervised training method for salient object detection, thereby resolving the performance bottleneck problem in existing technologies caused by the limited number of images in the dataset.
[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0007] In the implementation of this invention, the following steps are provided:
[0008] Multiple salient object detection models with different network structures were trained using finely labeled data images. The different models were then used as discriminators to distinguish between reliable and unreliable pixels in unlabeled images.
[0009] Unlabeled and finely labeled images are simultaneously loaded into the salient object detection model to be trained as training data.
[0010] For labeled data images, data augmentation methods are used to increase the diversity of data samples. For unlabeled data images, multiple discriminators trained are called to obtain the saliency prediction results corresponding to the images.
[0011] Reliable pixels in the image are marked using a pixel reliability segmentation method;
[0012] The model is trained with mixed supervision by using cross-entropy or binary cross-entropy loss function on reliable pixels in labeled and unlabeled data.
[0013] Furthermore, the data augmentation method comprises the following steps:
[0014] For any pair of data samples, given the foreground and background of the image, only the foreground of the image is adjusted using a color enhancement method;
[0015] The background color remains unchanged, thereby further expanding the number of data samples.
[0016] Furthermore, the data sample pair includes image I and its label alpha. Image processing methods are used to enhance the color of image I, and foreground color transformation is achieved using an image fusion formula, which is as follows:
[0017] Merge=I*(1-alpha)+C*alpha
[0018] In the formula: C represents the image after overall color enhancement, and Merge represents the image after color enhancement only for the foreground.
[0019] Furthermore, the color enhancement processing method employs, but is not limited to, the Brightness, Contrast, and Sharpness classes of the ImageEnhance module. The Brightness class represents brightness enhancement and is used to adjust the brightness of the image; the Contrast class represents contrast enhancement and is used to adjust the contrast of the image; and the Sharpness class represents sharpness enhancement and is used to adjust the sharpness of the image.
[0020] Furthermore, the pixel reliability partitioning method comprises the following steps:
[0021] For any pixel in an unlabeled image, compare the saliency prediction results obtained by different discriminators.
[0022] If the saliency prediction values of a pixel obtained by different discriminators are the same, then the pixel is considered a reliable pixel; otherwise, it is classified as an unreliable pixel.
[0023] Furthermore, to achieve the above objectives, the present invention also provides an electronic device for hybrid supervised training of salient object detection, including an input device and an output device, and further comprising...
[0024] Processor, suitable for implementing one or more hybrid supervised training instructions for salient object detection; and
[0025] A computer storage medium, wherein the computer-readable storage medium stores hybrid supervised training instructions for salient target detection, the instructions implementing the steps of any one of the methods described above during execution by a processor.
[0026] Furthermore, to further achieve the above objectives, the present invention also provides a computer-readable storage medium storing hybrid supervised training instructions for salient target detection, characterized in that the instructions implement the steps of any one of the above methods during execution by a processor.
[0027] According to the embodiments of the present invention, compared with the prior art, it has the following advantages: It trains multiple salient object detection models with different network structures using finely labeled data images, treating the resulting different models as discriminators to classify reliable and unreliable pixels in unlabeled images; it simultaneously loads unlabeled and finely labeled data images into the salient object detection model to be trained as training data; for labeled data images, it uses data augmentation to increase the diversity of data samples; for unlabeled data images, it calls the trained multiple discriminators to obtain the corresponding salient prediction results; it marks reliable pixels in the image using a pixel reliability classification method; and it uses cross-entropy or binary cross-entropy loss function to perform mixed-supervised training on reliable pixels in labeled and unlabeled data. By labeling the data and using the model to discriminate the corresponding data, it increases the diversity of data samples through mixed-supervised training of all data, improving the model's generalization ability, reducing data acquisition costs, and achieving deep learning of data more efficiently. Attached Figure Description
[0028] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.
[0029] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.
[0030] Figure 1 A flowchart of a hybrid supervised training method for salient target detection provided in an embodiment of the present invention;
[0031] Figure 2 A structural diagram illustrating the implementation of the hybrid supervised training method for salient target detection provided in this embodiment of the invention;
[0032] Figure 3 This is a structural diagram illustrating the implementation of the data augmentation method in the hybrid supervised training method for salient target detection provided in this embodiment of the invention.
[0033] Figure 4 This is a structural diagram illustrating the implementation of the pixel reliability partitioning method in a hybrid supervised training method for salient target detection, provided as another embodiment of the present invention. Detailed Implementation
[0034] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] The terms such as "upper," "lower," "left," "right," and "middle" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.
[0036] Currently, common data augmentation methods for salient object detection are limited to affine transformations, random cropping, and random flipping. Furthermore, supervised learning methods rely on large-scale labeled data, which is prohibitively expensive to acquire in practice. To alleviate this problem, assigning pseudo-labels to unlabeled pixels and making reasonable use of unlabeled data samples becomes crucial.
[0037] To address this, the present invention proposes a novel data augmentation method and a hybrid label-supervised training method device to further enrich data diversity, improve the generalization ability of the model, reduce data acquisition costs, and achieve deep learning of data more efficiently.
[0038] Exemplary device
[0039] In this example, an electronic device capable of salient object detection with hybrid supervised training, comprising an input device and an output device, is described as being able to achieve the aforementioned objective.
[0040] A processor suitable for implementing instructions for a hybrid supervised training method for one or more salient object detection methods; and
[0041] A computer storage medium storing hybrid supervised training instructions for salient object detection, wherein the steps of a hybrid supervised training method for salient object detection are implemented during the execution of the instructions by a processor.
[0042] Exemplary methods
[0043] like Figure 1 As shown, it illustrates a hybrid supervised training method for salient target detection provided by an embodiment of the present invention, with the following steps:
[0044] Step 11: Train multiple salient object detection models with different network structures using finely labeled data images, and use the different models obtained as discriminators to distinguish reliable pixels from unreliable pixels in unlabeled images;
[0045] To ensure the quality of the labeled dataset, fine annotation is performed by multiple experienced staff using tools such as Photoshop. In this embodiment, nine staff members annotate the corresponding data forming a 9-grid. Then, multiple rounds of detection are used to check each labeled data, resulting in a high-quality, finely annotated salient target detection dataset.
[0046] Step 12: Load both the unlabeled and finely labeled data images into the salient object detection model to be trained as training data;
[0047] The salient object detection model to be trained can be a salient object detection network built on deep learning, or it can be an open source model such as U2Net, ICON, or an improved version of such current best models.
[0048] Taking ICON as an example: Based on the encoder architecture, it consists of a Diverse Feature Aggregation (DFA) module, an Integrity Channel Enhancement (ICE) module, and a Part-Whole Validation (PWV) module. The encoder uses ResNet-50 as the backbone to extract multi-level features. First, the backbone features are passed to the Diverse Feature Aggregation (DFA) module, which consists of multiple convolutional blocks, to enhance the backbone features. Second, the Integrity Channel Enhancement (ICE) module is used to mine the integrity information hidden in different feature channels. Finally, to further refine the saliency map, the Part-Whole Validation (PWV) module is used to verify the consistency between partial and whole salient regions.
[0049] Specifically, the aforementioned salient object detection models with different network structures reproduce several of the most accurate network models in the field. The starting point of this method is to improve the performance of end-to-end deep learning-based network models, which can be achieved not only by designing a better network structure but also by improving performance from a data perspective.
[0050] Step 13: For labeled data images, use data augmentation methods to increase the diversity of data samples; for unlabeled data images, call multiple trained discriminators to obtain the saliency prediction results corresponding to the images.
[0051] The core idea of data augmentation is as follows: for any pair of data samples, given the foreground and background of the image, only the foreground is adjusted using color enhancement, while the background color remains unchanged, thereby further expanding the number of data samples.
[0052] Conventional data augmentation methods typically enhance the color of the entire image. However, considering the practical application scenarios of salient object detection, this invention proposes to enhance the color of only the foreground of the image while leaving the background color unchanged, thus further enriching the data samples. Figure 3 As shown, the method steps are as follows:
[0053] For any pair of data samples, given the foreground and background of the image, only the foreground of the image is adjusted using a color enhancement method;
[0054] The background color remains unchanged, thereby further expanding the number of data samples.
[0055] In this example, the data sample pair includes image I and its label alpha. Image processing methods are used to enhance the color of image I, and foreground color transformation is achieved using an image fusion formula, which is as follows:
[0056] Merge=I*(1-alpha)+C*alpha
[0057] In the formula: C represents the image after overall color enhancement, and Merge represents the image after color enhancement only for the foreground.
[0058] In specific embodiments, the color enhancement processing method uses, but is not limited to, the Brightness, Contrast, and Sharpness classes of the ImageEnhance module. The Brightness class represents brightness enhancement and is used to adjust the brightness of the image; the Contrast class represents contrast enhancement and is used to adjust the contrast of the image; and the Sharpness class represents sharpness enhancement and is used to adjust the sharpness of the image.
[0059] Step 14: Mark reliable pixels in the image using a pixel reliability segmentation method;
[0060] like Figure 4 As shown, the steps of the pixel reliability partitioning method are as follows:
[0061] For any pixel in an unlabeled image, compare the saliency prediction results obtained by different discriminators.
[0062] If the saliency prediction values of a pixel obtained by different discriminators are the same, then the pixel is considered a reliable pixel; otherwise, it is classified as an unreliable pixel.
[0063] Step 15: Use loss functions such as cross-entropy or binary cross-entropy to perform mixed-supervised training of reliable pixels in labeled and unlabeled data.
[0064] In the specific implementation process, in order to explain in detail the hybrid supervised training process for salient object detection, this invention starts from the practical application scenarios of salient object detection, such as... Figure 2 As shown, the main problem with training with a mixture of unlabeled and labeled data is how to make reasonable use of the unlabeled data. This invention effectively divides pixels in unlabeled data by comparing the discrimination results of multiple discriminators. The specific steps are as follows:
[0065] (1) Construct a finely labeled data sample set and an unlabeled data sample set.
[0066] (2) For the problem of salient target detection, N different network models are trained using a finely labeled data sample set.
[0067] (3) Use the unlabeled data sample set as input to the N salient object detection models trained, and output the corresponding salient prediction map of the image.
[0068] (4) Filter reliable pixels in the unlabeled data sample set.
[0069] Discrimination method: For any unlabeled image, N different saliency target detection models are used to obtain N saliency prediction results. If a pixel in the original image has the same predicted value in all N saliency prediction results, that pixel is marked as a reliable pixel.
[0070] (5) Supervised training of the salient target detection model is carried out using reliable pixels from finely labeled data sample sets and unlabeled data sample sets.
[0071] Furthermore, to further achieve the above objectives, the present invention also provides a computer-readable storage medium storing hybrid supervised training instructions for salient target detection, characterized in that the instructions implement the steps of any one of the above methods during execution by a processor.
[0072] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.
Claims
1. A hybrid supervised training method for salient object detection, characterized in that, Includes the following steps: Multiple salient object detection models with different network structures were trained using finely labeled data images. The different models were then used as discriminators to distinguish between reliable and unreliable pixels in unlabeled images. Unlabeled and finely labeled images are simultaneously loaded into the salient object detection model to be trained as training data. For labeled image data, data augmentation methods are used to increase the diversity of data samples; For unlabeled data images, multiple discriminators trained are invoked to obtain the saliency prediction results corresponding to the images. Reliable pixels in the images are marked by a pixel reliability partitioning method. The model is trained with mixed supervision by using cross-entropy or binary cross-entropy loss function on reliable pixels in labeled and unlabeled data. The data augmentation method comprises the following steps: For any pair of data samples, given the foreground and background of the image, only the foreground of the image is adjusted using a color enhancement method; The background color remains unchanged, thereby further expanding the number of data samples; The data sample pair includes image I and its label alpha. Image processing methods are used to enhance the color of image I, and foreground color transformation is achieved using an image fusion formula, which is as follows: Merge = I * (1 - alpha) + C * alpha In the formula: C represents the image after overall color enhancement, and Merge represents the image after color enhancement only for the foreground.
2. The hybrid supervised training method for salient target detection as described in claim 1, characterized in that, The color enhancement processing method uses the Brightness, Contrast, and Sharpness classes of the ImageEnhance module. The Brightness class represents brightness enhancement and is used to adjust the brightness of the image; the Contrast class represents contrast enhancement and is used to adjust the contrast of the image; and the Sharpness class represents sharpness enhancement and is used to adjust the sharpness of the image.
3. The hybrid supervised training method for salient target detection as described in claim 1, characterized in that, The steps of the pixel reliability partitioning method are as follows: For any pixel in an unlabeled image, compare the saliency prediction results obtained by different discriminators. If the saliency prediction values of a pixel obtained by different discriminators are the same, then the pixel is considered a reliable pixel; otherwise, it is classified as an unreliable pixel.
4. An electronic device for salient object detection using hybrid supervised training, comprising an input device and an output device, characterized in that, Also includes The processor is suitable for implementing one or more hybrid supervised training instructions for salient object detection; as well as A computer-readable storage medium storing hybrid supervised training instructions for salient object detection, wherein the instructions, when executed by a processor, implement the steps of any one of claims 1-3.
5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores hybrid supervised training instructions for salient target detection, characterized in that the instructions implement the steps of any one of claims 1-3 during execution by a processor.