A training method, device, equipment and storage medium for a pseudo-label model

Abstract

The disclosed embodiments provide a training method, apparatus, device, and storage medium for a pseudo-label model, wherein the method comprises: obtaining label information and edge information of a sample image; obtaining a first classification prediction result and a first edge prediction result of the sample image, wherein the first classification prediction result and the first edge prediction result are predicted by the pseudo-label model to be trained; determining a first network loss based on the difference between the first classification prediction result and the label information; determining a second network loss based on the difference between the first edge prediction result and the edge information; adjusting the network parameters of the pseudo-label model to be trained based on the first network loss and the second loss until the model training end condition is reached, thereby obtaining the pseudo-label model. The pseudo-label model trained by this method can use the predicted edge information to guide the correction of the predicted label information for unlabeled images, thereby obtaining a more reliable pseudo-label.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a training method, apparatus, device, and storage medium for a pseudo-label model. Background Art

[0002] With the development of deep learning, semantic segmentation has been widely studied. Supervised learning-based semantic segmentation algorithms continue to achieve record-breaking accuracy on benchmark datasets such as Cityscapes and Pascal VOC. However, for supervised learning-based semantic segmentation, a large number of sample images in the training datasets require high-quality pixel-level annotation, which translates to expensive and time-consuming annotation.

[0003] Semi-supervised learning (SSL) is a learning method that combines supervised learning with unsupervised learning. Semi-supervised learning uses a small number of labeled images and a large number of unlabeled images during training. Its core lies in the effective use of a large number of unlabeled images as a supplement to labeled images to improve the accuracy of the trained model. Semi-supervised semantic segmentation (SSL) applies semi-supervised learning to semantic segmentation tasks to alleviate the dependence of supervised learning-based semantic segmentation tasks on high-quality annotated training datasets. The key to semi-supervised semantic segmentation is to assign pseudo labels (PseudoLabel) to each pixel of the unlabeled images in the training dataset. The higher the accuracy and reliability of the pseudo labels, the higher the accuracy of the trained model.

[0004] In related technologies, a common practice is self-training, which first uses labeled images to train a model, and then uses this model to predict labels for unlabeled images, and selects highly reliable predictions as pseudo-labels. However, there are a certain number of unreliable pseudo-labels in the pseudo-labels obtained in this way. Using datasets annotated with these pseudo-labels for model training of semantic segmentation tasks will lead to model degradation and even cause the model to develop in the wrong direction, which restricts the accuracy of semi-supervised semantic segmentation algorithms. Summary of the Invention

[0005] In view of this, embodiments of the present disclosure provide at least one pseudo-label model training, apparatus, device, and storage medium.

[0006] Specifically, the embodiments of the present disclosure are implemented through the following technical solutions:

[0007] In a first aspect, a method for training a pseudo-label model is provided, the method comprising:

[0008] Obtaining label information and edge information of a sample image, wherein the label information includes information of a segmentation mask of the sample image, and the edge information includes information of an edge of the segmentation mask, wherein the segmentation mask is used to mark the category to which each pixel in the sample image belongs;

[0009] Obtaining a first classification prediction result and a first edge prediction result of the sample image, where the first classification prediction result and the first edge prediction result are obtained by predicting a pseudo label model to be trained;

[0010] determining a first network loss based on a difference between the first classification prediction result and the label information;

[0011] determining a second network loss based on a difference between the first edge prediction result and the edge information;

[0012] The network parameters of the pseudo-label model to be trained are adjusted according to the first network loss and the second loss until a model training end condition is reached, thereby obtaining the pseudo-label model.

[0013] In a second aspect, a method for training a semi-supervised semantic segmentation network is provided, wherein a training sample set of the semantic segmentation network includes: a plurality of unlabeled images; the method includes:

[0014] Obtain a pseudo-label model trained using the pseudo-label model training method described above;

[0015] Inputting the unlabeled image into the pseudo-label model to obtain a corresponding second prediction result and a second edge prediction result;

[0016] Correcting the second classification prediction result according to the second edge prediction result to obtain pseudo label information corresponding to the unlabeled image;

[0017] The semantic segmentation network is trained according to the unlabeled image and its corresponding pseudo-label information.

[0018] In a third aspect, a pseudo-label model training device is provided, the device comprising:

[0019] a training data acquisition module, configured to: acquire label information and edge information of a sample image, wherein the label information includes information of a segmentation mask of the sample image, and the edge information includes information of an edge of the segmentation mask, wherein the segmentation mask is used to label the category to which each pixel in the sample image belongs; and acquire a first classification prediction result and a first edge prediction result of the sample image, wherein the first classification prediction result and the first edge prediction result are predicted by a pseudo-label model to be trained;

[0020] a network loss determination module, configured to: determine a first network loss based on a difference between the first classification prediction result and the label information; and determine a second network loss based on a difference between the first edge prediction result and the edge information;

[0021] A network parameter adjustment module is used to adjust the network parameters of the pseudo-label model to be trained according to the first network loss and the second loss until the model training end condition is met, thereby obtaining the pseudo-label model.

[0022] In a fourth aspect, a training device for a semi-supervised semantic segmentation network is provided, wherein a training sample set of the semantic segmentation network includes: a plurality of unlabeled images; the device includes:

[0023] A model acquisition module is used to: acquire a pseudo-label model trained by the above-mentioned pseudo-label model training method;

[0024] A model prediction module, configured to: input an unlabeled image into the pseudo-label model to obtain a corresponding second prediction result and a second edge prediction result;

[0025] a label correction module, configured to correct the second classification prediction result according to the second edge prediction result to obtain pseudo label information corresponding to the unlabeled image;

[0026] The network training module is used to train the semantic segmentation network according to the unlabeled image and its corresponding pseudo-label information.

[0027] In a fifth aspect, an electronic device includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the training method of the pseudo-label model or the training method of the semi-supervised semantic segmentation network described in any embodiment of the present disclosure when executing the computer instructions.

[0028] In a sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the pseudo-label model or the training method of the semi-supervised semantic segmentation network described in any embodiment of the present disclosure is implemented.

[0029] The technical solution of the embodiment of the present disclosure provides a training method for a pseudo-label model, which uses the label information and edge information of the sample image as supervision information to train a pseudo-label model. The pseudo-label model is used to predict classification prediction results and edge prediction results, so that for unlabeled images, the predicted classification prediction results can be used to guide the correction of the predicted classification prediction results, thereby obtaining more reliable pseudo-labels, which can make full use of the information provided by the labels of limited sample images, improve the algorithm accuracy, and save a lot of labeling costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in one or more embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] Figure 1 is a schematic diagram of a semantic segmentation network training method shown in at least one embodiment of the present disclosure;

[0032] Figure 2 is a flowchart of a pseudo-label model training method shown in at least one embodiment of the present disclosure;

[0033] Figure 3 A sample image and its corresponding segmentation mask shown in at least one embodiment of the present disclosure;

[0034] Figure 4 A segmentation mask and its corresponding edge shown in at least one embodiment of the present disclosure;

[0035] Figure 5 is a schematic diagram of training a pseudo-label model based on an encoder-decoder network structure according to at least one embodiment of the present disclosure;

[0036] Figure 6 is a flowchart of a pseudo-label generation method according to at least one embodiment of the present disclosure;

[0037] Figure 7 is an unlabeled image and its corresponding pseudo label shown in at least one embodiment of the present disclosure;

[0038] Figure 8 is a flowchart of a training method for a semi-supervised semantic segmentation network according to at least one embodiment of the present disclosure;

[0039] Figure 9 is a block diagram of a pseudo-label model training device according to at least one embodiment of the present disclosure;

[0040] Figure 10 is a block diagram of a training device for a semi-supervised semantic segmentation network according to at least one embodiment of the present disclosure;

[0041] Figure 11 This is a schematic diagram of the hardware structure of an electronic device shown in at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0043] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. As used in this specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0044] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information without departing from the scope of this specification. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0045] Semantic segmentation is the pixel-level classification of an image, labeling each pixel with its corresponding category. Common semantic segmentation methods, such as Figure 1 As shown in Figure 1, the labeled images required for training semantic segmentation networks typically consist of two parts: a three-channel RGB image and the semantic segmentation ground truth labels. The ground truth labels include the segmentation mask of the labeled image. The segmentation mask annotates the category of each pixel in the labeled image and is typically used as supervision information, serving as the predicted output of the semantic segmentation network to supervise its training.

[0046] In semi-supervised semantic segmentation tasks, a large proportion of the image data in the training dataset is unlabeled. Unlabeled images lack true value labels, and only a small portion of labeled images contain true value labels. In view of this, at least one embodiment of the present disclosure provides a training method for a pseudo-label model and a training method for a semi-supervised semantic segmentation network. This method fully utilizes the label information of labeled sample images to assign high-quality pseudo-labels with higher accuracy and reliability to unlabeled images, thereby improving the performance of the entire semantic segmentation task.

[0047] like Figure 2 As shown, Figure 2 The flowchart of a method for training a pseudo-label model according to at least one embodiment of the present disclosure may include the following steps:

[0048] In step 102, label information and edge information of a sample image are obtained.

[0049] In this embodiment, the label information and edge information of the sample image are used as supervisory information to train the pseudo label model.

[0050] The sample image is an image annotated with a true value label. The true value label is a category label pre-annotated for each pixel in the image based on semantic understanding. The true value label can be represented by a segmentation mask. The label information of the true value label includes the segmentation mask information of the sample image, which is used to annotate the category of each pixel in the sample image.

[0051] Specifically, the segmentation mask can be a two-dimensional matrix with the same size as the sample image, and different categories of each pixel are represented by different colors. Figure 3 As shown, Figure 3 On the left is a sample image of a knight on horseback, and on the right is the segmentation mask for the sample image. The mask indicates the three categories of pixels in the sample image: knight, horse, and background. Different categories are represented by different grayscale values, and pixels with the same grayscale value belong to the same category.

[0052] Edge information includes information about the edge of the segmentation mask, and the edge information identifies the position of the real edge in the segmentation mask. On the segmentation mask, there is a dividing line, i.e., an edge, between regions composed of pixels of different categories.

[0053] This embodiment does not limit the specific method of obtaining the label information and edge information of the sample image. For example, it can be obtained by manually annotating the image, or it can be obtained from an existing training data set.

[0054] In one embodiment, the segmentation mask can be processed by performing an image algorithm to obtain edge information of the segmentation mask. To avoid introducing additional training overhead, this step can directly use image graphics methods without training costs to process the segmentation mask to generate edges. The edges are used to provide additional supervision information for the next step.

[0055] In one example, a traditional image processing algorithm can be used, for example, the segmentation mask can be filled graphically to obtain edge information of the segmentation mask. Specifically, a graphics filling method is used to perform image erosion and dilation operations on the segmentation mask through a function algorithm to generate edges. Alternatively, an existing edge extraction model can be used to input the segmentation mask and output the edge. Figure 4 An edge obtained by pattern filling a segmentation mask is shown.

[0056] In step 104, a first classification prediction result and a first edge prediction result of the sample image are obtained.

[0057] Among them, the first classification prediction result and the first edge prediction result are obtained by predicting the pseudo label model to be trained.

[0058] In this embodiment, the pseudo-label model to be trained can be obtained by improving the architecture of the existing semantic segmentation model. The difference is that compared with the output of the existing semantic segmentation model, which is the predicted true value label, that is, the classification prediction result of the sample image, the output of the pseudo-label model is two branches: the first classification prediction result of the sample image and the first edge prediction result.

[0059] This embodiment does not limit the specific network structure of the pseudo-label model. For example, an encoder-decoder network structure can be used, where the encoder can be a neural network of the type VGG (residual neural network) or ResNet (deep convolutional network), and the decoder can semantically project the discriminative features learned by the encoder into the pixel space to obtain dense classification, and can be a network such as DeepLab v3 (deep neural network). For another example, a fully convolutional neural network network structure can be used.

[0060] The following is an example of a pseudo-label model with an encoder-decoder network structure. Figure 5 The training process of the pseudo label model is shown to illustrate the training method of the pseudo label model of this embodiment.

[0061] In this step, the acquisition method can be: input the sample image into the pseudo-label model to be trained, and the pseudo-label model simultaneously performs segmentation prediction and edge prediction, outputting two prediction results, the segmentation prediction obtaining a first classification prediction result of the sample image, and the edge prediction obtaining a first edge prediction result of the sample image. Alternatively, the first classification prediction result and the first edge prediction result obtained by performing the above processing on the sample image by other equipment can be obtained.

[0062] The first classification prediction result includes information of a segmentation mask obtained by predicting the sample image, and the predicted segmentation mask is used to mark the predicted category of each pixel of the sample image. The first edge prediction result includes information of an edge of the segmentation mask obtained by predicting the sample image.

[0063] It should be noted that the position of the edge of the segmentation mask in the first edge prediction result is different from the actual position of the edge of the predicted segmentation mask, and both are predicted based on the sample image.

[0064] In step 106, a first network loss is determined based on the difference between the first classification prediction result and the label information.

[0065] For segmentation prediction, the label information of the true value label is used here to supervise it. The first network loss can be calculated by a loss function. The loss function is used to determine the gap between the first classification prediction result actually output by the network and the label information expected to be output. This embodiment does not limit the specific loss function used. For example, the quantile loss function, the mean square error loss function or the cross entropy loss function can be used. By optimizing the first network loss, the first classification prediction result predicted by the model can be gradually approached to the label information in the true value label.

[0066] In one example, a cross entropy loss function can be used for optimization. The first network loss The calculation formula is as follows:

[0067]

[0068] Among them, P ij (·) represents the probability that the pixel at row i and column j in the sample image belongs to a certain category of the segmentation mask, Y ij The label information representing the true value label of this pixel indicates the category to which the pixel actually belongs. H represents the height of the sample image, and W represents the width of the sample image, both in pixels.

[0069] In step 108, a second network loss is determined based on the difference between the first edge prediction result and the edge information.

[0070] For edge prediction, previously acquired edge information is used here to supervise it, and the edge information contains information about the real edges of the segmentation mask. The second network loss can be calculated by a loss function. The loss function is used to determine the gap between the first edge prediction result actually output by the network and the edge information expected to be output. This embodiment does not limit the specific loss function used. For example, the quantile loss function, the mean square error loss function, or the cross entropy loss function can be used. By optimizing the second network loss, the first edge prediction result predicted by the model can be gradually approached to the expected edge information.

[0071] In one embodiment, for each pixel of the sample image located at the edge of the segmentation mask, a second network loss is determined based on the difference between the first edge prediction result and the edge information of the pixel. In another embodiment, the second network loss may be determined based on the difference between the first edge prediction result and the edge information of other pixels outside the background category of the segmentation mask.

[0072] In one example, a modified version of the cross entropy loss function can be used to supervise it, where the supervision is on the pixels at the edge of the segmentation mask in the sample image. The calculation formula is as follows:

[0073]

[0074] Where H represents the height of the sample image, and W represents the width of the sample image, both in pixels. Unlike formula (1), the categories in formula (2) are not the categories identified by the segmentation mask, but refer to the edge category and the background category. The pixels at the edge position identified in the edge information in the sample image belong to the edge category, and the other pixels outside and inside the edge belong to the background category. This is different from the meaning of the background category in the segmentation mask. Here, C is used. background Represents the background category. ij (·) represents the probability that the pixel at row i and column j in the sample image belongs to the edge category, Y ij Indicates the category to which the pixel actually belongs, as identified by the edge information of the pixel, i.e., edge category or background category. I(·) is the indicator function. When the conditions in the brackets are met, i.e., Y ij The value is 1 if the current pixel is an edge, otherwise it is 0.

[0075] Through this indicator function, we limit the prediction error of the background category to not be calculated, and only focus on the prediction of the edge, avoiding the loss caused by too many pixels participating in the loss calculation. If the value is too large, it will cause the training to be unstable.

[0076] In step 110, the network parameters of the pseudo-label model to be trained are adjusted according to the first network loss and the second loss until the model training end condition is met, thereby obtaining the pseudo-label model.

[0077] In one example, in each round of training, a weighted sum is performed on the first network loss and the second network loss to obtain a total network loss; and network parameters of the pseudo label model are adjusted according to the total network loss.

[0078] For example, the first network loss and the second network loss can be weighted and summed according to a certain weight ratio to obtain a total loss value:

[0079]

[0080] Among them, λ is a coefficient used to balance the sizes of the two.

[0081] By optimizing Minimizing the overall loss enables the pseudo-label model to have preliminary prediction capabilities. For example, by iterating gradient descent through backpropagation, the network parameters of the pseudo-label model can be adjusted.

[0082] When the network iteration end condition is reached, network training ends and a trained pseudo-label model is obtained. The end condition can be that the number of iterations reaches a certain number or the loss value is less than a certain threshold.

[0083] In one example, in each round of training in the first stage of training, the network parameters of the pseudo-label model are adjusted according to the first network loss until the first stage of training is completed to obtain the adjusted pseudo-label model; in each round of training in the second stage of training, the network parameters of the pseudo-label model after the first stage of training are adjusted according to the second network loss until the second stage of training is completed.

[0084] For example, in each round of training in the first training phase, the network parameters are adjusted so that the loss value of the first network gradually decreases until it is lower than a preset loss value or reaches a preset number of iterations, thereby obtaining a pseudo-label model after the first phase of training, and using this pseudo-label model to continue the second phase of training. Then, in each round of training in the second training phase, the network parameters are adjusted so that the loss value of the second network gradually decreases until it is lower than a preset loss value or reaches a preset number of iterations, thereby obtaining a trained pseudo-label model.

[0085] In other examples, the network parameters may be adjusted in the first training phase to gradually reduce the loss value of the second network loss, and then the network parameters may be adjusted in the second training phase to gradually reduce the loss value of the first network loss.

[0086] The technical solution of the embodiment of the present disclosure provides a training method for a pseudo-label model, which uses the label information and edge information of the sample image as supervision information to train a pseudo-label model. The pseudo-label model is used to predict the classification prediction results and the edge prediction results, so that for unlabeled images, the predicted edge prediction results can be used to guide the correction of the predicted classification prediction results, thereby obtaining more reliable pseudo-labels, which can make full use of the information provided by the labels of limited sample images, improve the algorithm accuracy, and save a lot of labeling costs.

[0087] The following describes an application method for obtaining the pseudo label model in the above embodiment.

[0088] like Figure 6 As shown, Figure 6 This is a flowchart of a pseudo-label generation method according to at least one embodiment of the present disclosure. This method requires using a pseudo-label model trained using the training method in the previous embodiment, and may include the following steps:

[0089] In step 202, the unlabeled image is input into a pseudo-label model to obtain a classification prediction result and an edge prediction result of the unlabeled image.

[0090] The unlabeled image is input into the pseudo-label model, and the pseudo-label model can perform segmentation prediction and edge prediction on the unlabeled image, and obtain the classification prediction results and edge prediction results of the labeled image respectively.

[0091] The classification prediction result includes information about the predicted pseudo-segmentation mask for the unlabeled image. The pseudo-segmentation mask is used to label the category of each pixel in the unlabeled image. The edge prediction result includes information about the pseudo-edges of the predicted segmentation mask. It should be noted that the locations of the pseudo-edges of the segmentation mask in the edge prediction result are different from the locations of the edges of the predicted pseudo-segmentation mask; both are predicted based on the unlabeled image.

[0092] In step 204, the classification prediction result is corrected according to the edge prediction result to obtain pseudo label information of the unlabeled image.

[0093] The classification prediction results output by the pseudo-label model are not reliable enough. Here we use the classification prediction results to correct them, optimize and filter out unreasonable boundary areas, and obtain optimized pseudo-labels. Auxiliary edge guidance as additional supervision information can significantly improve the accuracy and reliability of pseudo-labels.

[0094] The edge of the pseudo segmentation mask is not consistent with the predicted pseudo edge. In this step, the information provided by the two can be combined for optimization.

[0095] For example, the portion of the pseudo segmentation mask of the classification prediction result that exceeds the pseudo edge of the edge prediction result is removed to obtain a modified pseudo segmentation mask, and the segmentation prediction is limited to the range of the edge prediction. Figure 7 The unlabeled image and optimized pseudo-label are shown. The unlabeled image on the left is an image of an airplane. In this figure, there is a white gap between the pseudo-edge and the pseudo-segmentation mask. When using the edge prediction result to correct the classification prediction result, the area within the pseudo-edge is retained, and the pseudo-label is closer to the shape of the airplane in the unlabeled image.

[0096] After obtaining the pseudo-labels, the pseudo-labels of the unlabeled images can be used as supervisory information to train the neural network model, treating the pseudo-labels of the unlabeled images as equivalent to the true labels of the labeled images. Generally speaking, unlabeled images are easier and cheaper to obtain than labeled images, allowing for a large number of training samples.

[0097] In one example, when training a neural network model, the optimization objective or loss function can be:

[0098]

[0099] in, Represents the loss value of the unlabeled image, P ij (·) represents the probability that the pixel at row i and column j in the unlabeled image belongs to a category of the segmentation mask, Y ij Indicates the category to which the pixel actually belongs, as identified by the pseudo-label information of this pixel. H represents the height of the unlabeled image, and W represents the width of the unlabeled image, both in pixels.

[0100] When actually training a neural network model, unlabeled images and labeled sample images can be mixed together for training.

[0101] The pseudo-label generation method provided by the technical solution of the embodiment of the present disclosure predicts the unlabeled image through the pseudo-label model to obtain classification prediction results and edge prediction results, and uses the edge prediction results to guide the correction of the classification prediction results, thereby obtaining more reliable pseudo-labels, giving the unlabeled image high-quality pseudo-labels with higher accuracy and reliability, thereby improving the performance of the entire semantic segmentation task.

[0102] The complete training method of the semantic segmentation network based on semi-supervised learning is described below.

[0103] Figure 8 A flowchart of a training method for a semi-supervised semantic segmentation network is shown, wherein a training sample set of the semantic segmentation network includes: a plurality of unlabeled images.

[0104] For semi-supervised learning, only a small number of sample images in the training sample set contain true value labels, while a large proportion of unlabeled images lack true value labels. In order to make full use of the small number of sample images, this embodiment uses edge guidance obtained from the true value labels as additional auxiliary supervision information to improve the performance of the entire semantic segmentation task.

[0105] The method may include the following steps:

[0106] In step 302, a pseudo label model trained by the pseudo label model training method of the above embodiment is obtained.

[0107] Among them, the pseudo-label model is used to predict the label information and edge information of the image.

[0108] The pseudo-label model can be obtained by improving the semantic segmentation network that is desired to be trained in this embodiment. Of course, other network models can also be used instead of this improvement method. For example, for a semantic segmentation network with an encoder-decoder network structure, a binary classification structure can be added to the output of the original decoder, thereby adding a branch that outputs edge information instead of the previous one that only outputs label information.

[0109] During the pseudo-label model training process in the above embodiment, both label information and edge information are used for supervision, and the network parameters of the pseudo-label model are continuously characterized, so that the first classification prediction result predicted by the pseudo-label model is increasingly close to the label information, and the first edge prediction result is increasingly close to the edge information. When the training conditions are met, the pseudo-label model is obtained, and the weight parameters of each layer of the pseudo-label model will be fixed and no longer changed.

[0110] In step 304, the unlabeled image is input into the pseudo-label model to obtain a corresponding second classification prediction result and a second edge prediction result.

[0111] Input the unlabeled image into the pseudo-label model obtained in the previous step, and output the second classification prediction result obtained by segmentation prediction and the second edge prediction result obtained by edge prediction.

[0112] The second classification prediction result includes information about a predicted pseudo segmentation mask of the unlabeled image, where the pseudo segmentation mask is used to label the category to which each pixel in the unlabeled image belongs. The second edge prediction result includes information about a predicted pseudo edge of the segmentation mask.

[0113] In step 306, the second classification prediction result is corrected according to the second edge prediction result to obtain pseudo label information corresponding to the unlabeled image.

[0114] The second classification prediction result output by the pseudo-label model is not reliable enough. This step uses the second classification prediction result to correct it, optimize and filter unreasonable boundary areas, and obtain optimized pseudo-label information. The auxiliary edge guidance serves as additional supervision information, which can significantly improve the accuracy and reliability of the pseudo-label.

[0115] The edges of the pseudo-segmentation mask and the predicted pseudo-edges are not consistent. In this step, we can combine the information provided by both to optimize the results. For example, we can remove the portion of the pseudo-segmentation mask from the classification prediction that exceeds the pseudo-edge in the edge prediction result, obtain a corrected pseudo-segmentation mask, and limit the segmentation prediction to the range of the edge prediction.

[0116] By combining the second edge prediction results with the second classification prediction results, the boundary areas of the unreasonable segmentation mask are optimized and filtered to obtain optimized pseudo-labels. This allows a large number of unlabeled images to obtain their corresponding more reliable pseudo-label information.

[0117] In step 308, the semantic segmentation network is trained based on the unlabeled image and its corresponding pseudo-label information.

[0118] The optimized pseudo labels are regarded as supervision information for unlabeled images for training.

[0119] For example, an unlabeled image is input into the semantic segmentation network to be trained to obtain a predicted segmentation result. The semantic segmentation network used in this embodiment can be selected by a person skilled in the art according to actual needs and is not limited here. According to the difference between the predicted segmentation result and the pseudo-label information, the network parameters of the semantic segmentation network are adjusted, and the difference between the predicted segmentation result and the pseudo-label information is calculated to obtain the network loss. The network parameters of the semantic segmentation network are continuously adjusted to make the network loss smaller and smaller until the end condition is reached. This process can make the predicted segmentation result predicted by the semantic segmentation network gradually close to the pseudo-label information.

[0120] The trained semantic segmentation network can be applied to any scenario where segmentation tasks exist, helping to improve the accuracy of the semantic segmentation algorithm while significantly saving labeling costs.

[0121] For example, in scenarios such as high-speed rail inspection and automobile quality inspection, the method of the disclosed embodiment can be used to train defect detection algorithms such as nut loosening detection and automobile stamping quality inspection, thereby improving the accuracy of the defect detection algorithm at a low cost.

[0122] like Figure 9 As shown, Figure 9 is a block diagram of a pseudo-label model training apparatus according to at least one embodiment of the present disclosure, the apparatus comprising:

[0123] The training data acquisition module 41 is used to: obtain label information and edge information of the sample image, wherein the label information includes information of the segmentation mask of the sample image, and the edge information includes information of the edge of the segmentation mask, and the segmentation mask is used to mark the category to which each pixel in the sample image belongs; obtain a first classification prediction result and a first edge prediction result of the sample image, wherein the first classification prediction result and the first edge prediction result are predicted by the pseudo-label model to be trained.

[0124] The network loss determination module 42 is configured to: determine a first network loss based on a difference between the first classification prediction result and the label information; and determine a second network loss based on a difference between the first edge prediction result and the edge information.

[0125] The network parameter adjustment module 43 is used to adjust the network parameters of the pseudo-label model to be trained according to the first network loss and the second loss until the model training end condition is met, thereby obtaining the pseudo-label model.

[0126] In some optional embodiments, the training data acquisition module 41 is further configured to fill the segmentation mask according to a pattern to obtain edge information of the segmentation mask.

[0127] In some optional embodiments, the network loss determination module 42, when used to determine the second network loss based on the difference between the first edge prediction result and the edge information, is specifically used to: for each pixel of the labeled image located at the edge of the segmentation mask, determine the second network loss based on the difference between the first edge prediction result and the edge information of the pixel.

[0128] In some optional embodiments, the first classification prediction result includes the predicted probability of each pixel in the labeled image belonging to each category in the category set; the network loss determination module 42, when used to determine the first network loss based on the difference between the first classification prediction result and the label information, is specifically used to: for each pixel in the labeled image, calculate the first network loss based on the category of the pixel marked in the label information and the probability of the pixel belonging to the category included in the first classification prediction result.

[0129] In some optional embodiments, the first edge prediction result includes a predicted probability of each pixel in the labeled image belonging to an edge category or a background category; the network loss determination module 42, when used to determine the second network loss based on the difference between the first edge prediction result and the edge information, is specifically used to: for each pixel in the labeled image, calculate the second network loss based on the edge category or background category to which the pixel belongs as marked by the edge information, and the probability of the pixel belonging to the edge category or background category included in the first edge prediction result.

[0130] In some optional embodiments, the network parameter adjustment module 43, when used to adjust the network parameters of the pseudo label model according to the first network loss and the second loss, is specifically used to: in each round of training, perform weighted summation of the first network loss and the second network loss to obtain a total network loss; adjust the network parameters of the pseudo label model according to the total network loss; or, in each round of training in the first stage of training, adjust the network parameters of the pseudo label model according to the first network loss until the first stage of training is completed and the adjusted pseudo label model is obtained; in each round of training in the second stage of training, adjust the network parameters of the pseudo label model after the first stage of training according to the second network loss until the second stage of training is completed.

[0131] like Figure 10 As shown, Figure 10 This is a block diagram of a training device for a semi-supervised semantic segmentation network according to at least one embodiment of the present disclosure, wherein a training sample set of the semantic segmentation network includes: a plurality of unlabeled images; the device includes:

[0132] The model acquisition module 51 is used to obtain a pseudo label model trained by the above-mentioned pseudo label model training method.

[0133] The model prediction module 52 is used to input the unlabeled image into the pseudo-label model to obtain a corresponding second prediction result and a second edge prediction result.

[0134] The label correction module 53 is configured to correct the second classification prediction result according to the second edge prediction result to obtain pseudo label information corresponding to the unlabeled image.

[0135] The network training module 54 is used to train the semantic segmentation network according to the unlabeled image and its corresponding pseudo-label information.

[0136] In some optional embodiments, the network training module 54, when used to train the semantic segmentation network based on the unlabeled image and its corresponding pseudo-label information, is specifically used to: input the unlabeled image into the semantic segmentation network to be trained to obtain a predicted segmentation result; and adjust the network parameters of the semantic segmentation network based on the difference between the predicted segmentation result and the pseudo-label information.

[0137] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0138] The present disclosure also provides an electronic device, such as Figure 11 As shown, the electronic device includes a memory 11 and a processor 12, wherein the memory 11 is used to store computer instructions that can be executed on the processor, and the processor 12 is used to implement the training method of the pseudo-label model or the training method of the semi-supervised semantic segmentation network described in any embodiment of the present disclosure when executing the computer instructions.

[0139] An embodiment of the present disclosure also provides a computer program product, which includes a computer program / instruction, which, when executed by a processor, implements the training method of the pseudo-label model or the training method of the semi-supervised semantic segmentation network described in any embodiment of the present disclosure.

[0140] An embodiment of the present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the pseudo-label model or the training method of the semi-supervised semantic segmentation network described in any embodiment of the present disclosure.

[0141] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0142] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] Other embodiments of the present invention will readily occur to those skilled in the art upon consideration of the present invention and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present invention being indicated by the following claims.

[0144] It should be understood that the present description is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0145] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A training method for a pseudo-label model, characterized in that: The method comprises: Obtaining label information and edge information of a sample image, wherein the label information includes information of a segmentation mask of the sample image, and the edge information includes the position of an edge of the segmentation mask, wherein the segmentation mask is used to mark the category to which each pixel in the sample image belongs; Obtaining a first classification prediction result and a first edge prediction result of the sample image, where the first classification prediction result and the first edge prediction result are obtained by predicting a pseudo label model to be trained; determining a first network loss based on a difference between the first classification prediction result and the label information; determining a second network loss based on a difference between the first edge prediction result and the edge information; The network parameters of the pseudo-label model to be trained are adjusted according to the first network loss and the second network loss until a model training end condition is reached, thereby obtaining the pseudo-label model.

2. The method according to claim 1, characterized in that The method further comprises: The segmentation mask is filled according to a pattern to obtain edge information of the segmentation mask.

3. The method according to claim 1, characterized in that The determining a second network loss according to a difference between the first edge prediction result and the edge information includes: For pixels located at the edge of the segmentation mask among the pixels of the sample image, a second network loss is determined according to a difference between the first edge prediction result and the edge information of the pixel.

4. The method according to claim 1, wherein The first classification prediction result includes the predicted probability of each pixel in the sample image belonging to each category in the category set; The determining a first network loss according to a difference between the first classification prediction result and the label information includes: For each pixel in the sample image, a first network loss is calculated based on the category of the pixel marked in the label information and the probability that the pixel belongs to the category included in the first classification prediction result.

5. The method according to claim 1, wherein The first edge prediction result includes the predicted probability of each pixel in the sample image belonging to the edge category or the background category; The determining a second network loss according to a difference between the first edge prediction result and the edge information includes: For each pixel in the sample image, a second network loss is calculated based on the edge category or background category to which the pixel belongs as marked by the edge information, and the probability that the pixel belongs to the edge category or background category contained in the first edge prediction result.

6. The method according to claim 1, characterized in that Adjusting the network parameters of the pseudo label model according to the first network loss and the second network loss includes: In each round of training, the first network loss and the second network loss are weighted summed to obtain the total network loss; Adjusting the network parameters of the pseudo-label model according to the total network loss; Alternatively, in each round of training in the first stage of training, the network parameters of the pseudo label model are adjusted according to the first network loss until the first stage of training is completed to obtain an adjusted pseudo label model; In each round of training in the second stage of training, the network parameters of the pseudo label model after the first stage of training are adjusted according to the second network loss until the second stage of training is completed.

7. A training method for a semi-supervised semantic segmentation network, characterized in that: The training sample set of the semantic segmentation network includes: multiple unlabeled images; The method comprises: Obtaining a pseudo label model trained by the pseudo label model training method according to any one of claims 1 to 6; Inputting the unlabeled image into the pseudo-label model to obtain a corresponding second classification prediction result and a second edge prediction result; Correcting the second classification prediction result according to the second edge prediction result to obtain pseudo label information corresponding to the unlabeled image; The semantic segmentation network is trained according to the unlabeled image and its corresponding pseudo-label information.

8. The method according to claim 7, characterized in that The step of training the semantic segmentation network according to the unlabeled image and its corresponding pseudo-label information includes: Inputting the unlabeled image into the semantic segmentation network to be trained to obtain a predicted segmentation result; According to the difference between the predicted segmentation result and the pseudo label information, the network parameters of the semantic segmentation network are adjusted.

9. A training device for a pseudo-label model, characterized in that: The device comprises: a training data acquisition module, configured to: acquire label information and edge information of a sample image, wherein the label information includes information of a segmentation mask of the sample image, the edge information includes the position of an edge of the segmentation mask, and the segmentation mask is used to mark the category to which each pixel in the sample image belongs; and acquire a first classification prediction result and a first edge prediction result of the sample image, wherein the first classification prediction result and the first edge prediction result are predicted by a pseudo-label model to be trained; a network loss determination module, configured to: determine a first network loss based on a difference between the first classification prediction result and the label information; and determine a second network loss based on a difference between the first edge prediction result and the edge information; A network parameter adjustment module is used to adjust the network parameters of the pseudo-label model to be trained according to the first network loss and the second network loss until the model training end condition is met, thereby obtaining the pseudo-label model.

10. A training device for a semi-supervised semantic segmentation network, characterized in that: The training sample set of the semantic segmentation network includes: multiple unlabeled images; The device comprises: A model acquisition module, configured to: acquire a pseudo-label model trained by the pseudo-label model training method according to any one of claims 1 to 6; A model prediction module is used to: input the unlabeled image into the pseudo-label model to obtain a corresponding second classification prediction result and a second edge prediction result; a label correction module, configured to correct the second classification prediction result according to the second edge prediction result to obtain pseudo label information corresponding to the unlabeled image; The network training module is used to train the semantic segmentation network according to the unlabeled image and its corresponding pseudo-label information.

11. An electronic device, characterized in that: The device includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the method according to any one of claims 1 to 6, or implement the method according to any one of claims 7 to 8 when executing the computer instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 8 is implemented.

Images