A network training method based on affinity coefficient

Abstract

An embodiment of the present disclosure provides a network training method based on an affinity coefficient. The method includes: obtaining the ground truth label and the ground truth affinity coefficient of a labeled image, where the ground truth label is used to label the category to which each pixel in the labeled image belongs, and the ground truth affinity coefficient is used to represent the similarity between image patches composed of different pixels in the labeled image; obtaining the prediction result and the predicted affinity coefficient of the labeled image; obtaining the predicted pseudo-label and the pseudo-label of the unlabeled image; adjusting the network parameters of the student network based on at least two of the first network loss, the second network loss, and the third network loss, and obtaining the student network when the training end condition is reached. This method can effectively utilize a large number of unlabeled images as a supplement to labeled images, improve the accuracy of the trained model, save computing resources, and greatly save the annotation cost.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and particularly relates to a network training method based on affinity coefficients. Background Art

[0002] With the development of deep learning, semantic segmentation tasks have been widely studied. For traditional supervised learning-based semantic segmentation tasks, the training data used is labeled images including ground truth labels, and the ground truth labels are used as supervision information to train a semantic segmentation network. Although the model trained by this method has high accuracy, it depends on a large number of high-quality pixel-level annotated labeled images.

[0003] For semi-supervised semantic segmentation, the training data set contains a large number of unlabeled images and a small number of labeled images with annotations. The common approach is to first train a model using the labeled images with annotations, and then use this model to predict the annotations of the unlabeled images and select highly credible predictions as pseudo-labels. Although this method alleviates the dependence of semantic segmentation tasks on high-quality annotations, it faces the problem of low accuracy of the trained model due to the unreliability of the pseudo-labels. Summary of the Invention

[0004] In view of this, embodiments of the present disclosure provide at least one network training method based on affinity coefficients.

[0005] Specifically, the embodiments of the present disclosure are implemented through the following technical solutions:

[0006] In a first aspect, a network training method based on affinity coefficients is provided, and the method includes:

[0007] Obtain the ground truth labels and ground truth affinity coefficients of the labeled images, where the ground truth labels are used to label the categories to which each pixel in the labeled image belongs, and the ground truth affinity coefficients are used to represent the similarity between image patches composed of different pixels in the labeled image;

[0008] Obtain the prediction results and prediction affinity coefficients of the labeled images, where the prediction results and prediction affinity coefficients are obtained by inputting the labeled images into the student network to be trained;

[0009] Obtain the predicted pseudo-labels and pseudo-labels of the unlabeled images, where the predicted pseudo-labels are obtained by inputting the unlabeled images into the student network to be trained, and the pseudo-labels are obtained by inputting the unlabeled images into the teacher network;

[0010] Adjust the network parameters of the student network based on at least two of the first network loss, the second network loss, and the third network loss, where the first network loss is obtained according to the difference between the prediction result and the true label, the second network loss is obtained according to the difference between the predicted affinity coefficient and the true affinity coefficient, and the third network loss is obtained according to the difference between the predicted pseudo-label and the pseudo-label; when the training end condition is reached, obtain the student network.

[0011] In a second aspect, a training method for a pseudo-label model based on an affinity coefficient is provided. The method includes:

[0012] Obtain the true label and the true affinity coefficient of the labeled image. The true label is used to label the category to which each pixel in the labeled image belongs, and the true affinity coefficient is used to represent the similarity between image patches composed of different pixels in the labeled image;

[0013] Input the labeled image into the pseudo-label model to be trained to obtain the predicted label and the predicted affinity coefficient of the labeled image;

[0014] Determine the first model loss according to the difference between the predicted label and the true label;

[0015] Determine the second model loss according to the difference between the predicted affinity coefficient and the true affinity coefficient;

[0016] Adjust the network parameters of the pseudo-label model according to the first model loss and the second model loss. When the model training end condition is reached, obtain the pseudo-label model.

[0017] In a third aspect, a network training device based on an affinity coefficient is provided. The device includes:

[0018] A training data acquisition module, configured to: obtain the true label and the true affinity coefficient of the labeled image. The true label is used to label the category to which each pixel in the labeled image belongs, and the true affinity coefficient is used to represent the similarity between image patches composed of different pixels in the labeled image; obtain the first prediction result and the predicted affinity coefficient of the labeled image, where the first prediction result and the predicted affinity coefficient are predicted by a student network to be trained; obtain the second prediction result and the pseudo-label of the unlabeled image, where the second prediction result is predicted by the student network to be trained, and the pseudo-label is predicted by a teacher network;

[0019] A network parameter adjustment module, configured to: adjust the network parameters of the student network based on at least two of the first network loss, the second network loss, and the third network loss;

[0020] Among them, the first network loss is obtained according to the difference between the first prediction result and the true value label, the second network loss is obtained according to the difference between the predicted affinity coefficient and the true value affinity coefficient, and the third network loss is obtained according to the difference between the second prediction result and the pseudo label.

[0021] Fourthly, a training device for a pseudo-label model based on an affinity coefficient is provided. The device includes:

[0022] A supervision information acquisition module, configured to: acquire the true value label and the true value affinity coefficient of a labeled image, where the true value label is used to label the category to which each pixel in the labeled image belongs, and the true value affinity coefficient is used to represent the similarity between image patches composed of different pixels in the labeled image;

[0023] A labeled image prediction module, configured to: input the labeled image into a pseudo-label model to be trained, and obtain the predicted label and the predicted affinity coefficient of the labeled image;

[0024] A model loss determination module, configured to: determine a first model loss according to the difference between the predicted label and the true value label; determine a second model loss according to the difference between the predicted affinity coefficient and the true value affinity coefficient;

[0025] A model parameter adjustment module, configured to adjust the network parameters of the pseudo-label model according to the first model loss and the second model loss, and obtain the pseudo-label model when the model training end condition is reached.

[0026] Fifthly, an electronic device is provided. The device includes a memory and a processor. The memory is used to store computer instructions that can be run on the processor, and the processor is configured to implement the network training method based on an affinity coefficient or the training method of a pseudo-label model based on an affinity coefficient according to any embodiment of the present disclosure when executing the computer instructions.

[0027] Sixthly, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it implements the network training method based on an affinity coefficient or the training method of a pseudo-label model based on an affinity coefficient according to any embodiment of the present disclosure.

[0028] The network training method based on affinity coefficient provided by the embodiments of the present disclosure, when performing semi-supervised learning, in addition to using the true labels as supervision information, also uses the affinity coefficients between image patches composed of different pixels in the labeled images for additional supervision to improve the generalization ability of the network; at the same time, using the information of labeled images and unlabeled images for end-to-end training can effectively utilize a large number of unlabeled images as a supplement to the labeled images, improve the accuracy of the trained model, save computing resources and significantly save the annotation cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To more clearly illustrate the technical solutions in one or more embodiments of the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments recorded in one or more embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0030] Figure 1 is a schematic diagram of a semantic segmentation network training method shown in at least one embodiment of the present disclosure;

[0031] Figure 2 is a flowchart of a network training method based on affinity coefficient shown in at least one embodiment of the present disclosure;

[0032] Figure 3 is a labeled image and its corresponding true label shown in at least one embodiment of the present disclosure;

[0033] Figure 4 is a visualization image of the affinity coefficient of a patch in a labeled image shown in at least one embodiment of the present disclosure;

[0034] Figure 5 is a training schematic diagram of a labeled image shown in at least one embodiment of the present disclosure;

[0035] Figure 6 is a training schematic diagram of an unlabeled image shown in at least one embodiment of the present disclosure;

[0036] Figure 7 is a flowchart of a training method of a pseudo-label model based on affinity coefficient shown in at least one embodiment of the present disclosure;

[0037] Figure 8 is a block diagram of a network training device based on affinity coefficient shown in at least one embodiment of the present disclosure;

[0038] Figure 9It is a block diagram of a training device for a pseudo-label model based on affinity coefficients shown in at least one embodiment of the present disclosure;

[0039] Figure 10 It is a schematic diagram of the hardware structure of an electronic device shown in at least one embodiment of the present disclosure. Detailed implementation manners

[0040] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0041] The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit this specification. The singular forms "a", "the", and "said" used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0042] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0043] Semantic segmentation is to classify an image at the pixel level and label each pixel with its corresponding category. Past semantic segmentation methods, for example, such as Figure 1 As shown, the labeled images required for training a semantic segmentation network usually consist of the following two parts: a three-channel RGB image and the ground truth label of the semantic segmentation. The ground truth label annotates the category of each pixel on the labeled image and is usually used as supervision information, that is, as the prediction result that the semantic segmentation network is expected to output, to supervise the training of the semantic segmentation network. In the figure, represents the loss function calculated from the ground truth label and the actual prediction result, and the dashed line represents the backpropagation of the gradient of this loss function.

[0044] In the semi-supervised semantic segmentation task, a large proportion of the image data in the training dataset is unlabeled images. The unlabeled images lack the component of ground truth labels, and only a small portion of the labeled images contain ground truth labels. For traditional semi-supervised semantic segmentation algorithms, the key lies in assigning pseudo-labels to each pixel of the unlabeled images. However, there are a certain number of unreliable pseudo-labels in the pseudo-labels obtained by traditional methods, which makes the semantic segmentation algorithm based on semi-supervised learning restricted by the existence of low-quality pseudo-labels and results in low accuracy.

[0045] In view of this, at least one embodiment of the present disclosure provides a network training method based on affinity coefficients, which introduces affinity coefficients for additional supervision to make full use of the information in the ground truth labels of labeled images, improve the accuracy of the trained network, and perform end-to-end training using both labeled images and unlabeled images during training, reducing the accumulation of errors while saving computational effort, thereby improving the performance of the entire semantic segmentation task.

[0046] As Figure 2 shown, Figure 2 FIG. is a flowchart of a network training method based on affinity coefficients shown in at least one embodiment of the present disclosure. The method may include the following steps:

[0047] In step 102, obtain the ground truth labels and ground truth affinity coefficients of the labeled images.

[0048] In this embodiment, different processing methods are adopted for labeled images and unlabeled images.

[0049] For labeled images, in this step, the already annotated ground truth labels of the labeled images can be obtained, and the ground truth affinity coefficients in the labeled images can be obtained according to the ground truth labels as supervision information.

[0050] Among them, the ground truth labels are used to annotate the category to which each pixel in the labeled image belongs. The ground truth labels are category labels pre-annotated for each pixel in the image based on semantic understanding and can be represented by a segmentation mask image. The segmentation mask image can be a two-dimensional matrix with the same size as the labeled image, and different colors are used to represent different categories to which each pixel belongs.

[0051] Exemplarily, as Figure 3 shown, Figure 3 In the left figure in, the content is a labeled image of a knight riding a horse, and the right figure is the ground truth label of the labeled image. The ground truth label annotates three categories of pixels in the labeled image: the knight category, the horse category, and the background category. Among them, different categories are represented by different gray values, and pixels with the same gray value belong to the same category.

[0052] The true affinity coefficient is used to represent the similarity between patches composed of different pixels in a labeled image. A labeled image can be considered to consist of multiple patches, and each patch contains multiple pixels. The distribution of true labels in each patch may vary. Here, we can set the pixel ratio of pixels belonging to different classes in a patch as the attribute of the patch, and can measure the similarity between any two patches based on the above attributes of the patch. The similarity between every two patches in a labeled image constitutes the true affinity coefficient of the labeled image.

[0053] This embodiment does not limit the method for measuring the similarity between two patches. For example, the Euclidean distance, Manhattan distance, or Pearson correlation coefficient between the pixel ratios of two patches can be used to calculate the similarity.

[0054] In one example, each patch contains the same number of pixels, and the similarity between two patches can be represented by the cosine similarity between the pixel ratios of the patches, that is, the similarity is obtained by calculating the cosine value of the vector between the pixel ratios. Specifically, it includes: determining the pixel ratio of pixels belonging to different classes in each patch according to the true labels of the labeled image; obtaining the true affinity coefficient according to the cosine similarity between the pixel ratios of every two patches in the labeled image.

[0055] In this example, a patch corresponds to 64 pixels (8*8) of the labeled image. The size of the patch is related to the feature size of the predicted affinity coefficient output by the student network. Here, the predicted affinity coefficient output by the student network we set is the feature after 8-fold downsampling. Correspondingly, when calculating the true affinity coefficient, the patch used is 8*8 in size. In other examples, the number of pixels contained in the patch can also be set to other values according to the actual output of the network.

[0056] For ease of understanding, coordinates are used here to represent different pixels and different patches. In a labeled image, the pixel at the coordinate position (1, 1) belongs to patch (1, 1); the pixel at the coordinate position (9, 9) belongs to patch (2, 2), and so on. For a patch with coordinates (k, l), we can calculate the distribution of true labels within this patch, that is, the pixel ratio of the patch mentioned above:

[0057]

[0058] where D kl(c) represents the proportion of pixels belonging to class c in the patch with coordinates (k, l) among the total number of pixels in this patch. Here, we set the total number of classes in the training dataset to C, and class c can be any one of these classes. Y ij represents the actual class to which this pixel belongs as identified by its ground-truth label. II[·] is the indicator function, when the condition inside the brackets is satisfied, that is, when Y ij represents 1 when the current pixel belongs to class c, and 0 otherwise.

[0059] Then, we define the ground-truth affinity coefficient between two patches with coordinates (k 1 , l 1 ) and (k 2 , l 2 ) as follows:

[0060]

[0061] where, represents and 's cosine similarity, and the calculation method is as follows:

[0062]

[0063] where, (k 1 , l 1 ) and (k 2 , l 2 ) respectively represent the coordinates of any two different patches in a labeled image.

[0064] As Figure 4 shown, Figure 4 visualizes the affinity coefficients between a patch in the labeled image and all other patches. This patch is represented by a small white square, and the dark area around the small square represents the area where the patches with high affinity coefficients to this patch are located.

[0065] In step 104, obtain the first prediction result and the predicted affinity coefficient of the labeled image.

[0066] Among them, the first prediction result and the predicted affinity coefficient are predicted by the student network to be trained.

[0067] In this embodiment, the student network used can be considered as an architecture improvement of the existing semantic segmentation model. The difference is that, compared with the output of the existing semantic segmentation model being the predicted segmentation result, that is, the prediction of the ground-truth label, the output of the student network has two branches: the segmentation prediction result (i.e., the first prediction result) P and the high-level feature z.

[0068] Among them, the segmentation prediction result P is the predicted label of the labeled image, and the high-level feature z is the prediction of the distribution of the true labels of each patch in the labeled image, or the prediction of the pixel ratio. The predicted affinity coefficient is obtained by calculating the similarity between every two image patches according to the high-level feature z. Here, the calculation method of the predicted affinity coefficient is the same as that of the aforementioned true affinity coefficient, and will not be elaborated here.

[0069] This embodiment does not limit the specific network structure of the student network.

[0070] For example, an encoder-decoder network structure can be adopted. Among them, the encoder can be a neural network of types such as VGG (residual neural network), ResNet (deep convolutional network), etc., and the decoder can be a network such as DeepLab v3 (deep neural network). This decoder has two branches, which respectively output the segmentation prediction result and the high-level feature. Another example is that a fully convolutional neural network structure can be adopted, and there are also two branches at the end of the network.

[0071] In step 106, obtain the second prediction result and the pseudo-label of the unlabeled image.

[0072] Among them, the second prediction result is predicted by the student network to be trained, and the pseudo-label is predicted by the teacher network.

[0073] In this embodiment, the teacher network can be a network with the same structure as the student network, or it can also be a network with a structure that is not completely the same as the student network. This will not be elaborated here for the time being, and reference can be made to the description in the following embodiments.

[0074] The teacher network can perform semantic segmentation prediction on the unlabeled image to obtain a semantic segmentation map, that is, a pseudo-label. Here, we use the prediction result P of the teacher network on the unlabeled image 1 as the supervision information during the training of the student network, that is, the pseudo-label.

[0075] During the entire training process, we use the supervision information to supervise the second prediction result output by the student network, and then continuously optimize the network parameters of the student network. At the same time, we use the optimized network parameters of the student network to update the network parameters of the teacher network. The speeds at which the two adjust the network parameters are different.

[0076] In one implementation manner, in order to further improve the quality of the pseudo-labels inferred by the model on the unlabeled image, a pixel screening mechanism can also be introduced in this step. After screening and correcting the prediction result output by the teacher network, the pseudo-label of the unlabeled image is obtained. Specifically, it includes:

[0077] Input the unlabeled image into the teacher network to output the third prediction result of the unlabeled image. The third prediction result includes the probability that each pixel in the unlabeled image belongs to each category in the category set. For each pixel in the unlabeled image, calculate the entropy of the pixel according to the probability that the pixel belongs to each category in the category set in the third prediction result. In response to the entropy of the pixel meeting the reliability requirement, determine that the pixel is a reliable pixel. Obtain the pseudo-label of the unlabeled image according to the category corresponding to the determined reliable pixel.

[0078] After inputting the unlabeled image into the teacher network, the teacher network generates a third prediction result on the unlabeled image, that is, a pseudo-label with unreliable quality. The prediction result of each pixel can be measured by entropy. The lower the entropy, the higher the reliability, and the higher the entropy, the lower the reliability. For the pixel with coordinates (i, j), the calculation formula of the entropy of this pixel is as follows:

[0079]

[0080] where P ij represents the probability that the pixel in the i-th row and j-th column belongs to the category c in the category set C.

[0081] The reliability requirement is a requirement for the value of entropy. For example, pixels with entropy exceeding 20% can be considered unreliable pixels and should not supervise the student network. Pixels with entropy not exceeding 20% are considered reliable pixels. All reliable pixels form the pseudo-label of the unlabeled image, and each pixel in the pseudo-label contains its corresponding category.

[0082] In this step, when using the student network to predict the unlabeled image, only the second prediction result output by the branch for segmentation prediction of the student network can be used, and the branch for outputting high-level features is not used.

[0083] In step 108, based on at least two of the first network loss, the second network loss, and the third network loss, adjust the network parameters of the student network.

[0084] Among them, the first network loss is obtained according to the difference between the first prediction result and the true value label, the second network loss is obtained according to the difference between the predicted affinity coefficient and the true value affinity coefficient, and the third network loss is obtained according to the difference between the second prediction result and the pseudo-label.

[0085] In this embodiment, the first network loss can be determined according to the difference between the first prediction result and the true value label. For the first prediction result output by the student network, we use the true value label as its optimization target. The first network loss can be calculated by a loss function. The loss function is used to determine the gap between the actually output first prediction result of the network and the expected true value label. This embodiment does not limit the specific loss function to be used. For example, a quantile loss function, a mean squared error loss function, or a cross-entropy loss function can be used. By optimizing the first network loss, the first prediction result predicted by the model can be gradually approximated to the true value label.

[0086] In one example, the classical cross-entropy loss function can be used for optimization, and the first network loss is calculated as follows:

[0087]

[0088] where P ij (·) represents the probability that the pixel at the i-th row and j-th column of the labeled image belongs to a certain category, and Y ij represents the category to which the true value label of this pixel belongs. H represents the height of the labeled image, and W represents the width of the labeled image, both in pixels.

[0089] In this embodiment, the second network loss can be determined according to the difference between the predicted affinity coefficient and the true value affinity coefficient. For the predicted affinity coefficient calculated based on the high-level feature z output by the student network, we use the true value affinity coefficient as its optimization target. The second network loss can be calculated by a loss function. The loss function is used to determine the gap between the actually obtained predicted affinity coefficient and the expected true value affinity coefficient. This embodiment does not limit the specific loss function to be used. For example, a quantile loss function, a mean squared error loss function, or a cross-entropy loss function can be used. By optimizing the second network loss, the predicted affinity coefficient predicted by the model can be gradually approximated to the expected true value affinity coefficient.

[0090] In one example, we use the mean squared error loss function for optimization. When the cosine similarity is used to represent the true value affinity coefficient and the patch used is 8*8 in size, the first network loss is calculated as follows:

[0091]

[0092] where k 1 , l 1 and k 2 , l 2 respectively represent the coordinates of two different patches in a labeled image. and respectively represent the high-level features of two patches with coordinates (k 1 , l 1 ) and (k 2 , l 2 ). represents the predicted affinity coefficient between two patches with calculated coordinates (k 1 , l 1 ) and (k 2 , l 2 ). is the ground-truth affinity coefficient between two patches with coordinates (k 1 , l 1 ) and (k 2 , l 2 ). H represents the height of the labeled image, and W represents the width of the labeled image, both in pixels.

[0093] In this embodiment, the third network loss can be determined according to the difference between the second prediction result and the pseudo-label. For unlabeled images, the pseudo-label obtained through the teacher network is used as the optimization target for the second prediction result output by the student network. The third network loss can be calculated through a loss function. The loss function is used to determine the gap between the second prediction result and the pseudo-label. In this embodiment, there is no limitation on the specific loss function used, and it can be the same loss function as the first loss function. By optimizing the third network loss, the second prediction result predicted by the student network can be gradually approximated to the pseudo-label obtained according to the teacher network.

[0094] For example, the third network loss can be calculated using the cross-entropy loss function.

[0095] In this step, the student network is trained using at least two of the above-obtained losses simultaneously, that is, end-to-end training. When the training end condition is reached, the student network is obtained.

[0096] In one example, in each round of training, the first network loss, the second network loss, and the third network loss are weighted and summed to obtain the total network loss; according to the total network loss, the network parameters of the student network are adjusted.

[0097] For example, the first network loss, the second network loss, and the third network loss are weighted and summed according to a certain weight ratio to obtain the total objective function as where x, y, and z are coefficients used to balance the magnitudes of the three. By optimizing the network parameters of the student network are updated by the gradient descent method to minimize the overall loss Gradually decreasing it can make the prediction ability of the student network for semantic segmentation better and better.

[0098] In one example, in each round of the first-stage training, according to the first network loss and the second network loss, the network parameters of the student network are adjusted until the first-stage training is completed, obtaining the adjusted student network; in each round of the second-stage training, according to the third network loss, the network parameters of the student network after the first-stage training are adjusted until the second-stage training is completed; wherein, the second prediction result of the unlabeled image is predicted using the student network after the first-stage training.

[0099] For example, during the first-stage training, the first network loss and the second network loss are weighted and summed according to a certain weight ratio to obtain the total objective function as where a and b are coefficients used to balance their magnitudes. By optimizing using the gradient descent method to update the network parameters of the student network to make the overall loss gradually decrease, obtaining the student network after the first-stage training, and using this network to continue the second-stage training. Before the second-stage training, use the obtained student network to predict the unlabeled image to obtain the second prediction result, and use the difference between this second prediction result and the pseudo-label to obtain the third network loss By optimizing using the gradient descent method to update the network parameters of the student network to make the loss gradually decrease, which can make the prediction ability of the student network for semantic segmentation better and better.

[0100] In one example, it can also be to adjust the network parameters of the student network according to the first network loss and the third network loss. When calculating the total loss value, the weighted sum value of the first network loss and the third network loss is used.

[0101] For each iteration during training, randomly select a labeled image and its ground truth label, and an unlabeled image from the training dataset to perform the above steps. When the training end condition is reached, end the network training to obtain the trained student network. Wherein, this end condition can be that the iteration reaches a certain number of times, or the overall loss is less than a certain threshold.

[0102] The network training method based on affinity coefficient provided by the embodiments of the present disclosure uses, in addition to the ground truth labels as supervision information, the affinity coefficients between image patches composed of different pixels in the labeled images for additional supervision during semi-supervised learning to improve the generalization ability of the network. At the same time, the information of labeled images and unlabeled images is used for end-to-end training, which can effectively utilize a large number of unlabeled images as a supplement to the labeled images, improve the accuracy of the trained model, save computing resources, and significantly reduce the annotation cost.

[0103] It should be noted that the network training method based on affinity coefficient provided by the embodiments of the present disclosure obtains not only a student network for performing semantic segmentation tasks, but also a teacher network that can achieve the same functions as the student network. The teacher network and the student network can be used to generate high-quality pseudo-labels for other deep learning tasks, or can be used to perform semantic segmentation of the input image and output the results of semantic segmentation. Regarding the two output branches of the student network, we use the branch that outputs the segmentation result and the branch that outputs the high-level features during the training phase. In the application phase, only the branch that outputs the segmentation result can be used.

[0104] In one implementation, the steps of the above embodiment further include: updating the network parameters of the teacher network according to the adjusted network parameters of the student network, and using the updated teacher network to train the student network; when the training end condition is reached, in addition to obtaining the student network, the teacher network is also obtained.

[0105] Here, our training strategy is to slowly update the network parameters of the teacher network after updating the network parameters of the student network. In this embodiment, the teacher network can be a network that is exactly the same as the student network in network structure, except for the network parameters; or, it can be a network that is not exactly the same as the student network in network structure. The difference is that the student network has two output branches at the end, while the teacher network can have only one branch that outputs the predicted segmentation result. For the content of the network structure, please refer to the next embodiment for details.

[0106] For example, denote the network parameters of the student network and the teacher network at the t-th iteration as θ s (t) and θ t (t). The subscripts represent student and teacher respectively.

[0107] At the t-th iteration, update the parameters of the student network by the gradient descent method:

[0108]

[0109] Among them, η represents the learning rate, which is set by those skilled in the art according to actual needs and can be taken as 0.01 here.

[0110] After the student network obtains parameter updates through one gradient descent at the t-th iteration, we can slowly update the parameters of the teacher network using the parameters of the student network before and after the update. For example, the EMA (Exponential Moving Average) algorithm can be used. The specific method for updating the teacher network parameters is as follows:

[0111] θ s (t) = mθ s (t - 1)+(1 - m)θ t (t) (8)

[0112] Among them, m is the momentum parameter, which is set by those skilled in the art according to actual needs and can be taken as 0.999 here.

[0113] In this way, after one iteration of training the student network, the teacher network will also be updated accordingly. In the next iteration of training, the updated teacher network is used to continue training, that is, the updated teacher network is used to predict and generate pseudo-labels for unlabeled images in step 104.

[0114] When the training of the student network ends, semantic segmentation tasks can be performed based on the obtained student network and / or teacher network.

[0115] In one implementation, to further improve the training effect of the network training method in this embodiment, we designed a network structure with multiple decoders here, which can simultaneously output segmentation results and high-level features without affecting the inference speed.

[0116] Next, the network structures of the student network and the teacher network will be described in sequence.

[0117] The network structure used in traditional semantic segmentation methods is an encoder and a decoder. As Figure 1 shown, the decoder outputs the prediction results. In this embodiment, on the basis of the original network structure, an additional decoder is added to output high-level features.

[0118] As Figure 5 shown, for the training process of the student network using the multi-decoder architecture for labeled images, the student network consists of a student encoder, a student main decoder, and a student auxiliary decoder. In the above step 106, inputting the labeled images into the student network to be trained to obtain the predicted labels and predicted affinity coefficients of the labeled images may include the following processing:

[0119] The student encoder extracts features from the labeled image to obtain a feature map of the labeled image; the student main decoder makes a prediction based on the feature map and outputs a first prediction result of the labeled image; the student auxiliary decoder makes a prediction based on the feature map to obtain a predicted affinity coefficient of the labeled image.

[0120] Exemplarily, after inputting an RGB labeled image with a shape of 3*H*W into the student encoder, a low-level feature (i.e., the feature map) with a shape of 2048*H / 8*W / 8 is obtained, where H and W respectively represent the original height and width of the labeled image. Then, this low-level feature is input into the student main decoder and the student auxiliary decoder, obtaining a first prediction result P with a shape of C*H*W and a high-level feature z with a shape of 256*H / 8*W / 8 after 8-fold downsampling respectively, where C is the total number of categories in the training dataset. Among them, the first prediction result P is the predicted label, with the true label as the supervision information, and the high-level feature z is calculated to obtain the predicted affinity coefficient, with the true affinity coefficient as the supervision information. Figure 5 In it, the gray levels of patches with similar pixel ratios in the true value visualization map of the affinity coefficient are similar. And since the shape of the high-level feature output by the auxiliary decoder is 256*H / 8*W / 8, when calculating the true affinity coefficient, the patch shape size used is 64 pixels (8*8). In other examples, when the shape of the high-level feature is of other sizes, the size of the patch needs to be adaptively adjusted.

[0121] During each iteration of training, it is necessary to simultaneously adjust the network parameters of the student encoder, the student main decoder, and the student auxiliary decoder in the student network.

[0122] For the teacher network, it can use the same network structure as the student network. For example, it consists of a teacher encoder, a teacher main decoder, and a teacher main decoder, or it can also not use the multi-decoder network structure and consist of a teacher encoder and a teacher main decoder. The teacher encoder and the teacher main decoder can be the same as the student encoder and the student main decoder, only with different values of network parameters.

[0123] As Figure 6 shown in the training process for the unlabeled image, where the teacher network consists of a teacher encoder and a teacher main decoder. In the above step 104, the teacher network makes a prediction on the unlabeled image to obtain a pseudo-label of the unlabeled image, which may include the following processing: the teacher encoder extracts features from the unlabeled image to obtain a feature map of the unlabeled image, and the teacher main decoder makes a prediction based on the feature map to obtain a pseudo-label of the unlabeled image. Among them, the teacher main decoder may first output an unreliable third prediction result, and after screening, obtain the pseudo-label. As Figure 6As shown by the pseudo-labels in , the white parts are unreliable pixels that have been filtered out.

[0124] Similarly, input the unlabeled image into the student network. The student encoder of the student network extracts features from the unlabeled image to obtain the feature map of the unlabeled image. The student main decoder makes predictions based on the feature map to obtain the second prediction result of the unlabeled image, and calculates the third network loss according to the difference between the second prediction result and the pseudo-labels.

[0125] At least one embodiment of the present disclosure further provides a training method for a pseudo-label model based on affinity coefficients, as Figure 7 shown. This method may include the following processes, and the steps that are the same as those in the above embodiments will not be described in detail.

[0126] In step 202, obtain the ground-truth labels and ground-truth affinity coefficients of the labeled images.

[0127] In this embodiment, the ground-truth labels and ground-truth affinity coefficients of the labeled images are used as supervision information for training the pseudo-label model.

[0128] Among them, the ground-truth labels are used to label the categories to which each pixel in the labeled image belongs, and the ground-truth affinity coefficients are used to represent the similarity between image patches composed of different pixels in the labeled image.

[0129] Before this step, the ground-truth affinity coefficients can be calculated from the ground-truth labels of the labeled images. For example, the affinity coefficient can be represented by the cosine similarity between image patches. Each image patch contains the same number of pixels. According to the ground-truth labels of the labeled image, determine the pixel ratio of pixels belonging to different categories in each image patch, and obtain the ground-truth affinity coefficient according to the cosine similarity between the pixel ratios of every two image patches in the labeled image. The specific calculation method can refer to Figure 1 the embodiments of

[0130] In step 204, input the labeled images into the pseudo-label model to be trained, and obtain the predicted labels and predicted affinity coefficients of the labeled images.

[0131] In this embodiment, the pseudo-label model to be trained can be obtained by improving the architecture of an existing semantic segmentation model. The difference is that, compared with the output of the existing semantic segmentation model being the prediction result of the ground-truth labels, the output of the pseudo-label model has two branches: the predicted labels of the labeled images and the high-level features used to calculate the predicted affinity coefficients. The high-level features can be understood as the prediction of the pixel ratio of image patches in the labeled image belonging to each category.

[0132] To further improve the accuracy of pseudo-labels, the predicted labels output by the pseudo-label model can be screened: input the labeled images into the pseudo-label model to be trained, and output the fourth prediction result and the predicted affinity coefficient of the labeled images. The fourth prediction result includes the probabilities that each pixel in the labeled image belongs to each category in the category set; for each pixel in the labeled image, calculate the entropy of the pixel according to the probabilities that the pixel belongs to each category in the category set; in response to the entropy of the pixel meeting the reliability requirement, determine the pixel as a reliable pixel; and obtain the predicted label of the labeled image according to the category corresponding to the determined reliable pixel. The specific screening method can refer to Figure 1 the embodiments of

[0133] In step 206, according to the difference between the predicted label and the ground-truth label, determine the first model loss.

[0134] For the predicted label obtained by segmentation prediction, the ground-truth label is used to supervise it here. The first model loss can be calculated by a loss function. The loss function is used to determine the gap between the predicted label actually output by the network and the expected ground-truth label. This embodiment does not limit the specific loss function to be used. For example, a quantile loss function, a mean squared error loss function, or a cross-entropy loss function can be used. By optimizing the first model loss, the predicted label predicted by the model can be gradually approximated to the ground-truth label.

[0135] In step 208, according to the difference between the predicted affinity coefficient and the ground-truth affinity coefficient, determine the second model loss.

[0136] For the predicted affinity coefficient calculated from the predicted high-level features, we use the ground-truth affinity coefficient as its optimization target. The second model loss can be calculated by a loss function. The loss function is used to determine the gap between the actually obtained predicted affinity coefficient and the expected ground-truth affinity coefficient. This embodiment does not limit the specific loss function to be used. For example, a quantile loss function, a mean squared error loss function, or a mean squared error loss function can be used. By optimizing the second model loss, the predicted affinity coefficient predicted by the model can be gradually approximated to the expected ground-truth affinity coefficient.

[0137] In step 210, adjust the network parameters of the pseudo-label model according to the first model loss and the second model loss. When the model training end condition is reached, obtain the pseudo-label model.

[0138] In this step, the first model loss and the second model loss can be weighted and summed according to a certain weight ratio to obtain a total loss value. By continuously training to minimize the overall loss, the pseudo-label model can be made to have the ability of preliminary segmentation prediction. For example, through backpropagation for gradient descent iteration to adjust the network parameters of the pseudo-label model. When the network iteration end condition is reached, the network training is terminated, and the trained pseudo-label model is obtained. Among them, the end condition can be that the iteration reaches a certain number of times, or the loss value is less than a certain threshold.

[0139] A training method for a pseudo-label model based on an affinity coefficient provided by the technical solution of the embodiment of the present disclosure uses the ground truth label and the ground truth affinity coefficient of the labeled image as supervision information to train the pseudo-label model. The ground truth affinity coefficient can improve the generalization ability of the model as additional supervision information, and thus can generate more reliable pseudo-labels. This method can make full use of the information provided by the ground truth labels of limited labeled images, improve the algorithm accuracy, and save a large amount of annotation costs at the same time.

[0140] Different from the method in the above embodiment, the training method in this embodiment no longer adopts a complete end-to-end training, but first trains a pseudo-label model, and then uses the pseudo-labels generated by the pseudo-label model to train a neural network for performing semantic segmentation tasks.

[0141] Based on the above embodiment, we designed a network structure with multiple decoders, which can output the segmentation result and pixel features at the same time without affecting the inference speed.

[0142] Among them, the pseudo-label model is composed of an encoder, a main decoder, and an auxiliary decoder. In step 204, the labeled image is input into the pseudo-label model to be trained, and the predicted label and predicted affinity coefficient of the labeled image are obtained. Specifically, the encoder extracts features from the labeled image to obtain the feature map of the labeled image; the main decoder makes a prediction based on the feature map and outputs the predicted label of the labeled image; the auxiliary decoder makes a prediction based on the feature map to obtain the predicted affinity coefficient of the labeled image.

[0143] In the above training process, for the auxiliary decoder, during the optimization process of the predicted affinity coefficient calculated from its output high-level features, not only the network parameters in the auxiliary encoder will be adjusted, but also the network parameters in the encoder will be adjusted, thereby improving the accuracy of the segmentation result output by the main encoder, and also improving the reliability of the pseudo-labels predicted by the pseudo-label model.

[0144] At least one embodiment of the present disclosure further provides a pseudo-label generation method. This method requires using the pseudo-label model trained by the training method in the previous embodiment, and inputting the unlabeled image into this pseudo-label model to obtain the pseudo-label of the unlabeled image.

[0145] Among them, the above pseudo-label model is a model trained by the method in the above embodiment.

[0146] The predicted pseudo-label is used to label the category to which each pixel in the predicted unlabeled image belongs.

[0147] In one implementation manner, the pseudo-labels output for the unlabeled image can also be screened. Input the unlabeled image into the pseudo-label model, and output the prediction result of the unlabeled image. The prediction result includes the probability that each pixel in the predicted unlabeled image belongs to each category in the category set. For each pixel in the unlabeled image, calculate the entropy of the pixel according to the probability that the pixel belongs to each category in the category set in the prediction result; in response to the entropy of the pixel meeting the reliability requirement, determine that the pixel is a reliable pixel; according to the category corresponding to the determined reliable pixel, obtain the pseudo-label of the unlabeled image. For the specific screening method, refer to the above embodiment.

[0148] For semi-supervised learning, only a small part of the labeled images with true labels in the training sample set are used, while a large proportion of the unlabeled images lacking true labels are used. The method in this embodiment can use the pseudo-label model trained by the true labels to generate the pseudo-labels of the unlabeled images as supervision information for training the neural network model.

[0149] As Figure 8 shown, Figure 8 is a block diagram of a network training device based on affinity coefficients shown in at least one embodiment of the present disclosure. The device includes:

[0150] A training data acquisition module 81, configured to: acquire the true label and true affinity coefficient of the labeled image, where the true label is used to label the category to which each pixel in the labeled image belongs, and the true affinity coefficient is used to represent the similarity between image patches composed of different pixels in the labeled image; acquire the first prediction result and prediction affinity coefficient of the labeled image, where the first prediction result and prediction affinity coefficient are predicted by the student network to be trained; acquire the second prediction result and pseudo-label of the unlabeled image, where the second prediction result is predicted by the student network to be trained, and the pseudo-label is predicted by the teacher network.

[0151] A network parameter adjustment module 82, configured to: adjust network parameters of the student network based on at least two of a first network loss, a second network loss, and a third network loss; wherein, the first network loss is obtained according to a difference between the first prediction result and the true value label, the second network loss is obtained according to a difference between the predicted affinity coefficient and the true value affinity coefficient, and the third network loss is obtained according to a difference between the second prediction result and the pseudo label.

[0152] In some optional embodiments, each of the image patches includes the same number of pixels; the training data acquisition module 81 is further configured to: determine a pixel ratio of pixels belonging to different classes in each of the image patches according to the true value label of the labeled image; obtain the true value affinity coefficient according to a cosine similarity between the pixel ratios of every two image patches in the labeled image.

[0153] In some optional embodiments, when the training data acquisition module 81 is configured to obtain the pseudo label of the unlabeled image, it is specifically configured to: input the unlabeled image into a teacher network, and output a third prediction result of the unlabeled image, where the third prediction result includes probabilities of each pixel in the unlabeled image belonging to each class in a class set; for each pixel in the unlabeled image, calculate an entropy of the pixel according to probabilities of the pixel belonging to each class in the class set in the third prediction result; in response to the entropy of the pixel meeting a reliability requirement, determine the pixel as a reliable pixel; and determine the pseudo label of the unlabeled image according to the class corresponding to the determined reliable pixel.

[0154] In some optional embodiments, the student network is composed of a student encoder, a student main decoder, and a student auxiliary decoder; when the training data acquisition module 81 is configured to obtain the first prediction result and the predicted affinity coefficient of the labeled image, it is specifically configured to: extract features from the labeled image by the student encoder to obtain a feature map of the labeled image; predict by the student main decoder according to the feature map, and output a first prediction result of the labeled image; and predict by the student auxiliary decoder according to the feature map to obtain the predicted affinity coefficient of the labeled image.

[0155] In some optional embodiments, the teacher network is composed of a teacher encoder and a teacher main decoder; when the training data acquisition module 81 is configured to obtain the pseudo label of the unlabeled image, it is specifically configured to: extract features from the unlabeled image by the teacher encoder to obtain a feature map of the unlabeled image; and predict by the teacher main decoder according to the feature map, and output the pseudo label of the unlabeled image.

[0156] In some alternative embodiments, after adjusting the network parameters of the student network based on at least two of the first network loss, the second network loss, and the third network loss, the network parameter adjustment module 82 is further configured to: update the network parameters of the teacher network according to the adjusted network parameters of the student network, and use the updated teacher network to train the student network; when the training end condition is reached, obtain the student network and also obtain the teacher network.

[0157] In some alternative embodiments, the apparatus further includes a network application module, configured to perform a semantic segmentation task based on the obtained student network and / or the teacher network.

[0158] In some alternative embodiments, when the network parameter adjustment module 82 is configured to adjust the network parameters of the student network based on at least two of the first network loss, the second network loss, and the third network loss, it is specifically configured to: in each round of training, perform a weighted sum of the first network loss, the second network loss, and the third network loss to obtain a total network loss; adjust the network parameters of the student network according to the total network loss; or, in each round of training in the first-stage training, adjust the network parameters of the student network according to the first network loss and the second network loss until the first-stage training is completed to obtain an adjusted student network; in each round of training in the second-stage training, adjust the network parameters of the student network after the first-stage training according to the third network loss until the second-stage training is completed; wherein, the second prediction result of the unlabeled image is predicted by using the student network after the first-stage training.

[0159] As Figure 9 shown, Figure 9 is a block diagram of a training apparatus for a pseudo-label model based on an affinity coefficient shown in at least one embodiment of the present disclosure. The apparatus includes:

[0160] A supervised information acquisition module 91, configured to: acquire a ground truth label and a ground truth affinity coefficient of a labeled image, where the ground truth label is used to label the category to which each pixel in the labeled image belongs, and the ground truth affinity coefficient is used to represent the similarity between image patches composed of different pixels in the labeled image.

[0161] A labeled image prediction module 92, configured to: input the labeled image into a pseudo-label model to be trained, and obtain a predicted label and a predicted affinity coefficient of the labeled image.

[0162] A model loss determination module 93, configured to: determine a first model loss according to the difference between the predicted label and the ground truth label; determine a second model loss according to the difference between the predicted affinity coefficient and the ground truth affinity coefficient.

[0163] A model parameter adjustment module 94, configured to adjust network parameters of the pseudo-label model according to the first model loss and the second model loss, and obtain the pseudo-label model when a model training end condition is reached.

[0164] In some optional embodiments, each of the image patches includes the same number of pixels;

[0165] When obtaining the true affinity coefficient of the labeled image, the supervision information acquisition module 91 is specifically configured to: determine the pixel ratio of pixels belonging to different classes in each of the image patches according to the true labels of the labeled image; obtain the true affinity coefficient according to the cosine similarity between the pixel ratios of every two image patches in the labeled image.

[0166] In some optional embodiments, the labeled image prediction module 92 is specifically configured to: input the labeled image into a pseudo-label model to be trained, and output a fourth prediction result and a predicted affinity coefficient of the labeled image, where the fourth prediction result includes the probability that each pixel in the labeled image belongs to each class in the class set; for each pixel in the labeled image, calculate the entropy of the pixel according to the probability that the pixel belongs to each class in the class set in the fourth prediction result; in response to the entropy of the pixel meeting the reliability requirement, determine the pixel as a reliable pixel; obtain the predicted label of the labeled image according to the class corresponding to the determined reliable pixel.

[0167] For the implementation processes of the functions and effects of each module in the above device, refer to the implementation processes of the corresponding steps in the above method for details, which will not be elaborated here.

[0168] An embodiment of the present disclosure further provides an electronic device, as Figure 10 shown, the electronic device includes a memory 11 and a processor 12. The memory 11 is used to store computer instructions that can be run on the processor, and the processor 12 is used to implement the network training method based on affinity coefficient or the training method of the pseudo-label model based on affinity coefficient according to any embodiment of the present disclosure when executing the computer instructions.

[0169] An embodiment of the present disclosure further provides a computer program product, which includes computer programs / instructions. When the computer programs / instructions are executed by a processor, the network training method based on affinity coefficient or the training method of the pseudo-label model based on affinity coefficient according to any embodiment of the present disclosure is implemented.

[0170] Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the network training method based on affinity coefficient or the training method of the pseudo-label model based on affinity coefficient described in any embodiment of the present disclosure.

[0171] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial descriptions of the method embodiments. The apparatus embodiments described above are only illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in this specification. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0172] The specific embodiments of this specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0173] Those skilled in the art will readily conceive of other embodiments of this specification after considering the specification and practicing the invention herein. This specification is intended to cover any variations, uses, or adaptations of this specification, which follow the general principles of this specification and include common general knowledge or conventional technical means in the technical field not claimed in this application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of this specification are pointed out by the following claims.

[0174] It should be understood that this specification is not limited to the exact structures described above and shown in the figures, and various modifications and changes can be made without departing from its scope. The scope of this specification is only limited by the appended claims.

[0175] The above are only the preferred embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this specification shall be included within the scope of protection of this specification.

Claims

1. A network training method based on affinity coefficient, characterized in that, the method includes: Obtain the true label and true affinity coefficient of the labeled image, where the true label is used to label the category to which each pixel in the labeled image belongs, and the true affinity coefficient is used to represent the similarity between the pixel ratios of every two image patches in the labeled image. Among them, the pixel ratio is the ratio of the pixels of different categories in the image patch; Obtain the first prediction result and predicted affinity coefficient of the labeled image, where the first prediction result and predicted affinity coefficient are predicted by the student network to be trained; Obtain the second prediction result and pseudo-label of the unlabeled image, where the second prediction result is predicted by the student network to be trained, and the pseudo-label is predicted by the teacher network; Adjust the network parameters of the student network based on at least two of the first network loss, the second network loss, and the third network loss; Among them, the first network loss is obtained according to the difference between the first prediction result and the true label, the second network loss is obtained according to the difference between the predicted affinity coefficient and the true affinity coefficient, and the third network loss is obtained according to the difference between the second prediction result and the pseudo-label.

2. The method according to claim 1, characterized in that, Each of the image patches contains the same number of pixels; The obtaining of the true affinity coefficient of the labeled image includes: According to the true label of the labeled image, determine the pixel ratio of the pixels of different categories in each of the image patches; According to the cosine similarity between the pixel ratios of every two image patches in the labeled image, obtain the true affinity coefficient.

3. The method according to claim 1, characterized in that, The pseudo-label is predicted by the teacher network, including: Input the unlabeled image into the teacher network, and output the third prediction result of the unlabeled image, where the third prediction result includes the probability that each pixel in the unlabeled image belongs to each category in the category set; For each pixel in the unlabeled image, calculate the entropy of the pixel according to the probability that the pixel belongs to each category in the category set in the third prediction result; In response to the entropy of the pixel meeting the reliability requirement, determine that the pixel is a reliable pixel; According to the category corresponding to the determined reliable pixel, determine the pseudo-label of the unlabeled image.

4. The method according to claim 1, characterized in that, The student network consists of a student encoder, a student main decoder, and a student auxiliary decoder; The obtaining of the first prediction result and predicted affinity coefficient of the labeled image includes: The student encoder extracts features from the labeled image to obtain the feature map of the labeled image; The student main decoder makes a prediction according to the feature map and outputs the first prediction result of the labeled image; The student auxiliary decoder makes a prediction according to the feature map to obtain the predicted affinity coefficient of the labeled image.

5. The method according to claim 1, characterized in that, The teacher network consists of a teacher encoder and a teacher main decoder; Obtaining the pseudo-labels of the unlabeled images includes: Performing feature extraction on the unlabeled images by the teacher encoder to obtain the feature maps of the unlabeled images; Performing prediction by the teacher main decoder according to the feature maps and outputting the pseudo-labels of the unlabeled images.

6. The method according to claim 1, wherein, after adjusting the network parameters of the student network based on at least two of the first network loss, the second network loss, and the third network loss, the method further includes: Updating the network parameters of the teacher network according to the adjusted network parameters of the student network, and using the updated teacher network to train the student network; When the training end condition is reached, obtaining the student network and also obtaining the teacher network.

7. The method according to claim 6, wherein, Performing a semantic segmentation task based on the obtained student network and / or the teacher network.

8. The method according to any one of claims 1-7, wherein: Adjusting the network parameters of the student network based on at least two of the first network loss, the second network loss, and the third network loss includes: In each round of training, performing weighted summation on the first network loss, the second network loss, and the third network loss to obtain the total network loss; Adjusting the network parameters of the student network according to the total network loss; Or, In each round of training in the first stage of training, adjusting the network parameters of the student network according to the first network loss and the second network loss until the first stage of training is completed to obtain the adjusted student network; In each round of training in the second stage of training, adjusting the network parameters of the student network after the first stage of training according to the third network loss until the second stage of training is completed; wherein, the second prediction result of the unlabeled images is predicted by the student network after the first stage of training.

9. A training method for a pseudo-label model based on affinity coefficients, wherein, the method includes: Obtaining the true labels and true affinity coefficients of the labeled images, where the true labels are used to label the categories to which each pixel in the labeled images belongs, and the true affinity coefficients are used to represent the similarity between the pixel ratios of every two image patches in the labeled images, where the pixel ratio is the ratio of the pixels of different categories belonging to the image patch; Inputting the labeled images into the pseudo-label model to be trained to obtain the predicted labels and predicted affinity coefficients of the labeled images; Determining the first model loss according to the difference between the predicted labels and the true labels; Determining the second model loss according to the difference between the predicted affinity coefficients and the true affinity coefficients; Adjusting the network parameters of the pseudo-label model according to the first model loss and the second model loss, and when the model training end condition is reached, obtaining the pseudo-label model.

10. The method according to claim 9, wherein, Each of the image patches contains the same number of pixels; The obtaining of the true affinity coefficient of the labeled image includes: Determining, according to the true label of the labeled image, the pixel ratio of pixels belonging to different categories in each of the image patches; Obtaining the true affinity coefficient according to the cosine similarity between the pixel ratios of every two image patches in the labeled image.

11. According to the method described in claim 9, wherein, The inputting of the labeled image into the pseudo-label model to be trained to obtain the predicted label and predicted affinity coefficient of the labeled image includes: Inputting the labeled image into the pseudo-label model to be trained, and outputting a fourth prediction result and a predicted affinity coefficient of the labeled image, where the fourth prediction result includes the probabilities of each pixel in the labeled image belonging to each category in the category set; For each pixel in the labeled image, calculating the entropy of the pixel according to the probabilities of the pixel belonging to each category in the category set in the fourth prediction result; Determining the pixel as a reliable pixel in response to the entropy of the pixel meeting the reliability requirement; Obtaining the predicted label of the labeled image according to the category corresponding to the determined reliable pixel.

12. An apparatus for training a network based on an affinity coefficient, wherein, The apparatus includes: A training data acquisition module, configured to: acquire the true label and true affinity coefficient of a labeled image, where the true label is used to label the category to which each pixel in the labeled image belongs, and the true affinity coefficient is used to represent the similarity between the pixel ratios of every two image patches in the labeled image, where the pixel ratio is the ratio of the pixels belonging to different categories in the image patch; acquire a first prediction result and a predicted affinity coefficient of the labeled image, where the first prediction result and the predicted affinity coefficient are predicted by a student network to be trained; acquire a second prediction result and a pseudo-label of an unlabeled image, where the second prediction result is predicted by the student network to be trained, and the pseudo-label is predicted by a teacher network; A network parameter adjustment module, configured to: adjust the network parameters of the student network based on at least two of a first network loss, a second network loss, and a third network loss; wherein, the first network loss is obtained according to the difference between the first prediction result and the true label, the second network loss is obtained according to the difference between the predicted affinity coefficient and the true affinity coefficient, and the third network loss is obtained according to the difference between the second prediction result and the pseudo-label.

13. An apparatus for training a pseudo-label model based on an affinity coefficient, wherein, The apparatus includes: A supervision information acquisition module, configured to: acquire the true label and true affinity coefficient of a labeled image, where the true label is used to label the category to which each pixel in the labeled image belongs, and the true affinity coefficient is used to represent the similarity between the pixel ratios of every two image patches in the labeled image, where the pixel ratio is the ratio of the pixels belonging to different categories in the image patch; A labeled image prediction module, configured to: input the labeled image into a pseudo-label model to be trained, and obtain a predicted label and a predicted affinity coefficient of the labeled image; A model loss determination module, configured to: determine a first model loss according to the difference between the predicted label and the true label; determine a second model loss according to the difference between the predicted affinity coefficient and the true affinity coefficient; A model parameter adjustment module, configured to adjust network parameters of the pseudo-label model according to the first model loss and the second model loss, and obtain the pseudo-label model when a model training end condition is reached.

14. An electronic device, characterized in that the device includes a memory and a processor, the memory is used to store computer instructions that can run on the processor, and the processor is used to implement the method according to any one of claims 1 to 8, or implement the method according to any one of claims 9 to 11 when executing the computer instructions.

15. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when executed by a processor, implements the method according to any one of claims 1 to 8, or implements the method according to any one of claims 9 to 11.

Images