Image processing method, device and equipment

Abstract

The embodiments of the present disclosure provide an image processing method, device, and electronic device. Taking into account that for unreliable samples, although it is impossible to accurately determine that the sample belongs to a certain category based on the prediction results, it is possible to determine the target category to which the unreliable sample is most likely not based on the prediction results, and then a negative sample can be constructed based on the target category to which the unreliable sample does not belong. Then, the negative samples can be fully utilized to train the model, thereby improving the utilization rate of unlabeled samples, increasing the number of available training samples, and thus improving the accuracy of the trained model. The trained high-precision model is then used to predict the category to which the target image to be processed belongs, or the category to which the pixels in the target image belong, so that a more accurate prediction result can be obtained for subsequent applications.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, and device. Background Art

[0002] In many scenarios, images need to be classified, for example, to determine whether an image is a person or a landscape. Alternatively, pixels in an image need to be categorized, for example, to determine whether each pixel in an image belongs to a category such as sky, grass, or a person (i.e., semantic segmentation). Pre-trained models are often used to classify or semantically segment images.

[0003] When training a model, you can use semi-supervised learning. Semi-supervised learning combines supervised and unsupervised learning, using a small number of labeled samples and a large number of unlabeled samples to train the model. By using a large number of unlabeled samples to supplement labeled samples, the accuracy of the trained model can be improved. However, current semi-supervised learning methods still fail to fully and maximize the use of unlabeled samples, resulting in a limited number of available training samples and a need to improve the accuracy of the trained model. Summary of the Invention

[0004] Embodiments of the present disclosure provide an image processing method, apparatus, and device.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Obtain the target image to be processed;

[0007] Determine classification information of the target image using a pre-trained classification model, the classification information including the category to which each pixel of the target image belongs and at least one of the categories to which the target image belongs, wherein the classification model is trained based on the following method:

[0008] Performing prediction on the unlabeled first sample image to obtain a prediction result including the classification information;

[0009] Determining, from the first sample image, unreliable samples whose prediction result reliability is lower than a preset threshold, and determining a target category to which the unreliable samples do not belong, the unreliable samples including pixels or images;

[0010] A preset model is trained based on negative samples to obtain the classification model, wherein the negative samples are constructed based on the unreliable samples and the target categories to which they do not belong.

[0011] In some embodiments, the prediction result includes a probability that a pixel point in the first sample image belongs to each of a plurality of preset categories, and determining an unreliable sample whose reliability of the prediction result is lower than a preset threshold from the first sample image includes:

[0012] For each pixel of the first sample image, based on the predicted probability that each pixel belongs to each category, determining the information entropy of each pixel, wherein the information entropy is used to represent the reliability of the prediction result of each pixel;

[0013] Selecting a plurality of target pixels from the pixels of the first sample image, wherein the information entropy of the selected plurality of target pixels is greater than the information entropy of other pixels in the first sample image except the plurality of target pixels;

[0014] Taking the plurality of target pixels as the unreliable samples; or

[0015] The first sample image includes a plurality of frames, the prediction result includes a probability that each frame of the first sample image belongs to each of a plurality of preset categories, and determining, from the first sample image, an unreliable sample whose reliability of the prediction result is lower than a preset threshold, comprising:

[0016] Determining the information entropy of each frame of the first sample image based on the probability that each frame of the first sample image belongs to each of the categories, wherein the information entropy is used to characterize the reliability of the prediction result of each frame of the first sample image;

[0017] Selecting a plurality of target image frames from the plurality of first sample image frames, wherein the information entropy of the selected target image frames is greater than the information entropy of other images in the plurality of first sample image frames except the plurality of target image frames;

[0018] The several frames of target images are used as the unreliable samples.

[0019] In some embodiments, a preset model is trained based on the negative samples, including

[0020] Performing multiple rounds of training on a preset model based on the negative samples;

[0021] In a case where the unreliable samples are pixels in the first sample image, the ratio of the number of the unreliable samples to the total number of pixels in the first sample image decreases as the number of training rounds increases; and / or

[0022] In a case where the unreliable samples are the first sample images, the ratio of the number of the unreliable samples to the total number of the first sample images decreases as the number of training rounds increases.

[0023] In some embodiments, the proportion is determined based on a ratio of a current number of training rounds to a total number of training rounds.

[0024] In some embodiments, determining the target category to which the unreliable sample does not belong includes:

[0025] For each unreliable sample, sorting the multiple categories based on the probability that each unreliable sample belongs to each category of the preset multiple categories;

[0026] The categories other than the top several categories in the multiple categories are used as target categories to which the unreliable sample does not belong; or the middle several categories are selected from the multiple categories as target categories to which the unreliable sample does not belong.

[0027] In some embodiments, training a preset model based on the negative samples to obtain the classification model includes:

[0028] Inputting the first sample image into a preset model, and determining a first target loss based on a probability output by the model that the negative sample belongs to the target category;

[0029] The preset model is trained using the first target loss to obtain the classification model.

[0030] In some embodiments, training a preset model based on the negative samples to obtain the classification model includes:

[0031] For any target category, determine at least two positive samples belonging to the target category, where the positive samples are pixels or images;

[0032] A preset model is trained based on the at least two positive samples and the negative samples to obtain the classification model.

[0033] In some embodiments, training a preset model based on the at least two positive samples and the negative sample of the target category to obtain the classification model includes:

[0034] Performing feature extraction on the negative sample of the target category and the at least two positive samples respectively to obtain target features of the negative sample of the target category and target features of the at least two positive samples, wherein the target features are related to the category to which the samples belong;

[0035] Determining a second target loss based on a similarity between a target feature of a negative sample of the target category and a target feature of any one of the at least two positive samples, and a similarity between the target features of the at least two positive samples;

[0036] The preset model is trained using the second target loss to obtain the classification model.

[0037] In some embodiments, the method further comprises:

[0038] Determining a reliable sample from the first sample image based on the prediction result, and determining a pseudo label for the reliable sample based on the prediction result, wherein the reliable sample is a pixel point or image whose reliability of the prediction result is higher than a preset threshold; and training a preset model based on the negative sample, comprising:

[0039] The preset model is trained based on the negative samples, the reliable samples carrying pseudo labels, and the second sample images carrying true labels, wherein the true labels are manually annotated labels indicating the classification information of the second sample images.

[0040] In some embodiments, training the preset model based on the negative sample, the reliable sample carrying the pseudo label, and the second sample image carrying the real label includes:

[0041] determining a first loss based on the negative sample;

[0042] Determine a second loss based on a difference between a prediction result of the reliable sample in the first sample image output by the model and the pseudo label;

[0043] Determine a third loss based on a difference between a prediction result of the second sample image output by the model and the true label;

[0044] A third target loss is obtained based on the first loss, the second loss, and the third loss, and the preset model is trained using the third target loss.

[0045] In some embodiments, the step of predicting the classification information in the unlabeled first sample image to obtain the prediction result is performed by the teacher model.

[0046] The training of a preset model based on the negative samples to obtain the classification model includes:

[0047] The student model is trained based on the negative samples, and the classification model is obtained based on the student model obtained through training, wherein the structure of the teacher model is consistent with the structure of the student model, and the parameters of the teacher model are at least partially different from the parameters of the student model.

[0048] In some embodiments, the parameters of the teacher model are updated using the parameters of the trained student model to determine the teacher model with updated parameters as the classification model, including: repeatedly using the teacher model to predict the classification information of the unlabeled first sample image to obtain a prediction result, determining from the first sample image an unreliable sample whose reliability of the prediction result is lower than a preset threshold, and determining the target category to which the unreliable sample does not belong, constructing a negative sample based on the unreliable sample and the corresponding target category, training a preset model based on the negative sample to obtain the classification model, and using the parameters of the trained student model to update the parameters of the teacher model until the preset conditions are met, and using the teacher model with updated parameters as the classification model.

[0049] In some embodiments, after constructing negative samples based on the unreliable samples and the target categories to which the samples do not belong, the method further includes:

[0050] The target category and the target feature of the negative sample are stored in correspondence.

[0051] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0052] Acquisition module: used to acquire the target image to be processed;

[0053] Prediction module: used to determine the classification information of the target image using a pre-trained classification model, the classification information including the category to which each pixel of the target image belongs and at least one of the categories to which the target image belongs, wherein the classification model is trained based on the following method: predicting a first unlabeled sample image to obtain a prediction result containing the classification information; determining from the first sample image an unreliable sample whose reliability of the prediction result is lower than a preset threshold, and determining the target category to which the unreliable sample does not belong, the unreliable sample including a pixel or an image; training a preset model based on negative samples to obtain the classification model, wherein the negative samples are constructed based on the unreliable samples and the corresponding target category to which they do not belong.

[0054] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and computer instructions stored in the memory for execution by the processor. When the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.

[0055] In the disclosed embodiment, a prediction can be performed on an unlabeled first sample image to obtain a prediction result including classification information of the first sample image. Based on the prediction result, unreliable samples with a reliability below a preset threshold are identified from the first sample image, and the target category to which the unreliable samples do not belong is determined. Negative samples are constructed based on the unreliable samples and the target category to which the unreliable samples do not belong to train a preset model to obtain a classification model. The classification model is then used to classify the target image to be processed. Considering that, although it is impossible to accurately determine that the unreliable samples belong to a certain category based on the prediction result, it is possible to determine, based on the prediction result, that the unreliable samples have a high probability of not belonging to the target category. Negative samples can then be fully utilized to train the model, thereby improving the utilization rate of unlabeled samples, increasing the number of available training samples, and thus improving the accuracy of the trained model. The trained high-precision model is then used to predict the category to which the target image to be processed belongs, or the category to which the pixels in the target image belong, to obtain more accurate prediction results for subsequent applications.

[0056] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0058] Figure 1 Schematic diagram of a method for training a model based on semi-supervised learning according to an embodiment of the present disclosure.

[0059] Figure 2 This is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0060] Figure 3 2 is a schematic diagram of an embodiment of the present disclosure for cyclically training a model using negative samples.

[0061] Figure 4 This is a schematic diagram of extracting target features using a model according to an embodiment of the present disclosure.

[0062] Figure 5 This is a schematic diagram of an embodiment of the present disclosure for training a model using a teacher model-student model approach.

[0063] Figure 6 Schematic diagram of an image processing device according to an embodiment of the present disclosure.

[0064] Figure 7It is a schematic diagram of the logical structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0066] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0067] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0068] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure are further described in detail below with reference to the accompanying drawings.

[0069] In many scenarios, images need to be classified, for example, to determine whether an image is a person or a landscape. Alternatively, pixels within an image need to be categorized (i.e., semantic segmentation), for example, to determine whether each pixel in an image belongs to a category such as sky, grass, or a person. Or, in road inspection scenarios, to determine whether each pixel in an image belongs to a category such as cracks or potholes. Pre-trained models are often used for image classification or semantic segmentation.

[0070] When training the model, semi-supervised learning can be used. Semi-supervised learning is a method that combines supervised and unsupervised learning. During the training process, the model is trained using a small number of labeled samples and a large number of unlabeled samples. By using a large number of unlabeled samples as a supplement to the labeled samples, the accuracy of the trained model can be improved. Labeled samples are samples that have been manually labeled with the category to which each pixel in the image belongs, or the category to which the image belongs. For example, the pixels in the image are manually labeled as belonging to category A, or the image is manually labeled as belonging to category B. Unlabeled samples are samples that do not have such manually labeled labels.

[0071] For ease of distinction, the manually annotated label indicating the category to which an image or a pixel in the image belongs is referred to as a true label; the label indicating whether an image or a pixel in the image belongs to a certain category or not obtained based on the prediction result is referred to as a pseudo label.

[0072] In related technologies, when training a model using semi-supervised learning, such as Figure 1 As shown, an initial model 1 can be trained using labeled samples to obtain a model 2 that has a certain classification ability for images or pixels in images. Unlabeled samples are then input into model 2, and model 2 is used to predict the classification information of the unlabeled samples. The unlabeled samples can then be divided into reliable samples and unreliable samples based on the reliability of the unlabeled sample prediction results. If the trained model is used for image classification, the reliable samples and unreliable samples are images. If the trained model is used for semantic segmentation of images, the reliable samples and unreliable samples are pixels.

[0073] Among them, reliable samples can be samples whose category can be accurately determined based on the prediction results of model 2 with a high probability. For example, if model 2 predicts that the probability of a certain pixel point being in category A is 90%, the probability of being in category B is 5%, and the probability of being in category C is 5%, then the pixel point is likely to belong to category A, and thus the pixel point can be used as a reliable sample. Unreliable samples are samples whose category cannot be accurately determined based on the prediction results of model 2. For example, if model 2 predicts that the probability of a certain pixel point being in category A is 50%, the probability of being in category B is 48%, and the probability of being in category C is 2%, then the prediction results based on model 2 cannot determine whether the pixel point belongs to category A or category B, and thus the pixel point can be used as an unreliable sample.

[0074] Currently, after determining reliable and unreliable samples, the pseudo-labels for the reliable samples can be further determined based on the prediction results. For example, if the prediction results for a reliable sample are: 70% probability of belonging to category A, 20% probability of belonging to category B, 10% probability of belonging to category C, and 0 probability of belonging to category D, then the pseudo-label for the reliable sample can be determined to be category A, or the pseudo-label is that it does not belong to category D.

[0075] Then, the reliable samples with pseudo labels can be input into Model 2 to train Model 2 and obtain Model 3 with higher accuracy.

[0076] It can be seen that in the relevant technology, only reliable samples among the unlabeled samples are used to train the model, while unreliable samples are directly discarded. The unlabeled samples are not fully and maximized, resulting in a limited number of available training samples. The accuracy of the trained model still needs to be improved.

[0077] Moreover, in the process of improving semi-supervised learning with current technologies, the overall improvement trend is to make good use of reliable samples among unlabeled samples, for example, how to determine more accurate pseudo-labels for reliable samples, and there is no consideration of using unreliable samples in model training.

[0078] Based on this, the embodiment of the present disclosure provides an image processing method, which takes into account that for unreliable samples, although it is impossible to accurately determine that the sample belongs to a certain category based on the prediction results, it can be determined based on the prediction results that the unreliable sample is likely not to belong to certain target categories. Then, negative samples can be constructed based on the unreliable samples and the target categories to which the unreliable samples do not belong, and the model is trained using the negative samples, thereby improving the utilization rate of unlabeled samples, increasing the number of available training samples, and thus improving the accuracy of the trained model. The trained high-precision model is then used to predict the category to which the target image to be processed belongs, or the category to which the pixels in the target image belong, so that more accurate prediction results can be obtained for subsequent applications.

[0079] The image processing method provided by the embodiments of the present disclosure can be executed by various electronic devices that have deployed pre-trained classification models, such as mobile phones, computers, cloud servers, etc., and the embodiments of the present disclosure do not limit this.

[0080] Specifically, the image processing method of the embodiment of the present disclosure is as follows Figure 2 As shown, the following steps may be included:

[0081] S210, obtaining a target image to be processed;

[0082] The target image can be any image that needs to be classified to determine its category, or it can be an image that needs semantic segmentation, that is, to determine the category of each pixel in the image. It can be an image collected in real time or a pre-stored image. The target image can be an RGB image, a grayscale image, an infrared image, etc., and the embodiments of this disclosure do not impose any restrictions.

[0083] S220: Determine classification information of the target image using a pre-trained classification model, where the classification information includes the category to which each pixel of the target image belongs and at least one of the categories to which the target image belongs. To achieve higher accuracy for the classification model, the classification model may be trained based on the following steps S221-S223:

[0084] S221, predicting the first sample image without labels to obtain a prediction result containing classification information;

[0085] In step S221 , classification information of the unlabeled first sample image may be predicted. For example, the category to which each pixel in the first sample image belongs may be predicted, or the category to which the first sample image belongs may be predicted.

[0086] For example, taking the prediction of the category of each pixel in the first sample image as an example, it can be predicted whether each pixel in the first sample image is a "person", "animal", "plant", "building", etc.; for example, the category to which pixel A belongs is predicted, and the prediction result can be: the probability of pixel A being a "person" is 80%, the probability of being an "animal" is 15%, the probability of being a "plant" is 5%, and the probability of being a "building" is 0.

[0087] For example, predicting the category of a first sample image can predict whether it is an "image of a building," "image of a person," and so on. For example, predicting the category of image a can yield a 40% probability of image a being an "image of a building" and a 60% probability of being an "image of a person." Of course, the aforementioned categories and numbers are merely illustrative, and the prediction results are also illustrative and do not limit the present disclosure.

[0088] In some embodiments, the classification information of the first sample image can be predicted using a pre-trained model with certain classification capabilities, or other methods can be used for prediction. The above-mentioned model with certain classification capabilities can be trained based on samples with real labels, or can be trained based on samples with pseudo labels, or can be trained based on samples with real labels and samples with pseudo labels at the same time, which is not limited in the present embodiment.

[0089] S222: Based on the prediction results, unreliable samples whose prediction reliability is lower than a preset threshold are determined from the first sample image, target categories to which the unreliable samples do not belong are determined, and negative samples are constructed based on the unreliable samples and the target categories to which the unreliable samples do not belong. The unreliable samples can be pixels or images.

[0090] Generally speaking, if the category of the sample can be accurately determined based on the prediction results, the sample is a reliable sample; if the category of the sample cannot be accurately determined based on the prediction results, the sample is an unreliable sample.

[0091] When predicting the category of each pixel in the first sample image, an unreliable sample is a pixel whose prediction reliability is lower than a preset threshold. When predicting the category of the first sample image, an unreliable sample is an image whose prediction reliability is lower than a preset threshold. The reliability of the prediction results can be measured in various ways, which are explained in detail below.

[0092] After determining the unreliable samples, it is possible to determine the target categories that the unreliable samples do not belong to in order to construct negative samples. Taking the prediction of the classification information of each pixel in the first sample image as an example, the prediction result of the unreliable pixel B is: the probability of pixel B being a "person" is 50%, the probability of being an "animal" is 40%, the probability of being a "plant" is 5%, and the probability of being a "building" is 5%. Based on the prediction result, although it is impossible to determine whether pixel B is a "person" or an "animal", it can be determined that pixel B is highly likely not to belong to the target category "plant" and is also highly likely not to belong to the target category "building". Therefore, the unreliable pixel B can be used as a negative sample for the two categories of "plant" and "building". The above embodiments are merely exemplary and are not intended to be limiting of the present disclosure.

[0093] Of course, for example, if the probability of an unreliable pixel C being a "person" is 25%, the probability of being an "animal" is 25%, the probability of being a "plant" is 25%, and the probability of being a "building" is 25%, then based on the prediction results, it is impossible to determine what the unreliable pixel C is, nor what it is most likely not to be. In this case, it is impossible to determine the target category that the unreliable pixel C does not belong to, and no negative sample will be constructed based on the unreliable pixel C.

[0094] S223: Training a preset model based on negative samples to obtain a classification model.

[0095] In step S223, after the negative samples are constructed, the preset model is trained using the negative samples to obtain a classification model, which can then be used to determine classification information of the target image to be processed.

[0096] In some embodiments, the preset model and the model used to predict the prediction result of the first sample image can be the same model. For example, the model J can be trained using the sample image with the real label, and then the model J is used to predict the unlabeled first sample image to obtain the prediction result. Based on the prediction result, the unreliable sample and the target category to which the unreliable sample does not belong are determined, and then the negative sample is constructed, and then the negative sample is continued to be used to train the model J, and the classification model is obtained using the trained model J.

[0097] In some embodiments, the preset model and the model used to predict the prediction result of the first sample image can also be different models. For example, a model J can be trained using sample images with real labels, and then the model J can be used to predict the unlabeled first sample image to obtain a prediction result. Based on the prediction result, unreliable samples and target categories that the unreliable samples do not belong to are determined to construct negative samples. The negative samples are then used to train the model K, and the trained model K is used to obtain a classification model.

[0098] Using negative samples to train the preset model can be to train the preset model using only negative samples, or to train the model in combination with negative samples and at least one of reliable samples carrying pseudo labels and samples carrying real labels. The specific settings can be based on actual needs.

[0099] In some embodiments, when determining unreliable samples based on the reliability of the prediction results, the category to which each pixel in the unlabeled first sample image belongs can be predicted. The resulting prediction results include the probability that the pixel in the first sample image belongs to each of the multiple preset categories. The reliability of the prediction results can then be determined based on the probability that each pixel belongs to each of the multiple preset categories, and the unreliable samples can be determined based on the reliability. There are many specific ways to determine unreliable samples.

[0100] For example, in some embodiments, the prediction results of the first sample image or the pixel points of the first sample image can be used to characterize the reliability of the prediction results using the probability value of the category with the highest probability, and whether the first sample image or the pixel points of the first sample image are unreliable samples can be determined based on whether the probability value of the category with the highest probability exceeds a threshold.

[0101] Taking the prediction of the category to which each pixel in the first sample image belongs as an example, if the probability value of the highest probability of the category in the prediction result of pixel A exceeds a preset threshold, for example, 80%, then the reliability of the prediction result of pixel A is determined to be higher than the preset threshold, and pixel A is a reliable sample; otherwise, the reliability of the prediction result of pixel A is determined to be lower than the preset threshold, and pixel A is an unreliable sample. Taking the prediction of the category to which the first sample image belongs as an example, if the probability value of the highest probability of the category in the prediction result of image a exceeds a preset threshold of 80%, then the reliability of the prediction result of image a is determined to be higher than the preset threshold, and pixel a is a reliable sample; otherwise, the reliability of the prediction result of image a is determined to be lower than the preset threshold, and pixel a is an unreliable sample. In the above embodiment, the numerical value of the probability value of the highest probability of the category in the prediction result to represent the reliability of the prediction result is only exemplary, and the numerical value of the preset threshold is also only exemplary and does not serve as a limitation of the present disclosure.

[0102] In some embodiments, the reliability of the prediction results can also be characterized by the closeness of the probabilities corresponding to the two categories with the highest probabilities in the prediction results to determine unreliable samples. For example, if the probabilities corresponding to the two categories with the highest probabilities are relatively close, it is impossible to determine which category the sample belongs to. Therefore, it can be determined as an unreliable sample. Otherwise, it can be determined as a reliable sample.

[0103] In some embodiments, it is also possible to determine whether a sample is an unreliable sample or a reliable sample based on the information entropy of the prediction result. For example, taking the prediction of the classification information of each pixel in the first sample image as an example, the information entropy of each pixel used to characterize the reliability of the prediction result can be determined based on the probability of the category to which each pixel in the first sample image belongs. Since the larger the information entropy, the less reliable the prediction result. Therefore, in some embodiments, if the information entropy is greater than a preset threshold, it is determined to be an unreliable sample; if the information entropy is less than the preset threshold, it is determined to be a reliable sample. In some implementations, it is also possible to select a number of target pixels from the first sample image, and the information entropy of the selected target pixels is greater than the information entropy of other pixels in the first sample image except the target pixels, that is, the target pixels are the pixels in the first sample image that have a larger information entropy, and then these target pixels are used as unreliable samples.

[0104] For example, the information entropy of the prediction results of each pixel can be determined, the information entropy can be sorted in descending order, and the first M pixels can be determined as unreliable samples, where M is a positive integer. Alternatively, the pixel with the largest information entropy in the first sample image can be first selected as the target pixel, and then the pixel with the largest information entropy can be selected from the remaining pixels as the target pixel, and this step can be repeated until the number of selected target pixels meets the preset conditions.

[0105] Similarly, when the unreliable samples are images, the information entropy of each first sample frame can be determined based on the probability that each first sample frame belongs to each category. This information entropy is used to characterize the reliability of the prediction results for each first sample frame. Then, several target images are selected from the multiple first sample frames, and the information entropy of the selected target images is greater than the information entropy of all other images in the multiple first sample frames except for the selected target images. For example, the information entropy of the prediction results of each image can be determined and sorted in descending order. The first N images are then determined as the target images, where N is a positive integer. Alternatively, the image with the largest information entropy from the multiple first sample frames can be selected as the target image, and then the image with the largest information entropy from the remaining first sample images can be selected as the target image. This step can be repeated until the number of selected target images meets the preset conditions.

[0106] Of course, an image whose information entropy is greater than a preset threshold may also be used as a target image, and then the target image is used as an unreliable sample.

[0107] In some implementations, the information entropy of a pixel can be calculated using the following formula (1):

[0108]

[0109] Where H(pij) represents the value of information entropy, c represents the current category, C represents the total number of preset categories, and pij represents the pixel p in the i-th row and j-th column of the image.

[0110] Determining whether a sample is an unreliable sample based on information entropy can improve the accuracy of the determined unreliable sample. In addition, when selecting unreliable samples, a preset number of unreliable samples can be selected, or unreliable samples can be selected from the first sample image according to a preset number ratio.

[0111] In some embodiments, to improve the accuracy of the trained model, the preset model can be trained multiple times based on negative samples. As the number of training rounds increases, the accuracy and performance of the trained model will increase, and the model's prediction results for unlabeled samples will become more and more accurate. Therefore, the ratio of unreliable samples to the total number of unlabeled samples should also decrease.

[0112] Therefore, in order to improve the accuracy of the determined unreliable samples, the proportion of unreliable samples in the total number of unlabeled samples can be dynamically adjusted during the training process. For example, when the unreliable samples are pixels, the proportion of the number of unreliable samples to the total number of pixels in the first sample image can decrease as the number of training rounds increases. For example: during the first round of training, the first M1 pixels with the largest information entropy are determined as unreliable samples; during the tenth round of training, the first M2 pixels with the largest information entropy are determined as unreliable samples, where M1>M2.

[0113] Similarly, when the unreliable samples are images, the ratio of the number of unreliable samples to the total number of first sample images decreases as the number of training rounds increases. For example, during the first round of training, the first N1 frames of first sample images with the highest information entropy are determined to be unreliable samples; during the tenth round of training, the first N2 frames of first sample images with the highest information entropy are determined to be unreliable samples, where N1>N2.

[0114] By dynamically adjusting the proportion of unreliable samples, the accuracy of identifying unreliable samples can be improved, and thus the accuracy of the model trained using unreliable samples will also be improved.

[0115] In some embodiments, in order to more accurately determine the number of unreliable samples, the number of unreliable samples may be determined based on a ratio of the current number of training rounds to the total number of training rounds.

[0116] For example, assuming that it is preset that the preset model needs to be trained for 100 rounds, when predicting the category to which each pixel in the unlabeled first sample image belongs, the number of unreliable samples in the first round of training can be one tenth of the total number of pixels in the first sample image; the number of unreliable samples in the tenth round of training can be one hundredth of the total number of pixels in the first sample image; and the number of unreliable samples in the 100th round of training can be one thousandth of the total number of pixels in the first sample image. Similarly, when predicting the category to which the unlabeled first sample image belongs, the ratio of the number of unreliable samples to the total number of the first sample images can be determined based on the ratio of the current number of training rounds to the total number of training rounds, which will not be further described in the embodiments of this specification. Using the above method to determine unreliable samples can improve the accuracy of the unreliable samples used to train the model, thereby improving the accuracy of the trained model. Of course, the above embodiments are merely exemplary and are not intended to be limiting of the present disclosure.

[0117] In some real-time examples, the proportion can also be calculated using the following formula (2):

[0118]

[0119] Among them, α0 represents the initial proportion of unreliable samples, α t Indicates the proportion of unreliable samples in the tth round of training, the total number of training rounds is total_epoch, and the current number of training rounds is t. In some embodiments, in order to ensure the accuracy and practicality of negative samples, the method of determining negative samples can be to sort each unreliable sample based on the probability that each unreliable sample belongs to each of the preset multiple categories, for example, sorting them in descending order according to the probability of each category, and removing several categories in the front, and using the remaining categories as the target categories to which the unreliable sample does not belong, or after sorting the multiple categories, selecting several categories in the middle as the target categories to which the unreliable sample does not belong; and then constructing corresponding negative samples based on the target categories to which the unreliable sample does not belong.

[0120] The categories with the highest probability (i.e., the categories listed at the top) are removed because they have a high probability of being the target category for unreliable samples. These categories are less accurate as the target category for which the unreliable samples do not belong, making them unsuitable for constructing negative samples. For example, if the probability that unreliable sample X belongs to category Y is 60%, then the probability of unreliable sample X belonging to category Y is high, and it is not appropriate to use unreliable sample X as a negative sample for category Y.

[0121] Among them, selecting several categories in the middle as the target categories that unreliable samples do not belong to is equivalent to removing several categories in the front and several categories in the back. The reason for removing several categories with the lowest probability (several categories in the back) is that for several categories with the lowest probability, the model can already predict relatively accurately that unreliable samples do not belong to these categories. If these unreliable samples are used as negative samples for these categories to train the model, the performance of the model will not be greatly improved. In other words, the negative samples constructed in this way are worthless negative samples. For example, the probability that unreliable sample X belongs to category Y is 0. Then the prediction result can accurately determine that unreliable sample X is most likely not in category Y. If the unreliable sample X is used as a negative sample of the target category Y to train the model, it will not help improve the accuracy of the model.

[0122] To illustrate this with a specific example, when predicting the category of each pixel in the unlabeled first sample image, the prediction result for unreliable pixel D is: the probability of pixel D being "person" is 30%, the probability of being "tree" is 30%, the probability of being "building" is 30%, the probability of being "flower" is 5%, the probability of being "grass" is 5%, the probability of being "sun" is 0, and the probability of being "moon" is 0. Sorting the probabilities of the multiple categories to which pixel D belongs, pixel D has the highest probabilities of being "person," "tree," and "building," all at 30%. This means that the prediction result cannot accurately determine whether pixel D is a "person," "tree," or "building." Since these three categories have the highest probabilities, pixel D is not suitable as a negative sample for the "person," "tree," or "building" categories. Therefore, these three categories can be removed when determining the target category of pixel D.

[0123] In some embodiments, the probability that pixel D is "sun" or "moon" is 0, which means that the prediction result can accurately determine that pixel D is not of the two categories of "sun" or "moon". Therefore, if pixel D is used as a negative sample of "sun" or "moon" to train the preset model, the performance of the model will not be greatly improved. Therefore, when determining the target category, the two categories of "sun" and "moon" can also be removed, and negative samples of pixel D that do not belong to "sun" or "moon" are not constructed. Furthermore, it can be determined that the unreliable pixel D is a negative sample of the target category "flower" and "grass". That is, through the above method, it is possible to accurately determine the target category to which the unreliable sample does not belong, and then construct a negative sample based on the unreliable sample and the target category to which the unreliable sample does not belong; further, the model is trained using negative samples to improve the accuracy of the trained model.

[0124] Of course, the above embodiments are only exemplary. In actual applications, the number of categories removed from the first and last positions can be determined according to actual conditions. For example, there are 21 categories preset. In the prediction results of a certain sample, each category corresponds to a probability value. The 21 categories can be sorted from large to small according to the size of their corresponding probability values. The target category can be determined as the category ranked 3rd to 15th, or it can be determined as the category ranked 4th to 12th, so as to select the most appropriate target category. This is not limited in the present disclosure.

[0125] In some embodiments, after constructing negative samples, there are multiple ways to train a preset model using negative samples. For example, in some embodiments, a first sample image can be input into a preset model, and a first target loss can be determined based on the difference between the probability that the negative sample output by the model belongs to the target category and the actual probability of the negative sample (i.e., 0); then the preset model is trained using the first target loss to obtain a classification model. For example: the first sample image is input into a preset model, and the preset model outputs that the probability that the unreliable pixel E belongs to the target category "person" is 10%, wherein the unreliable pixel E is a negative sample of the "person" category (i.e., the probability of being a "person" is 0). Therefore, based on the difference between the above "10%" and "0", the first target loss can be determined, and then the parameters of the model are continuously adjusted based on the first target loss to train the preset model to obtain a classification model.

[0126] Of course, the above embodiment is also applicable to the embodiment in which each frame image in the first sample image is used as an unreliable sample, and the embodiment of the present disclosure will not be described in detail here.

[0127] In other embodiments, after determining the negative samples of the target category, the negative samples of the target category and the positive samples belonging to the target category can also be used to construct comparative learning samples, and the constructed comparative learning samples can be used to train the model. Among them, the positive samples can be determined based on manually annotated labels or based on pseudo labels determined based on the prediction results. For example, for any target category, at least two positive samples belonging to the target category are determined, and the positive samples can be pixels or images; the negative samples of the target category and the at least two positive samples are used to construct comparative learning samples, and the preset model is trained to obtain a classification model.

[0128] For example, for the target class "person," we can determine at least two target pixels that "belong to a person" (positive samples) and unreliable pixels that "do not belong to a person" (negative samples). Based on these two target pixels and the unreliable pixels that "do not belong to a person," we construct comparative learning samples and train the model to obtain a classification model. The determined target pixels can be pixels with real labels or pixels with pseudo labels determined based on the prediction results.

[0129] Of course, the above embodiment is also applicable to the scene where the sample is an image, and this embodiment of the specification will not be repeated. Based on the above embodiment, the preset model can be trained in combination with positive samples and negative samples of the target category to improve the accuracy of the model.

[0130] In some embodiments, to further improve the accuracy of the trained model, when training a preset model based on a negative sample of the target category and at least two positive samples, feature extraction can be performed on the negative samples and the positive samples, respectively, to obtain target features for the negative samples and target features for the positive samples; wherein the target features are related to the category to which the samples belong, that is, the target features are features that are helpful in classifying the samples. For example, when the target category is "person," the positive samples are samples that "belong to people," and the negative samples are samples that "do not belong to people." The extracted target features are features that characterize the category to which the samples belong, that is, they can be used to assist in determining the category of the samples.

[0131] Since the similarity of the target features of two images or pixels belonging to the same category must be greater than the similarity of the target features of two images or pixels that do not belong to the same category, after extracting the target features, the second target loss can be determined based on the similarity between the target features of the negative sample and the target features of any one of the above-mentioned at least two positive samples, and the similarity between the target features of the above-mentioned at least two positive samples; the preset model is trained using the second target loss to obtain a classification model.

[0132] For example, a negative sample is a sample that "does not belong to a person," while the two positive samples are samples that "belong to a person." The similarity between the target features of the negative sample and the target features of the positive sample will be low, while the similarity between the target features of the two positive samples will be high. Therefore, the two similarities can be used to determine a second target loss, which can then be used to train the preset model to obtain a classification model.

[0133] In some embodiments, the second target loss may also be calculated using the following formula (3):

[0134]

[0135] Among them, C represents the total number of categories, and M represents the total number of current positive samples. ci Represents the target feature of the current pixel or current image, z + ci represents the target feature of the positive sample, z - cij represents the negative sample target feature.

[0136] Of course, when the model is trained using negative samples and at least two positive samples, the negative samples and the two positive samples can also be directly input into the model to be trained, and the second target loss can be constructed based on the similarity between the probabilities of the at least two positive samples belonging to the target category, or the similarity between the probability of any positive sample belonging to the target category and the probability of the negative sample belonging to the target category.

[0137] In some embodiments, in addition to determining unreliable samples in the first sample image, reliable samples can also be determined from the first sample image based on the prediction results, wherein the reliable samples are pixels or images whose reliability of the prediction results is higher than a preset threshold. The pseudo-labels of the reliable samples can then be determined based on the prediction results. For example, the pseudo-labels can be used to indicate that the category to which the reliable samples belong is the category with the highest probability in the prediction results. When training a preset model based on negative samples, in order to improve the accuracy of the model, the preset model can be trained by combining negative samples, reliable samples carrying pseudo-labels, and second sample images carrying real labels, wherein the real labels are manually labeled and used to indicate the category to which the second sample image belongs, or the category to which the pixels in the second sample image belong.

[0138] Of course, the preset model can also be trained in combination with negative samples and reliable samples carrying pseudo labels, or the preset model can be trained in combination with negative samples and second sample images carrying real labels. The specific settings can be made according to actual needs.

[0139] When a preset model is trained by combining negative samples, reliable samples carrying pseudo labels, and second sample images carrying real labels, a first loss can be determined based on the negative samples. For example, the first sample image can be input into the preset model, and the first loss can be determined based on the difference between the probability of the negative sample belonging to the target category output by the model and the actual probability of the negative sample (i.e., 0). Alternatively, for the target category to which the negative sample does not belong, at least two positive samples belonging to the target category can be determined, and a comparative learning sample can be constructed using the at least two positive samples and the negative sample. The first loss can then be constructed using the comparative learning sample. The specific method can be referred to the description in the above embodiment and will not be repeated here.

[0140] A second loss can then be determined based on the difference between the predicted results of the first sample image and the pseudo labels output by the preset model. A third loss can be determined based on the difference between the predicted results of the second sample image and the true labels output by the preset model. Combining the first, second, and third losses yields a third target loss, which is then used to train the preset model.

[0141] There are many ways to obtain the third target loss by combining the first loss, the second loss, and the third loss. For example, the sum of the three or the weighted average of the three can be taken as the third target loss. In some embodiments, the third target loss can be determined by the following formula (4):

[0142] L=λsLs+λuLu+λcLc Formula (4)

[0143] Wherein, L represents the third target loss, Ls represents the first loss, Lu represents the second loss, and Lc represents the third loss. λs, λu, and λc are coefficients of Ls, Lu, and Lc, respectively. The values ​​of these coefficients can be determined based on actual conditions. Of course, the third target loss can also be calculated using the first, second, and third losses using other calculation methods, and this disclosure is not limited to this.

[0144] Based on this, the reliable and unreliable samples in the labeled samples and unlabeled samples are fully utilized to train the model, which increases the number of training samples and further improves the accuracy of the model.

[0145] In the embodiments of this specification, the classification information of the first sample image is predicted, and the model used to obtain the prediction result can be the same model as the model used to train the model based on the negative sample. However, under this implementation method, the trained result may not be accurate enough. Using a single model to make predictions and determine negative samples based on the results of its own predictions to perform self-training based on the negative samples may cause the model to make the subsequent training direction incorrect when making prediction errors, ultimately resulting in the trained model having low accuracy, and even the model classification result may be opposite to the actual value.

[0146] Therefore, in some embodiments, the model structure can follow the Mean-Teacher model, and the model training process can adopt a method of training two models simultaneously, that is, a student-teacher model training method. The teacher model predicts the classification information in the unlabeled first sample image to obtain a prediction result; the student model is trained based on the negative sample, and then the classification model is obtained based on the student model obtained by the final training. Among them, the student model and the teacher model can be models with the same structure but different parameters. Of course, in some embodiments, the structure and parameters of the student model and the teacher model can also be inconsistent.

[0147] In some embodiments, a teacher model and a student model with the same structure but different parameters can be used for model training. A more aggressive approach can be used to train the student model, for example, by adjusting the student model parameters with greater amplitude. The teacher model parameters are derived based on the student model parameters. For example, the teacher model parameters can be updated by combining the current teacher model parameters with the student model parameters. Therefore, the teacher model parameters change less than the student model parameters, thereby ensuring that the teacher model parameters change steadily and slowly.

[0148] By using a teacher model with smoothly adjusted parameters to predict the classification information of the unlabeled first sample, the prediction result is made more reliable; then the more reliable prediction result is used to determine the negative sample, and then the negative sample is used to train the student model, which can ensure that the trained student model is more accurate, so that a more accurate classification model can be obtained based on the student model.

[0149] In some embodiments, since the trained student model has the ability to classify, the trained student model can be directly used as a classification model. Of course, the parameters of the trained student model can also be used to update the parameters of the teacher model, so that the teacher model with updated parameters can be used as a classification model. Since the parameters of the student model change more radically and the parameters of the teacher model change more gradually, the accuracy of the teacher model will be relatively higher. Therefore, using the teacher model as a classification model can make the classification model more accurate.

[0150] The training of the teacher-student model can be a single round of training using all sample data (for example, negative samples, reliable samples with pseudo labels, and second sample images with real labels) or multiple rounds of training. In order to improve the accuracy of the model, the teacher-student model is generally trained for multiple rounds. For example, the teacher model can be used to predict the classification information of the unlabeled first sample image to obtain a prediction result; based on the prediction result, unreliable samples are determined from the first sample image, wherein the proportion of unreliable samples can also gradually decrease as the number of training rounds changes; then, unreliable samples are determined based on the prediction result, and the target category to which the unreliable samples do not belong is determined, negative samples are constructed based on the unreliable samples and the target category to which the unreliable samples do not belong, the student model is trained based on the negative samples, and the parameters of the teacher model are updated based on the parameters of the trained student model. The above steps are then repeated until the preset conditions are met, and the teacher model with updated parameters is used as the classification model.

[0151] Among them, the preset conditions can be met, for example, that the number of training rounds reaches the preset total number of rounds P, or that the parameters of the teacher model are close enough to those of the student model. Figure 3 , Figure 3This is a schematic diagram of the above-mentioned cyclic training. First, the unlabeled first sample image is input into the teacher model. After the teacher model's prediction, the unreliable samples in the unlabeled first sample image are determined, and negative samples are constructed based on the unreliable samples and the target categories to which the unreliable samples do not belong. The negative samples are input into the student model to train the student model; the parameters of the trained student model can then be used to update the parameters of the teacher model. Then repeat the above steps. It can be preset to repeat the above steps for P rounds, that is, when the number of repeated rounds reaches P, stop training, and then use the teacher model with updated parameters as the classification model. It can also be that when the parameters of the teacher model are close enough to the parameters of the student model, for example, when the difference between the parameters of the teacher model and the parameters of the student model is less than a certain threshold, stop training, and then use the teacher model with updated parameters as the classification model.

[0152] During the above-mentioned multiple rounds of training, in addition to using negative samples to train the student model, the student model can also be trained in combination with reliable samples carrying pseudo labels and / or second sample images carrying real labels, which is not limited in the embodiments of the present disclosure.

[0153] Through multiple rounds of training of the teacher-student model, the accuracy of the final trained teacher-student model can be further improved, thereby improving the accuracy of the obtained classification model.

[0154] The student model can be obtained by training the initial model with labeled samples. For example, the initial model can be trained based on a second sample image with real labels. The trained model is the student model. After obtaining the student model, the parameters of the initial model are updated using the parameters of the student model to obtain the teacher model.

[0155] When training the student model using sample images, the student model's parameters can be updated using a back-propagation (BP) algorithm. The student model's parameters are then used to update the teacher model's parameters. During the teacher model update, the teacher model's parameters can be updated using an EMA (Exponential Moving Average) algorithm, combining the teacher model's current parameters with the student model's parameters.

[0156] Specifically, using the parameters of the student model to update the parameters of the teacher model can be to perform weighted averaging on the parameters of the student model and the parameters of the teacher model to obtain target parameters, and use the target parameters to replace the current parameters of the teacher model.

[0157] In some embodiments, the target parameter can be calculated using the following calculation formula (5):

[0158] P = αp1 + (1-α)p2 (Formula 5)

[0159] Among them, p1 represents the parameters of the current student model, p2 represents the parameters of the current teacher model, α represents the preset coefficient value, and P represents the target parameter.

[0160] By using the above-mentioned weighted average method to calculate the target parameters, the parameter changes of the teacher model can be made more stable, and the accuracy of the teacher model can be made higher. Furthermore, the classification model obtained based on the teacher model has higher accuracy. In the embodiments of this specification, other calculation methods can also be used to calculate the target parameters. The specific values ​​of the above-mentioned coefficients can be determined according to actual conditions, and this disclosure does not limit this.

[0161] In some embodiments, the structure of the teacher model or the student model may include a backbone network, a first network branch, and a second network branch. The backbone network is used to extract features from an image input to the teacher model or the student model. The first network branch is used to predict the classification information of the image based on the features extracted by the backbone network to obtain a prediction result. The second network branch is used to obtain target features of the image or its pixels based on the features extracted by the backbone network. The target features are features related to the category to which the image or its pixels belong.

[0162] Combine Figure 4 As shown in the figure, taking the pixels of an image as an example, the backbone network extracts the features of pixel 1, pixel 2, pixel 3, and pixel 4. The first network branch predicts the classification information of the four pixels based on the features extracted by the backbone network, and obtains the probability that the pixels belong to each category. The second network branch determines the target features of the pixels based on the features extracted by the backbone network.

[0163] Of course, the structure of the student model and the teacher model is not limited to the above structure. For example, determining the target features of an image or pixel point can also be an operation performed by another model. That is to say, the target features of an image or pixel point can be determined by a model X that is neither a student model nor a teacher model, while the student model and the teacher model only predict the classification information. However, in the embodiment of the present disclosure, the student model or the teacher model includes the structure of the above-mentioned backbone network, the first network branch and the second network branch, so that the student model and the teacher model can not only predict the classification information of the image, but also, in the process of predicting the classification information of the image, the target features of the image or the pixel points in the image can be extracted, and then the loss can be constructed based on the similarity of the target features of the constructed comparative learning samples to train the model, so that unreliable samples can be fully utilized.

[0164] In some embodiments, after the teacher model or the student model extracts the target features of the negative samples that do not belong to the target category, the target features of the negative samples can also be stored corresponding to the target category. For example, they can be stored in a memory bank, such as Figure 4 As shown. Specifically, taking the negative sample pixel 3 as an example, the target category A that pixel 3 does not belong to can be stored corresponding to the target feature of pixel 3. Of course, for reliable samples and samples carrying true labels, the teacher model or student model can also be used to extract their target features and store them in the Memory bank. Then, the data stored in the Memory bank can be used to construct comparative learning samples for training the model. For example, in addition to storing the target categories that negative sample pixel 3 and negative sample pixel 4 do not belong to (i.e., they do not belong to category A), and their target features, the Memory bank can also store the target categories that pixel 1 and pixel 2 belong to (i.e., they belong to category A), and their target features. Of course, pixel 1 or pixel 2 can be a reliable sample carrying a pseudo-label, or it can be a second sample image carrying a true label. Figure 4 The storage content of the Memory bank is only exemplary and is not intended to limit the embodiments of the present disclosure.

[0165] Based on this stored data, when training a model, one or more negative samples of the target category and their target features can be determined from the memory bank, as well as one or more samples with true or false labels and their target features. Based on these samples, comparative learning samples can be constructed for model training.

[0166] The above embodiment uses image pixels as samples. Of course, images can also be used as samples, and this embodiment of the present disclosure will not be repeated. Of course, in addition to using pixels and images as samples, pixel blocks can also be used as samples, and this embodiment of the present disclosure does not limit this.

[0167] It should be noted that the above embodiments can be freely combined without conflict, and due to space constraints, they will not be described in detail here.

[0168] In order to further explain the image processing method provided by the embodiment of the present disclosure, the following is explained in conjunction with a specific embodiment. Figure 5 , Figure 5 The flowchart of a specific embodiment of the present disclosure is shown in FIG.

[0169] First, in S501, the initial model is trained using a second sample image with a true label to obtain a student model. The true label indicates the classification information of the second sample image. The true label can be a manually annotated label, for example, a label that labels the category of the second sample image or a label that labels the category of each pixel in the second sample image. By training the initial model using the second sample image with a true label, the resulting student model has a certain degree of classification capability.

[0170] In S502, the parameters of the initial model are updated using the parameters of the student model to obtain the teacher model. The updating method can be to perform weighted averaging of the parameters of the student model and the parameters of the initial model, and the specific coefficients can be pre-set.

[0171] In S503 , the unlabeled first sample image is input into the teacher model obtained in the above steps. The unlabeled first sample image may be a first sample image that has not been manually labeled.

[0172] In S504, the teacher model is then used to predict the classification information of the unlabeled first sample image to obtain a prediction result. After the teacher model predicts the first sample image, it can obtain classification information for each category to which the first sample image belongs. For example, this can be the category to which each pixel in the first sample image belongs, or the category to which the first sample image belongs. Based on the above prediction results, the unlabeled first sample image is classified as a reliable sample or an unreliable sample.

[0173] When determining whether a sample in the first sample image is a reliable sample or an unreliable sample, the information entropy of each pixel or image can be determined based on the probability of each category to which each pixel or image belongs in the prediction result of the teacher model. Furthermore, the top M pixels with the largest information entropy can be regarded as unreliable image samples, or the top N images with the largest information entropy can be regarded as unreliable samples. In addition to the unreliable samples, other samples can be reliable samples.

[0174] In S505, the pseudo label of the reliable sample is determined. For example, the probability that the reliable sample belongs to category A is 5%, the probability that it belongs to category B is 10%, and the probability that it belongs to category C is 85%. That is, it can be determined that the reliable sample belongs to category C with a high probability, and the pseudo label of the reliable sample is used to characterize that the reliable sample belongs to category C. At the same time, the model can also be used to determine the target feature of the reliable sample, wherein the target feature is a feature related to the category to which the sample belongs. Furthermore, the category C to which the reliable sample belongs can be stored corresponding to the target feature of the reliable sample, for example, it can be stored in a memory library.

[0175] In S506, negative samples are constructed based on the unreliable samples and the target categories to which the unreliable samples do not belong. For example, the probability that the unreliable sample belongs to category A is 50%, the probability that it belongs to category B is 40%, the probability that it belongs to category C is 10%, and the probability that it belongs to category D is 0%. That is, the probabilities can be sorted from large to small, and several categories at the top and bottom can be eliminated. For example, the unreliable sample may belong to one of categories A and B. Therefore, categories A and B can be eliminated when determining the target category. In addition, since the model can already accurately determine that the unreliable sample does not belong to category D, if the unreliable sample is used as a negative sample of category D, the performance of the model will not be greatly improved. Therefore, category D can be eliminated when determining the target category. Finally, category C can be used as the target category to which the unreliable sample does not belong, and the unreliable sample can be used as a negative sample of category C.

[0176] At the same time, the model can also be used to determine the target feature of the negative sample, wherein the target feature is a feature related to category C. Furthermore, the target category to which the negative sample does not belong can be stored correspondingly to the target feature of the negative sample, for example, in a memory library.

[0177] In S507, the student model may be trained using the reliable samples with pseudo labels, the negative samples, and the second sample images with real labels. The memory bank may also store the categories of the samples with real labels and the target features of the samples with real labels.

[0178] The teacher model and the student model can be models with the same structure but different parameters. The structure of the teacher model or the student model may include a backbone network, a first network branch, and a second network branch. The backbone network is used to extract features from the image input to the teacher model or the student model. The first network branch is used to predict the classification information of the image based on the features extracted by the backbone network to obtain a prediction result. The second network branch is used to obtain the target features of the image or the pixels of the image based on the features extracted by the backbone network.

[0179] Comparative learning samples are constructed using negative samples stored in the memory library, reliable samples with pseudo labels, or second sample images with real labels, as well as the target features of each sample. For example, for any category, at least two positive samples (positive samples include reliable samples with pseudo labels and samples with real labels) and at least one negative sample of the category can be determined based on the data stored in the memory library. Then, similarity 1 of the target features between the positive samples and similarity 2 of the target features between the positive and negative samples are determined. Then, a first loss is constructed based on similarity 1 and similarity 2.

[0180] The student model outputs the prediction result 1 of the classification information of the reliable sample with the pseudo label, and the second loss is determined according to the difference between the prediction result 1 and the pseudo label;

[0181] The student model outputs a prediction result 2 for the classification information of the second sample image, and the third loss is determined based on the difference between the prediction result 2 and the true label;

[0182] The total loss is calculated by combining the first, second, and third losses, and the parameters of the student model are adjusted by backpropagation using the total loss. The total loss can be calculated by weighting the first, second, and third losses separately and then adding them together, or by other calculation methods.

[0183] In the disclosed embodiment, multiple rounds of training can be performed on the student model. After each training, the parameters of the student model and the parameters of the teacher model are weighted averaged to update the parameters of the teacher model, and then the final teacher model is used as the classification model.

[0184] As the training process progresses, the accuracy and performance of the student and teacher models continue to improve, and the proportion of unreliable samples should continue to decrease. Therefore, when determining unreliable samples among unlabeled samples, the proportion of unreliable samples can be gradually reduced as the number of training rounds increases.

[0185] The classification model obtained based on the above training method can make full use of unreliable samples in unlabeled samples, thereby improving the accuracy of the classification model.

[0186] Accordingly, the present disclosure also provides an image processing device, see Figure 6 , the device comprises:

[0187] An acquisition module 61 is used to acquire a target image to be processed;

[0188] The prediction module 62 is used to determine the classification information of the target image using a pre-trained classification model, wherein the classification information includes the category to which each pixel of the target image belongs and at least one of the categories to which the target image belongs, wherein the classification model is trained based on the following method: predicting a first unlabeled sample image to obtain a prediction result containing the classification information; determining an unreliable sample whose reliability of the prediction result is lower than a preset threshold from the first sample image, and determining the target category to which the unreliable sample does not belong, wherein the unreliable sample includes a pixel or an image; and training a preset model based on a negative sample to obtain the classification model, wherein the negative sample is constructed based on the unreliable sample and the corresponding target category to which it does not belong.

[0189] The specific steps of the image processing method executed by the above-mentioned device can be referred to the description in the above-mentioned method embodiment, which will not be repeated here.

[0190] Furthermore, the present disclosure also provides an electronic device, see Figure 7 The electronic device 70 includes a processor 71, a memory 72, and a computer program stored in the memory for execution by the processor. When the processor executes the computer program, the method described in any one of the above embodiments is implemented.

[0191] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in any of the aforementioned embodiments when the program is executed by a processor.

[0192] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0193] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that the embodiments of this specification can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the embodiments of this specification, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments of this specification.

[0194] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0195] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and when implementing the embodiment scheme of this specification, the functions of each module can be implemented in the same one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0196] The above is only a specific implementation of the embodiment of this specification. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of this specification. These improvements and modifications should also be regarded as the scope of protection of the embodiment of this specification.

Claims

1. An image processing method, characterized in that: The method comprises: Obtain the target image to be processed; Determine classification information of the target image using a pre-trained classification model, the classification information including the category to which each pixel of the target image belongs and at least one of the categories to which the target image belongs, wherein the classification model is trained based on the following method: Performing prediction on the unlabeled first sample image to obtain a prediction result including the classification information; Determining, from the first sample image, unreliable samples whose prediction result reliability is lower than a preset threshold, and determining a target category to which the unreliable samples do not belong, the unreliable samples including pixels or images; A preset model is trained based on negative samples to obtain the classification model, wherein the negative samples are constructed based on the unreliable samples and the target categories to which they do not belong.

2. The method according to claim 1, characterized in that The prediction result includes a probability that a pixel point in the first sample image belongs to each of a plurality of preset categories, and determining an unreliable sample whose reliability of the prediction result is lower than a preset threshold from the first sample image includes: For each pixel of the first sample image, based on the predicted probability that each pixel belongs to each category, determining the information entropy of each pixel, wherein the information entropy is used to represent the reliability of the prediction result of each pixel; Selecting a plurality of target pixels from the pixels of the first sample image, wherein the information entropy of the selected plurality of target pixels is greater than the information entropy of other pixels in the first sample image except the plurality of target pixels; Taking the plurality of target pixels as the unreliable samples; or The first sample image includes a plurality of frames, the prediction result includes a probability that each frame of the first sample image belongs to each of a plurality of preset categories, and determining, from the first sample image, an unreliable sample whose reliability of the prediction result is lower than a preset threshold, comprising: Determining the information entropy of each frame of the first sample image based on the probability that each frame of the first sample image belongs to each of the categories, wherein the information entropy is used to characterize the reliability of the prediction result of each frame of the first sample image; Selecting a plurality of target image frames from the plurality of first sample image frames, wherein the information entropy of the selected target image frames is greater than the information entropy of other images in the plurality of first sample image frames except the plurality of target image frames; The target image is used as the unreliable sample.

3. The method according to claim 1 or 2, characterized in that Training a preset model based on the negative sample includes: Performing multiple rounds of training on a preset model based on the negative samples; In a case where the unreliable samples are pixels in the first sample image, the ratio of the number of the unreliable samples to the total number of pixels in the first sample image decreases as the number of training rounds increases; and / or In a case where the unreliable samples are the first sample images, the ratio of the number of the unreliable samples to the total number of the first sample images decreases as the number of training rounds increases.

4. The method according to claim 3, characterized in that The proportion is determined based on the ratio of the current number of training rounds to the total number of training rounds.

5. The method according to any one of claims 1 to 4, characterized in that Determining the target category to which the unreliable sample does not belong, including: For each unreliable sample, sorting the multiple categories based on the probability that each unreliable sample belongs to each category of the preset multiple categories; The categories other than the top several categories in the multiple categories are used as target categories to which the unreliable sample does not belong; or, the middle several categories are selected from the multiple categories as target categories to which the unreliable sample does not belong.

6. The method according to any one of claims 1 to 5, characterized in that The training of a preset model based on the negative samples to obtain the classification model includes: Inputting the first sample image into a preset model, and determining a first target loss based on a probability output by the model that the negative sample belongs to the target category; The preset model is trained using the first target loss to obtain the classification model.

7. The method according to any one of claims 1 to 5, characterized in that The training of a preset model based on the negative samples to obtain the classification model includes: For any target category, determine at least two positive samples belonging to the target category, where the positive samples are pixels or images; A preset model is trained based on the at least two positive samples and the negative sample of the target category to obtain the classification model.

8. The method according to claim 7, characterized in that Training a preset model based on the at least two positive samples and the negative sample of the target category to obtain the classification model includes: Performing feature extraction on the negative sample of the target category and the at least two positive samples respectively to obtain target features of the negative sample of the target category and target features of the at least two positive samples, wherein the target features are related to the category to which the samples belong; Determining a second target loss based on a similarity between a target feature of a negative sample of the target category and a target feature of any one of the at least two positive samples, and a similarity between the target features of the at least two positive samples; The preset model is trained using the second target loss to obtain the classification model.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Determining a reliable sample from the first sample image based on the prediction result, and determining a pseudo label for the reliable sample based on the prediction result, wherein the reliable sample is a pixel point or image whose reliability of the prediction result is higher than a preset threshold; The training of a preset model based on the negative sample includes: The preset model is trained based on the negative samples, the reliable samples carrying pseudo labels, and the second sample images carrying true labels, wherein the true labels are manually annotated labels indicating the classification information of the second sample images.

10. The method according to claim 9, characterized in that The preset model is trained based on the negative sample, the reliable sample carrying the pseudo label, and the second sample image carrying the real label, including: determining a first loss based on the negative sample; Determine a second loss based on a difference between a prediction result of the reliable sample in the first sample image output by the model and the pseudo label; Determine a third loss based on a difference between a prediction result of the second sample image output by the model and the true label; A third target loss is obtained based on the first loss, the second loss, and the third loss, and the preset model is trained using the third target loss.

11. The method according to any one of claims 1 to 10, characterized in that The step of predicting the classification information in the unlabeled first sample image to obtain a prediction result is performed by the teacher model. The training of a preset model based on the negative samples to obtain the classification model includes: The student model is trained based on the negative samples, and the classification model is obtained based on the student model obtained through training, wherein the structure of the teacher model is consistent with the structure of the student model, and the parameters of the teacher model are at least partially different from the parameters of the student model.

12. The method according to claim 11, characterized in that The parameters of the teacher model are updated using the parameters of the trained student model to determine the teacher model with updated parameters as the classification model, including: repeatedly using the teacher model to predict the classification information of the unlabeled first sample image to obtain a prediction result, determining from the first sample image an unreliable sample whose reliability of the prediction result is lower than a preset threshold, and determining the target category to which the unreliable sample does not belong, constructing a negative sample based on the unreliable sample and the corresponding target category, training the preset model based on the negative sample to obtain the classification model, and using the parameters of the trained student model to update the parameters of the teacher model until the preset conditions are met, and using the teacher model with updated parameters as the classification model.

13. The method according to any one of claims 1 to 12, characterized in that After constructing negative samples based on the unreliable samples and the target categories to which they do not belong, the method further includes: The target category and the target feature of the negative sample are stored in correspondence.

14. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a target image to be processed; A prediction module is used to determine the classification information of the target image using a pre-trained classification model, wherein the classification information includes the category to which each pixel of the target image belongs and at least one of the categories to which the target image belongs, wherein the classification model is trained based on the following method: predicting a first unlabeled sample image to obtain a prediction result containing the classification information; determining from the first sample image an unreliable sample whose reliability of the prediction result is lower than a preset threshold, and determining the target category to which the unreliable sample does not belong, wherein the unreliable sample includes a pixel or an image; and training a preset model based on negative samples to obtain the classification model, wherein the negative samples are constructed based on the unreliable samples and the corresponding target category to which they do not belong.

15. An electronic device, characterized in that: The device includes a processor, a memory, and computer instructions stored in the memory and executable by the processor. When the processor executes the computer instructions, the method according to any one of claims 1 to 13 is implemented.

Images