Semantic Pseudo-Label Error Correction Method, Device, Equipment and Storage Medium

Abstract

The present application discloses a semantic pseudo-label error correction method, device, equipment and storage medium, relating to the technical field of machine vision. The method includes: obtaining an initial semantic pseudo-label of a sample image in a training dataset, performing binary segmentation on the initial semantic pseudo-label to obtain a binary image, and performing superpixel segmentation on the sample image to obtain a plurality of superpixels; based on the binary image, performing preliminary classification on each superpixel to obtain a first superpixel set and a second superpixel set, and using a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; for any non-defective superpixel, based on the gray-scale distribution of each superpixel, performing image reconstruction on the non-defective superpixel; based on each non-defective superpixel after image reconstruction, correcting the initial semantic pseudo-label to obtain a target semantic pseudo-label. Thus, the correction of the semantic pseudo-label can be realized.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular, to a method, device, equipment and storage medium for correcting semantic pseudo-label errors. Background Art

[0002] Semantic segmentation is an image processing technology that divides an image into several characteristic and non-overlapping regions and assigns semantic categories based on image feature information. Currently, when training a semantic segmentation model, it is usually trained using a labeled real dataset, and this training method is called fully supervised learning. Its training result is relatively objective and the performance of the obtained model is good. However, this pixel-level image annotation is very labor-consuming.

[0003] Therefore, some relevant scholars have proposed weakly supervised learning to solve the high dependence of fully supervised learning on manual labeling. Weakly supervised learning is trained using data with unreliable label quality. Although it can solve problems such as scarce labeled data and high cost of accurate labeling, weakly supervised learning inevitably uses unreliable semantic pseudo-labels (mislabeled or missing labels), so the performance of the trained model is poor. Summary of the Invention

[0004] The main purpose of the present application is to provide a method, device, equipment and storage medium for correcting semantic pseudo-label errors, so as to correct semantic pseudo-labels and solve the problem of the decline in model performance caused by weakly supervised learning.

[0005] To achieve the above object, the present application provides a method for correcting semantic pseudo-label errors, including:

[0006] Obtain the initial semantic pseudo-label of the sample image in the training dataset, perform binary segmentation on the initial semantic pseudo-label to obtain a binary image, and perform superpixel segmentation on the sample image to obtain a plurality of superpixels;

[0007] Based on the binary image, perform preliminary classification on each of the superpixels to obtain a first superpixel set and a second superpixel set, and use a preset classifier to perform secondary classification on each of the superpixels in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; wherein, the defect probability of the first superpixel set is higher than that of the second superpixel set;

[0008] For any non-defective superpixel in the non-defective superpixel set, perform image reconstruction on the non-defective superpixel based on the gray-scale distribution of each of the superpixels;

[0009] Based on each non-defective superpixel after image reconstruction, correct the initial semantic pseudo-label to obtain a target semantic pseudo-label.

[0010] Optionally, the image reconstruction of the non-defective superpixels based on the gray-scale distribution of each of the superpixels includes: determining the center point distances between the non-defective superpixels and their respective adjacent superpixels, and determining a preset number of relevant superpixels from the respective adjacent superpixels based on the center point distances; performing fitting, discretization, and normalization processing on the gray-scale distributions of the relevant superpixels to obtain a normalization function; determining a cumulative probability function based on the normalization function, generating a random number, and using a preset gray-scale calculation formula to determine the reconstructed gray-scale values of the pixels in the non-defective superpixels based on the cumulative probability function and the random number; and performing image reconstruction on the non-defective superpixels based on the reconstructed gray-scale values of the pixels in the non-defective superpixels.

[0011] Optionally, the performing fitting, discretization, and normalization processing on the gray-scale distributions of the relevant superpixels to obtain a normalization function includes: fitting the gray-scale distributions of the relevant superpixels using an asymmetric generalized Gaussian distribution function to obtain a fitting function; performing a rounding operation on each point on the fitting function to obtain a discretized function; each point on the discretized function being an integer point within a preset range; and performing normalization processing on the discretized function to obtain the normalization function.

[0012] Optionally, the preset gray-scale calculation formula is:

[0013] ;

[0014] wherein, is the reconstructed gray-scale value of the i-th pixel in the non-defective superpixel, is the cumulative probability function, k is a variable, N represents a natural number, is the random number.

[0015] Optionally, the preliminary classification of each of the superpixels based on the binary image to obtain a first superpixel set and a second superpixel set includes: determining a target region corresponding to each of the superpixels in the binary image, the position of the target region in the binary image being the same as the position of the corresponding superpixel in the sample image; for any superpixel, if it is determined that the corresponding target region meets a preset condition, then dividing the superpixel into the first superpixel set; the preset condition being that the proportion of the pixel points with a first preset value in the target region is greater than or equal to a preset threshold; for any superpixel, if it is determined that the corresponding target region does not meet the preset condition, then dividing the superpixel into the second superpixel set.

[0016] Optionally, the secondary classification of each of the superpixels in the second superpixel set using the preset classifier to obtain a defective superpixel set and a non-defective superpixel set includes: training the preset classifier using the first superpixel set and the second superpixel set; and performing secondary classification on each of the superpixels in the second superpixel set using the trained preset classifier to obtain the defective superpixel set and the non-defective superpixel set.

[0017] Optionally, the obtaining of the initial semantic pseudo-label of the sample image in the training dataset includes: inputting the sample image in the training dataset into a pre-trained defect detection model to obtain a class activation map; and performing upsampling on the class activation map to obtain the initial semantic pseudo-label.

[0018] In addition, to achieve the above object, the present application further provides a semantic pseudo-label error correction device, including: an acquisition module, configured to acquire the initial semantic pseudo-label of the sample image in the training dataset, perform binary segmentation on the initial semantic pseudo-label to obtain a binary image, and perform superpixel segmentation on the sample image to obtain a plurality of superpixels; a classification module, configured to perform preliminary classification on each of the superpixels based on the binary image to obtain a first superpixel set and a second superpixel set, and perform secondary classification on each of the superpixels in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than that of the second superpixel set; an image reconstruction module, configured to perform image reconstruction on any non-defective superpixel in the non-defective superpixel set based on the gray-scale distribution of each of the superpixels; and a correction module, configured to correct the initial semantic pseudo-label based on each of the image-reconstructed non-defective superpixels to obtain a target semantic pseudo-label.

[0019] The present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, the semantic pseudo-label error correction method as described in any one of the above is implemented.

[0020] The present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the semantic pseudo-label error correction method as described in any one of the above is implemented.

[0021] The semantic pseudo-label error correction method of the present application obtains a binary image by binarizing and segmenting an initial semantic pseudo-label, and performs superpixel segmentation on a sample image to obtain multiple superpixels; then, the binary image is used to perform a primary classification on each superpixel, that is, the superpixels are divided into a first superpixel set and a second superpixel set according to the annotation results of the initial semantic pseudo-label, where the first superpixel set represents the area with annotation defects in the initial semantic pseudo-label, and the second superpixel set represents the area without annotation defects in the initial semantic pseudo-label; then, a preset classifier is used to perform a secondary classification on the superpixels in the second superpixel set, so as to divide the superpixels that may have defects but are not annotated by the initial semantic pseudo-label into the defective superpixel set, and divide the superpixels that do not have defects but are not annotated by the initial semantic pseudo-label into the non-defective superpixel set; further, image reconstruction is performed on each non-defective superpixel in the non-defective superpixel set to eliminate the defects mislabeled by the initial semantic pseudo-label in the non-defective superpixel; finally, the initial semantic pseudo-label is corrected based on each non-defective superpixel after image reconstruction to obtain a target semantic pseudo-label, realizing the correction of the semantic pseudo-label and solving the problem of the decline in model performance caused by weak supervision learning. Description of the Drawings

[0022] Figure 1 is one of the flowcharts of the semantic pseudo-label error correction method according to an embodiment of the present application;

[0023] Figure 2 is the second flowchart of the semantic pseudo-label error correction method according to an embodiment of the present application;

[0024] Figure 3 is the third flowchart of the semantic pseudo-label error correction method according to an embodiment of the present application;

[0025] Figure 4 is the fourth flowchart of the semantic pseudo-label error correction method according to an embodiment of the present application;

[0026] Figure 5 is the fifth flowchart of the semantic pseudo-label error correction method according to an embodiment of the present application;

[0027] Figure 6 is a schematic diagram of the semantic pseudo-label error correction device according to an embodiment of the present application;

[0028] Figure 7 illustrates a schematic diagram of the physical structure of an electronic device;

[0029] In the figure: 600, semantic pseudo-label error correction device; 610, acquisition module; 620, classification module; 630, image reconstruction module; 640, correction module; 710, processor; 720, communication interface; 730, memory; 740, communication bus.

[0030] The realization, functional features and advantages of the present application will be further described in conjunction with embodiments with reference to the accompanying drawings. Detailed implementation manners

[0031] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts shall fall within the protection scope of the present application.

[0032] Semantic segmentation is an image processing technology that divides an image into several non-overlapping regions with characteristics and assigns semantic categories based on image feature information. Semantic segmentation simplifies the image, extracts the image information, facilitates scene understanding and high-level vision tasks, and is an important processing technology in the fields of computer vision and image recognition.

[0033] Currently, deep learning models are mainly trained using well-annotated real datasets, and this training method is called fully supervised learning. Traditional fully supervised learning methods require accurate manual annotation labels for each training sample, which not only consumes a large amount of time and resources, but also in some professional fields, such as medical image analysis, the cost of expert annotation is particularly high. Therefore, how to reduce the dependence on high-quality annotated data has become a research hotspot.

[0034] In this context, weakly supervised learning emerged as an alternative. Weakly supervised learning allows using labels with incomplete information or errors to train the model. This type of label is usually called a "pseudo-label", and they can be provided by non-experts or even automatically generated through some heuristic rules. Nevertheless, the quality of pseudo-labels often varies, including misannotations (i.e., wrongly assigning a label to a certain category) and missed annotations (failing to correctly identify a certain category), which makes model training complex and may ultimately lead to a decline in the performance of the trained model. Therefore, in the framework of weakly supervised learning, how to effectively reduce the noise impact introduced by pseudo-labels has become an urgent technical challenge to be solved.

[0035] Based on this, the embodiments of the present application provide a method, device, equipment and storage medium for correcting semantic pseudo-label errors. By using the method of image local reconstruction to correct semantic pseudo-label errors, the dependence of machine vision algorithms on pixel-level semantic labels is effectively reduced, and accurate detection and recognition of product surface defects are realized on the basis of low manual annotation cost.

[0036] Figure 1It is one of the flowcharts of the semantic pseudo-label error correction method according to an embodiment of the present application. The semantic pseudo-label error correction method can be executed by a processor of an electronic device, such as Figure 1 shown. The semantic pseudo-label error correction method may include the following steps:

[0037] Step 110: Obtain the initial semantic pseudo-labels of the sample images in the training dataset, perform binary segmentation on the initial semantic pseudo-labels to obtain a binary image, and perform superpixel segmentation on the sample images to obtain a plurality of superpixels.

[0038] Step 120: Based on the binary image, perform preliminary classification on each superpixel to obtain a first superpixel set and a second superpixel set, and use a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; wherein, the defective probability of the first superpixel set is higher than that of the second superpixel set.

[0039] Step 130: For any non-defective superpixel in the non-defective superpixel set, perform image reconstruction on the non-defective superpixel based on the gray-scale distribution of each superpixel.

[0040] Step 140: Based on each non-defective superpixel after image reconstruction, correct the initial semantic pseudo-labels to obtain target semantic pseudo-labels.

[0041] First of all, it should be noted that the semantic pseudo-label error correction method according to an embodiment of the present application can be applied to any surface defect detection scenario, and the application scenario is not specifically limited here. For example, it can be applied to the scenario of defect detection on the uniform texture surface of an airplane.

[0042] In step 110, a variety of sample images can be collected first to form a training dataset. These sample images can be object surface images, and some sample images may contain object surface defects. Then, the training dataset is divided into a training set and a test set. The sample images in the training set can be labeled images, and the labels can include information such as whether there are defects and defect categories, etc.; the model can be trained using the training set and tested using the test set. It should be noted that the embodiments of the present application mainly detect surface defects. Therefore, the models described in the embodiments of the present application can all be defect detection models, and the defect detection models can be trained based on the existing Resnet model.

[0043] In this embodiment, the initial semantic pseudo-label may refer to a semantic pseudo-label obtained by a conventional method or a specific method without error correction. The method for obtaining the initial semantic pseudo-label of the sample images in the training dataset may be: obtained by using the self-labeling method, that is, using a pre-trained model to predict unlabeled sample images and taking the prediction result as the semantic pseudo-label. This method depends on the quality of the initial model. If the initial model is good enough, the generated semantic pseudo-label can be used for further training to improve the model. The method for obtaining the initial semantic pseudo-label of the sample images in the training dataset may also be: obtained by using the clustering method, that is, using an unsupervised learning algorithm (such as K-means) to cluster unlabeled data, and then assigning a label to each cluster. This label can be an inference based on the known class with the closest features or a newly defined class according to the characteristics of the data within the cluster. Specifically, other methods can also be used to obtain the initial semantic pseudo-label of the sample images in the training dataset, which is not specifically limited here.

[0044] After obtaining the initial semantic pseudo-label of the sample image, adaptive binary segmentation can be performed on the initial semantic pseudo-label to divide the pixel values of each pixel point of the semantic pseudo-label into two categories for subsequent analysis of the initial semantic pseudo-label. Specifically, the OTSU algorithm can be used to perform binary segmentation on the initial semantic pseudo-label to convert the semantic pseudo-label into a black-and-white image. After conversion, the pixel values of some pixel points of the semantic pseudo-label are 0 (black), and the pixel values of the other pixel points are 1 (white). Among them, the area where the white pixel points are located represents the defective area in the semantic pseudo-label, and the area where the black pixel points are located represents the non-defective area in the semantic pseudo-label. Thus, the defective area and the non-defective area labeled in the semantic pseudo-label can be distinguished by the pixel values.

[0045] It should be noted that the basic principle of the OTSU algorithm is to find a threshold in a single-band grayscale image. After the image is segmented into two parts according to this threshold, the between-class variance between these two parts is maximized. The purpose of doing this is to maximize the contrast between the foreground and the background, so as to obtain a better segmentation effect. The process of using the OTSU algorithm to perform binary segmentation on the initial semantic pseudo-label can be as follows: First, calculate the grayscale histogram of the initial semantic pseudo-label to obtain the number of pixels for each grayscale level; then set some initial parameters, including the total number of pixels, the total pixel intensity of the image, etc.; further traverse all possible grayscale levels as candidate thresholds, and for each candidate threshold, calculate the following parameters: the number of foreground and background pixels, the average grayscale values of the foreground and background, and the between-class variance; further calculate the between-class variance, which is a measure of the difference between the foreground and the background; record the threshold with the maximum between-class variance during the traversal process, and this threshold is the optimal threshold obtained by the OTSU algorithm; finally, use the found optimal threshold to perform binary processing on the original initial semantic pseudo-label, that is, set the pixel points smaller than the threshold as the background (usually black 0), and set the pixel points greater than or equal to the threshold as the foreground (usually white 1).

[0046] In this embodiment, the sample image can also be subjected to superpixel segmentation to obtain a plurality of superpixels. Superpixel segmentation is to subdivide the sample image into multiple small regions according to the similarity of features such as color, brightness, and texture among the pixels of the sample image. In this embodiment, the method of linear iterative clustering (SLIC) can be used to perform superpixel segmentation on the sample image.

[0047] In step 120, after obtaining the binary image corresponding to the semantic pseudo-label and several superpixels corresponding to the sample image, these superpixels can be initially classified according to the binary image to obtain a first superpixel set and a second superpixel set. It should be noted that the first superpixel set contains all superpixels with a higher defect probability, and the second superpixel set contains all superpixels with a lower defect probability. It can be understood that the obtained binary image can clearly reflect the defect results labeled by the initial semantic pseudo-label. Therefore, initially classifying all superpixels according to the binary image, that is, classifying all superpixels according to the labeling results of the initial semantic pseudo-label, and the result of the initial classification can reflect the labeling results of the initial semantic pseudo-label.

[0048] After the initial classification, further, a preset classifier can be used to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set. It can be understood that the second superpixel set is composed of superpixels with a relatively small probability of defects determined based on the initial semantic pseudo-label. However, due to the unreliability of the initial semantic pseudo-label, there may be regions with missed labels and mislabeled regions in the second superpixel set. Therefore, in this embodiment, the second superpixel set that is not labeled in the semantic pseudo-label is subjected to secondary classification to find possible regions with missed labels and possible mislabeled regions. After the secondary division, the region where the superpixel belonging to the defective superpixel set is located may be the region with a missed label in the semantic pseudo-label, and there may be mislabeling in the region where the superpixel belonging to the non-defective superpixel set is located.

[0049] For the region with a missed label, it is possible to directly select to label the region with the missed label to correct the semantic pseudo-label. For the mislabeled region, in the embodiment of the present application, image reconstruction is performed on the region image where the superpixel in the defective superpixel set is located, and the non-defective superpixel is reconstructed using the gray-scale distribution of its surrounding background, which is equivalent to performing median filtering to reduce the interference of outliers, thereby eliminating the mislabeling error in the region where the non-defective superpixel is located. Finally, the sample image after image reconstruction can be used to re-determine the semantic pseudo-label once, and the re-determined semantic pseudo-label is the corrected target semantic pseudo-label.

[0050] Thus, by using the local image reconstruction method to correct the semantic pseudo-label, the problem that the performance of the model obtained by training decreases due to errors (mislabeling and / or missed labeling) in the semantic pseudo-label in the existing semantic segmentation model based on weakly supervised learning is solved. And by generating pixel-level semantic pseudo-labels through class labels, manual annotation is replaced, effectively reducing the cost of manual labeling, accelerating the model deployment efficiency, and also helping to improve the detection accuracy and accuracy of the defective detection model obtained by training.

[0051] The above steps will be introduced in detail below.

[0052] Figure 2 is the second flowchart of the semantic pseudo-label error correction method according to the embodiment of the present application. As Figure 2 shown, in step 110, obtaining the initial semantic pseudo-label of the sample image in the training dataset may include the following steps:

[0053] Step 210: Input the sample image in the training dataset into a pre-trained defective detection model to obtain a class activation map.

[0054] Step 220: Upsample the class activation map to obtain the initial semantic pseudo-label.

[0055] Specifically, after obtaining the training data set, the defect detection model can be first trained using the training set to obtain a fine-tuned defect detection model, so that the defect detection model can determine whether there are defects and the types of defects in the sample image. It should be noted here that the model structure of the defect detection model can directly adopt the model structure of the Resnet model, that is, the Resnet model can be directly trained using the training set to obtain a fine-tuned defect detection model.

[0056] Further, the sample image can be input into the pre-trained defect detection model for processing. There are multiple convolutional layers, pooling layers, etc. inside the defect detection model, which are used to extract multi-level features of the sample image. In this embodiment, there is usually a fully connected layer at the end of the network of the defect detection model for final classification. The weight matrix can be obtained from the fully connected layer, and these weights represent the importance of different features for the final output. In addition, different levels of feature maps can be obtained from the convolutional layer part, and the feature maps can reflect the intermediate representation obtained after the sample image is subjected to the filter convolution operation.

[0057] After obtaining the weight matrix and the feature maps, the weight matrix and the feature maps can be multiplied. Here, the multiplication operation can be element-wise multiplication of corresponding positions, or other forms of fusion methods, such as matrix multiplication. After multiplying the weight matrix and the feature maps, the class activation map can be obtained. The obtained class activation map may be smaller than the original sample image. Therefore, the class activation map also needs to be upsampled to obtain a class activation map with the same size as the sample image. This class activation map is the initial semantic pseudo-label. The upsampled class activation map can be normalized to ensure that the element values in the class activation map are between 0 and 1. In the finally obtained class activation map, each element value represents the probability that the corresponding pixel position belongs to the defect area. The larger the element value, the more likely it is that the position is the location of the defect.

[0058] It should be noted that the Class Activation Mapping (CAM) is a visualization technique used to show which parts of an image are most critical when a neural network makes a certain class prediction. Simply put, the CAM technique can generate heat maps that highlight the regions in the input image that are most relevant to the model's prediction. In this embodiment, the defect positions predicted by the defect detection model in the sample image can be marked using the class activation map as the initial semantic pseudo-label.

[0059] After obtaining the initial semantic pseudo-label, the semantic pseudo-label is adaptively binarized and segmented to obtain a binary image. At the same time, the sample image can be superpixel segmented to obtain a number of superpixels, and each superpixel has an independent number.

[0060] Figure 3This is the third flowchart of the semantic pseudo-label error correction method according to the embodiments of the present application. As Figure 3 shown, in step 120, based on the binary image, each superpixel is preliminarily classified to obtain a first superpixel set and a second superpixel set, which may include the following steps:

[0061] Step 310: Determine the target area corresponding to each superpixel in the binary image, and the position of the target area in the binary image is the same as the position of the corresponding superpixel in the sample image.

[0062] Step 320: For any superpixel, if it is determined that the corresponding target area meets the preset condition, the superpixel is classified into the first superpixel set; the preset condition is that the proportion of pixel points with the first preset value in the target area is greater than or equal to the preset threshold.

[0063] Step 330: For any superpixel, if it is determined that the corresponding target area does not meet the preset condition, the superpixel is classified into the second superpixel set.

[0064] It should be noted that the first preset value can be determined according to the binary image, and the first preset value can take any one of the two pixel values in the binary image. In this embodiment, since the binary image includes two pixel values, black 0 and white 1, the first preset value can be set to 1. In addition, the preset threshold can be set by the staff according to actual needs. For example, the preset threshold can be set to 50%, and the preset threshold is not specifically limited here.

[0065] Specifically, each superpixel can be positionally corresponding to the binary image to find the corresponding position area of each superpixel in the binary image, and the corresponding position area is the target area corresponding to the superpixel. The method of positionally corresponding each superpixel to the binary image can be: establish the same coordinate system on the binary image and the sample image after superpixel segmentation, determine the coordinates of each pixel point in the superpixel, and then find the corresponding area in the binary image based on the coordinates of each pixel point in the superpixel, and this area is the target area corresponding to the superpixel.

[0066] Taking the first preset value as 1 and the preset threshold as 50% as an example, if the proportion of pixel points with the pixel value of 1 in the target area corresponding to the superpixel is greater than or equal to 50%, the superpixel can be classified into the first superpixel set. If the proportion of pixel points with the pixel value of 1 in the target area corresponding to the superpixel is less than 50%, the superpixel can be classified into the second superpixel set. The superpixels in the first superpixel set are high-probability defective superpixels, and the superpixels in the second superpixel set are low-probability defective superpixels.

[0067] Figure 4 This is the fourth flowchart of the semantic pseudo-label error correction method according to the embodiments of the present application. AsFigure 4 As shown, in step 120, a preset classifier is used to perform secondary classification on each superpixel in the second superpixel set, and a defective superpixel set and a non-defective superpixel set can be obtained, which may include the following steps:

[0068] Step 410: Use the first superpixel set and the second superpixel set to train a preset classifier.

[0069] Step 420: Use the trained preset classifier to perform secondary classification on each superpixel in the second superpixel set, and obtain a defective superpixel set and a non-defective superpixel set.

[0070] It should be noted that the preset classifier can be a Multiple Instance Learning (MIL) classifier, which is a special supervised learning framework. Different from traditional single-instance learning classifiers, in MIL, the training data is given in the form of "bags", each bag contains multiple "instances", and the label is usually given to the whole bag rather than a single instance. In this embodiment, a key instance support vector machine can be selected as the multiple instance learning classifier. Taking the multiple instance learning classifier as an example, the secondary classification process will be introduced below.

[0071] In this embodiment, the first superpixel set can be used as a negative bag, and the second superpixel set can be used as a positive bag to train the multiple instance learning classifier. Specifically, the multiple instance learning classifier can extract features from the superpixels in the positive bag and the negative bag. The features can include (but are not limited to): color histogram, texture feature, shape descriptor, etc.; further, the superpixels can be converted into the form of feature vectors, and it is ensured that the information contained in the bag is sufficient to describe the category (positive / negative) of the bag; use a multiple instance learning algorithm (such as MILBoost algorithm, Diverse Density algorithm, MI-SVM algorithm, etc.) to train the model. In this setting, the goal of the model is to learn to distinguish between positive bags and negative bags.

[0072] After the multi-instance learning classifier is trained using positive bags and negative bags, the trained multi-instance learning classifier can be used to classify the superpixels in the second superpixel set. At this time, the multi-instance learning classifier can distinguish positive bags and negative bags, that is, distinguish which superpixels can be superpixels of the defective area and which superpixels can be superpixels of the non-defective area. By using the trained multi-instance learning classifier to further divide the superpixels in the second superpixel set, the areas in the initial semantic pseudo-label that are not labeled as defective can be further divided to obtain a defective superpixel set and a non-defective superpixel set; the area corresponding to the superpixels divided into the defective superpixel set is the missed-labeled area, and the area corresponding to the superpixels divided into the non-defective superpixel set can be almost certainly defect-free, but there may be mislabels in the initial semantic pseudo-label, that is, the area corresponding to the superpixels in the non-defective superpixel set is also labeled as defective. Therefore, in this embodiment, image reconstruction is performed on the superpixels in the non-defective superpixel set, which is equivalent to performing median filtering, to reduce the interference of outliers.

[0073] Figure 5 It is the fifth flowchart of the semantic pseudo-label error correction method according to the embodiment of the present application. As Figure 5 shown, in step 130, based on the gray-scale distribution of each superpixel, image reconstruction of non-defective superpixels may include the following steps:

[0074] Step 510: Determine the central point distance between the non-defective superpixel and each adjacent superpixel, and determine a preset number of relevant superpixels from each adjacent superpixel based on each central point distance.

[0075] Step 520: Perform fitting, discretization, and normalization processing on the gray-scale distributions of the relevant superpixels to obtain a normalization function.

[0076] Step 530: Determine a cumulative probability function based on the normalization function, generate a random number, and determine the reconstructed gray-scale value of each pixel point in the non-defective superpixel based on the cumulative probability function and the random number using a preset gray-scale calculation formula.

[0077] Step 540: Perform image reconstruction on the non-defective superpixel based on the reconstructed gray-scale values of each pixel point in the non-defective superpixel.

[0078] It should be noted that the superpixels in the non-defective superpixel set are all denoted as non-defective superpixels for subsequent description. In this embodiment, all non-defective superpixels need to be subjected to image reconstruction. Here, taking a non-defective superpixel as an example, the process of image reconstruction of non-defective superpixels is introduced.

[0079] In this embodiment, the relevant superpixels corresponding to the non-defective superpixels can be determined first. The relevant superpixels refer to the superpixels that are relatively close to the non-defective superpixels and have a certain correlation with the non-defective superpixels. Specifically, the central point distance between the non-defective superpixel and all superpixels within a certain range around it can be calculated, and then the relevant superpixels can be determined based on each central point distance. If the central point of the non-defective superpixel is denoted as and the central points of the remaining superpixels are denoted as , where n is the total number of superpixels within a certain range around the non-defective superpixel, can be connected to each , and the length of each connection line can be calculated. This length is the central point distance. Further, a preset number of superpixels can be selected, and the central point distances of these superpixels are the smallest. These superpixels are used as the above-mentioned relevant superpixels.

[0080] It should be noted that the preset number can be set by the staff according to actual needs. For example, the preset number can be 3, that is, 3 superpixels with the smallest central point distances are selected as the relevant superpixels. The preset number is not specifically limited here.

[0081] After determining the relevant superpixels corresponding to the non-defective superpixels, the gray-level distributions of the relevant superpixels can be fitted first. In some embodiments, the gray-level distributions of the relevant superpixels are fitted using an asymmetric generalized Gaussian distribution function to obtain a fitting function.

[0082] Specifically, considering the non-uniformity of the local gray-level distribution of the image background, in this embodiment, an asymmetric generalized Gaussian distribution function is selected to fit the gray-level distributions of the relevant superpixels, and the following fitting function can be obtained :

[0083] ;

[0084] where are the function parameters of , is the gamma function, and x is the variable.

[0085] It should be noted here that the fitting function is obtained by fitting the gray-level distribution histogram of each relevant superpixel. The value range of the abscissa X in the gray-level distribution histogram is 0 - 255, and X is an integer. For each gray value X, the value of the ordinate Y in the gray-level distribution histogram represents the number of pixel points with this gray value in the image. For example, if the number of pixel points with a gray value of 125 in the surrounding superpixels (i.e., the relevant superpixels) is 1000, then the corresponding function value in the gray-level distribution histogram is (125, 1000), and the asymmetric generalized Gaussian distribution function fits the gray-level distribution histogram.

[0086] Further, the fitting function is discretized. In some embodiments, rounding operations are performed on each point of the fitting function to obtain a discretized function; each point on the discretized function is an integer point within a preset range. Specifically, the fitting function can be integer-valued within a preset range of 0 - 255 for each point thereon to obtain a discretized function, and the discretized function can be denoted as .

[0087] Furthermore, the discretized function can be normalized to obtain a normalized function , and the normalized function is as follows:

[0088] ;

[0089] wherein, is a variable with a value range of any integer point from 0 to 255; represents the gray level.

[0090] In this embodiment, after obtaining the normalized function, the cumulative probability function of the normalized function can be calculated. The cumulative probability function is used to describe the probability that a random variable takes a value less than or equal to a certain specific value. Specifically, the cumulative probability function can be calculated by the following formula :

[0091] ;

[0092] wherein, k is a variable with a value range of any natural number from 0 to 255, and N represents a natural number.

[0093] It should be noted that the normalized function represents the probability that each pixel point in the relevant superpixel takes the corresponding gray value, while the cumulative probability function refers to the sum of probabilities of all gray values less than or equal to this gray value. By normalizing the discretized function, the values of the discretized function can be converted to between 0 and 1, and the one-to-one correspondence between X and Y can be ensured.

[0094] Furthermore, a random number can be generated within the range of , and then the reconstructed gray value of each pixel point in the non-defective superpixel can be determined using the following preset gray calculation formula:

[0095] ;

[0096] wherein, is the reconstructed gray value of the i-th pixel point in the non-defective superpixel, is the cumulative probability function, k is a variable, N represents a natural number, is a random number.

[0097] As an example, assume that the probabilities of the gray values equal to 125 and 200 in the relevant superpixel are both 0.15. If a random number generated is 0.16, it is impossible to determine whether the reconstructed gray value is 125 or 200. After processing the probabilities of the corresponding gray values of the relevant superpixel through the cumulative probability function, assuming that the cumulative probabilities of the gray values 125 and 200 are 0.2 and 0.8 respectively, the reconstructed gray value can be determined to be 125 through the above formula for reconstructing the gray value (at this time, it is necessary to assume that when the gray values are 124 and 126, the cumulative probability function values are 0.1 and 0.3, that is, the gray value corresponding to the smallest difference after subtraction is 125).

[0098] After traversing all the pixel points in the current non-defective superpixel using the preset gray value calculation formula, the reconstructed gray values of all the pixel points in the non-defective superpixel can be obtained. Replacing the corresponding original gray values with the reconstructed gray values of all the pixel points, the non-defective superpixel after image reconstruction can be obtained.

[0099] Continuously repeat the above image reconstruction process to perform local image reconstruction on all non-defective superpixels. After all non-defective superpixels undergo image reconstruction, the pixel values of the non-defective superpixels are relatively uniform and there will be no abnormal pixel values. In this way, when re-determining the semantic pseudo-label, the probability of mislabeling can be reduced, thereby improving the reliability of the semantic pseudo-label.

[0100] Thus, by combining the initial semantic pseudo-label with superpixel segmentation, all superpixels are initially segmented based on the initial semantic pseudo-label to distinguish the defects marked in the initial semantic pseudo-label and the unmarked defects; then, the preset classifier is used to perform secondary classification on all superpixels with unmarked defects in the initial semantic pseudo-label to distinguish the superpixels that may have defects and the superpixels that hardly have defects. The area corresponding to the superpixels that may have defects determined through the secondary classification is the missed label area, and the area corresponding to the superpixels that hardly have defects determined through the secondary classification is the mislabeled area. By performing image reconstruction on non-defective superpixels, the probability of mislabeling in the initial semantic pseudo-label can be effectively reduced; at the same time, by labeling the area corresponding to the defective superpixels, the probability of missed labeling in the initial semantic pseudo-label can be effectively reduced. Using the corrected semantic pseudo-label to train the defect detection model can effectively improve the model performance, improve the accuracy and precision of defect detection, and effectively reduce the manual labeling cost and speed up the model deployment efficiency.

[0101] Based on the above embodiments, the embodiments of the present application further provide a semantic pseudo-label error correction device. Figure 6 It is a schematic diagram of the semantic pseudo-label error correction device of the embodiments of the present application, as Figure 6As shown in the figure, the semantic pseudo-label error correction device 600 may include an acquisition module 610, a classification module 620, an image reconstruction module 630, and a correction module 640.

[0102] Among them, the acquisition module 610 is used to obtain the initial semantic pseudo-labels of the sample images in the training dataset, perform binary segmentation on the initial semantic pseudo-labels to obtain binary images, and perform superpixel segmentation on the sample images to obtain multiple superpixels; the classification module 620 is used to perform preliminary classification on each superpixel based on the binary image to obtain a first superpixel set and a second superpixel set, and use a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; among them, the defective probability of the first superpixel set is higher than that of the second superpixel set; the image reconstruction module 630 is used to perform image reconstruction on any non-defective superpixel in the non-defective superpixel set based on the gray-scale distribution of each superpixel; the correction module 640 is used to correct the initial semantic pseudo-labels based on each non-defective superpixel after image reconstruction to obtain target semantic pseudo-labels.

[0103] Thus, the acquisition module 610 performs binary segmentation on the initial semantic pseudo-labels to obtain binary images, and performs superpixel segmentation on the sample images to obtain multiple superpixels; the classification module 620 then performs primary classification on each superpixel through the binary image, that is, divides the superpixels into a first superpixel set and a second superpixel set through the annotation results of the initial semantic pseudo-labels. Among them, the first superpixel set represents the area where defects are annotated in the initial semantic pseudo-labels, and the second superpixel set represents the area where no defects are annotated in the initial semantic pseudo-labels; then, the preset classifier is used to perform secondary classification on the superpixels in the second superpixel set to divide the superpixels that may have defects but are not annotated in the initial semantic pseudo-labels into the defective superpixel set, and divide the superpixels that have no defects and are not annotated in the initial semantic pseudo-labels into the non-defective superpixel set; further, the image reconstruction module 630 reconstructs each non-defective superpixel in the non-defective superpixel set to eliminate the defects mislabeled by the initial semantic pseudo-labels in the non-defective superpixels; finally, the correction module 640 corrects the initial semantic pseudo-labels based on each non-defective superpixel after image reconstruction to obtain target semantic pseudo-labels, realizing the correction of semantic pseudo-labels and solving the problem of the decline in model performance caused by weak supervision learning.

[0104] In some embodiments, the image reconstruction module 630 is specifically configured to: determine the central point distances between non-defective superpixels and their respective adjacent superpixels, and determine a preset number of relevant superpixels from the respective adjacent superpixels based on the central point distances; perform fitting, discretization, and normalization processing on the gray-level distributions of the respective relevant superpixels to obtain a normalization function; determine a cumulative probability function based on the normalization function, generate a random number, and use a preset gray-level calculation formula to determine the reconstructed gray-level values of each pixel point in the non-defective superpixels based on the cumulative probability function and the random number; perform image reconstruction on the non-defective superpixels based on the reconstructed gray-level values of each pixel point in the non-defective superpixels.

[0105] In some embodiments, the image reconstruction module 630 is further specifically configured to: fit the gray-level distributions of the respective relevant superpixels using an asymmetric generalized Gaussian distribution function to obtain a fitting function; perform a rounding operation on each point on the fitting function to obtain a discretization function; each point on the discretization function is an integer point within a preset range; perform normalization processing on the discretization function to obtain a normalization function.

[0106] In some embodiments, the preset gray-level calculation formula is:

[0107] ;

[0108] In the formula, is the reconstructed gray-level value of the i-th pixel point in the non-defective superpixel, is the cumulative probability function, k is a variable, N represents a natural number, is a random number.

[0109] In some embodiments, the classification module 620 is specifically configured to: determine a target region corresponding to each superpixel in the binary image, and the position of the target region in the binary image is the same as the position of the corresponding superpixel in the sample image; for any superpixel, if it is determined that the corresponding target region meets a preset condition, then divide the superpixel into the first superpixel set; the preset condition is that the proportion of pixel points with a first preset value in the target region is greater than or equal to a preset threshold; for any superpixel, if it is determined that the corresponding target region does not meet the preset condition, then divide the superpixel into the second superpixel set.

[0110] In some embodiments, the classification module 620 is further specifically configured to: train a preset classifier using the first superpixel set and the second superpixel set; perform secondary classification on each superpixel in the second superpixel set using the trained preset classifier to obtain a defective superpixel set and a non-defective superpixel set.

[0111] In some embodiments, the acquisition module 610 is specifically configured to: input the sample image in the training dataset into a pre-trained defect detection model to obtain a class activation map; perform upsampling on the class activation map to obtain an initial semantic pseudo-label.

[0112] It should be noted that for the details not disclosed in the semantic pseudo-label error correction device of this embodiment, please refer to the details disclosed in the embodiment of the semantic pseudo-label error correction method in this specification, and will not be elaborated here.

[0113] Based on the above embodiments, Figure 7 An example of the physical structure diagram of an electronic device is shown as Figure 7 shown. The electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communications interface 720, and the memory 730 complete communication with each other through the communication bus 740. The processor 710 may call the logical instructions in the memory 730 to execute the semantic pseudo-label error correction method, which includes: obtaining the initial semantic pseudo-labels of the sample images in the training dataset, performing binary segmentation on the initial semantic pseudo-labels to obtain a binary image, and performing superpixel segmentation on the sample images to obtain a plurality of superpixels; based on the binary image, performing preliminary classification on each superpixel to obtain a first superpixel set and a second superpixel set, and using a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; wherein, the defective probability of the first superpixel set is higher than that of the second superpixel set; for any non-defective superpixel in the non-defective superpixel set, based on the gray-scale distribution of each superpixel, performing image reconstruction on the non-defective superpixel; based on each non-defective superpixel after image reconstruction, correcting the initial semantic pseudo-labels to obtain the target semantic pseudo-labels.

[0114] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical disks, etc., which can store program codes.

[0115] Based on the above embodiments, on the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the semantic pseudo-label error correction method provided by the above-mentioned various methods. The method includes: obtaining an initial semantic pseudo-label of a sample image in a training dataset, performing binary segmentation on the initial semantic pseudo-label to obtain a binary image, and performing superpixel segmentation on the sample image to obtain a plurality of superpixels; based on the binary image, performing preliminary classification on each superpixel to obtain a first superpixel set and a second superpixel set, and using a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; wherein the defective probability of the first superpixel set is higher than that of the second superpixel set; for any non-defective superpixel in the non-defective superpixel set, based on the gray-scale distribution of each superpixel, performing image reconstruction on the non-defective superpixel; based on each non-defective superpixel after image reconstruction, correcting the initial semantic pseudo-label to obtain a target semantic pseudo-label.

[0116] Based on the above embodiments, on another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the semantic pseudo-label error correction method provided by the above-mentioned various methods. The method includes: obtaining an initial semantic pseudo-label of a sample image in a training dataset, performing binary segmentation on the initial semantic pseudo-label to obtain a binary image, and performing superpixel segmentation on the sample image to obtain a plurality of superpixels; based on the binary image, performing preliminary classification on each superpixel to obtain a first superpixel set and a second superpixel set, and using a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; wherein the defective probability of the first superpixel set is higher than that of the second superpixel set; for any non-defective superpixel in the non-defective superpixel set, based on the gray-scale distribution of each superpixel, performing image reconstruction on the non-defective superpixel; based on each non-defective superpixel after image reconstruction, correcting the initial semantic pseudo-label to obtain a target semantic pseudo-label.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0118] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Claims

1. A semantic pseudo-label error correction method, characterized in that Including: Obtaining an initial semantic pseudo-label of a sample image in a training dataset, performing binary segmentation on the initial semantic pseudo-label to obtain a binary image, and performing superpixel segmentation on the sample image to obtain a plurality of superpixels; Based on the binary image, performing preliminary classification on each of the superpixels to obtain a first superpixel set and a second superpixel set, and performing secondary classification on each of the superpixels in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein, the defect probability of the first superpixel set is higher than that of the second superpixel set; For any non-defective superpixel in the non-defective superpixel set, based on the gray-scale distribution of each of the superpixels, performing image reconstruction on the non-defective superpixel; Based on each of the non-defective superpixels after image reconstruction, correcting the initial semantic pseudo-label to obtain a target semantic pseudo-label.

2. The semantic pseudo-label error correction method according to claim 1, wherein The performing image reconstruction on the non-defective superpixel based on the gray-scale distribution of each of the superpixels includes: Determining the center point distance between the non-defective superpixel and each adjacent superpixel, and determining a preset number of relevant superpixels from each of the adjacent superpixels based on each of the center point distances; Performing fitting, discretization, and normalization processing on the gray-scale distribution of each of the relevant superpixels to obtain a normalization function; Based on the normalization function, determining a cumulative probability function, generating a random number, and determining the reconstructed gray-scale value of each pixel point in the non-defective superpixel based on the cumulative probability function and the random number using a preset gray-scale calculation formula; Based on the reconstructed gray-scale values of each pixel point in the non-defective superpixel, performing image reconstruction on the non-defective superpixel.

3. The semantic pseudo-label error correction method according to claim 2, characterized in that The performing fitting, discretization, and normalization processing on the gray-scale distribution of each of the relevant superpixels to obtain a normalization function includes: Using an asymmetric generalized Gaussian distribution function to fit the gray-scale distribution of each of the relevant superpixels to obtain a fitting function; Performing a rounding operation on each point on the fitting function to obtain a discretization function; each point on the discretization function is an integer point within a preset range; Performing normalization processing on the discretization function to obtain the normalization function.

4. The semantic pseudo-label error correction method according to claim 2, wherein The preset gray-scale calculation formula is: ； In the formula, is the reconstructed gray value of the i-th pixel point in the non-defective superpixel, is the cumulative probability function, k is a variable, and N represents a natural number, is the random number.

5. The semantic pseudo-label error correction method according to claim 1, wherein, The performing preliminary classification on each of the superpixels based on the binary image to obtain a first superpixel set and a second superpixel set includes: Determining a target region corresponding to each of the superpixels in the binary image, and the position of the target region in the binary image is the same as the position of the corresponding superpixel in the sample image; For any superpixel, if it is determined that the corresponding target region meets a preset condition, then dividing the superpixel into the first superpixel set; the preset condition is that the proportion of pixel points with a first preset value in the target region is greater than or equal to a preset threshold; For any superpixel, if it is determined that the corresponding target region does not meet the preset condition, then dividing the superpixel into the second superpixel set.

6. The semantic pseudo-label error correction method according to claim 1, characterized in that Performing secondary classification on each of the superpixels in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set includes: Training the preset classifier using the first superpixel set and the second superpixel set; Performing secondary classification on each of the superpixels in the second superpixel set using the trained preset classifier to obtain the defective superpixel set and the non-defective superpixel set.

7. The semantic pseudo-label error correction method according to claim 1, characterized in that, Obtaining the initial semantic pseudo-label of the sample image in the training dataset includes: Inputting the sample image in the training dataset into a pre-trained defect detection model to obtain a class activation map; Performing upsampling on the class activation map to obtain the initial semantic pseudo-label.

8. A semantic pseudo-label error correction device, characterized in that, Includes: An acquisition module for obtaining the initial semantic pseudo-label of the sample image in the training dataset, performing binary segmentation on the initial semantic pseudo-label to obtain a binary image, and performing superpixel segmentation on the sample image to obtain a plurality of superpixels; A classification module for performing preliminary classification on each of the superpixels based on the binary image to obtain a first superpixel set and a second superpixel set, and performing secondary classification on each of the superpixels in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein, the defect probability of the first superpixel set is higher than that of the second superpixel set; An image reconstruction module for reconstructing an image of any non-defective superpixel in the non-defective superpixel set based on the gray-scale distribution of each of the superpixels; A correction module for correcting the initial semantic pseudo-label based on each of the image-reconstructed non-defective superpixels to obtain a target semantic pseudo-label.

9. An electronic device, comprising a memory and a processor, wherein a computer program is stored on the memory, characterized in that, When the processor executes the computer program, it implements the semantic pseudo-label error correction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the semantic pseudo-label error correction method according to any one of claims 1 to 7.

Images