Semantic pseudo tag error correction method and device, equipment and storage medium

Abstract

The invention discloses a semantic pseudo tag error correction method and device, equipment and a storage medium, and relates to the technical field of machine vision. The method comprises the steps of obtaining an initial semantic pseudo tag of a sample image in a training data set, performing binarization segmentation on the initial semantic pseudo tag to obtain a binarized image, and performing superpixel segmentation on the sample image to obtain a plurality of superpixels; performing preliminary classification on each super-pixel based on the binarized image to obtain a first super-pixel set and a second super-pixel set, and performing secondary classification on each super-pixel in the second super-pixel set by using a preset classifier to obtain a defect super-pixel set and a non-defect super-pixel set; for any non-defect superpixel, performing image reconstruction on the non-defect superpixel based on the gray level distribution of each superpixel; and correcting the initial semantic pseudo-tag based on the reconstructed non-defect super-pixel of each image to obtain a target semantic pseudo-tag. Therefore, correction of the semantic pseudo tag is realized.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a semantic pseudo-label error correction method, device, equipment and storage medium. Background Art

[0002] Semantic segmentation is an image processing technology that divides an image into several characteristic and non-overlapping regions and assigns semantic categories based on image feature information. Currently, when training semantic segmentation models, it is usually trained using annotated real data sets. This training method is called fully supervised learning. Its training results are relatively objective and the obtained model performance is good, but this pixel-level image annotation consumes a lot of labor costs.

[0003] Therefore, some scholars have proposed weakly supervised learning to solve the problem of the high reliance of fully supervised learning on manual labeling. Weakly supervised learning uses data with unreliable label quality for training. Although it can solve the problems of scarce labeled data and high cost of accurate labeling, weakly supervised learning will inevitably use unreliable semantic pseudo-labels (wrong or missing labels), which will lead to poor performance of the trained model. Summary of the invention

[0004] The main purpose of this application is to provide a semantic pseudo-label error correction method, device, equipment and storage medium to achieve the correction of semantic pseudo-labels and solve the problem of model performance degradation caused by weakly supervised learning.

[0005] To achieve the above objectives, the present application provides a semantic pseudo-label error correction method, comprising: Obtaining initial semantic pseudo labels of sample images in a training data set, performing binarization segmentation on the initial semantic pseudo labels to obtain a binarized image, and performing superpixel segmentation on the sample images to obtain a plurality of superpixels; Preliminarily classifying each of the superpixels based on the binary image to obtain a first superpixel set and a second superpixel set, and performing secondary classification on each of the superpixels in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set; For any non-defective superpixel in the non-defective superpixel set, reconstructing the image of the non-defective superpixel based on the grayscale distribution of each superpixel; The initial semantic pseudo-label is corrected based on the non-defective superpixels reconstructed from each image to obtain a target semantic pseudo-label.

[0006] Optionally, the image reconstruction of the non-defective superpixel based on the grayscale distribution of each superpixel includes: determining the center point distance between the non-defective superpixel and each adjacent superpixel, and determining a preset number of related superpixels from each adjacent superpixel based on each center point distance; fitting, discretizing and normalizing the grayscale distribution of each related superpixel to obtain a normalized function; determining a cumulative probability function based on the normalized function, generating a random number, and determining the reconstructed grayscale value of each pixel point in the non-defective superpixel based on the cumulative probability function and the random number using a preset grayscale calculation formula; and image reconstruction of the non-defective superpixel based on the reconstructed grayscale value of each pixel point in the non-defective superpixel.

[0007] Optionally, the grayscale distribution of each of the related superpixels is fitted, discretized and normalized to obtain a normalized function, including: using an asymmetric generalized Gaussian distribution function to fit the grayscale distribution of each of the related superpixels to obtain a fitting function; performing rounding operation on each point on the fitting function to obtain a discretization function; each point on the discretization function is an integer point within a preset range; and normalizing the discretization function to obtain the normalized function.

[0008] Optionally, the preset grayscale calculation formula is: ; In the formula, is the reconstructed grayscale value of the i-th pixel in the non-defective superpixel, is the cumulative probability function, k is a variable, N represents a natural number, is the random number.

[0009] Optionally, the preliminary classification of each superpixel based on the binary image to obtain a first superpixel set and a second superpixel set includes: determining a target area corresponding to each superpixel in the binary image, the position of the target area in the binary image being the same as the position of the corresponding superpixel in the sample image; for any superpixel, if it is determined that the corresponding target area meets a preset condition, the superpixel is divided into the first superpixel set; the preset condition is that the proportion of pixels in the target area whose pixel value is a first preset value is greater than or equal to a preset threshold; for any superpixel, if it is determined that the corresponding target area does not meet the preset condition, the superpixel is divided into the second superpixel set.

[0010] Optionally, the use of a preset classifier to perform secondary classification on each of the superpixels in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set includes: training the preset classifier using the first superpixel set and the second superpixel set; and using the trained preset classifier to perform secondary classification on each of the superpixels in the second superpixel set to obtain the defective superpixel set and the non-defective superpixel set.

[0011] Optionally, obtaining initial semantic pseudo labels of sample images in the training data set includes: inputting the sample images in the training data set into a pre-trained defect detection model to obtain a class activation map; and upsampling the class activation map to obtain the initial semantic pseudo labels.

[0012] In addition, to achieve the above-mentioned purpose, the present application also provides a semantic pseudo-label error correction device, including: an acquisition module, used to obtain the initial semantic pseudo-label of a sample image in a training data set, and to perform binarization segmentation on the initial semantic pseudo-label to obtain a binarized image, and to perform superpixel segmentation on the sample image to obtain multiple superpixels; a classification module, used to perform preliminary classification of each of the superpixels based on the binarized image to obtain a first superpixel set and a second superpixel set, and to perform secondary classification on each of the superpixels in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set; an image reconstruction module, used to perform image reconstruction of the non-defective superpixel for any non-defective superpixel in the non-defective superpixel set based on the grayscale distribution of each superpixel; a correction module, used to correct the initial semantic pseudo-label based on the non-defective superpixel after each image reconstruction to obtain a target semantic pseudo-label.

[0013] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for correcting semantic pseudo-label errors as described above is implemented.

[0014] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for correcting semantic pseudo-label errors as described in any one of the above is implemented.

[0015] The semantic pseudo-label error correction method of the present application obtains a binary image by binarizing the initial semantic pseudo-label, and obtains multiple superpixels by superpixel segmentation of the sample image; then, each superpixel is initially classified through the binary image, that is, the superpixel is divided into a first superpixel set and a second superpixel set according to the labeling result of the initial semantic pseudo-label, wherein the first superpixel set represents the area with labeled defects in the initial semantic pseudo-label, and the second superpixel set represents the area with unlabeled defects in the initial semantic pseudo-label; then, the superpixels in the second superpixel set are secondary classified by using a preset classifier Classification is performed to divide superpixels that are not labeled with initial semantic pseudo-labels and may have defects into a defective superpixel set, and to divide superpixels that are not labeled with initial semantic pseudo-labels and do not have defects into a non-defective superpixel set; further, each non-defective superpixel in the non-defective superpixel set is image reconstructed to eliminate the defects of the non-defective superpixels that are mislabeled by the initial semantic pseudo-labels; finally, the initial semantic pseudo-labels are corrected based on the non-defective superpixels after image reconstruction to obtain the target semantic pseudo-labels, thereby realizing the correction of the semantic pseudo-labels and solving the problem of model performance degradation caused by weak supervision learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is one of the flow charts of the semantic pseudo-label error correction method of the embodiment of the present application; Figure 2 This is the second flowchart of the semantic pseudo-label error correction method of the embodiment of the present application; Figure 3 This is the third flowchart of the semantic pseudo-label error correction method of the embodiment of the present application; Figure 4 This is the fourth flowchart of the semantic pseudo-label error correction method of the embodiment of the present application; Figure 5 This is the fifth flow chart of the semantic pseudo-label error correction method of the embodiment of the present application; Figure 6 is a schematic diagram of a semantic pseudo-label error correction device according to an embodiment of the present application; Figure 7 An example of a physical structure diagram of an electronic device is provided; In the figure: 600, semantic pseudo-label error correction device; 610, acquisition module; 620, classification module; 630, image reconstruction module; 640, correction module; 710, processor; 720, communication interface; 730, memory; 740, communication bus.

[0017] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] Semantic segmentation is an image processing technology that divides an image into several characteristic and non-overlapping regions and assigns semantic categories based on image feature information. Semantic segmentation simplifies the image, refines the image information, facilitates scene understanding and high-level visual tasks, and is an important processing technology in the fields of computer vision and image recognition.

[0020] Currently, deep learning models are mainly trained using well-labeled real data sets, a training method called fully supervised learning. Traditional fully supervised learning methods require each training sample to have an accurate manual label, which not only consumes a lot of time and resources, but also in some professional fields, such as medical image analysis, the cost of expert labeling is particularly expensive. Therefore, how to reduce the dependence on high-quality labeled data has become a research hotspot.

[0021] In this context, weakly supervised learning has emerged as an alternative. Weakly supervised learning allows the use of labels with incomplete information or errors for model training. This type of label is usually called "pseudo-label", which can be provided by non-experts or even automatically generated through some heuristic rules. Nevertheless, the quality of pseudo-labels is often uneven, including mislabeling (i.e., labels that are incorrectly assigned to a certain category) and missing labels (failure to correctly identify a certain category), which makes model training complicated and may ultimately lead to a decrease in the performance of the trained model. Therefore, in the framework of weakly supervised learning, how to effectively reduce the noise introduced by pseudo-labels has become a technical challenge that needs to be solved urgently.

[0022] Based on this, the embodiments of the present application provide a semantic pseudo-label error correction method, device, equipment and storage medium. By using the method of local image reconstruction to correct the semantic pseudo-label error, the dependence of the machine vision algorithm on pixel-level semantic labels is effectively reduced, and accurate detection and identification of product surface defects are achieved on the basis of low manual labeling costs.

[0023] Figure 1 is one of the flow charts of the semantic pseudo-label error correction method of the embodiment of the present application. The semantic pseudo-label error correction method can be executed by a processor of an electronic device, such as Figure 1 As shown, the semantic pseudo-label error correction method may include the following steps: Step 110: Obtain initial semantic pseudo labels of sample images in the training data set, perform binarization segmentation on the initial semantic pseudo labels to obtain a binarized image, and perform superpixel segmentation on the sample images to obtain multiple superpixels.

[0024] Step 120: Preliminarily classify each superpixel based on the binary image to obtain a first superpixel set and a second superpixel set, and use a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set.

[0025] Step 130: For any non-defective superpixel in the non-defective superpixel set, reconstruct the image of the non-defective superpixel based on the grayscale distribution of each superpixel.

[0026] Step 140: Correct the initial semantic pseudo-label based on the non-defective superpixels reconstructed from each image to obtain the target semantic pseudo-label.

[0027] First of all, it should be noted that the semantic pseudo-label error correction method of the embodiment of the present application can be applied to any surface defect detection scenario, and the application scenario is not specifically limited here, for example, the scenario of defect detection on the uniform texture surface of an aircraft.

[0028] In step 110, a variety of sample images can be collected to form a training data set. These sample images can be surface images of objects, and some sample images can contain surface defects of objects. The training data set is then divided into a training set and a test set. The sample images in the training set can be labeled images, and the labels can contain information such as the presence or absence of defects and defect categories. The model can be trained using the training set, and the model can be tested using the test set. It should be noted that the embodiments of the present application are mainly for detecting surface defects. Therefore, the models described in the embodiments of the present application can all be defect detection models, and the defect detection models can be obtained by training based on the existing Resnet model.

[0029] In this embodiment, the initial semantic pseudo-label may refer to a semantic pseudo-label obtained by a conventional method or a specific method and without error correction. The method for obtaining the initial semantic pseudo-label of the sample image in the training data set may be: obtaining it by self-labeling, that is, using a pre-trained model to predict the unlabeled sample image, and using the prediction result as the semantic pseudo-label. This method depends on the quality of the initial model. If the initial model is good enough, the generated semantic pseudo-label can be used for further training to improve the model. The method for obtaining the initial semantic pseudo-label of the sample image in the training data set may also be: obtaining it by clustering, that is, clustering the unlabeled data using an unsupervised learning algorithm (such as K-means), and then assigning a label to each cluster. This label may be an inference based on the known category with the closest feature, or a newly defined category based on the characteristics of the data in the cluster. Other methods may also be used to obtain the initial semantic pseudo-label of the sample image in the training data set, which are not specifically limited here.

[0030] After obtaining the initial semantic pseudo-label of the sample image, the initial semantic pseudo-label can be adaptively binarized and segmented, and the pixel values ​​of each pixel of the semantic pseudo-label can be divided into two categories for subsequent analysis of the initial semantic pseudo-label. Specifically, the OTSU algorithm can be used to binarize the initial semantic pseudo-label and convert the semantic pseudo-label into a black and white image. After the conversion, the pixel values ​​of some pixels of the semantic pseudo-label are 0 (black), and the pixel values ​​of the other pixels are 1 (white). The area where the white pixels are located represents the area with defects in the semantic pseudo-label, and the area where the black pixels are located represents the area without defects in the semantic pseudo-label. Therefore, the defective area and non-defective area marked in the semantic pseudo-label can be distinguished by pixel values.

[0031] It should be noted that the basic principle of the OTSU algorithm is to find a threshold in a single-band grayscale image so that after the image is segmented into two parts according to this threshold, the between-class variance between the two parts is maximized. The purpose of this is to maximize the contrast between the foreground and the background, thereby obtaining a better segmentation effect. The process of using the OTSU algorithm to perform binary segmentation on the initial semantic pseudo-label can be as follows: first calculate the grayscale histogram of the initial semantic pseudo-label to obtain the number of pixels at each grayscale level; then set some initial parameters, including the total number of pixels, the total pixel intensity of the image, etc.; further traverse all possible grayscale levels as candidate thresholds, and for each candidate threshold, calculate the following parameters: the number of pixels of the foreground and background, the average grayscale value of the foreground and background, and the inter-class variance; then further calculate the inter-class variance, which is a measure of the difference between the foreground and background; during the traversal process, record the threshold with the largest inter-class variance, which is the optimal threshold sought by the OTSU algorithm; finally, use the found optimal threshold to binarize the original initial semantic pseudo-label, that is, pixels less than the threshold are set as the background (usually black 0), and pixels greater than or equal to the threshold are set as the foreground (usually white 1).

[0032] In this embodiment, the sample image can also be segmented into superpixels to obtain multiple superpixels. Superpixel segmentation is to subdivide the sample image into multiple small areas based on the similarity of features such as color, brightness, texture, etc. between pixels of the sample image. In this embodiment, the sample image can be segmented into superpixels using the SLIC method.

[0033] In step 120, after obtaining the binarized image corresponding to the semantic pseudo-label and several superpixels corresponding to the sample image, these superpixels can be initially classified according to the binarized image to obtain a first superpixel set and a second superpixel set. It should be noted that the first superpixel set contains superpixels with a higher defect probability, and the second superpixel set contains superpixels with a lower defect probability. It can be understood that the obtained binarized image can clearly reflect the defect results marked by the initial semantic pseudo-label. Therefore, all superpixels are initially classified according to the binarized image, that is, all superpixels are classified according to the marking results of the initial semantic pseudo-label, and the results of the initial classification can reflect the marking results of the initial semantic pseudo-label.

[0034] After the initial classification, the preset classifier can be further used to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set. It can be understood that the second superpixel set contains superpixels with a lower probability of defects that are determined based on the initial semantic pseudo-labels. However, due to the unreliability of the initial semantic pseudo-labels, there may be missed areas and mislabeled areas in the second superpixel set. Therefore, this embodiment finds possible missed areas and possible mislabeled areas by performing secondary classification on the second superpixel set that is not labeled in the semantic pseudo-label. After the secondary division, the area where the superpixels in the defective superpixel set are located may be an area where the semantic pseudo-label is missed, and the area where the superpixels in the non-defective superpixel set are located may be mislabeled.

[0035] For the missing areas, you can directly choose to mark the missing areas to correct the semantic pseudo-labels. For the mislabeled areas, the embodiment of the present application reconstructs the image of the area where the superpixels in the defective superpixel set are located, and reconstructs the non-defective superpixels using the grayscale distribution of the surrounding background, which is equivalent to performing median filtering to reduce the interference of outliers, thereby eliminating the mislabeling errors in the area where the non-defective superpixels are located. Finally, the semantic pseudo-label can be re-determined using the sample image after image reconstruction, and the re-determined semantic pseudo-label is the corrected target semantic pseudo-label.

[0036] Therefore, the semantic pseudo-labels are corrected by local image reconstruction methods, which solves the problem of performance degradation of the trained model due to errors (wrong labels and / or missing labels) in the semantic pseudo-labels in the existing semantic segmentation model based on weakly supervised learning. In addition, pixel-level semantic pseudo-labels are generated by category labels to replace manual labeling, which effectively reduces the cost of manual labeling (annotation), speeds up model deployment efficiency, and helps improve the detection precision and accuracy of the trained defect detection model.

[0037] The above steps are introduced in detail below.

[0038] Figure 2 This is the second flowchart of the semantic pseudo-label error correction method of the embodiment of the present application. Figure 2 As shown, obtaining the initial semantic pseudo labels of the sample images in the training data set in step 110 may include the following steps: Step 210: Input the sample images in the training data set into the pre-trained defect detection model to obtain a class activation map.

[0039] Step 220: Upsample the class activation map to obtain an initial semantic pseudo label.

[0040] Specifically, after obtaining the training data set, the defect detection model can be trained using the training set to obtain a fine-tuned defect detection model, so that the defect detection model can judge whether there are defects in the sample image and the defect type. It should be noted here that the model structure of the defect detection model can directly adopt the model structure of the Resnet model, that is, the training set can be directly used to train the Resnet model to obtain a fine-tuned defect detection model.

[0041] Furthermore, the sample image can be input into a pre-trained defect detection model for processing. The defect detection model has multiple convolutional layers, pooling layers, etc., which are used to extract multi-level features of the sample image. In this embodiment, the network end of the defect detection model usually has a fully connected layer for final classification, and the weight matrix can be obtained from the fully connected layer. These weights represent the importance of different features to the final output; in addition, feature maps of different levels can be obtained from the convolution layer part, and the feature map can reflect the intermediate representation obtained after the sample image is subjected to the filter convolution operation.

[0042] After obtaining the weight matrix and feature map, the weight matrix and feature map can be multiplied. The multiplication operation here can be the multiplication of the elements at the corresponding positions, or other forms of fusion methods, such as matrix multiplication. After multiplying the weight matrix and the feature map, the class activation map can be obtained. The obtained class activation map may be smaller than the original sample image, so it is also necessary to upsample the class activation map to obtain a class activation map of the same size as the sample image. The class activation map is the initial semantic pseudo-label. The upsampled class activation map can be normalized to ensure that the element values ​​in the class activation map are between 0 and 1. In the final class activation map, each element value represents the probability that the corresponding pixel position belongs to the defect area. The larger the element value, the more likely it is that the position is the defect location.

[0043] It should be noted that Class Activation Mapping (CAM) is a visualization technique used to show which parts of an image are most critical when a neural network makes a prediction of a certain category. In simple terms, CAM technology can generate heat maps that highlight the areas in the input image that are most relevant to the model's prediction. In this embodiment, the class activation map can be used to mark the defect locations predicted by the defect detection model in the sample image as initial semantic pseudo-labels.

[0044] After obtaining the initial semantic pseudo-label, the semantic pseudo-label is adaptively binarized to obtain a binary image. At the same time, the sample image can be segmented into superpixels to obtain several superpixels, each of which has an independent number.

[0045] Figure 3This is the third flow chart of the semantic pseudo-label error correction method of the embodiment of the present application. Figure 3 As shown, in step 120, each superpixel is preliminarily classified based on the binary image to obtain a first superpixel set and a second superpixel set, which may include the following steps: Step 310: Determine a target region corresponding to each superpixel in the binary image, wherein the position of the target region in the binary image is the same as the position of the corresponding superpixel in the sample image.

[0046] Step 320: For any superpixel, if it is determined that the corresponding target area meets the preset condition, the superpixel is divided into a first superpixel set; the preset condition is that the proportion of pixel points in the target area whose pixel value is the first preset value is greater than or equal to the preset threshold.

[0047] Step 330: For any superpixel, if it is determined that the corresponding target area does not meet the preset condition, the superpixel is divided into a second superpixel set.

[0048] It should be noted that the first preset value can be determined according to the binary image, and the first preset value can be any one of the two pixel values ​​in the binary image. In this embodiment, since the binary image includes two pixel values ​​of black 0 and white 1, the first preset value can be set to 1. In addition, the preset threshold value can be set by the staff according to actual needs. For example, the preset threshold value can be set to 50%, and the preset threshold value is not specifically limited here.

[0049] Specifically, each superpixel can be positionally matched with the binary image, and the position area corresponding to each superpixel in the binary image can be found, and the corresponding position area is the target area corresponding to the superpixel. The method of positionally matching each superpixel with the binary image can be: establishing the same coordinate system on the binary image and the sample image after superpixel segmentation, determining the coordinates of each pixel in the superpixel, and then finding the corresponding area in the binary image based on the coordinates of each pixel in the superpixel, and the area is the target area corresponding to the superpixel.

[0050] Taking the first preset value as 1 and the preset threshold as 50% as an example, if the proportion of pixels with a pixel value of 1 in the target area corresponding to the superpixel is greater than or equal to 50%, the superpixel can be divided into the first superpixel set. If the proportion of pixels with a pixel value of 1 in the target area corresponding to the superpixel is less than 50%, the superpixel can be divided into the second superpixel set. The superpixels in the first superpixel set are high-probability defect superpixels, and the superpixels in the second superpixel set are low-probability defect superpixels.

[0051] Figure 4 This is the fourth flow chart of the semantic pseudo-label error correction method of the embodiment of the present application. Figure 4As shown, in step 120, the preset classifier is used to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set, which may include the following steps: Step 410: Train a preset classifier using the first superpixel set and the second superpixel set.

[0052] Step 420: perform secondary classification on each superpixel in the second superpixel set using the trained preset classifier to obtain a defective superpixel set and a non-defective superpixel set.

[0053] It should be noted that the preset classifier can be a multiple instance learning (MIL) classifier, which is a special supervised learning framework. Different from the traditional single instance learning classifier, in MIL, the training data is given in the form of "packages", each package contains multiple "instances", and the label is usually given to the entire package, not a single instance. In this embodiment, a key instance support vector machine can be selected as a multiple instance learning classifier. The following takes the multiple instance learning classifier as an example to introduce the secondary classification process.

[0054] In this embodiment, the first superpixel set can be used as a negative bag, and the second superpixel set can be used as a positive bag, and the multi-instance learning classifier can be trained using the positive bag and the negative bag. Specifically, the multi-instance learning classifier can extract features from superpixels in the positive bag and the negative bag, and the features can include (but are not limited to): color histogram, texture features, shape descriptors, etc.; further, the superpixels can be converted into the form of feature vectors, and it is ensured that the information contained in the bag is sufficient to describe the category of the bag (positive / negative); a multi-instance learning algorithm (for example, MILBoost algorithm, Diverse Density algorithm, MI-SVM algorithm, etc.) is used to train the model. In this setting, the goal of the model is to learn to distinguish between positive bags and negative bags.

[0055] After the multi-instance learning classifier is trained using positive and negative packages, the trained multi-instance learning classifier can be used to classify the superpixels in the second superpixel set. At this time, the multi-instance learning classifier can distinguish positive and negative packages, that is, distinguish which superpixels can be defective area superpixels and which superpixels can be non-defective area superpixels. The superpixels in the second superpixel set are further divided using the trained multi-instance learning classifier, and the areas not marked as defects in the initial semantic pseudo-label can be further divided to obtain defective superpixel sets and non-defective superpixel sets; the area corresponding to the superpixels divided into the defective superpixel set is the missed label area, and the area corresponding to the superpixels divided into the non-defective superpixel set can be almost determined to be defect-free, but there may be mislabeling in the initial semantic pseudo-label, that is, the area corresponding to the superpixels in the non-defective superpixel set is also marked as defective. Therefore, this embodiment reduces outlier interference by reconstructing the image of the superpixels in the non-defective superpixel set, which is equivalent to performing median filtering.

[0056] Figure 5 This is the fifth flow chart of the semantic pseudo-label error correction method of the embodiment of the present application. Figure 5 As shown, in step 130, image reconstruction of non-defective superpixels based on the grayscale distribution of each superpixel may include the following steps: Step 510: Determine the center point distance between the non-defective superpixel and each adjacent superpixel, and determine a preset number of related superpixels from each adjacent superpixel based on each center point distance.

[0057] Step 520: Fit, discretize and normalize the grayscale distribution of each relevant superpixel to obtain a normalized function.

[0058] Step 530: Determine a cumulative probability function based on the normalization function, generate a random number, and determine the reconstructed grayscale value of each pixel in the non-defective superpixel based on the cumulative probability function and the random number using a preset grayscale calculation formula.

[0059] Step 540: reconstruct the image of the non-defective superpixel based on the reconstructed grayscale value of each pixel in the non-defective superpixel.

[0060] It should be noted that all superpixels in the non-defective superpixel set are recorded as non-defective superpixels for the sake of subsequent description. In this embodiment, all non-defective superpixels need to be reconstructed. The following takes a non-defective superpixel as an example to introduce the process of non-defective superpixel image reconstruction.

[0061] In this embodiment, the related superpixels corresponding to the non-defective superpixels can be determined first. The related superpixels refer to superpixels that are close to the non-defective superpixels and have a certain correlation with the non-defective superpixels. Specifically, the center point distances between the non-defective superpixel and all superpixels within a certain range around it can be calculated, and then the related superpixels can be determined based on the center point distances. If the center point of the non-defective superpixel is recorded as , and the center points of the remaining superpixels are recorded as , n is the total number of superpixels within a certain range around the non-defective superpixel, which can be With each Lines are connected and the length of each line is calculated, which is the center point distance; further, a preset number of superpixels with the smallest center point distances can be selected, and these superpixels are used as the above-mentioned related superpixels.

[0062] It should be noted that the preset number can be set by the staff according to actual needs. For example, the preset number can be 3, that is, the three superpixels with the smallest center point distance are selected as relevant superpixels. The preset number is not specifically limited here.

[0063] After determining the relevant superpixels corresponding to the non-defective superpixels, the grayscale distribution of the relevant superpixels may be fitted. In some embodiments, an asymmetric generalized Gaussian distribution function is used to fit the grayscale distribution of the relevant superpixels to obtain a fitting function.

[0064] Specifically, considering the unevenness of the local grayscale distribution of the image background, this embodiment chooses to use an asymmetric generalized Gaussian distribution function to fit the grayscale distribution of each related superpixel, and the following fitting function can be obtained: : ; in, for The function parameters, is the gamma function and x is the variable.

[0065] It should be noted here that the fitting function It is obtained by fitting the grayscale distribution histogram of each related superpixel. The value range of the horizontal coordinate X in the grayscale distribution histogram is 0-255, and X is an integer; for each grayscale value X, the value of the vertical coordinate Y in the grayscale distribution histogram represents the number of pixels of this grayscale value in the image. For example, if the number of pixels with a grayscale value of 125 in the surrounding superpixels (i.e., the related superpixels) is 1000, then the corresponding function value in the grayscale distribution histogram is (125, 1000), and the asymmetric generalized Gaussian distribution function is fitted to the grayscale distribution histogram.

[0066] The fitting function is further discretized. In some implementations, each point on the fitting function is rounded to obtain a discretized function; each point on the discretized function is an integer point within a preset range. Specifically, the fitting function can be made Each point on the discretization function is obtained by taking an integer value in the preset range of 0-255. The discretization function can be recorded as .

[0067] The discretized function can be further normalized to obtain the normalized function , normalization function as follows: ; in, is a variable, and its value range is any integer point from 0 to 255; Represents grayscale.

[0068] In this embodiment, after obtaining the normalized function, the cumulative probability function of the normalized function can be calculated. The cumulative probability function is used to describe the probability that the random variable takes a value less than or equal to a certain value. Specifically, the cumulative probability function can be calculated by the following formula: : ; Among them, k is a variable, the value range is any natural number from 0 to 255, and N represents a natural number.

[0069] It should be noted that the normalization function represents the probability of each pixel in the relevant superpixel taking the corresponding grayscale value, while the cumulative probability function refers to the sum of all probabilities less than or equal to the grayscale value. By normalizing the discretization function, the discretization function value can be converted to between 0 and 1, and a one-to-one correspondence between X and Y is guaranteed.

[0070] Furthermore, in Generate a random number in the range , the following preset grayscale calculation formula can be used to determine the reconstructed grayscale value of each pixel in the non-defective superpixel: ; in, is the reconstructed gray value of the i-th pixel in the non-defective superpixel, is the cumulative probability function, k is a variable, N represents a natural number, Is a random number.

[0071] As an example, assuming that the probability that the grayscale value in the relevant superpixel is equal to 125 and 200 is both 0.15, if a random number of 0.16 is generated, it is impossible to determine whether the reconstructed grayscale value is 125 or 200. After the probability of the corresponding grayscale values ​​of the relevant superpixels is processed by the cumulative probability function, assuming that the cumulative probabilities of the grayscale values ​​​​are 125 and 200 are 0.2 and 0.8, the reconstructed grayscale value can be determined to be 125 through the above-mentioned formula for reconstructing grayscale values ​​(at this time, it is necessary to assume that when the grayscale values ​​are 124 and 126, the cumulative probability function values ​​​​are 0.1 and 0.3, that is, the grayscale value corresponding to the minimum difference after subtraction is 125).

[0072] After traversing all the pixels in the current non-defective superpixel using the preset grayscale calculation formula, the reconstructed grayscale values ​​of all the pixels in the non-defective superpixel can be obtained. The reconstructed grayscale values ​​of all the pixels replace the corresponding original grayscale values ​​to obtain the non-defective superpixel after image reconstruction.

[0073] The above image reconstruction process is continuously repeated to perform local image reconstruction on all non-defective superpixels. After image reconstruction, the pixel values ​​of all non-defective superpixels are relatively uniform, and there will be no abnormal pixel values. This can reduce the probability of mislabeling when redetermining the semantic pseudo-labels, thereby improving the reliability of the semantic pseudo-labels.

[0074] Therefore, by combining the initial semantic pseudo-label with superpixel segmentation, all superpixels are segmented initially based on the initial semantic pseudo-label to distinguish the defects marked in the initial semantic pseudo-label from the defects not marked; then, all superpixels with defects not marked in the initial semantic pseudo-label are secondary classified using a preset classifier to distinguish the superpixels that may have defects from the superpixels that almost do not have defects. The corresponding area of ​​the superpixels that may have defects determined by the secondary classification is the missed area, and the corresponding area of ​​the superpixels that almost do not have defects determined by the secondary classification is the mislabeled area. By reconstructing the image of non-defective superpixels, the probability of mislabeling in the initial semantic pseudo-label can be effectively reduced; at the same time, labeling the area corresponding to the defective superpixel can effectively reduce the probability of missing labels in the initial semantic pseudo-label. Using the corrected semantic pseudo-label to train the defect detection model can effectively improve the model performance, improve the precision and accuracy of defect detection, and effectively reduce the cost of manual labeling (labeling), and speed up the efficiency of model deployment.

[0075] Based on the above embodiments, the embodiments of the present application further provide a semantic pseudo-label error correction device. Figure 6 is a schematic diagram of a semantic pseudo-label error correction device according to an embodiment of the present application. Figure 6 As shown, the semantic pseudo-label error correction device 600 may include an acquisition module 610 , a classification module 620 , an image reconstruction module 630 and a correction module 640 .

[0076] Among them, the acquisition module 610 is used to obtain the initial semantic pseudo-label of the sample image in the training data set, and to perform binary segmentation on the initial semantic pseudo-label to obtain a binary image, and to perform superpixel segmentation on the sample image to obtain multiple superpixels; the classification module 620 is used to perform preliminary classification of each superpixel based on the binary image to obtain a first superpixel set and a second superpixel set, and to perform secondary classification on each superpixel in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set; the image reconstruction module 630 is used to reconstruct the image of the non-defective superpixel based on the grayscale distribution of each superpixel for any non-defective superpixel in the non-defective superpixel set; the correction module 640 is used to correct the initial semantic pseudo-label based on the non-defective superpixel after each image reconstruction to obtain a target semantic pseudo-label.

[0077] Thus, the initial semantic pseudo-label is binarized and segmented by the acquisition module 610 to obtain a binary image, and the sample image is segmented into superpixels to obtain multiple superpixels; the classification module 620 then performs a preliminary classification on each superpixel through the binary image, that is, the superpixel is divided into a first superpixel set and a second superpixel set according to the labeling result of the initial semantic pseudo-label, wherein the first superpixel set represents the area with defects labeled in the initial semantic pseudo-label, and the second superpixel set represents the area with no defects labeled in the initial semantic pseudo-label; and then the superpixels in the second superpixel set are secondary classified using a preset classifier to classify the superpixels in the initial semantic pseudo-label into the first superpixel set and the second superpixel set into the second superpixel set. The superpixels that are not labeled with the initial semantic pseudo-label and may have defects are divided into the defective superpixel set, and the superpixels that are not labeled with the initial semantic pseudo-label and do not have defects are divided into the non-defective superpixel set; further, the image reconstruction module 630 reconstructs the image of each non-defective superpixel in the non-defective superpixel set to eliminate the defects of the initial semantic pseudo-label in the non-defective superpixel; finally, the correction module 640 corrects the initial semantic pseudo-label based on the non-defective superpixels after each image reconstruction to obtain the target semantic pseudo-label, thereby realizing the correction of the semantic pseudo-label and solving the problem of model performance degradation caused by weak supervision learning.

[0078] In some embodiments, the image reconstruction module 630 is specifically used to: determine the center point distance between a non-defective superpixel and each adjacent superpixel, and determine a preset number of related superpixels from each adjacent superpixel based on each center point distance; fit, discretize and normalize the grayscale distribution of each related superpixel to obtain a normalized function; determine a cumulative probability function based on the normalized function, generate a random number, and determine the reconstructed grayscale value of each pixel in the non-defective superpixel based on the cumulative probability function and the random number using a preset grayscale calculation formula; and reconstruct the image of the non-defective superpixel based on the reconstructed grayscale value of each pixel in the non-defective superpixel.

[0079] In some embodiments, the image reconstruction module 630 is also specifically used to: fit the grayscale distribution of each related superpixel using an asymmetric generalized Gaussian distribution function to obtain a fitting function; perform an integer operation on each point on the fitting function to obtain a discretized function; each point on the discretized function is an integer point within a preset range; and normalize the discretized function to obtain a normalized function.

[0080] In some implementations, the preset grayscale calculation formula is: ; In the formula, is the reconstructed gray value of the i-th pixel in the non-defective superpixel, is the cumulative probability function, k is a variable, N represents a natural number, Is a random number.

[0081] In some embodiments, the classification module 620 is specifically used to: determine a target area corresponding to each superpixel in the binary image, the position of the target area in the binary image is the same as the position of the corresponding superpixel in the sample image; for any superpixel, if it is determined that the corresponding target area meets a preset condition, the superpixel is divided into a first superpixel set; the preset condition is that the proportion of pixels in the target area whose pixel value is the first preset value is greater than or equal to a preset threshold; for any superpixel, if it is determined that the corresponding target area does not meet the preset condition, the superpixel is divided into a second superpixel set.

[0082] In some embodiments, the classification module 620 is also specifically used to: train a preset classifier using the first superpixel set and the second superpixel set; and perform secondary classification on each superpixel in the second superpixel set using the trained preset classifier to obtain a defective superpixel set and a non-defective superpixel set.

[0083] In some implementations, the acquisition module 610 is specifically used to: input sample images in the training data set into a pre-trained defect detection model to obtain a class activation map; and upsample the class activation map to obtain an initial semantic pseudo-label.

[0084] It should be noted that for details not disclosed in the semantic pseudo-label error correction device of this embodiment, please refer to the details disclosed in the embodiment of the semantic pseudo-label error correction method in the embodiments of this specification, and will not be repeated here.

[0085] Based on the above embodiments, Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor (processor) 710, a communication interface (Communications Interface) 720, a memory (memory) 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logic instructions in the memory 730 to execute the semantic pseudo-label error correction method, which includes: obtaining the initial semantic pseudo-label of the sample image in the training data set, and binarizing the initial semantic pseudo-label to obtain a binarized image, and superpixel segmenting the sample image to obtain multiple superpixels; preliminarily classifying each superpixel based on the binarized image to obtain a first superpixel set and a second superpixel set, and using a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set; for any non-defective superpixel in the non-defective superpixel set, based on the grayscale distribution of each superpixel, reconstructing the image of the non-defective superpixel; and correcting the initial semantic pseudo-label based on the non-defective superpixel after each image reconstruction to obtain a target semantic pseudo-label.

[0086] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] On the basis of the above embodiments, on the other hand, the present invention further provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the semantic pseudo-label error correction method provided by the above methods, which method includes: obtaining an initial semantic pseudo-label of a sample image in a training data set, and binarizing the initial semantic pseudo-label to obtain a binarized image, and superpixel segmenting the sample image to obtain multiple superpixels; preliminarily classifying each superpixel based on the binarized image to obtain a first superpixel set and a second superpixel set, and using a preset classifier to perform secondary classification on each superpixel in the second superpixel set to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set; for any non-defective superpixel in the non-defective superpixel set, based on the grayscale distribution of each superpixel, reconstructing the image of the non-defective superpixel; correcting the initial semantic pseudo-label based on the non-defective superpixel after each image reconstruction to obtain a target semantic pseudo-label.

[0088] On the basis of the above embodiments, on another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the semantic pseudo-label error correction method provided by the above methods, the method comprising: obtaining an initial semantic pseudo-label of a sample image in a training data set, and performing binarization segmentation on the initial semantic pseudo-label to obtain a binarized image, and performing superpixel segmentation on the sample image to obtain a plurality of superpixels; performing a preliminary classification of each superpixel based on the binarized image to obtain a first superpixel set and a second superpixel set, and performing secondary classification of each superpixel in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set; for any non-defective superpixel in the non-defective superpixel set, performing image reconstruction on the non-defective superpixel based on the grayscale distribution of each superpixel; and correcting the initial semantic pseudo-label based on the non-defective superpixel after each image reconstruction to obtain a target semantic pseudo-label.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Claims

1. A semantic pseudo-label error correction method, characterized in that: include: Obtaining initial semantic pseudo labels of sample images in a training data set, performing binarization segmentation on the initial semantic pseudo labels to obtain a binarized image, and performing superpixel segmentation on the sample images to obtain a plurality of superpixels; Preliminarily classifying each of the superpixels based on the binary image to obtain a first superpixel set and a second superpixel set, and performing secondary classification on each of the superpixels in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set; For any non-defective superpixel in the non-defective superpixel set, reconstructing the image of the non-defective superpixel based on the grayscale distribution of each superpixel; The initial semantic pseudo-label is corrected based on the non-defective superpixels reconstructed from each image to obtain a target semantic pseudo-label.

2. The semantic pseudo-label error correction method according to claim 1, characterized in that: The image reconstruction of the non-defective super-pixel based on the grayscale distribution of each super-pixel comprises: Determining the center point distance between the non-defective superpixel and each adjacent superpixel, and determining a preset number of related superpixels from each adjacent superpixel based on each center point distance; Fitting, discretizing and normalizing the grayscale distribution of each of the related superpixels to obtain a normalized function; Determine a cumulative probability function based on the normalization function, generate a random number, and determine a reconstructed grayscale value of each pixel in the non-defective superpixel based on the cumulative probability function and the random number using a preset grayscale calculation formula; The image of the non-defective super-pixel is reconstructed based on the reconstructed grayscale value of each pixel in the non-defective super-pixel.

3. The semantic pseudo-label error correction method according to claim 2, characterized in that: The fitting, discretization and normalization processing are performed on the grayscale distribution of each of the related superpixels to obtain a normalization function, including: Using an asymmetric generalized Gaussian distribution function to fit the grayscale distribution of each of the related superpixels to obtain a fitting function; Performing a rounding operation on each point on the fitting function to obtain a discretized function; each point on the discretized function is an integer point within a preset range; The discretization function is normalized to obtain the normalized function.

4. The semantic pseudo-label error correction method according to claim 2, characterized in that: The preset grayscale calculation formula is: ； In the formula, is the reconstructed grayscale value of the i-th pixel in the non-defective superpixel, is the cumulative probability function, k is a variable, N represents a natural number, is the random number.

5. The semantic pseudo-label error correction method according to claim 1, characterized in that: The preliminarily classifying each of the superpixels based on the binary image to obtain a first superpixel set and a second superpixel set comprises: Determining a target region corresponding to each superpixel in the binary image, wherein a position of the target region in the binary image is the same as a position of the corresponding superpixel in the sample image; For any superpixel, if it is determined that the corresponding target area meets a preset condition, the superpixel is divided into the first superpixel set; the preset condition is that the proportion of pixels in the target area whose pixel value is the first preset value is greater than or equal to a preset threshold; For any superpixel, if it is determined that the corresponding target area does not satisfy the preset condition, the superpixel is divided into the second superpixel set.

6. The semantic pseudo-label error correction method according to claim 1, characterized in that: The second superpixel set is subjected to secondary classification by using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set, including: Training the preset classifier using the first superpixel set and the second superpixel set; A trained preset classifier is used to perform secondary classification on each of the superpixels in the second superpixel set to obtain the defective superpixel set and the non-defective superpixel set.

7. The semantic pseudo-label error correction method according to claim 1, characterized in that: The step of obtaining initial semantic pseudo labels of sample images in the training data set includes: Input the sample images in the training data set into the pre-trained defect detection model to obtain the class activation map; The class activation map is upsampled to obtain an initial semantic pseudo label.

8. A semantic pseudo-label error correction device, characterized in that: include: An acquisition module is used to acquire initial semantic pseudo labels of sample images in a training data set, and to perform binarization segmentation on the initial semantic pseudo labels to obtain a binarized image, and to perform superpixel segmentation on the sample images to obtain a plurality of superpixels; A classification module, configured to perform preliminary classification of each of the superpixels based on the binary image to obtain a first superpixel set and a second superpixel set, and perform secondary classification of each of the superpixels in the second superpixel set using a preset classifier to obtain a defective superpixel set and a non-defective superpixel set; wherein the defect probability of the first superpixel set is higher than the defect probability of the second superpixel set; An image reconstruction module, configured to reconstruct an image of any non-defective superpixel in the non-defective superpixel set based on the grayscale distribution of each superpixel; The correction module is used to correct the initial semantic pseudo-label based on the non-defective superpixels reconstructed from each image to obtain a target semantic pseudo-label.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the semantic pseudo-label error correction method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the semantic pseudo-label error correction method according to any one of claims 1 to 7 is implemented.

Images