Image defect segmentation method based on superpixel active learning and semi-supervised learning strategy

Abstract

The present invention relates to the field of industrial machine vision, and discloses an image defect segmentation method based on superpixel active learning and semi-supervised learning strategies, comprising the following steps: S1. Obtain an image data set; S2. Perform superpixel processing on the image data set, and cluster the pixels of each image into pixel groups; S3. Select pixel groups with valuable annotations through an active learning strategy, and have experts annotate the images corresponding to the pixel groups with valuable annotations; S4. Construct a dual-branch semi-supervised semantic segmentation model, and input the annotated image set and the unannotated image set into the dual-branch semi-supervised semantic segmentation model for training respectively; S5. Input the image to be predicted into the trained dual-branch semi-supervised semantic segmentation model to obtain the defect segmentation result of the image to be predicted. The present invention solves the problem that it is difficult to obtain a large data set with complete annotations for full supervised learning in the prior art, and has the characteristics of small sample requirements and high precision.

Description

Technical Field

[0001] The present invention relates to the field of industrial machine vision, and in particular to an image defect segmentation method based on superpixel active learning and semi-supervised learning strategies. Background Art

[0002] In recent years, traditional machine learning methods have made some progress in the application of object appearance defect segmentation. For example, a detection method for identifying appearance defects of deformed materials is proposed using the wavelet method; the frequency domain filter technology is used to process texture defect images to weaken the influence of background texture and realize the extraction of defect features. Although traditional machine learning methods have achieved certain results in the detection of surface defects of various devices, they are far from meeting the requirements of industrial automation production for defect detection accuracy, speed and generalization ability. With the rapid development of deep learning in the field of images, the object appearance defect segmentation method based on deep learning has achieved considerable results to a certain extent. However, most of the existing deep learning methods adopt the strategy of fully supervised learning, that is, all training data are fully labeled. Although the fully supervised learning method can achieve good results, this method is highly dependent on data and data annotation. The larger the amount of data and the finer the data annotation, the better the effect of fully supervised learning can be achieved. On the contrary, the smaller the amount of data and the rougher the annotation, the effect of fully supervised learning will drop sharply.

[0003] However, in practical applications, there are often problems such as high labeling costs and difficulty in obtaining labels in large quantities. Especially in the field of industrial machine vision, data acquisition and labeling will add a lot of costs to actual production, and the cost of making a large and fully labeled data set is often unacceptable. In the field of industrial machine vision, unlabeled data is relatively easy to obtain. In this case, semi-supervised learning is more suitable for actual production applications. Semi-supervised learning methods are between fully supervised learning and unsupervised learning. Unlike fully supervised learning methods, semi-supervised learning methods only require a small number of labeled samples and a large number of unlabeled samples to achieve the effect of fully supervised learning. However, most of the existing semi-supervised learning methods do not consider the problem of which samples should be selected for a small amount of labeling, resulting in unstable quality of training samples and difficulty in obtaining good results in the field of industrial machine vision.

[0004] The core idea of ​​the active learning strategy is to assume that different samples have different importance for a specific task, so the performance improvement they bring is also different. By selecting more important samples from the samples and annotating them in small quantities, the effect of fully supervised learning can be achieved. In this process, the essence of active learning is to evaluate the importance of samples. Similar to semi-supervised learning, the active learning strategy can enable the current model to achieve better performance with a smaller number of annotated samples. Therefore, it is of great significance to combine the advantages of active learning strategies and semi-supervised learning and apply them in the field of industrial vision.

[0005] In the technical field, there is a semi-supervised industrial product defect detection method and system based on positive sample learning. The image restoration network and the defect segmentation prediction network are trained by positive sample images. An image containing the defect to be detected is input into the image restoration network to obtain a restored genuine image. After calculating the absolute value of the difference between the two, the three images are spliced ​​to obtain a detection tensor. A segmentation mask binary image is generated according to the detection tensor through a segmentation prediction network to obtain a defect area.

[0006] However, the existing technology has the problem that it is difficult to obtain a complete labeled large data set for fully supervised learning. Therefore, how to invent an image defect segmentation method based on superpixel active learning and semi-supervised learning strategies is an urgent problem to be solved in this technical field. Summary of the invention

[0007] In order to solve the problem that it is difficult to obtain a complete labeled large data set for fully supervised learning in the prior art, the present invention provides an image defect segmentation method based on superpixel active learning and semi-supervised learning strategies, which has the characteristics of small sample requirement and high precision.

[0008] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0009] The image defect segmentation method based on superpixel active learning and semi-supervised learning strategy includes the following steps:

[0010] S1. Collect several images with defects and make them into an image dataset;

[0011] S2. Perform superpixel processing on the image dataset and cluster the pixels of each image into pixel groups;

[0012] S3. Selecting pixel groups with annotation value through active learning strategy, and having experts annotate images corresponding to the pixel groups with annotation value to obtain an annotated image set;

[0013] S4. construct a dual-branch semi-supervised semantic segmentation model, and input the labeled image set and the unlabeled image set into the dual-branch semi-supervised semantic segmentation model for training;

[0014] S5. Input the image to be predicted into the trained dual-branch semi-supervised semantic segmentation model to obtain the defect segmentation result of the image to be predicted.

[0015] Preferably, the active learning strategy is specifically:

[0016] A1. Randomly select a portion of images in the image dataset for annotation, and form the annotated images into a seed image set

[0017] A2. Train a fully supervised semantic segmentation model and a cost prediction model using the seed image set.

[0018] A3. Input the unlabeled images in the image dataset into the trained fully supervised semantic segmentation model and cost prediction model in sequence, and output the defect segmentation result and cost prediction result map of each image;

[0019] A4. Obtain a distribution map of the labeled values ​​of the images in the image data set by estimating the value of the defect segmentation results, and fuse it with the corresponding cost prediction result map to obtain a fused information map;

[0020] A5. Select pixel groups with information content higher than a preset threshold in each fusion information map, mark them on the original image and hand them over to experts for annotation, and add the annotated images to the seed image set;

[0021] A6. Repeat steps A2 to A5 until the expert annotation budget is exhausted, and use the seed image set at this time as the annotated image set.

[0022] Furthermore, the defect segmentation results are estimated through value estimation to obtain the labeled value distribution map of the images in the image data set, and then fused with the corresponding cost prediction result map to obtain the fused information map. The specific steps are as follows:

[0023] B1. Obtain the information entropy of the pixels in the image data set through the value estimation formula:

[0024]

[0025] Among them, (x, y) is the pixel point, I i is the image in the image dataset, Seg(I i ) is the defect segmentation result, P def (x ,y) (Seg(I i )) is the probability that the pixel (x, y) is a defect, H (x,y) is the information entropy of pixel point (x, y);

[0026] B2. Combine each pixel in the image in the image dataset to obtain the annotation value distribution map

[0027]

[0028] Among them, V i To mark the value distribution map;

[0029] B3. Fuse the corresponding annotated value distribution map with the cost prediction result map to obtain a fused information map.

[0030] Furthermore, the corresponding annotation value distribution map is integrated with the cost prediction result map, specifically:

[0031]

[0032] Among them, F i To integrate information graphics, C i This is the cost prediction result diagram.

[0033] Furthermore, the first branch of the dual-branch semi-supervised semantic segmentation model is a fully supervised convolutional neural network for training a labeled image set, and the second branch is a self-supervised convolutional neural network for training an unlabeled image set; the first branch and the second branch share an encoder and a decoder.

[0034] Furthermore, the labeled image set and the unlabeled image set are respectively input into the dual-branch semi-supervised semantic segmentation model for training. Specifically, the labeled images are input into the fully supervised convolutional neural network for fully supervised training, and the unlabeled images are input into the self-supervised convolutional neural network for self-supervised training. During training, the loss function of the fully supervised convolutional neural network and the loss function of the self-supervised convolutional neural network are merged into the total loss function.

[0035] Furthermore, the image to be predicted is input into the trained two-branch semi-supervised semantic segmentation model to obtain the defect segmentation result. The specific steps are as follows:

[0036] K1. Close the input channel of the self-supervised convolutional neural network of the dual-branch semi-supervised semantic segmentation model;

[0037] K2. Input the image to be predicted into the encoder of the fully supervised convolutional neural network to obtain the underlying feature map and the fused feature map;

[0038] K3. Input the underlying feature map and the fused feature map into the decoder of the fully supervised convolutional neural network to obtain the final prediction result map.

[0039] Furthermore, the image to be predicted is input into the encoder of the fully supervised convolutional neural network to obtain the underlying feature map and the fused feature map. The specific steps are as follows:

[0040] K201. Extract features of the image to be predicted through the deep convolutional neural network Resnet-101 to obtain the underlying feature map M1;

[0041] K202. Input the bottom feature map M1 into the pyramid pooling module for multi-scale feature extraction to obtain a multi-scale feature map M2;

[0042] K203. Input the multi-scale feature map M2 into a 1×1 convolution kernel for feature fusion to obtain the fused feature map M3.

[0043] Furthermore, the underlying feature map and the fused feature map are input into the decoder of the fully supervised convolutional neural network to obtain the final prediction result map. The specific steps are as follows:

[0044] K301. Input the underlying feature map M1 into a 1×1 convolution kernel to obtain feature map M4;

[0045] K302. Input the fused feature map M3 into a 4-fold upsampling module to obtain a feature map M5;

[0046] K303. Splice the feature map M4 and the feature map M5 together to fuse the underlying features and the multi-scale features to obtain a spliced ​​feature map M6;

[0047] K304. After passing the concatenated feature map M6 through a 3×3 convolution kernel and a 4x upsampling module, the final prediction result map is output.

[0048] An image defect segmentation system based on superpixel active learning and semi-supervised learning strategies includes a data acquisition module for acquiring a number of images containing defects and making them into an image data set, a data processing module for performing superpixel processing on the image data set and clustering the pixels of each image into pixel groups, an active learning strategy module for selecting and labeling images in pixel groups with labeling value, and a dual-branch semi-supervised semantic segmentation model for performing defect segmentation on an input predicted image.

[0049] The beneficial effects of the present invention are as follows:

[0050] The present invention processes superpixels of images, selects and annotates images in pixel groups through an active learning strategy, constructs and trains a dual-branch semi-supervised semantic segmentation model through the annotated images, and performs defect segmentation on the predicted image through the trained dual-branch semi-supervised semantic segmentation model. Thus, the present invention solves the problem that it is difficult to obtain a fully annotated large data set for full supervised learning in the prior art, and has the characteristics of small sample requirements and high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1This is a flowchart of the image defect segmentation method based on superpixel active learning and semi-supervised learning strategies.

[0052] Figure 2 It is a flowchart of the active learning strategy.

[0053] Figure 3 It is a schematic diagram of the training process of the dual-branch semi-supervised semantic segmentation model.

[0054] Figure 4 It is a schematic diagram of the structure of the encoder-decoder of the semantic segmentation model. DETAILED DESCRIPTION

[0055] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Example 1

[0057] like Figure 1 As shown, the image defect segmentation method based on superpixel active learning and semi-supervised learning strategy includes the following steps:

[0058] S1. Collect several images with defects and make them into an image dataset;

[0059] S2. Perform superpixel processing on the image dataset and cluster the pixels of each image into pixel groups;

[0060] S3. Selecting pixel groups with annotation value through active learning strategy, and having experts annotate images corresponding to the pixel groups with annotation value to obtain an annotated image set;

[0061] S4. construct a dual-branch semi-supervised semantic segmentation model, and input the labeled image set and the unlabeled image set into the dual-branch semi-supervised semantic segmentation model for training;

[0062] S5. Input the image to be predicted into the trained dual-branch semi-supervised semantic segmentation model to obtain the defect segmentation result of the image to be predicted.

[0063] Example 2

[0064] like Figure 1 As shown, the image defect segmentation method based on superpixel active learning and semi-supervised learning strategy includes the following steps:

[0065] S1. Collect several images with defects and make them into an image dataset;

[0066] In this embodiment, the image is a corrugated paper image, and N high-precision printed corrugated paper sample images with defects and a resolution of w×h are collected from a corrugated paper printing production line as a data set. The data set is denoted as I = {I1, I2, ..., IN}.

[0067] S2. Perform superpixel processing on the image dataset and cluster the pixels of each image into pixel groups;

[0068] S3. Selecting pixel groups with annotation value through active learning strategy, and having experts annotate images corresponding to the pixel groups with annotation value to obtain an annotated image set;

[0069] S4. construct a dual-branch semi-supervised semantic segmentation model, and input the labeled image set and the unlabeled image set into the dual-branch semi-supervised semantic segmentation model for training;

[0070] S5. Input the image to be predicted into the trained dual-branch semi-supervised semantic segmentation model to obtain the defect segmentation result of the image to be predicted.

[0071] In a specific embodiment, the superpixel processing method is a Seeds segmentation algorithm. In this embodiment, after the processing is completed, each image I i The corresponding pixel group can be recorded as P i = {P i1 ,P i2 ,……,P iN The pixel group will be used as the smallest unit in subsequent processing. Since the pixel group itself has information such as contours and pixel similarity, the active learning method based on superpixels can obtain the image annotation value more precisely than the sliding window-based method. In subsequent expert annotation, only the pixel group that needs to be annotated needs to be selected to complete the annotation, without the need to manually annotate each pixel.

[0072] In a specific embodiment, the active learning strategy is specifically:

[0073] A1. Randomly select a portion of images in the image dataset for annotation, and form the annotated images into a seed image set

[0074] A2. Train a fully supervised semantic segmentation model and a cost prediction model using the seed image set.

[0075] A3. Input the unlabeled images in the image dataset into the trained fully supervised semantic segmentation model and cost prediction model in sequence, and output the defect segmentation result and cost prediction result map of each image;

[0076] In this embodiment, the fully supervised semantic segmentation model is FCN, and this part of the model can also be replaced by other semantic segmentation models. The defect segmentation results of the seed image set and the seed image set are used as training sets to train a cost prediction model Cost(·) to output a cost prediction result graph of the cost required to annotate the image. The cost required for annotation is defined as the number of operations required for annotation.

[0077] A4. Obtain a distribution map of the labeled values ​​of the images in the image data set by estimating the value of the defect segmentation results, and fuse it with the corresponding cost prediction result map to obtain a fused information map;

[0078] A5. Select pixel groups with information content higher than a preset threshold in each fusion information map, mark them on the original image and hand them over to experts for annotation, and add the annotated images to the seed image set;

[0079] A6. Repeat steps A2 to A5 until the expert annotation budget is exhausted, and use the seed image set at this time as the annotated image set.

[0080] In a specific embodiment, the defect segmentation result is used to obtain a labeled value distribution map of the image in the image data set through value estimation, and is fused with the corresponding cost prediction result map to obtain a fused information map. The specific steps are:

[0081] B1. Obtain the information entropy of the pixels in the image data set through the value estimation formula:

[0082]

[0083] Among them, (x, y) is the pixel point, I i is the image in the image dataset, Seg(I i ) is the defect segmentation result, P def (x ,y) (Seg(I i )) is the probability that the pixel (x, y) is a defect, H (x,y) is the information entropy of pixel point (x, y);

[0084] B2. Combine each pixel in the image in the image dataset to obtain the annotation value distribution map

[0085]

[0086] Among them, V i To mark the value distribution map;

[0087] B3. Fuse the corresponding annotated value distribution map with the cost prediction result map to obtain a fused information map.

[0088] In a specific embodiment, the corresponding annotation value distribution map is merged with the cost prediction result map, specifically:

[0089]

[0090] Among them, F i To integrate information graphics, C i This is the cost prediction result diagram.

[0091] In a specific embodiment, the first branch of the dual-branch semi-supervised semantic segmentation model is a fully supervised convolutional neural network for training a labeled image set, and the second branch is a self-supervised convolutional neural network for training an unlabeled image set.

[0092] In a specific embodiment, the labeled image set and the unlabeled image set are respectively input into a dual-branch semi-supervised semantic segmentation model for training, specifically: the labeled images are input into a fully supervised convolutional neural network for fully supervised training, and the unlabeled images are input into a self-supervised convolutional neural network for self-supervised training; during training, the loss function of the fully supervised convolutional neural network and the loss function of the self-supervised convolutional neural network are combined into a total loss function.

[0093] Example 3

[0094] like Figure 1 As shown, the image defect segmentation method based on superpixel active learning and semi-supervised learning strategy includes the following steps:

[0095] S1. Collect several images with defects and make them into an image dataset;

[0096] S2. Perform superpixel processing on the image dataset and cluster the pixels of each image into pixel groups;

[0097] S3. Selecting pixel groups with annotation value through active learning strategy, and having experts annotate images corresponding to the pixel groups with annotation value to obtain an annotated image set;

[0098] S4. construct a dual-branch semi-supervised semantic segmentation model, and input the labeled image set and the unlabeled image set into the dual-branch semi-supervised semantic segmentation model for training;

[0099] S5. Input the image to be predicted into the trained dual-branch semi-supervised semantic segmentation model to obtain the defect segmentation result of the image to be predicted.

[0100] In a specific embodiment, the superpixel processing method is a Seeds segmentation algorithm.

[0101] like Figure 2As shown, in a specific embodiment, the active learning strategy is specifically:

[0102] A1. Randomly select a portion of images in the image dataset for annotation, and form the annotated images into a seed image set

[0103] A2. Train a fully supervised semantic segmentation model and a cost prediction model using the seed image set.

[0104] A3. Input the unlabeled images in the image dataset into the trained fully supervised semantic segmentation model and cost prediction model in sequence, and output the defect segmentation result and cost prediction result map of each image;

[0105] A4. Obtain a distribution map of the labeled values ​​of the images in the image data set by estimating the value of the defect segmentation results, and fuse it with the corresponding cost prediction result map to obtain a fused information map;

[0106] A5. Select pixel groups with information content higher than a preset threshold in each fusion information map, mark them on the original image and hand them over to experts for annotation, and add the annotated images to the seed image set;

[0107] A6. Repeat steps A2 to A5 until the expert annotation budget is exhausted, and use the seed image set at this time as the annotated image set. In this embodiment, the expert budget is the total number of annotated images, the total number of operations in the annotation process, and the total working hours of the annotation process.

[0108] In a specific embodiment, before the expert annotation budget is exhausted, if the annotated image set obtained by the active learning strategy in this round achieves the expected effect in the subsequent test, that is, the annotated image set obtained by the active learning strategy reaches 95% of the full supervised learning performance of the same data set under complete annotation, the loop is stopped and the output is

[0109] In a specific embodiment, the defect segmentation result is used to obtain a labeled value distribution map of the image in the image data set through value estimation, and is fused with the corresponding cost prediction result map to obtain a fused information map. The specific steps are:

[0110] B1. Obtain the information entropy of the pixels in the image data set through the value estimation formula:

[0111]

[0112] Among them, (x, y) is the pixel point, I i is the image in the image dataset, Seg(I i ) is the defect segmentation result, P def (x ,y) iSeg(I i)) is the probability that the pixel (x, y) is a defect, H (x,y) is the information entropy of pixel point (x, y);

[0113] B2. Combine each pixel in the image in the image dataset to obtain the annotation value distribution map

[0114]

[0115] Among them, V i To mark the value distribution map;

[0116] B3. Fuse the corresponding annotated value distribution map with the cost prediction result map to obtain a fused information map.

[0117] In a specific embodiment, the corresponding annotation value distribution map is merged with the cost prediction result map, specifically:

[0118]

[0119] Among them, F i To integrate information graphics, C i This is the cost prediction result diagram.

[0120] In a specific embodiment, the first branch of the dual-branch semi-supervised semantic segmentation model is a fully supervised convolutional neural network for training a labeled image set, and the second branch is a self-supervised convolutional neural network for training an unlabeled image set; the first branch and the second branch share an encoder and a decoder.

[0121] In a specific embodiment, the labeled image set and the unlabeled image set are respectively input into a dual-branch semi-supervised semantic segmentation model for training, specifically: the labeled images are input into a fully supervised convolutional neural network for fully supervised training, and the unlabeled images are input into a self-supervised convolutional neural network for self-supervised training; during training, the loss function of the fully supervised convolutional neural network and the loss function of the self-supervised convolutional neural network are combined into a total loss function.

[0122] In a specific embodiment, the loss function of the first branch is

[0123] L label =-∑p i log(q i )

[0124] In the above formula, p i Indicates annotation information. i Represents the probability predicted by the two-branch semi-supervised semantic segmentation model, obtained by the softmax function.

[0125] In one embodiment, the second branch includes an image enhancement module.

[0126] In a specific embodiment, the image enhancement module is designed to transform the original unlabeled image I a By using the image enhancement method, a similar image I is obtained b , including but not limited to cropping, rotating, translating, color enhancement, etc. of the image; after enhancement, the original image I a Similar to image I b Combined into a positive sample pair, the original image I a Different from randomly selected images I c constitute a negative sample pair.

[0127] like Figure 3 As shown in the figure, the first branch inputs the labeled image set into the encoder and performs fully supervised training through the encoder. The second branch inputs the unlabeled image set into the image enhancement module to obtain positive sample pairs and negative sample pairs, and inputs the positive sample pairs and negative samples into the encoder and performs self-supervised training through the decoder.

[0128] In a specific embodiment, the image to be predicted is input into the trained two-branch semi-supervised semantic segmentation model to obtain the defect segmentation result. The specific steps are as follows: the loss function used by the second branch is

[0129] L unlabel =∑[‖f(I a )-f(I b )‖-‖f(I a )-f(I c )‖]

[0130] In the above formula, f(·) represents the result after model processing, ‖f(I a )-f(I b )‖ represents the Euclidean distance between positive sample pairs, ‖f(I a )-f(I c )‖ represents the Euclidean distance between negative sample pairs. The closer the distance between positive sample pairs and the farther the distance between negative sample pairs, the better the self-supervision effect and the better the semantic segmentation effect.

[0131] In a specific embodiment, the two branches are trained simultaneously, and the total loss function is

[0132] L total =L label +L unlabel .

[0133] In a specific embodiment, the image to be predicted is input into a trained dual-branch semi-supervised semantic segmentation model to obtain a defect segmentation result. The specific steps are as follows:

[0134] K1. Close the input channel of the self-supervised convolutional neural network of the dual-branch semi-supervised semantic segmentation model;

[0135] K2. Input the image to be predicted into the encoder of the fully supervised convolutional neural network to obtain the underlying feature map and the fused feature map;

[0136] K3. Input the underlying feature map and the fused feature map into the decoder of the fully supervised convolutional neural network to obtain the final prediction result map.

[0137] like Figure 4 As shown, in a specific embodiment, the image to be predicted is input into the encoder of the fully supervised convolutional neural network to obtain the underlying feature map and the fused feature map. The specific steps are:

[0138] K201. Extract features of the image to be predicted through the deep convolutional neural network Resnet-101 to obtain the underlying feature map M1;

[0139] K202. Input the bottom feature map M1 into the pyramid pooling module for multi-scale feature extraction to obtain a multi-scale feature map M2;

[0140] K203. Input the multi-scale feature map M2 into a 1×1 convolution kernel for feature fusion to obtain the fused feature map M3.

[0141] In this embodiment, the pyramid pooling module is composed of multiple parallel convolution kernels and pooling layers of different scales. The five parts of the module run in parallel, and the feature maps are output separately and then spliced ​​into a complete multi-scale feature map.

[0142] In this embodiment, multi-scale feature extraction can be expressed as:

[0143] M21=Conv 1×1 (M1)

[0144] M22=Conv 3×3 (M1)

[0145] M23=Conv 3×3 (M1)

[0146] M24=Conv 3×3 (M1)

[0147] M25=Pool(M1)

[0148] M2=Concat(M21, M22, M23, M24, M25).

[0149] In a specific embodiment, the bottom feature map and the fused feature map are input into the decoder of the fully supervised convolutional neural network to obtain the final prediction result map. The specific steps are as follows:

[0150] K301. Input the underlying feature map M1 into a 1×1 convolution kernel to obtain feature map M4;

[0151] K302. Input the fused feature map M3 into a 4-fold upsampling module to obtain a feature map M5;

[0152] K303. Splice the feature map M4 and the feature map M5 together to fuse the underlying features and the multi-scale features to obtain a spliced ​​feature map M6;

[0153] K304. After passing the concatenated feature map M6 through a 3×3 convolution kernel and a 4x upsampling module, the final prediction result map is output.

[0154] The present invention processes superpixels of images, selects and annotates images in pixel groups through an active learning strategy, constructs and trains a dual-branch semi-supervised semantic segmentation model through the annotated images, and performs defect segmentation on the predicted image through the trained dual-branch semi-supervised semantic segmentation model. Thus, the present invention solves the problem that it is difficult to obtain a fully annotated large data set for full supervised learning in the prior art, and has the characteristics of small sample requirements and high precision.

[0155] Example 4

[0156] An image defect segmentation system based on superpixel active learning and semi-supervised learning strategies includes a data acquisition module for acquiring a number of images containing defects and making them into an image data set, a data processing module for performing superpixel processing on the image data set and clustering the pixels of each image into pixel groups, an active learning strategy module for selecting and labeling images in pixel groups with labeling value, and a dual-branch semi-supervised semantic segmentation model for performing defect segmentation on an input predicted image.

[0157] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. An image defect segmentation method based on superpixel active learning and semi-supervised learning strategy, characterized by: The following steps are involved: S1. Collect several images with defects and make them into an image dataset; S2. Perform superpixel processing on the image dataset and cluster the pixels of each image into pixel groups; the active learning strategy is specifically: A1. Randomly select a portion of images in the image dataset for annotation, and form the annotated images into a seed image set A2. Train a fully supervised semantic segmentation model and a cost prediction model using the seed image set. A3. Input the unlabeled images in the image dataset into the trained fully supervised semantic segmentation model and cost prediction model in sequence, and output the defect segmentation result and cost prediction result map of each image; A4. Obtain a distribution map of the labeled values ​​of the images in the image dataset by value estimation based on the defect segmentation results, and fuse it with the corresponding cost prediction result map to obtain a fused information map; A5. Select the pixel groups with information content higher than the preset threshold in each fusion information image, mark them on the original image and hand them over to experts for annotation, and add the annotated images to the seed image set; A6. Repeat steps A2 to A5 until the expert annotation budget is exhausted, and use the seed image set at this time as the annotated image set S3. Selecting pixel groups with annotation value through active learning strategy, and having experts annotate images corresponding to the pixel groups with annotation value to obtain an annotated image set; The defect segmentation results are estimated through value estimation to obtain the labeled value distribution map of the images in the image dataset, and then fused with the corresponding cost prediction result map to obtain the fused information map. The specific steps are as follows: B1. Obtain the information entropy of the pixels in the image dataset using the value estimation formula: Among them, (x, y) is the pixel point, is the image in the image dataset, is the defect segmentation result, Pixels The probability of being a defect, Pixel Information entropy of B2. Combine each pixel in the image in the image dataset to obtain the annotation value distribution map Among them, V i To mark the value distribution map; B3. Fuse the corresponding annotated value distribution map with the cost prediction result map to obtain a fused information map; S4. Construct a dual-branch semi-supervised semantic segmentation model, and input the labeled image set and the unlabeled image set into the dual-branch semi-supervised semantic segmentation model for training; S5. Input the image to be predicted into the trained dual-branch semi-supervised semantic segmentation model to obtain the defect segmentation result of the image to be predicted.

2. The image defect segmentation method based on superpixel active learning and semi-supervised learning strategy according to claim 1 is characterized in that: The corresponding annotated value distribution map is integrated with the cost prediction result map, specifically: in, To integrate information graphics, This is the cost prediction result diagram.

3. The image defect segmentation method based on superpixel active learning and semi-supervised learning strategy according to claim 1, characterized in that: The first branch of the dual-branch semi-supervised semantic segmentation model is a fully supervised convolutional neural network for training labeled image sets, and the second branch is a self-supervised convolutional neural network for training unlabeled image sets; the first branch and the second branch share an encoder and a decoder.

4. The image defect segmentation method based on superpixel active learning and semi-supervised learning strategy according to claim 3 is characterized in that: The labeled image set and the unlabeled image set are respectively input into the dual-branch semi-supervised semantic segmentation model for training. Specifically, the labeled images are input into the fully supervised convolutional neural network for fully supervised training, and the unlabeled images are input into the self-supervised convolutional neural network for self-supervised training. During training, the loss function of the fully supervised convolutional neural network and the loss function of the self-supervised convolutional neural network are combined into the total loss function.

5. The image defect segmentation method based on superpixel active learning and semi-supervised learning strategy according to claim 4 is characterized in that: The image to be predicted is input into the trained dual-branch semi-supervised semantic segmentation model to obtain the defect segmentation result. The specific steps are as follows: K1. Close the input channel of the self-supervised convolutional neural network of the two-branch semi-supervised semantic segmentation model; K2. Input the image to be predicted into the encoder of the fully supervised convolutional neural network to obtain the underlying feature map and the fused feature map; K3. Input the underlying feature map and the fused feature map into the decoder of the fully supervised convolutional neural network to obtain the final prediction result map.

6. The image defect segmentation method based on superpixel active learning and semi-supervised learning strategy according to claim 5, characterized in that: The image to be predicted is input into the encoder of the fully supervised convolutional neural network to obtain the underlying feature map and the fused feature map. The specific steps are as follows: K201. Extract features of the image to be predicted through the deep convolutional neural network Resnet-101 to obtain the underlying feature map ; K202. The underlying feature map Input to the pyramid pooling module for multi-scale feature extraction to obtain a multi-scale feature map ; K203. Multi-scale feature maps Enter a The convolution kernel is used to perform feature fusion to obtain the fusion feature map .

7. The image defect segmentation method based on superpixel active learning and semi-supervised learning strategy according to claim 6 is characterized in that: Input the underlying feature map and the fused feature map into the decoder of the fully supervised convolutional neural network to obtain the final prediction result map. The specific steps are: K301. The underlying feature map Enter a The convolution kernel obtains the feature map ; K302. Fusion feature map Input a 4x upsampling module to get the feature map ; K303. The feature map And feature map Splice together to fuse the underlying features and multi-scale features to obtain a spliced ​​feature map ; K304. Concatenate feature maps After a After the convolution kernel and a 4-fold upsampling module, the final prediction result map is output.

8. An image defect segmentation system based on superpixel active learning and semi-supervised learning strategy, characterized by: Used to execute the image defect segmentation method as described in any one of claims 1 to 7, including a data acquisition module for collecting a number of images containing defects and making them into an image data set, a data processing module for performing superpixel processing on the image data set and clustering the pixels of each image into pixel groups, an active learning strategy module for selecting and labeling images in pixel groups with labeling value, and a dual-branch semi-supervised semantic segmentation model for performing defect segmentation on an input predicted image.

Images