Self-Calibration Method for Small-Sample Image Segmentation in Industrial Product Quality Inspection

Abstract

The present invention proposes a small-sample image segmentation self-calibration method for industrial product quality inspection, including: training image acquisition, dividing the training set and the test set, selecting guiding pictures and query pictures to construct the input, initializing the algorithm model, training the self-calibration algorithm, and testing the algorithm performance. The calibration method combines the small-sample learning algorithm with industrial product quality inspection, and uses a dual-branch structure to solve the problem that deep learning methods cannot be applied to industrial product quality inspection in the field of industrial product quality inspection due to the lack of data and the lack of annotation of relevant data.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly relates to a small-sample image segmentation self-calibration method for industrial product quality inspection. Background Art

[0002] Defect detection of industrial products is an important branch in the industrial field. Due to the variety of defects, it is difficult for traditional machine vision algorithms to completely model and transfer the features of defects. Deep learning has relatively good results in feature extraction and localization. Therefore, using image semantic segmentation algorithms based on deep learning for defect detection of industrial products is one of the main research methods at present.

[0003] Image semantic segmentation is a basic task in computer vision, which aims to divide each pixel in the input image into a clear category. In recent years, with the development of deep learning technology and the emergence of large-scale labeled datasets, the accuracy of image semantic segmentation has been significantly improved, and image semantic segmentation technology is also playing an increasingly important role in fields such as defect detection, medical image processing, and autonomous driving. Currently, image semantic segmentation algorithms based on deep learning are mainly implemented through deep convolutional neural networks (Deep Convolution Neural Network). Although such algorithms have excellent performance, they have two problems:

[0004] First, image semantic segmentation algorithms based on deep learning need to use a large number of pixel-by-pixel labeled pictures for training to generate a model with good performance. Taking the commonly used dataset for image semantic segmentation, Common Objects in Context (hereinafter referred to as COCO) as an example, the COCO dataset provides 120,000 pictures for image semantic segmentation, and each picture is manually labeled pixel by pixel (tens of thousands or even hundreds of thousands of pixels per picture) to ensure that the models involved in the algorithm can be fully trained and converge.

[0005] Second, after the training of the image semantic segmentation algorithm based on deep learning is completed, for new image categories, that is, categories that have not participated in the training of the model, unless the model is retrained together with the original data, it is impossible to meet the image segmentation requirements of new categories based on the existing model. For example, the COCO dataset contains a total of 80 object categories, so the image semantic segmentation model generated by training with the COCO dataset can only perform semantic segmentation on images containing these 80 categories. If the semantic segmentation requirements for new categories (such as industrial metal oxidation block areas and light industrial fabric damage areas) are added at this time, it can only be achieved by manually labeling a large number of new category images and retraining the model. To solve problems such as insufficient number of new category images and lack of high-quality per-pixel annotations, this requires a huge amount of time and labor costs. At the same time, in industrial fields such as defect detection, the publicity of relevant data is not high, and the acquisition cost is high, and the differences between different defects are large, which increases the difficulty of using deep learning methods in defect detection.

[0006] Based on the above problems existing in the prior art, the present invention provides a small-sample image segmentation self-calibration method for industrial product quality inspection. Summary of the Invention

[0007] The present invention proposes a small-sample image segmentation self-calibration method for industrial product quality inspection.

[0008] The present invention adopts the following technical solutions:

[0009] A small-sample image segmentation self-calibration method for industrial product quality inspection, comprising:

[0010] Step 1, collect training images to obtain a dataset, and divide the dataset into a training set C base and a test set C novel into two categories. The training set C base is the type with sample defects, and the training set C base and the test set C novel have no intersection;

[0011] Step 2, construct the input data of the self-calibration algorithm. The input data includes a number of triples (I s , M s , I q ). The triple is used as the basic data unit during the training or testing process and is called a segment. The self-calibration algorithm includes a guidance branch and a query branch. Among them, the guidance branch inputs the guidance image I s and the per-pixel annotation image M s of the guidance image I s . The guidance branch is used to learn and extract prior knowledge from the fully annotated guidance image I s . The input of the query branch is the query image Iq , the query branch is used to draw on the knowledge learned from the guidance picture I s to generate the predicted result of the query picture I q ;

[0012] Step 3, extract the features of the input picture. For each segment, the guidance picture I s and the query picture I q respectively pass through an encoder composed of a convolutional neural network to extract their deep features in the high-dimensional space. The high-dimensional features of the guidance picture I s are denoted as F s , and the high-dimensional features of the query picture I q are denoted as F q ;

[0013] Step 4, generate the feature representation of the guidance picture. Input the high-dimensional features F s of the guidance picture I s into the guidance branch to generate the corresponding feature representation; multiply the high-dimensional features F s of the guidance picture I s pixel by pixel with its annotation M s to remove the background part in the high-dimensional features F s of the guidance picture I s ; generate the feature representation V s of the guidance picture I s through global average pooling and input it into the query branch;

[0014] Step 5, generate the predicted probability distribution map. Input the high-dimensional features F q of the query picture I q into the query branch, and generate the preliminary segmentation prediction of the query picture I s together with the feature representation V s of the guidance picture I q ; calculate the cosine similarity pixel by pixel between the feature representation V s of the guidance picture I s and the high-dimensional features F q of the query picture I q to obtain a probability distribution map P, where the value at each position represents the probability that the pixel at that position belongs to the foreground class;

[0015] Step 6, generate the preliminary labeled mask. Select a threshold, set the positions in the probability distribution map P greater than the threshold to 1, and the positions less than the threshold to 0 to obtain the preliminary segmentation prediction M init composed of 0 and 1, where 0 represents that the pixel at that position belongs to the background class and 1 represents that the pixel at that position belongs to the foreground class;

[0016] Step 7, generate for the preliminary segmentation prediction Minit Self - calibrating feature representation V q , due to the problem of intra - class differences, there are differences in the foreground parts of the guiding image and the query image. In the preliminary segmentation prediction M init , there are parts with prediction errors or missed predictions, and further calibration is required;

[0017] Step 8, let the high - level feature F q of the query image I q and V q calculate the cosine similarity pixel - by - pixel to obtain the intermediate result P′. Let P′ and the high - dimensional feature F q of the query image I q be multiplied and input into the ASPP module to obtain the calibrated segmentation prediction result M rec corresponding probability distribution map P r , which is used as the output of the algorithm;

[0018] Step 9, calculate the self - calibration loss of the algorithm to supervise the training of the model;

[0019] Step 10, generate the prediction result.

[0020] Furthermore, in step 7, the self - calibration includes: let the high - dimensional feature F q of the query image I q and M init be multiplied to remove the regions determined as the background, and then generate the feature representation V q of the query image I q through global average pooling.

[0021] Furthermore, in step 9, the calibration loss includes a first loss and a second loss. The first loss is the cross - entropy loss between the probability distribution map P init corresponding to the preliminary segmentation prediction M and the true annotation M q of the query image I q , and the calculation formula is as follows:

[0022]

[0023] The second loss is the cross - entropy loss between the calibrated segmentation prediction probability distribution map P r and the true annotation M q of the query image I q , and the calculation formula is as follows:

[0024]

[0025] In the above formulas (1) and (2), N represents the total number of pixel points, x and y are the spatial position coordinates of each pixel point in the image respectively, and the total loss function is and The weighted sum is calculated by the following formula:

[0026]

[0027] The optimization goal of the algorithm is to minimize the loss function

[0028] Furthermore, step 10 includes:

[0029] Weightedly fuse the probability distribution maps P and P r to obtain a new probability distribution map P fuse ; normalize P fuse to map all the internal values to the range between 0 and 1; select a threshold, set the positions in the probability distribution map P fuse that are greater than the threshold to 1 and the positions less than the threshold to 0 to obtain the segmentation prediction M fuse consisting of 0s and 1s, where 0 represents that the pixel at that position belongs to the background class and 1 represents that the pixel at that position belongs to the foreground class.

[0030] Compared with the prior art, the superior effects of the present invention are as follows:

[0031] 1. The small-sample image segmentation self-calibration method for industrial product quality inspection according to the present invention combines the small-sample learning algorithm with industrial product quality inspection and uses a dual-branch structure to solve the problem that in the field of industrial product quality inspection, due to the lack of data and the lack of annotations related to the data, deep learning methods cannot be applied to industrial product quality inspection;

[0032] 2. Based on the dual-branch structure, the present invention proposes a self-calibration algorithm to further improve the accuracy of the algorithm for image segmentation. Due to the widespread existence of intra-class differences, there may be significant differences between the objects in the guidance image and the objects in the query image. If the dual-branch structure is simply used for image segmentation, there is likely to be missegmentation or omission. Through the self-calibration algorithm, the missegmented parts of the dual-branch structure can be effectively corrected and the omitted parts can be supplemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic flowchart of the small-sample image segmentation self-calibration method for industrial product quality inspection in an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of the guidance branch and the query branch of the image segmentation self-calibration algorithm in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] To better understand the above objects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0036] Embodiment

[0037] As Figure 1 shown, the self-calibration method includes two steps: forward and reverse. The forward process is as follows: First, the features of the guidance image are multiplied pixel by pixel with the pixel-by-pixel annotation of the guidance image (the value of the foreground area is 1, and the value of the background area is 0), and the background area of the guidance image is set to 0; then, the foreground feature representation of the guidance image is obtained through global average pooling; the cosine similarity between the guidance image and the query image features is calculated, and after normalization and binarization, a preliminary segmentation prediction is obtained. The reverse process is as follows: First, the preliminary segmentation prediction obtained in the forward step is multiplied by the query image features, and the part of the query image preliminarily predicted as the background is set to 0; then, the current foreground feature representation of the query image is obtained through global average pooling. After calculating the cosine similarity between this feature representation and the query image features, the result is multiplied by the query image features and sent to the ASPP module; the output of the ASPP module undergoes a binarization operation to obtain the calibrated segmentation result.

[0038] Specifically, the self-calibration method includes: training image acquisition, dividing the training set and the test set, selecting the guidance image and the query image to construct the input, initializing the algorithm model, training the self-calibration algorithm, and testing the algorithm performance:

[0039] Training image acquisition. In the implementation process of the method in this embodiment, the publicly available dataset Magnetic Tiles is mainly used. This dataset collects images of 6 common magnetic tile defects, a total of 1344 images, and performs pixel-level marking on them; at the same time, the commonly used datasets Pascal VOC 2012 and Common Objects in Context (COCO) datasets for small-sample semantic segmentation algorithms are adopted. The Pascal VOC 2012 dataset has a total of 11530 images, including 20 categories, including people, birds, cats, airplanes, bicycles, bottles, chairs, dining tables, etc. The COCO dataset has more than 10 images with pixel-by-pixel annotations, including 80 categories. In addition to the 20 categories in Pascal VOC 2012, it also includes categories such as zebras, giraffes, traffic lights, etc., and the differences between categories are greater. Therefore, it is a more challenging dataset;

[0040] Divide the training set and the test set. First, divide the categories in the dataset into 4 groups. After dividing the 6 categories in the Magnetic Tiles dataset into 4 groups, each group contains 3 categories (since the number of categories in this dataset is small, the sliding window method is used for selection); after dividing the 20 categories in the Pascal VOC 2012 dataset into 4 groups, each group contains 5 out of the 20 categories; after dividing the 80 categories in the COCO dataset into 4 groups, each group contains 20 out of the 80 categories. During each training and testing process, select 3 groups of categories to form the training set, and the remaining 1 group of categories forms the test set. To ensure the universality of the experimental results, 4 groups of cross-validation experiments are conducted during the experiment, that is, during the i-th group of experiments, select the i-th group of categories to construct the test set, and the remaining 3 groups of categories to construct the training set. Finally, evaluate the quality of the algorithm results based on the comprehensive indicators of the 4 groups of experiments;

[0041] Construct the basic data unit "segment" on the training set and the test set, that is, select the guiding image, the pixel-level annotation image, and the query image respectively to construct the input to form a triple. The specific construction method is as follows: In the training stage, first randomly select a category from the selected three groups of categories (if using Magnetic Tiles, there are 3 categories in total; if using the Pascal VOC 2012 dataset, there are 15 categories in total; if using the COCO dataset, there are 60 categories in total), and then randomly sample two images from the images of this category. One image is used as the guiding image (at the same time, obtain the pixel-by-pixel annotation image of this image from the dataset), and the other image is used as the query image (do not use the pixel-by-pixel annotation image, and this annotation image is only used to calculate the difference with the predicted annotation image and use this difference to measure the effectiveness of the prediction). The two images are jointly input into the algorithm. During the training process of the algorithm, a total of 30,000 iterations are performed, that is, according to the above triple selection principle including the guiding image and the query image, a total of 30,000 groups of "guiding image - query image" triples are constructed, input into the algorithm in sequence, and calculate the final predicted image as the segmentation result of the query image;

[0042] As Figure 2 shown, the guiding branch and the query branch form a dual-branch structure. The specific process is as follows: First, the guiding image and the query image are respectively input into the feature encoder to obtain the guiding image feature and the query image feature; the guiding image feature and the pixel-by-pixel annotation of the guiding image are sent to the guiding branch to generate the feature representation of the guiding image; the feature representation of the guiding image and the query image feature are input into the query branch to generate the segmentation result of the query image; the segmentation result of the query image and its true segmentation label are used to calculate the loss to supervise the training of the entire algorithm. The algorithm process for training the model with the constructed dual-branch structure is as follows:

[0043] 1. Model initialization. In the experiment, ResNet-50 is selected as the feature encoder. To achieve better training results, the pre-trained parameters of ResNet-50 on a large-scale dataset are downloaded from the Internet and loaded into the current algorithm model;

[0044] 2. Self-calibration algorithm. According to the preliminary segmentation prediction M output by the constructed dual-branch model structure init for self-calibration. Due to the problem of intra-class differences, there may be large differences in the foreground parts of the guiding image and the query image. There will be many parts with prediction errors or missed predictions in the preliminary segmentation prediction M init . Therefore, further calibration is required. The specific method is as follows: Multiply the high-dimensional feature F q of the query image and M init to remove the areas determined to be the background, and then generate the feature representation V q of the query image through global average pooling;

[0045] 3. Let the feature F q output by the ResNet-50 feature encoder of the query image I q and V q calculate the cosine similarity pixel by pixel to obtain the feature P' of the intermediate result. Let P' and F q be multiplied again, and the result is input into the ASPP module to output the probability distribution map P r with the same size as the query image as the output of the model. Each pixel value of P r represents the probability value that the pixel value at the corresponding position of the query image belongs to the category of the query image;

[0046] 4. Subsequently, construct a loss function for solving the model parameters, and use the gradient descent method to solve the minimum value of the loss function. The parameter value corresponding to the minimum value is the final model parameter. Among them, the loss function includes the differences between the preliminary segmentation prediction result and the calibrated segmentation prediction result and the true per-pixel annotation map of the query image, that is and two items. Among them, the fusion weight λ of the loss function and the loss function is set to 0.4, that is,

[0047] 5. The parameters of the algorithm model are updated through backpropagation. Stochastic gradient descent (SGD) is selected as the optimizer, the initial learning rate is set to 1×10 -5 the momentum size is set to 0.9, and the weight decay is set to 5×10 -5 -4, and perform 30,000 iterations. After all iterations are completed, save the parameters of the algorithm to a configuration file for calling during testing.

[0048] The test of the performance of the constructed training model with a dual-branch structure is as follows:

[0049] In the test phase, it is necessary to run the training model saved after the algorithm is trained on the test set constructed by "dividing the training set and the test set" in this embodiment, and calculate the difference between the finally derived predicted segmentation result and the true per-pixel label to measure the model performance. The specific process is as follows: First, load the model generated after training is completed, then randomly and uniformly sample one category from the 5 categories included in the test data, and then randomly select a picture and the corresponding annotation from this category as the guiding picture and its pixel-level annotation, and randomly select another picture of this category as the query picture, jointly constituting a segment represented by a basic triple, and input it into the model, as Figure 2 shown; secondly, according to the structure of the training model and the optimal parameter values obtained from training, the model will finally generate a preliminary segmentation prediction M init and a calibrated segmentation prediction M rec , and the predicted output result is the per-pixel weighted sum of the two segmentation prediction results, that is, M = λ·M init + M rec , where the value of λ is 0.4; finally, repeat the process of randomly extracting data and calculating the prediction result 1000 times in total, and calculate the average segmentation accuracy as the test effect measurement value of the entire test data set. In this experiment and the training process, the mean Intersection over Union (mIoU) between the model prediction segmentation map and the true segmentation map is used to measure the segmentation prediction performance of the model. The calculation formula is:

[0050]

[0051] In the above formula, p ij represents the number of pixel points where the picture belongs to the i-th category but the prediction result is j. Since mIoU comprehensively considers the performance of the algorithm in predicting each category, it can more comprehensively represent the prediction ability of the algorithm.

[0052] The present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims.

Claims

1. A small-sample image segmentation self-calibration method for industrial product quality inspection, characterized in that Including: Step 1, collect training images to obtain a dataset, and divide the dataset into a training set C base and a test set C novel into two categories. The training set C base is the category with sample defects. The training set C base and the test set C novel have no intersection; Step 2, construct the input data of the self-calibration algorithm. The input data includes a number of triples (I s , M s , I q ). The triple is used as the basic data unit during the training or testing process and is called a segment. The self-calibration algorithm includes a guidance branch and a query branch. The guidance branch inputs the guidance image I s and the per-pixel annotation image M s of the guidance image I s . The guidance branch is used to learn and extract prior knowledge from the fully annotated guidance image I s . The input of the query branch is the query picture I q , and the query branch is used to draw on the knowledge learned from the guidance picture I s to generate the prediction result of the query picture I q ; Step 3, extract the features of the input image. For each segment, guide image I s and query image I q respectively pass through an encoder composed of a convolutional neural network to extract their deep features in the high-dimensional space. The high-dimensional feature of guide image I s is denoted as F s , and the high-dimensional feature of query image I q is denoted as F q ; Step 4, generate the feature representation of the guidance image, and input the high-dimensional feature F s of the guidance image I s into the guidance branch to generate the corresponding feature representation; multiply the high-dimensional feature F s of the guidance image I s pixel by pixel with its annotation M s to remove the background part in the high-dimensional feature F s of the guidance image I s ; generate the feature representation V s of the guidance image I s through global average pooling and input it into the query branch; Step 5, generate a predicted probability distribution map. Input the high-dimensional feature F of the query image I q into the query branch, and generate a preliminary segmentation prediction of the query image I together with the feature representation V of the guidance image I q s s q s s q q Calculate the cosine similarity pixel by pixel between the feature representation V of the guidance image I and the high-dimensional feature F of the query image I, and a probability distribution map P can be obtained, where the value at each position represents the probability that the pixel at that position belongs to the foreground class;​​​​​​​ Step 6: Generate a preliminary marking mask. Select a threshold, set the positions in the probability distribution map P that are greater than the threshold to 1, and set the positions less than the threshold to 0, obtaining a preliminary segmentation prediction M composed of 0s and 1s init , where 0 indicates that the pixel at that position belongs to the background category, and 1 indicates that the pixel at that position belongs to the foreground category; Step 7, generate a feature representation V for self-calibrating the preliminary segmentation prediction M init Since there is a problem of intra-class difference, there are differences in the foreground parts of the guiding image and the query image, and there are parts with prediction errors or missed predictions in the preliminary segmentation prediction M q , which need to be further calibrated; init ​ Step 8, let the query image be I q calculate the cosine similarity pixel by pixel for the high-level feature F q and V q to obtain an intermediate result P′. Multiply P′ by the high-dimensional feature F q of the query image I q and input it into the ASPP module to obtain the calibrated segmentation prediction result M rec and the corresponding probability distribution map P r , which is used as the output of the algorithm; Step 9, calculating the self-calibration loss of the algorithm and supervising the training of the model; Step 10, generating a prediction result.

2. The small-sample image segmentation self-calibration method for industrial product quality inspection according to claim 1, wherein In step 7, the self-calibration includes: multiplying the high-dimensional feature F q of the query picture I q by M init , removing the areas determined to be the background, and then generating the feature representation V q of the query picture I q .

3. The small-sample image segmentation self-calibration method for industrial product quality inspection according to claim 1, wherein In step 9, the calibration loss includes a first loss and a second loss. The first loss is the cross-entropy loss between the probability distribution map P and the true annotation M of the query image I q as follows: q The calculation formula is The second loss is the cross-entropy loss between the probability distribution map P r and the query image I q with the true annotation M q and is calculated as follows: In the above equations (1) and (2), N represents the total number of pixel points, x and y are the spatial position coordinates of each pixel point in the image respectively, and the total loss function is and a weighted sum of, and the calculation formula is: The optimization objective of the algorithm is to minimize the loss function 4. The small-sample image segmentation self-calibration method for industrial product quality inspection according to claim 1, characterized in that Step 10 includes: Fuse the probability distribution map P and the probability distribution map P r with weights to obtain a new probability distribution map P fuse ; normalize P fuse so that all the values inside it are mapped between 0 and 1; select a threshold, set the positions in the probability distribution map P fuse that are greater than the threshold to 1 and the positions less than the threshold to 0 to obtain the segmentation prediction M fuse consisting of 0s and 1s, where 0 represents that the pixel at that position belongs to the background class and 1 represents that the pixel at that position belongs to the foreground class.

Images