A semi-supervised semantic segmentation defect detection method and system based on industrial scenarios

Abstract

The present application relates to the field of image analysis technology, specifically to a semi-supervised semantic segmentation defect detection method and system based on industrial scenarios; in order to solve the problem of low defect detection accuracy in the prior art, the present application designs a semantic segmentation model that can effectively capture tiny defects and defect edge features of an image, and trains it, dynamically adjusts the training quality of the semantic segmentation model, and obtains a trained semantic segmentation model; then, in the formal defect detection process, in addition to performing image-level perturbations based on image-level enhancement processing on the image to be detected, a branch is added to perform feature space perturbations based on feature-level enhancement processing on the image to be detected, and the prediction results of the two branches in the training semantic segmentation model processing are cross-constrained and learned, the cross-constraint learning quality is dynamically supervised, and the most accurate pixel-by-pixel features are output, which can be applied in the field of industrial scenarios to achieve accurate detection of defective parts.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a semi-supervised semantic segmentation defect detection method and system based on industrial scenarios. Background Art

[0002] Traditional defect detection methods rely on manual image annotation and then use supervised learning algorithms for training and detection. However, due to the complexity and variability of industrial scenarios, traditional supervised learning algorithms often have difficulty achieving good detection results in the absence of sufficient labeled data.

[0003] Semi-supervised semantic segmentation defect detection methods can improve the accuracy of defect detection to a certain extent by using the learning model of unlabeled data. For example, the reconstruction model generated by the autoencoder (AE) method or the generative adversarial network (GAN) method often reconstructs the normal image with the smallest error and locates the defect anomaly based on the reconstructed image error. However, due to the complexity and variability of industrial scene images, these methods tend to ignore small defects and defect edges, resulting in low defect detection accuracy. Summary of the invention

[0004] In order to solve the problem that the above-mentioned background technology easily ignores tiny defects and defect edges in industrial images, resulting in low defect detection accuracy, the present invention provides a semi-supervised semantic segmentation defect detection method and system based on industrial scenarios.

[0005] The technical solution of the present invention is as follows:

[0006] A semi-supervised semantic segmentation defect detection method based on industrial scenarios includes the following operations:

[0007] S1. A semantic segmentation model is trained using several images with defect labels. When the cross entropy loss value of the semantic segmentation model is not greater than the cross entropy loss threshold, and the corresponding defect learning score is not less than the score threshold, a trained semantic segmentation model is obtained; the processing operation of the semantic segmentation model is as follows: the input image is processed by a convolutional neural network to obtain a resolution enhancement feature map; the shallow feature map and the middle feature map of the resolution enhancement feature map are subjected to channel fusion processing and convolution processing to obtain a channel feature map; the channel feature map is subjected to feature fusion processing with the spatial feature aggregation map of the resolution enhancement feature map to obtain a semantic segmentation map for obtaining a cross entropy loss value and / or a defect learning score; the shallow feature map and the middle feature map are resolution enhancement feature maps of a first scale size and a second scale size, respectively; the first scale size is larger than the second scale size;

[0008] S2. The image to be detected is subjected to image-level enhancement processing and feature-level enhancement processing respectively to obtain an image-level enhancement map to be detected and a feature-level enhancement map to be detected; the image-level enhancement map to be detected and the feature-level enhancement map to be detected are processed by the trained semantic segmentation model respectively to obtain an image-level pseudo-label prediction map and a feature-level pseudo-label prediction map in real time; the image-level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the feature-level pseudo-label prediction map to obtain a first constraint value; the feature-level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the image-level pseudo-label prediction map to obtain a second constraint value; based on the first constraint value and the second constraint value, a bidirectional constraint total value is obtained; the sum of the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is used as the constrained defect score; when the bidirectional constraint total value is not greater than the constraint threshold, and the constrained defect score is not less than the constraint score threshold, the current image-level pseudo-label prediction map or the feature-level pseudo-label prediction map is output as the defect detection result.

[0009] The first constraint value in S2 is: the ratio of the predicted values ​​of all pixels in the feature-level pseudo-label prediction map to the average of the smoothed cross entropy loss values ​​of the segmentation map label values ​​of the pixels at their corresponding positions in the image-level pseudo-label prediction map, and the size of the feature-level pseudo-label prediction map; in the image-level pseudo-label prediction map, the segmentation map label value of the current pixel is obtained based on the size comparison relationship between the predicted value of the current pixel and the prediction threshold.

[0010] The method for obtaining the defect learning score is specifically as follows: obtaining the predicted values ​​and corresponding segmentation label values ​​of all pixel values ​​in the current feature map; among the segmentation label values ​​of all pixel values, the segmentation label value with the largest number corresponds to the defect type as the scored defect category; taking the sum of the pixels whose predicted value is greater than the preset score threshold and whose segmentation label value is the scored defect category as the initial defect score; taking the product of the normalized initial defect score and the preset score threshold as the defect learning score; the current feature map is an image with a defect label, or an enhanced image at the image level to be detected, or an enhanced image at the feature level to be detected; in the process of training the semantic segmentation model in S1, the average of the initial defect learning scores of all images with defect labels is the initial defect learning score of the semantic segmentation model.

[0011] The channel fusion processing operation in S1 is as follows: the shallow feature map is processed by dilated convolution and feature channel to obtain the first channel feature map; the middle feature map is processed by feature channel to obtain the second channel feature map; the second channel feature map and the first channel feature map are spliced ​​to obtain a channel splicing map for performing convolution operations; the feature channel processing operation is as follows: the input is processed by global average pooling, one-dimensional convolution and activation function in sequence, and then weighted with the input to obtain the output as the first channel feature map or the second channel feature map.

[0012] The method for obtaining the spatial feature aggregation map of the resolution enhanced feature map in S1 is specifically as follows: the resolution enhanced feature map is subjected to dilated convolution, channel attention processing, spatial attention processing, convolution and upsampling processing to obtain a spatial feature aggregation map; the operation of dilated convolution is specifically as follows: the resolution enhanced feature map is subjected to several convolution and pooling processing respectively, and then spliced ​​to obtain a dilated convolution map for performing the channel attention processing operation; in the channel attention processing, feature mapping is implemented based on full connection; in the spatial attention processing, spatial feature aggregation is implemented based on dilated convolution.

[0013] The operations of the image level enhancement processing in S2 are specifically as follows: obtaining the complexity value of the image to be detected, and determining whether the complexity value exceeds the complexity threshold; if not, performing strong perturbation processing on the image to be detected to obtain a strongly perturbed image as the image level enhancement map to be detected; strong perturbation processing can be achieved by injecting noise, or / and random image erasing; if exceeded, performing weak perturbation processing on the image to be detected to obtain a weakly perturbated image as the image level enhancement map to be detected; weak perturbation processing can be achieved by randomly selecting rotation, or / and translation, or / and shearing, or / and flipping, or / and cropping.

[0014] The complexity value of the image to be detected is obtained based on shape information, or / and color information, or / and texture information, or / and contrast information of the image to be detected.

[0015] A semi-supervised semantic segmentation defect detection system based on industrial scenarios, including:

[0016] A training semantic segmentation model generation module is used to train the semantic segmentation model using a number of images with defect labels. When the cross entropy loss value of the semantic segmentation model is not greater than the cross entropy loss threshold, and the corresponding defect learning score is not less than the score threshold, a training semantic segmentation model is obtained; the processing operation of the semantic segmentation model is as follows: the input image is processed by a convolutional neural network to obtain a resolution enhancement feature map; the shallow feature map and the middle feature map of the resolution enhancement feature map are subjected to channel fusion processing and convolution processing to obtain a channel feature map; the channel feature map is subjected to feature fusion processing with the spatial feature aggregation map of the resolution enhancement feature map to obtain a semantic segmentation map for obtaining a cross entropy loss value and / or a defect learning score; the shallow feature map and the middle feature map are resolution enhancement feature maps of a first scale size and a second scale size, respectively; the first scale size is larger than the second scale size;

[0017] The defect detection result generation module is used to perform image enhancement processing and feature enhancement processing on the image to be detected, respectively, to obtain an enhanced image map at the image level to be detected and an enhanced feature map at the feature level to be detected; the enhanced image map at the image level to be detected and the enhanced feature map at the feature level to be detected are processed by the trained semantic segmentation model respectively, and the image level pseudo-label prediction map and the feature level pseudo-label prediction map are obtained in real time; the image level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the feature level pseudo-label prediction map to obtain a first constraint value; the feature level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the image level pseudo-label prediction map to obtain a second constraint value; based on the first constraint value and the second constraint value, a total value of the bidirectional constraint is obtained; the sum of the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is used as the constrained defect score; when the total value of the bidirectional constraint is not greater than the constraint threshold, and the constrained defect score is not less than the constraint score threshold, the current image level pseudo-label prediction map or the feature level pseudo-label prediction map is output as the defect detection result.

[0018] A semi-supervised semantic segmentation defect detection device based on industrial scenarios includes a processor and a memory, wherein the processor implements the above-mentioned semi-supervised semantic segmentation defect detection method based on industrial scenarios when executing a computer program stored in the memory.

[0019] A computer-readable storage medium is used to store a computer program, wherein when the computer program is executed by a processor, the above-mentioned semi-supervised semantic segmentation defect detection method based on industrial scenarios is implemented.

[0020] The beneficial effects of the present invention are:

[0021] The present invention provides a semi-supervised semantic segmentation defect detection method based on industrial scenarios. A semantic segmentation model that can effectively capture tiny defects and defect edge features of an image is designed and trained. During the training process, the cross entropy loss value and defect learning score of the semantic segmentation model are obtained in real time, and the training quality of the semantic segmentation model is dynamically adjusted, so as to finally obtain a trained semantic segmentation model with high defect detection accuracy, good stability and high computational efficiency. Then, in the process of formally performing defect detection on an image to be detected, in addition to performing image-level perturbation based on image-level enhancement processing on the image to be detected, a branch is added to perform feature space perturbation based on feature-level enhancement processing on the image to be detected, and cross-constraint learning is performed on the prediction results of the two branches in the processing of the training semantic segmentation model, and the quality of cross-constraint learning is dynamically supervised, and the most accurate pixel-by-pixel features of tiny defects and defect edge parts are output, so as to achieve accurate detection of defective parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By reading the detailed description of the preferred embodiment below, the scheme and advantages of the present application will become clear to those skilled in the art. The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0023] In the attached picture:

[0024] Figure 1 : is a defect detection result diagram of different industrial scene images in the embodiment; wherein, Figure 1 In a, the upper picture is the original picture of the wine bottle, and the lower picture is the defect detection picture of the wine bottle; Figure 1 In b, the upper picture is the original picture of the wine bottle bottom, and the lower picture is the defect detection picture of the wine bottle bottom; Figure 1 In c, the upper picture is the original picture of the wine bottle label, and the lower picture is the defect detection picture of the wine bottle label; Figure 1 In d, the upper picture is the original picture of the edge area of ​​instant noodles, and the lower picture is the defect detection picture of the edge area of ​​instant noodles; Figure 1 In e, the upper picture is the original picture of the middle area of ​​instant noodles, and the lower picture is the defect detection picture of the middle area of ​​instant noodles; Figure 1 In f, the upper picture is the original eggshell picture, and the lower picture is the eggshell defect detection picture. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0026] This embodiment provides a semi-supervised semantic segmentation defect detection method based on industrial scenarios, including the following operations:

[0027] S1. A semantic segmentation model is trained using several images with defect labels. When the cross entropy loss value of the semantic segmentation model is not greater than the cross entropy loss threshold, and the corresponding defect learning score is not less than the score threshold, a trained semantic segmentation model is obtained; the processing operation of the semantic segmentation model is as follows: the input image is processed by a convolutional neural network to obtain a resolution enhancement feature map; the shallow feature map and the middle feature map of the resolution enhancement feature map are subjected to channel fusion processing and convolution processing to obtain a channel feature map; the channel feature map is subjected to feature fusion processing with the spatial feature aggregation map of the resolution enhancement feature map to obtain a semantic segmentation map for obtaining a cross entropy loss value and / or a defect learning score; the shallow feature map and the middle feature map are resolution enhancement feature maps of a first scale size and a second scale size, respectively; the first scale size is larger than the second scale size;

[0028] S2. The image to be detected is subjected to image-level enhancement processing and feature-level enhancement processing respectively to obtain an image-level enhancement map to be detected and a feature-level enhancement map to be detected; the image-level enhancement map to be detected and the feature-level enhancement map to be detected are processed by the trained semantic segmentation model respectively to obtain an image-level pseudo-label prediction map and a feature-level pseudo-label prediction map in real time; the image-level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the feature-level pseudo-label prediction map to obtain a first constraint value; the feature-level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the image-level pseudo-label prediction map to obtain a second constraint value; based on the first constraint value and the second constraint value, a bidirectional constraint total value is obtained; the sum of the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is used as the constrained defect score; when the bidirectional constraint total value is not greater than the constraint threshold, and the constrained defect score is not less than the constraint score threshold, the current image-level pseudo-label prediction map or the feature-level pseudo-label prediction map is output as the defect detection result.

[0029] S1. Use several images with defect labels to train the semantic segmentation model. When the cross entropy loss value of the semantic segmentation model is not greater than the cross entropy loss threshold, and the corresponding defect learning score is not less than the score threshold, the trained semantic segmentation model is obtained.

[0030] A semantic segmentation model that can effectively capture tiny defects and defect edge features in images is designed and trained. During the training process, the cross entropy loss value and defect learning score of the semantic segmentation model are obtained in real time, and the training quality of the semantic segmentation model is dynamically adjusted. Finally, a trained semantic segmentation model with high defect detection accuracy, good stability and high computational efficiency is obtained.

[0031] First, a number of standard defect images with industrial target defects are obtained, and the industrial target defects are marked in the standard defect images to form a number of images with defect labels.

[0032] Then, the semantic segmentation model is trained using several images with defect labels.

[0033] Among them, the processing operations of the semantic segmentation model are specifically as follows: the input image (image with defect labels) is processed by a convolutional neural network to obtain a resolution enhanced feature map; the shallow feature map and the middle feature map of the resolution enhanced feature map are subjected to channel fusion processing and convolution processing to obtain a channel feature map; the channel feature map is subjected to feature fusion processing with the spatial feature aggregation map of the resolution enhanced feature map to obtain a semantic segmentation map, which is used to obtain a cross entropy loss value and / or a defect learning score.

[0034] The above convolutional neural network consists of a convolutional layer and several Bottleneck layers. The specific structure is shown in Table 1.

[0035] Table 1 Convolutional neural network structure

[0036] Network layer number Input size Operation Name Number of output channels Operation usage times Convolution stride 1 224*224*3 Conv2d 32 1 2 2 112*112*32 Bottleneck 16 1 1 3 112*112*16 Bottleneck 24 2 2 4 56*56*24 Bottleneck 32 3 2 5 28*28*32 Bottleneck 64 4 1 6 28*28*64 Bottleneck 96 3 1 7 28*28*96 Bottleneck 160 3 1 8 28*28*160 Bottleneck 320 1 1 .

[0037] The convolutional neural network of this embodiment is an improvement on the MobileNetv2 network structure. In order to improve the resolution of the final output image, the step size of the 5th and 7th Bottleneck layers is set to 1, which can increase the resolution of the output image by 4 times compared with the original MobileNetv2 network and retain more detailed information in the image. In addition, the convolutional neural network of this embodiment discards the last 3 layers of the original MobileNetv2 network structure used to implement classification, which can effectively solve the problem of excessive parameters after the 8th layer without reducing the detection accuracy.

[0038] At the same time, in this embodiment, the shallow feature map and the middle feature map are resolution enhancement feature maps of the first scale size and the second scale size respectively; the first scale size is larger than the second scale size. The first scale is 1 / 2, and the second scale is 1 / 4.

[0039] In addition, the operation of channel fusion processing of the shallow feature map and the middle feature map of the resolution enhancement feature map is specifically as follows: the shallow feature map is processed by dilated convolution and feature channel to obtain a first channel feature map; the middle feature map is processed by feature channel to obtain a second channel feature map; the second channel feature map and the first channel feature map are spliced ​​to obtain a channel splicing map for performing the convolution operation.

[0040] The specific operation of feature channel processing is as follows: the input is processed by global average pooling, one-dimensional convolution and activation function in sequence, and then weighted with the input to obtain the output as the first channel feature map or the second channel feature map.

[0041] The resolution enhancement feature map of the first scale size and the resolution enhancement feature map of the second scale size are subjected to channel fusion processing on shallow features and middle features of the resolution enhancement feature map, which not only facilitates capturing tiny details and local changes in the image, such as tiny defects and edge information, but also can capture more complex shapes and local structural information in industrial images, making it easier to detect the shape, contour and local structure of the defect, thereby improving the defect detection effect.

[0042] At the same time, the method for obtaining the spatial feature aggregation map of the resolution enhanced feature map is specifically as follows: the resolution enhanced feature map is subjected to hole convolution, channel attention processing, spatial attention processing, convolution and upsampling processing to obtain the spatial feature aggregation map.

[0043] The specific operation of the dilated convolution is as follows: the resolution-enhanced feature map is processed by several convolutions and pooling respectively, and then spliced ​​to obtain a dilated convolution map, which is used to perform channel attention processing operations.

[0044] In channel attention processing, feature mapping is implemented based on full connection. Compared with the feature mapping processing of multi-layer perceptron in the original channel attention processing, full connection processing can improve the attention to the features of key areas.

[0045] This can effectively extract defective parts and reduce information duplication, thereby improving defect detection accuracy and computing efficiency.

[0046] In spatial attention processing, spatial feature aggregation is achieved based on dilated convolution. Compared with the spliced ​​spatial feature aggregation in the original channel attention processing, dilated convolution can reduce the number of module parameters under the same receptive field size, solve the problem of poor model robustness caused by too many module parameters, and improve the computational efficiency while improving the stability of defect detection.

[0047] The feature fusion processing of the channel feature map and the spatial feature aggregation map of the resolution enhancement feature map is achieved through splicing, convolution and upsampling.

[0048] Finally, in the process of training the semantic segmentation model, the cross entropy loss value of the real-time semantic segmentation model and the corresponding defect learning score are obtained. When the cross entropy loss value of the semantic segmentation model is not greater than the cross entropy loss threshold and the corresponding defect learning score is not less than the score threshold, the trained semantic segmentation model is obtained.

[0049] During training, the cross entropy loss value can be realized by the following formula:

[0050] ,

[0051] L pis the cross entropy loss value, N is the total number of images with defect labels, n For the n images with defect labels, W , H are the width and height of the image, respectively. p m For the m The true label value of each pixel, q m For the m The predicted value of each pixel, M is the total number of image pixels.

[0052] During the training process, the method for obtaining the defect learning score is specifically as follows: obtain the predicted values ​​and corresponding segmentation label values ​​of all pixel values ​​in the image with defect labels; in the process of training the semantic segmentation model, the segmentation label value of the pixel value is the defect label of the pixel value, which is marked in advance; among the segmentation label values ​​of all pixel values, the segmentation label value with the largest number corresponds to the defect type, which is used as the scored defect category; the sum of all pixel points whose predicted value is greater than the preset score threshold and whose segmentation label value is the scored defect category is used as the initial defect score; the product of the initial defect score after normalization and the preset score threshold is used as the defect learning score; the average of the initial defect learning scores of all images with defect labels is the initial defect learning score of the semantic segmentation model.

[0053] Among them, the initial defect score can be achieved by the following formula:

[0054] ,

[0055] ,

[0056] S t for t The initial defect score of the round semantic segmentation model, s t,i for t Round i The initial defect scores of images with defect labels, I for t The total number of images with defect labels used for training in this round, i For the i images with defect labels, p t (u m ) No. m The predicted value of each pixel, c is the defect category, qt (u m ) No. m The segmentation label value of each pixel, τ is the preset score threshold, M is the total number of image pixels.

[0057] The normalization operation can be achieved through the following formula:

[0058] ,

[0059] S t,norm for t Normalized initial defect score of the semantic segmentation model for the round, maxs t,i for t The maximum value of the initial defect scores of the images with defect labels selected for training in the round, maxs j,t-1 for t The images that were not selected for training in this round are t -1 The maximum value of the initial defect score in round 1.

[0060] The defect learning score can be achieved by the following formula:

[0061] ,

[0062] S t for t Defect scores for the epochal semantic segmentation model.

[0063] S2. The image to be detected is subjected to image-level enhancement processing and feature-level enhancement processing respectively to obtain an image-level enhancement map to be detected and a feature-level enhancement map to be detected; the image-level enhancement map to be detected and the feature-level enhancement map to be detected are processed by the trained semantic segmentation model respectively to obtain an image-level pseudo-label prediction map and a feature-level pseudo-label prediction map in real time; the image-level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the feature-level pseudo-label prediction map to obtain a first constraint value; the feature-level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the image-level pseudo-label prediction map to obtain a second constraint value; based on the first constraint value and the second constraint value, a bidirectional constraint total value is obtained; the sum of the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is used as the constrained defect score; when the bidirectional constraint total value is not greater than the constraint threshold, and the constrained defect score is not less than the constraint score threshold, the current image-level pseudo-label prediction map or the feature-level pseudo-label prediction map is output as the defect detection result.

[0064] In the process of formal defect detection on the image to be detected, in addition to the image-level perturbation based on image-level enhancement processing on the image to be detected, a branch is added to perform feature space perturbation based on feature-level enhancement processing on the image to be detected. The prediction results of the two branches in the training semantic segmentation model are cross-constrained learned, and the quality of cross-constraint learning is dynamically supervised to output the most accurate pixel-by-pixel features of tiny defects and defect edges to achieve accurate detection of defective parts.

[0065] Firstly, before formally obtaining the defect detection result of the image to be detected, in order to improve the detection accuracy, the image to be detected is enhanced at the image level and at the feature level to obtain an enhanced image at the image level and an enhanced image at the feature level.

[0066] The operation of the image level enhancement processing is specifically as follows: obtaining the complexity value of the image to be detected, and determining whether the complexity value exceeds a complexity threshold. The complexity value is obtained based on the shape information, or / and color information, or / and texture information, or / and contrast information of the image to be detected, and is a weighted average of the shape information, or / and color information, or / and texture information, or / and contrast information of the image to be detected.

[0067] The shape information of the image to be detected is the ratio of the contour perimeter to the contour area of ​​the image to be detected. After the image to be detected is grayed and binarized, the target points are connected to form several regions. The average of the ratios of the contour perimeter to the contour area of ​​the several regions is the ratio of the contour perimeter to the contour area of ​​the image to be detected. The larger the ratio, the more complex the shape information of the image to be detected.

[0068] The color information of the image to be detected is the average color distance of the image to be detected, which is the average value of the distance between the color of each pixel in the image to be detected and the average color of the image.

[0069] The texture information of the image to be detected is the homogeneity of the gray value of the image to be detected, which is obtained from the gray level co-occurrence matrix of the image to be detected.

[0070] If it does not exceed, a strong disturbance processing is performed on the image to be detected to obtain a strong disturbance image as an enhanced image of the image layer to be detected; the strong disturbance processing can be achieved by injecting noise and / or random image erasing.

[0071] If it exceeds, it means that the image to be detected is relatively complex. In order to retain more image details and ensure detection accuracy, the image to be detected is weakly perturbed to obtain a weakly perturbed image as an enhanced image of the image layer to be detected; the weak perturbation processing can be achieved by randomly selecting rotation, or / and translation, or / and shearing, or / and flipping, or / and cropping.

[0072] At the same time, feature-level enhancement processing can be achieved by performing a Dropout layer processing on the image to be detected in a random channel dimension, thereby increasing the perturbation of the feature space and facilitating subsequent cross-supervised learning with the image-level enhancement map to be detected, thereby reducing missed detections and false detections and improving defect detection accuracy.

[0073] Then, the enhanced image layer to be detected and the enhanced feature layer to be detected are processed by the trained semantic segmentation model respectively, and the image layer pseudo-label prediction map and the feature layer pseudo-label prediction map are obtained in real time; and the image layer pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the feature layer pseudo-label prediction map to obtain the first constraint value; the feature layer pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the image layer pseudo-label prediction map to obtain the second constraint value.

[0074] Among them, the first constraint value is: the ratio of the predicted values ​​of all pixels in the feature-level pseudo-label prediction map to the average of the smoothed cross entropy loss values ​​of the pixel segmentation map label values ​​at their corresponding positions in the image-level pseudo-label prediction map, and the size of the feature-level pseudo-label prediction map; and in the image-level pseudo-label prediction map, the segmentation map label value of the current pixel is obtained based on the size comparison relationship between the predicted value of the current pixel and the prediction threshold.

[0075] The method for obtaining the second constraint value is similar to that of the first constraint value. To save space, it will not be described in detail here.

[0076] Subsequently, based on the first constraint value and the second constraint value, the total value of the bidirectional constraint is obtained; specifically, the weighted sum of the first constraint value and the second constraint value is taken as the total value of the bidirectional constraint; at the same time, the sum of the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is taken as the constraint defect score.

[0077] The method for obtaining the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is similar to that of the defect learning score when training the semantic segmentation model; however, the input is respectively an enhanced image of the image layer to be detected or an enhanced image of the feature layer to be detected; and in the operation of normalizing the initial defect score, the normalized defect learning score corresponding to the first constraint value and the normalized defect learning score corresponding to the second constraint value are obtained based on the maximum historical initial defect learning score and the minimum historical initial defect learning score.

[0078] Finally, when the total value of the bidirectional constraint is not greater than the constraint threshold and the constraint defect score is not less than the constraint score threshold, the current image-level pseudo-label prediction map or feature-level pseudo-label prediction map is output as the defect detection result (see Figure 1), realize the dynamic supervision cross-constraint learning quality and improve the defect detection accuracy of tiny defects and defect edges.

[0079] This embodiment also provides a semi-supervised semantic segmentation defect detection system based on industrial scenarios, including:

[0080] A training semantic segmentation model generation module is used to train the semantic segmentation model using a number of images with defect labels. When the cross entropy loss value of the semantic segmentation model is not greater than the cross entropy loss threshold, and the corresponding defect learning score is not less than the score threshold, a training semantic segmentation model is obtained; the processing operation of the semantic segmentation model is as follows: the input image is processed by a convolutional neural network to obtain a resolution enhancement feature map; the shallow feature map and the middle feature map of the resolution enhancement feature map are subjected to channel fusion processing and convolution processing to obtain a channel feature map; the channel feature map is subjected to feature fusion processing with the spatial feature aggregation map of the resolution enhancement feature map to obtain a semantic segmentation map for obtaining a cross entropy loss value and / or a defect learning score; the shallow feature map and the middle feature map are resolution enhancement feature maps of a first scale size and a second scale size, respectively; the first scale size is larger than the second scale size;

[0081] The defect detection result generation module is used to perform image enhancement processing and feature enhancement processing on the image to be detected, respectively, to obtain an enhanced image map at the image level to be detected and an enhanced feature map at the feature level to be detected; the enhanced image map at the image level to be detected and the enhanced feature map at the feature level to be detected are processed by the trained semantic segmentation model respectively, and the image level pseudo-label prediction map and the feature level pseudo-label prediction map are obtained in real time; the image level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the feature level pseudo-label prediction map to obtain a first constraint value; the feature level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the image level pseudo-label prediction map to obtain a second constraint value; based on the first constraint value and the second constraint value, a total value of the bidirectional constraint is obtained; the sum of the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is used as the constrained defect score; when the total value of the bidirectional constraint is not greater than the constraint threshold, and the constrained defect score is not less than the constraint score threshold, the current image level pseudo-label prediction map or the feature level pseudo-label prediction map is output as the defect detection result.

[0082] This embodiment also provides a semi-supervised semantic segmentation defect detection device based on industrial scenarios, including a processor and a memory, wherein the processor implements the above-mentioned semi-supervised semantic segmentation defect detection method based on industrial scenarios when executing a computer program stored in the memory.

[0083] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the above-mentioned semi-supervised semantic segmentation defect detection method based on industrial scenarios is implemented.

[0084] The present embodiment provides a semi-supervised semantic segmentation defect detection method based on an industrial scenario. A semantic segmentation model that can effectively capture tiny defects and defect edge features of an image is designed and trained. During the training process, the cross-entropy loss value and defect learning score of the semantic segmentation model are obtained in real time, and the training quality of the semantic segmentation model is dynamically adjusted, so as to finally obtain a trained semantic segmentation model with high defect detection accuracy, good stability, and high computational efficiency. Then, in the formal process of defect detection on the image to be detected, in addition to performing image-level perturbation based on image-level enhancement processing on the image to be detected, a branch is added to perform feature space perturbation based on feature-level enhancement processing on the image to be detected, and the prediction results of the two branches in the training semantic segmentation model processing are cross-constrained learned, the cross-constraint learning quality is dynamically supervised, and the most accurate pixel-by-pixel features of tiny defects and defect edge parts are output to achieve accurate detection of defective parts.

Claims

1. A semi-supervised semantic segmentation defect detection method based on industrial scenarios, characterized in that: The following operations are included: S1. Use several images with defect labels to train the semantic segmentation model. When the cross entropy loss value of the semantic segmentation model is not greater than the cross entropy loss threshold, and the corresponding defect learning score is not less than the score threshold, the trained semantic segmentation model is obtained; The processing operation of the semantic segmentation model is: the input image is processed by a convolutional neural network to obtain a resolution enhanced feature map; The shallow feature map and the middle feature map of the resolution enhancement feature map are subjected to channel fusion processing and convolution processing to obtain a channel feature map; the channel feature map is subjected to feature fusion processing with the spatial feature aggregation map of the resolution enhancement feature map to obtain a semantic segmentation map for obtaining a cross entropy loss value and / or a defect learning score; The shallow feature map and the middle feature map are resolution enhancement feature maps of a first scale size and a second scale size respectively; the first scale size is larger than the second scale size; During the training process, the defect learning score is obtained by: obtaining the predicted values ​​and corresponding segmentation label values ​​of all pixel values ​​in the image with defect labels; among the segmentation label values ​​of all pixel values, the segmentation label value with the largest number corresponds to the defect type, which is used as the score defect category; The sum of all pixels whose predicted values ​​are greater than the preset score threshold and whose segmentation label values ​​are the score defect category is taken as the initial defect score; The product of the normalized initial defect score and the preset score threshold is used as the defect learning score; The mean of the initial defect learning scores of all images with defect labels is the initial defect learning score of the semantic segmentation model; S2, the image to be detected is subjected to image-level enhancement processing and feature-level enhancement processing respectively, to obtain an image-level enhancement map to be detected and an enhanced map to be detected at the feature level; the image-level enhancement map to be detected and the enhanced map to be detected at the feature level are processed by the trained semantic segmentation model respectively, and an image-level pseudo-label prediction map and a feature-level pseudo-label prediction map are obtained in real time; Using the segmentation map label value corresponding to the pseudo-label prediction map at the feature level, constrain the image-level pseudo-label prediction map to obtain a first constraint value; using the segmentation map label value corresponding to the pseudo-label prediction map at the image level, constrain the feature-level pseudo-label prediction map to obtain a second constraint value; based on the first constraint value and the second constraint value, obtain a bidirectional constraint total value; and use the sum of the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value as the constrained defect score; The method for obtaining the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is similar to the defect learning score when training the semantic segmentation model; The input is an enhanced image of the image layer to be detected or an enhanced image of the feature layer to be detected; in the operation of normalizing the initial defect score, the normalized defect learning score corresponding to the first constraint value and the normalized defect learning score corresponding to the second constraint value are obtained based on the maximum value of the historical initial defect learning score and the minimum value of the historical initial defect learning score; When the total value of the bidirectional constraint is not greater than the constraint threshold and the constraint defect score is not less than the constraint score threshold, the current image-level pseudo-label prediction map or feature-level pseudo-label prediction map is output as the defect detection result.

2. The method for semi-supervised semantic segmentation defect detection based on industrial scenarios according to claim 1 is characterized in that: In S2, The first constraint value is: the ratio of the average of the smoothed cross entropy loss values ​​of all pixel prediction values ​​in the feature-level pseudo-label prediction map and the label values ​​of the pixel segmentation map at the corresponding positions in the image-level pseudo-label prediction map, and the size of the feature-level pseudo-label prediction map; In the image-level pseudo-label prediction map, the segmentation map label value of the current pixel is obtained based on the size comparison relationship between the predicted value of the current pixel and the prediction threshold.

3. The method for semi-supervised semantic segmentation defect detection based on industrial scenarios according to claim 1 is characterized in that: In S1, the channel fusion processing operation is specifically as follows: The shallow feature map is processed by dilated convolution and feature channel to obtain a first channel feature map; the middle feature map is processed by feature channel to obtain a second channel feature map; the second channel feature map and the first channel feature map are spliced ​​to obtain a channel splicing map for performing the convolution operation; The operation of the feature channel processing is specifically as follows: the input is processed by global average pooling, one-dimensional convolution and activation function in sequence, and then weighted processing is performed with the input to obtain an output as a first channel feature map or a second channel feature map.

4. The method for semi-supervised semantic segmentation defect detection based on industrial scenarios according to claim 1 is characterized in that: In S1, the method for obtaining the spatial feature aggregation map of the resolution enhancement feature map is specifically as follows: The resolution enhancement feature map is processed by hole convolution, channel attention processing, spatial attention processing, convolution and upsampling to obtain the spatial feature aggregation map; The operation of the dilated convolution is specifically as follows: the resolution enhancement feature map is subjected to a plurality of convolution processes and pooling processes respectively, and then is spliced ​​to obtain a dilated convolution map, which is used to perform the channel attention processing operation; In the channel attention processing, feature mapping is implemented based on full connection; In the spatial attention processing, spatial feature aggregation is achieved based on dilated convolution.

5. The method for semi-supervised semantic segmentation defect detection based on industrial scenarios according to claim 1 is characterized in that: In S2, the image level enhancement processing operation is specifically as follows: Obtaining a complexity value of the image to be detected, and determining whether the complexity value exceeds a complexity threshold; If it does not exceed, performing strong disturbance processing on the image to be detected to obtain a strong disturbance image as the layer enhancement image of the image to be detected; the strong disturbance processing can be achieved by injecting noise, and / or random image erasure; If it exceeds, a weak perturbation process is performed on the image to be detected to obtain a weak perturbation image as the layer enhancement image of the image to be detected; the weak perturbation process can be achieved by randomly selecting rotation, or / and translation, or / and shearing, or / and flipping, or / and cropping.

6. The method for semi-supervised semantic segmentation defect detection based on industrial scenarios according to claim 5 is characterized in that: The complexity value of the image to be detected is obtained based on shape information, or / and color information, or / and texture information, or / and contrast information of the image to be detected.

7. A semi-supervised semantic segmentation defect detection system based on industrial scenarios, used to implement the semi-supervised semantic segmentation defect detection method based on industrial scenarios according to claim 1, characterized in that: include: A training semantic segmentation model generation module is used to train the semantic segmentation model using a number of images with defect labels. When the cross entropy loss value of the semantic segmentation model is not greater than the cross entropy loss threshold, and the corresponding defect learning score is not less than the score threshold, a trained semantic segmentation model is obtained; the processing operation of the semantic segmentation model is: the input image is processed by a convolutional neural network to obtain a resolution enhancement feature map; The shallow feature map and the middle feature map of the resolution enhancement feature map are subjected to channel fusion processing and convolution processing to obtain a channel feature map; the channel feature map is subjected to feature fusion processing with the spatial feature aggregation map of the resolution enhancement feature map to obtain a semantic segmentation map for obtaining a cross entropy loss value and / or a defect learning score; the shallow feature map and the middle feature map are resolution enhancement feature maps of a first scale size and a second scale size, respectively; The first proportional dimension is greater than the second proportional dimension; The defect detection result generation module is used to perform image enhancement processing and feature enhancement processing on the image to be detected, respectively, to obtain an enhanced image level map to be detected and an enhanced feature level map to be detected; the enhanced image level map to be detected and the enhanced feature level map to be detected are respectively processed by the trained semantic segmentation model to obtain the image level pseudo-label prediction map and the feature level pseudo-label prediction map in real time; the image level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the feature level pseudo-label prediction map to obtain a first constraint value; the feature level pseudo-label prediction map is constrained by using the segmentation map label value corresponding to the image level pseudo-label prediction map to obtain a second constraint value; based on the first constraint value and the second constraint value, a bidirectional constraint total value is obtained; the sum of the defect learning score corresponding to the first constraint value and the defect learning score corresponding to the second constraint value is used as the constrained defect score; when the bidirectional constraint total value is not greater than the constraint threshold, and the constrained defect score is not less than the constraint score threshold, the current image level pseudo-label prediction map or the feature level pseudo-label prediction map is output as the defect detection result.

8. A semi-supervised semantic segmentation defect detection device based on industrial scenarios, characterized in that: It includes a processor and a memory, wherein when the processor executes the computer program stored in the memory, it implements the semi-supervised semantic segmentation defect detection method based on industrial scenarios as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the semi-supervised semantic segmentation defect detection method based on an industrial scenario as described in any one of claims 1 to 6 is implemented.

Images