Industrial product defect detection method and system based on probability mask

Abstract

The present invention provides an industrial product defect detection method and system based on probability mask, which relates to the technical field of defect detection. The method comprises: obtaining a defect image to be detected; inputting the defect image to be detected into a trained defect detection model for detection, and obtaining a defect detection result; the input of the defect detection model comprises a labeled defect image, an unlabeled defect image, and a mixed intermediate image, the mixed intermediate image is obtained by mixing the labeled defect image and the unlabeled defect image, the defect detection model extracts a feature map of the input image, obtains probability mask screening information corresponding to each feature map, obtains corresponding interactive information according to the probability mask screening information, and obtains corresponding defect detection results according to the interactive information. The present invention can improve the efficiency of model deployment while ensuring accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of defect detection, and in particular to an industrial product defect detection method and system based on probability mask. Background Art

[0002] In industrial scenarios, industrial products inevitably have defects during the production process, such as bubbles and oil stains in glass containers, and knots, cracks, peeling and other defects in wood. If these defective industrial products are not removed during the quality inspection process, they will bring huge safety hazards. Therefore, it is necessary to conduct comprehensive defect detection on industrial products.

[0003] Defect detection methods based on supervised learning of deep learning are gradually being applied in defect detection of industrial products. Such methods require a large number of defect images as training to be competent for defect detection tasks. However, in industrial scenarios, it is impossible to collect and annotate all defect information before training and model deployment, which will greatly increase the training cycle of the model and delay the defect detection task of industrial products. If unsupervised learning defect detection methods are used, it is difficult to meet the accuracy requirements of defect detection. Summary of the invention

[0004] In order to address the deficiencies of the prior art, the purpose of the present invention is to provide an industrial product defect detection method and system based on probability masking, so as to improve the efficiency of model deployment while ensuring accuracy.

[0005] In order to achieve the above object, according to some embodiments, a first aspect of the present invention provides an industrial product defect detection method based on probability mask, comprising:

[0006] Acquire an image of a defect to be detected;

[0007] The defect image to be detected is input into the trained defect detection model for detection to obtain the defect detection result;

[0008] Among them, the input of the defect detection model includes a labeled defect image, an unlabeled defect image and a mixed intermediate image, and the mixed intermediate image is obtained by mixing the labeled defect image and the unlabeled defect image; the defect detection model respectively extracts feature maps of the labeled defect image, the unlabeled defect image and the mixed intermediate image, obtains the probability mask filtering information corresponding to each feature map, obtains the corresponding interactive information according to the probability mask filtering information, and obtains the corresponding defect detection result according to the interactive information.

[0009] A second aspect of the present invention provides an industrial product defect detection system based on probability mask, comprising:

[0010] A data acquisition module is configured to acquire an image of a defect to be detected;

[0011] The defect detection module is configured to input the defect image to be detected into the trained defect detection model for detection to obtain the defect detection result;

[0012] Among them, the input of the defect detection model includes a labeled defect image, an unlabeled defect image and a mixed intermediate image, and the mixed intermediate image is obtained by mixing the labeled defect image and the unlabeled defect image; the defect detection model respectively extracts feature maps of the labeled defect image, the unlabeled defect image and the mixed intermediate image, obtains the probability mask filtering information corresponding to each feature map, obtains the corresponding interactive information according to the probability mask filtering information, and obtains the corresponding defect detection result according to the interactive information.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] The present invention provides an industrial product defect detection method and system based on probability mask, which realizes unsupervised adaptive feature alignment based on constructing a mixed intermediate image. The mixed intermediate image is obtained by fusing annotated defect images and unlabeled defect images. In the defect detection process, the model has absolute knowledge of the source domain, and the intermediate domain comes from the source domain, so the model also has knowledge of the intermediate domain. The image of the intermediate domain also comes from the target domain, so the model has knowledge of the target domain, thereby realizing unsupervised adaptive feature alignment for unlabeled defect images, improving the accuracy of unsupervised defect detection of industrial products, and having high robustness. In addition, the present invention also designs a probability mask screening module, which further improves the detection performance of industrial product defects by screening important information and noise information of defect features of industrial products.

[0015] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0017] Figure 1 is a schematic diagram of a defect detection model in Embodiment 1 of the present invention;

[0018] Figure 2 Schematic diagram of the generation principle of the mixed intermediate image;

[0019] Figure 3 Schematic diagram of the probabilistic mask filtering module. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0021] Embodiment 1

[0022] Embodiment 1 of the present invention provides an industrial product defect detection method based on probability masking, such as Figure 1-Figure 3 As shown, including:

[0023] Acquire an image of a defect to be detected;

[0024] The defect image to be detected is input into the trained defect detection model for detection to obtain the defect detection result;

[0025] Among them, the input of the defect detection model includes a labeled defect image, an unlabeled defect image and a mixed intermediate image, and the mixed intermediate image is obtained by mixing the labeled defect image and the unlabeled defect image; the defect detection model respectively extracts feature maps of the labeled defect image, the unlabeled defect image and the mixed intermediate image, obtains the probability mask filtering information corresponding to each feature map, obtains the corresponding interactive information according to the probability mask filtering information, and obtains the corresponding defect detection result according to the interactive information.

[0026] Specifically, the defect detection model includes a data set A input module, a data set B input module, a mixed intermediate image input module, an information extraction module, a probability mask screening module, a multi-source scale information interaction module, and a detection unit. The input data of the data set A input module is a labeled defect image, the input data of the data set B input module is an unlabeled defect image, and the input data of the mixed intermediate image input module is a mixed intermediate image obtained by fusing the labeled defect image and the unlabeled defect image; the information extraction module is used to extract the feature map of the image input by the data set A input module, the data set B input module, and the mixed intermediate image input module, and send it to the probability mask screening module; the probability mask screening module is used to obtain the probability mask screening information corresponding to each feature map, and send it to the multi-source scale information interaction module; the multi-source scale information interaction module is used to obtain the corresponding interaction information according to the probability mask screening information, and send it to the detection unit; the detection unit is used to obtain the corresponding defect detection result according to the interaction information.

[0027] In defect detection of industrial products, supervised learning has high requirements for training samples, and the training and deployment efficiency of the model is low. Traditional unsupervised learning has poor detection performance for unlabeled training samples, and it is difficult to meet the accuracy requirements of industrial product defect detection. In order to improve the efficiency of model deployment while ensuring accuracy, this embodiment proposes to construct a mixed intermediate image so that the model can also have certain knowledge of unlabeled training samples, thereby greatly improving the detection performance.

[0028] like Figure 1 As shown in the figure, the input of the model includes three parts: dataset A, dataset B, and mixed intermediate images. Among them, dataset A is a labeled defect image. The surface defect images of industrial products are obtained through industrial cameras, light sources, and image acquisition cards. The surface defect images of industrial products are annotated to obtain industrial product defect dataset A. ,in, is the defect image in dataset A, Respectively represent the 1st, 2nd, ..., Nth defect images in data set A, is the defect label corresponding to the defect image in dataset A, They respectively represent the defect labels corresponding to the 1st, 2nd, ..., Nth defect images in data set A.

[0029] In this embodiment, the industrial camera uses an array CCD camera, and the light source uses a strip light source. A rectangular frame is used to annotate the defect image, and then a corresponding label number is assigned to each defect to form a data set A. Specifically, taking a glass container as an example, Labelimg software is used to annotate the image of the bottle body of the glass container using a rectangular frame, and the bubble defect in the bottle body detection part is assigned a label number of 0, the stone defect is assigned a label number of 1, the oil stain defect is assigned a label number of 2, the vertical axis deviation defect is assigned a label number of 3, the bottle internal wire defect is assigned a label number of 4, the dead material defect is assigned a label number of 5, the quenching defect is assigned a label number of 6, the aircraft wing defect is assigned a label number of 7, and the mold seam line thick defect is assigned a label number of 8.

[0030] Dataset B is an unlabeled defect image. The surface defect images of industrial products are acquired through industrial cameras, light sources and image acquisition cards to obtain industrial product defect dataset B. ,in, They represent the 1st, 2nd, ..., Nth defect images in dataset B. This dataset only contains defect images of industrial products, but does not contain the defect labels corresponding to the defect images. At the same time, the images in dataset A and dataset B are not repeated, and in order to construct the mixed intermediate image, the number of images in dataset A and dataset B is equal.

[0031] Based on dataset A (source domain) and dataset B (target domain), a mixed intermediate image dataset H (intermediate domain) is constructed. The image samples in this dataset contain the foreground and background features of both dataset A and dataset B.

[0032] This embodiment constructs a strategy of mixing intermediate images based on probability masks to synthesize new pseudo detection training samples. The synthesized samples contain foreground and background features of two domains, thereby more effectively adapting to the target domain, rather than randomly selecting a part of the image for mixing.

[0033] Probabilistic masking is a commonly used technique in text generation, which helps to reduce repeated or irrelevant words in the generated text. The probability mask in this embodiment is a two-dimensional matrix, each value in the two-dimensional matrix is ​​a probability between 0 and 1, and the new feature matrix or feature tensor is obtained by multiplying the two-dimensional matrix with the original feature matrix (or feature tensor).

[0034] Specifically, if Figure 2 As shown, this embodiment uses two mask matrices Hs and Ht of size 640×640 to sample the source domain image (the labeled defect image in data set A) and the target domain image (the unlabeled defect image in data set B), and multiplies the source domain image and the target domain image with the mask matrices Hs and Ht respectively, and then adds them together to generate a mixed intermediate image. The mask matrix uses a binary mask, where the upper half 640×320 of Hs is all 1 and the lower half is all 0; Ht is the opposite, the lower half is all 1 and the upper half is all 0.

[0035] Dataset A, data set B, and mixed intermediate image data set H have the same number of images. The i-th image in data set A, data set B, and mixed intermediate image data set H is input into the information extraction module of the defect detection model through the data set A input module, data set B input module, and mixed intermediate image input module to extract feature maps. The information extraction module includes four sequentially connected information extraction structures: a first information extraction structure, a second information extraction structure, a third information extraction structure, and a fourth information extraction structure. The first information extraction structure, the second information extraction structure, the third information extraction structure, and the fourth information extraction structure have the same structure, and are all composed of a basic convolution mechanism and an enhanced extraction mechanism. The basic convolution mechanism consists of a convolution layer with a kernel size of 3 and a stride of 2, a BN (Batch Normalization) layer, and a SiLU activation function layer; the enhanced extraction mechanism consists of three sequentially connected residual convolution groups, and a single residual convolution group consists of a residual edge, a convolution layer with a kernel size of 1, a batch normalization layer, a SiLU activation function, a convolution layer with a kernel size of 3, a batch normalization layer, a SiLU activation function, a convolution layer with a kernel size of 1, a batch normalization layer, a sum layer, and a SiLU activation function layer connected in sequence. The image in the dataset A of the input information extraction module is called the first input image, the image in the dataset B of the input information extraction module is called the second input image, and the image in the mixed intermediate image dataset of the input information extraction module is called the third input image. The first input image is input into the information extraction module, and after the feature map is extracted by the first information extraction structure, the second information extraction structure, the third information extraction structure, and the fourth information extraction structure, the first feature map set is obtained. ,in, are feature maps extracted from the first input image by the first information extraction structure, the second information extraction structure, the third information extraction structure, and the fourth information extraction structure. Similarly, the second input image is extracted by the information extraction module to obtain the second feature map set , the third input image is passed through the information extraction module to obtain the third feature atlas .

[0036] The feature maps extracted by the second information extraction structure, the third information extraction structure, and the fourth information extraction structure are input into the probability mask screening module to obtain the probability mask screening information corresponding to each input image. Specifically, taking the first input image as an example, the first input image is extracted by the information extraction module to obtain the first feature map set , among which Input the probability mask filtering module respectively to obtain the corresponding probability mask filtering information, forming the first probability mask filtering information feature atlas ,in, According to The corresponding probability mask is used to filter information. Similarly, the second feature map extracted by the information extraction module from the second input image , after the probability mask filtering module, the second probability mask filtering information feature atlas is obtained ; The third feature atlas extracted by the information extraction module from the third input image After the probability mask filtering module, the third probability mask filtering information feature atlas is obtained .

[0037] The probability mask screening module is used to retain the optimal industrial product defect features and filter important information and noise information for the model. Specifically, Figure 3 As shown in the figure, the probability mask screening module includes a probability mask construction mechanism, an information filtering mechanism and a first feature extraction mechanism. The probability mask construction mechanism outputs a feature map of size C×H×W as a probability map of size 1×H×W, where C is the number of channels, H is the height of the image, and W is the width of the image. The construction process is as follows: suppose the feature map of the input probability mask construction mechanism is F, whose size is C×H×W, and the feature map is input to the mean layer and the probability layer; the mean layer averages the feature map in the C direction and outputs it as the first feature map F1 of size 1×H×W; then, the first feature map F1 is input to the probability layer, mapped to between (0,1), and output as a probability map F2 of size 1×H×W. The mapping process is shown below:

[0038] .

[0039] The information filtering mechanism multiplies the feature map input to the probability mask filtering module by the probability map F2 corresponding to the feature map to highlight the useful information, thereby obtaining the important information corresponding to the feature map; at the same time, the feature map is multiplied by (1-F2) to obtain the noise information corresponding to the feature map.

[0040] The first feature extraction mechanism consists of a convolution layer with a convolution kernel size of 3, a batch normalization layer, and a SiLU activation function layer connected in sequence. After the important information obtained by the information filtering mechanism is extracted by the first feature extraction mechanism, the probability mask filtering information is output.

[0041] The multi-source scale information interaction module is used to obtain the corresponding interactive information according to the probability mask filtering information, and send it to the detection unit for defect detection. The multi-source scale information interaction module includes a second feature extraction mechanism, an upsampling mechanism and a fusion mechanism. Among them, the second feature extraction mechanism is composed of a convolution layer with a convolution kernel size of 1, a batch normalization layer, a SiLU activation function layer, a convolution layer with a convolution kernel size of 3, a batch normalization layer and a SiLU activation function layer connected in sequence; the upsampling mechanism is composed of the nearest neighbor interpolation method, which upsamples the feature map output by the second feature extraction mechanism so that the upsampled feature map is consistent with the feature map input by the multi-source scale information interaction module, so that the fusion mechanism can effectively fuse the two; the fusion mechanism takes the probability mask filtering information output from the probability mask filtering module and the output features of the upsampling mechanism as input, and splices the features of the two sources in the channel dimension, so as to realize information sharing and obtain interactive information.

[0042] The first probability mask is used to filter the information feature atlas Each probability mask in the filter information Input the multi-source scale information interaction module respectively to obtain the corresponding interaction information , constituting the first mutual information feature atlas of the first input image Similarly, the second mutual information feature atlas corresponding to the second input image can be obtained , and the third interactive information feature atlas corresponding to the third input image Finally, the first interactive information feature map , the second interactive information feature atlas corresponding to the second input image , the third interactive information feature atlas corresponding to the third input image An input detection unit is used to perform defect detection, thereby obtaining a first defect detection result for the first input image, a second defect detection result for the second input image, and a third defect detection result for the third input image.

[0043] During the defect detection model training process, the true labels of the first input image and the third input image are used to calculate the loss, and the model is gradient optimized by the optimizer. The first input image from dataset A is a defect image with a defect label, and the third input image is obtained based on the first input image, so it has the same defect label.

[0044] According to the first defect detection result and the defect label corresponding to the first input image, the first loss is calculated. The first loss exists to ensure that the model maintains certain knowledge during the learning and feature alignment process. The calculation formula of the first loss is defined as Lcalculate.

[0045] According to the third defect detection result and the defect label corresponding to the first input image, the second loss is calculated. The second loss exists to merge the first defect detection result and the second defect detection result into a fused defect detection result, and calculate the loss between the fused defect detection result and the third defect detection result, so as to achieve the purpose of model optimization and penalizing the third defect detection result. The calculation formula of the second loss is consistent with the calculation formula of the first loss, and is also defined as Lcalculate.

[0046] Lcalculate is obtained by summing CIoU loss and BCE loss, and is adjusted by Focal Loss loss. The AdamW optimizer is used to perform gradient descent optimization on the model. The process of calculating the loss function and optimizing the model by gradient descent is also the process of unsupervised adaptive alignment of features. The synthesized mixed intermediate image is gradually fitted into the domain gap. The loss is calculated and the feature alignment is enhanced by the synthesized mixed intermediate image and the automatically generated pseudo-label. The pseudo-label is the defect label automatically generated by the defect detection model on the mixed intermediate image based on the learning result of the first input image. Finally, the accuracy of unsupervised defect detection of industrial products is improved.

[0047] Deploy the trained defect detection model to industrial scenarios to perform real-time defect detection on industrial products.

[0048] Embodiment 2

[0049] This embodiment provides an industrial product defect detection system based on probability mask, including:

[0050] A data acquisition module is configured to acquire an image of a defect to be detected;

[0051] The defect detection module is configured to input the defect image to be detected into the trained defect detection model for detection to obtain the defect detection result;

[0052] Among them, the input of the defect detection model includes a labeled defect image, an unlabeled defect image and a mixed intermediate image, and the mixed intermediate image is obtained by mixing the labeled defect image and the unlabeled defect image; the defect detection model respectively extracts feature maps of the labeled defect image, the unlabeled defect image and the mixed intermediate image, obtains the probability mask filtering information corresponding to each feature map, obtains the corresponding interactive information according to the probability mask filtering information, and obtains the corresponding defect detection result according to the interactive information.

[0053] It should be noted here that the various modules in this embodiment correspond one-to-one to the steps of the method in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for industrial product defect detection based on probability mask, characterized in that: include: Acquire an image of a defect to be detected; The defect image to be detected is input into the trained defect detection model for detection to obtain the defect detection result; The input of the defect detection model includes a labeled defect image, an unlabeled defect image and a mixed intermediate image, and the mixed intermediate image is obtained by mixing the labeled defect image and the unlabeled defect image; the defect detection model extracts feature maps of the labeled defect image, the unlabeled defect image and the mixed intermediate image respectively, obtains probability mask filtering information corresponding to each feature map, obtains corresponding interaction information according to the probability mask filtering information, and obtains corresponding defect detection results according to the interaction information; The mixed intermediate image is obtained by adding the dot product result of the labeled defect image and the first mask matrix, and the dot product result of the unlabeled defect image and the second mask matrix.

2. The method for industrial product defect detection based on probability mask according to claim 1, characterized in that: The upper half elements of the first mask matrix are 1, and the lower half elements are 0; the upper half elements of the second mask matrix are 0, and the lower half elements are 1.

3. The method for industrial product defect detection based on probability mask according to claim 1, characterized in that: The defect detection model extracts feature maps of labeled defect images, unlabeled defect images and mixed intermediate images through an information extraction module. The information extraction module includes a first information extraction structure, a second information extraction structure, a third information extraction structure and a fourth information extraction structure connected in sequence. The first information extraction structure, the second information extraction structure, the third information extraction structure and the fourth information extraction structure are all composed of a basic convolution mechanism and an enhanced extraction mechanism.

4. The method for industrial product defect detection based on probability mask according to claim 3, characterized in that: The defect detection model obtains probability mask filtering information corresponding to each feature map through a probability mask filtering module, and the probability mask filtering module obtains corresponding probability mask filtering information according to the feature maps output by the second information extraction structure, the third information extraction structure and the fourth information extraction structure in the information extraction module.

5. The method for industrial product defect detection based on probability mask according to claim 4, characterized in that: The probability mask screening module includes a probability mask construction mechanism, an information filtering mechanism and a first feature extraction mechanism. The probability mask construction mechanism generates a corresponding probability map according to the input feature map. The information filtering mechanism multiplies the input feature map with the probability map generated by the probability mask construction mechanism to obtain important information of the input feature map. The first feature extraction mechanism performs feature extraction on the important information to obtain probability mask screening information.

6. The method for industrial product defect detection based on probability mask according to claim 5, characterized in that: The probability mask construction mechanism includes a mean layer and a probability layer. The mean layer averages the input feature map in the direction of the number of channels and outputs a first feature map. The probability layer maps the first feature map to obtain a probability map. The mapping process is: ； Among them, F1 represents the first feature map, and F2 represents the probability map.

7. The method for industrial product defect detection based on probability mask according to claim 1, characterized in that: The defect detection model obtains corresponding interaction information according to the probability mask filtering information through a multi-source scale information interaction module. The multi-source scale information interaction module includes a second feature extraction mechanism, an upsampling mechanism and a fusion mechanism. The fusion mechanism fuses the probability mask filtering information and the output features of the probability mask filtering information after being processed by the second feature extraction mechanism and the upsampling mechanism to obtain the interaction information.

8. The method for industrial product defect detection based on probability mask according to claim 1, characterized in that: The loss of the defect detection model includes a first loss and a second loss, the first loss is obtained according to the first defect detection result and the defect label corresponding to the first input image, and the second loss is obtained according to the third defect detection result and the defect label corresponding to the first input image; The first defect detection result is the defect detection result of the first input image, which is a labeled defect image input by the data set A input module. The third defect detection result is the defect detection result of the third input image, which is a mixed intermediate image input by the mixed intermediate image input module.

9. An industrial product defect detection system based on probability mask, characterized in that: include: A data acquisition module is configured to acquire an image of a defect to be detected; The defect detection module is configured to input the defect image to be detected into the trained defect detection model for detection to obtain the defect detection result; The input of the defect detection model includes a labeled defect image, an unlabeled defect image and a mixed intermediate image, and the mixed intermediate image is obtained by mixing the labeled defect image and the unlabeled defect image; the defect detection model extracts feature maps of the labeled defect image, the unlabeled defect image and the mixed intermediate image respectively, obtains probability mask filtering information corresponding to each feature map, obtains corresponding interaction information according to the probability mask filtering information, and obtains corresponding defect detection results according to the interaction information; The mixed intermediate image is obtained by adding the dot product result of the labeled defect image and the first mask matrix, and the dot product result of the unlabeled defect image and the second mask matrix.

Images