Defect detection model training method, defect detection method and device
By simulating synthetic defects on OK images and using interactive attention processing and image reconstruction techniques, the problem of difficulty in collecting defect samples in defect detection model training in the prior art is solved, and the detection capability and robustness are improved.
Patent Information
- Application Number
- CN202411986630.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
In the prior art, it is difficult to obtain sufficient defect sample images for training of defect detection models, resulting in poor detection effect and low robustness.
By simulating the synthetic defect on the OK image without defects, generating defect synthetic images, and using interactive attention processing and image reconstruction techniques, the detection capability and robustness of the defect detection model are improved.
It effectively solves the problems of difficulty and high cost of image acquisition of defect samples, and significantly improves the detection capability and robustness of defect detection models in complex industrial anomaly detection scenarios.
Smart Images
Figure CN119919754A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology and industrial detection, and in particular relates to a defect detection model training method, a defect detection method and a device. Background Art
[0002] The training of defect detection models requires the collection of images of industrial production processes. However, there are only a few defective products, while the majority of products do not have defects. This will result in many images collected during the training phase being OK images of products without defects. The training of defect detection models requires NG images of products with defects, but NG images are difficult to collect and there is a long-tail defect sample phenomenon. This will result in poor detection effect and low robustness of the defect detection model obtained through training. Summary of the invention
[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a defect detection model training method, defect detection method and device, which solves the problem of difficulty and high acquisition cost of sample images with defective products in the prior art through defect simulation synthesis technology, and improves the detection capability and robustness of the defect detection model in complex industrial anomaly detection scenarios.
[0004] In a first aspect, the present application provides a method for training a defect detection model, the method comprising:
[0005] Each sample image is input into the initial defect detection model. The sample image is an OK image without defects. The following operations are performed by the defect detection model:
[0006] Simulate synthetic defects on each sample image to obtain a defect synthetic image of each sample image; the defect synthetic image has a defect supervision label;
[0007] Perform interactive attention processing on the feature sequence of each defect synthetic image and the corresponding auxiliary query feature sequence to obtain each new feature sequence;
[0008] Perform image reconstruction processing on each new feature sequence to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label;
[0009] Based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until the training stop condition is reached to obtain a trained defect detection model.
[0010] According to the training method of the defect detection model of the present application, a defect composite image of each sample image is obtained by simulating synthetic defects on each sample image; the defect composite image has a defect supervision label; interactive attention processing is performed on the feature sequence of each defect composite image and the corresponding auxiliary query feature sequence to obtain each new feature sequence; image reconstruction processing is performed on each new feature sequence to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label; based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until the training stop condition is reached to obtain a trained defect detection model, which solves the problem of difficulty and high cost in collecting sample images with defective products in the prior art. In addition, the auxiliary query feature sequence is used to guide and strengthen the attention mechanism for real defects, thereby improving the detection capability and robustness of the defect detection model in complex industrial anomaly detection scenarios.
[0011] According to one embodiment of the present application, a target area is selected from a sample image;
[0012] generating a binary image for indicating that a target area in the sample image is a defect area;
[0013] The pixel value of the binary image is multiplied by the pixel value of the defect texture image in the defect texture library to fill the defects in the target area and obtain a defect composite image.
[0014] According to one embodiment of the present application, before performing interactive attention processing on the feature sequence of each defect composite image and the corresponding auxiliary query feature sequence, the method further includes:
[0015] The auxiliary query feature sequence is obtained based on the following method:
[0016] Extracting and combining the feature maps of each target OK image in the target OK image group to obtain a feature memory library;
[0017] Match the sample image and the feature memory library to obtain a difference feature sequence;
[0018] Perform at least one of illumination processing and deformation disturbance processing on a group of NG images with defects collected from business scenarios to obtain a real defect sequence;
[0019] A feature sequence is selected from the difference feature sequence and the true defect sequence as an auxiliary query feature sequence.
[0020] According to an embodiment of the present application, based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until a training stop condition is reached, including:
[0021] Based on the defect supervision label and the corresponding defect prediction label of each sample image, the loss values of the initial defect detection model are obtained;
[0022] The initial defect detection model is trained by back-propagation error propagation based on each loss value to update the target model parameters of the initial defect detection model until the training stop condition is reached.
[0023] According to one embodiment of the present application, the defect detection model includes an encoder; the encoder is a tansforer structure that divides all SAM models; the encoder is used for a feature sequence of a defect synthesis image;
[0024] Before performing reverse error propagation training on the initial defect detection model based on each loss value to update the target model parameters of the initial defect detection model, the method further includes:
[0025] The encoder weights are frozen so that the encoder weights are not updated.
[0026] According to one embodiment of the present application, the defect detection model includes a decoder and multiple deconvolutions; the decoder and the multiple deconvolutions are used to perform interactive attention processing on the feature sequence of each defect synthetic image and the corresponding auxiliary query feature sequence to obtain each new feature sequence, and perform image reconstruction processing on each new feature sequence;
[0027] The target model parameters include the decoder weights and multiple deconvolution weights.
[0028] In a second aspect, the present application provides a defect detection method, the method comprising:
[0029] Acquire the image to be detected;
[0030] The defect detection model trained by the defect detection model training method provided by the first aspect performs defect detection processing on the image to be detected, and obtains a detection result output by the defect detection model; the detection result is the first detection result or the second detection result;
[0031] The first detection result indicates that the image to be detected is an OK image without defects;
[0032] The second detection result indicates that the image to be detected is at least one of a defective NG image and a defect label.
[0033] According to the defect detection method of the present application, by introducing a defect detection model trained by synthetic defect technology, the problems of difficulty and high cost in collecting NG samples with defects are solved, the ability of anomaly detection is significantly improved, and the industrial quality inspection level of deep learning models is enhanced.
[0034] In a third aspect, the present application provides a training device for a defect detection model, the device comprising:
[0035] The input module is used to input each sample image into the initial defect detection model, where the sample image is an OK image without defects; the following operations are performed through the defect detection model:
[0036] A synthesis module is used to simulate and synthesize defects on each sample image to obtain a defect synthesis image of each sample image; the defect synthesis image has a defect supervision label;
[0037] The first processing module is used to perform interactive attention processing on the feature sequence of each defect synthesis image and the corresponding auxiliary query feature sequence to obtain each new feature sequence;
[0038] The second processing module is used to perform image reconstruction processing on each new feature sequence to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label;
[0039] The training module is used to train the defect detection model based on the defect supervision label and the corresponding defect prediction label of each sample image until the training stop condition is reached to obtain the trained defect detection model.
[0040] According to the training device of the defect detection model of the present application, by simulating synthetic defects on each sample image, a defect synthetic image of each sample image is obtained; the defect synthetic image has a defect supervision label; interactive attention processing is performed on the feature sequence of each defect synthetic image and the corresponding auxiliary query feature sequence to obtain each new feature sequence; image reconstruction processing is performed on each new feature sequence to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label; based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until the training stop condition is reached to obtain a trained defect detection model, which solves the problem of difficulty and high cost in collecting sample images with defective products in the prior art. In addition, the auxiliary query feature sequence is used to guide and strengthen the attention mechanism for real defects, thereby improving the detection capability and robustness of the defect detection model in complex industrial anomaly detection scenarios.
[0041] In a fourth aspect, the present application provides a defect detection device, the device comprising:
[0042] An acquisition module, used for acquiring an image to be detected;
[0043] A detection module, which performs defect detection processing on the image to be detected using the defect detection model trained by the defect detection model training device provided by the third aspect, and obtains a detection result output by the defect detection model; the detection result is a first detection result or a second detection result;
[0044] The first detection result indicates that the image to be detected is an OK image without defects;
[0045] The second detection result indicates that the image to be detected is at least one of a defective NG image and a defect label.
[0046] According to the defect detection device of the present application, by introducing a defect detection model trained by synthetic defect technology, the problems of difficulty and high cost in collecting NG samples with defects are solved, the ability of anomaly detection is significantly improved, and the industrial quality inspection level of deep learning models is enhanced.
[0047] In a fifth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the training method for the defect detection model provided in the first aspect and / or the defect detection method provided in the second aspect.
[0048] In a sixth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the defect detection model training method provided in the first aspect and / or the defect detection method provided in the second aspect.
[0049] In the seventh aspect, the present application provides a chip, which includes a processor and a communication interface, which is coupled to the processor, and the processor is used to run programs or instructions to implement the training method of the defect detection model provided in the first aspect and / or the defect detection method provided in the second aspect.
[0050] In an eighth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the defect detection model training method provided in the first aspect and / or the defect detection method provided in the second aspect.
[0051] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
[0052] By simulating synthetic defects on each of the sample images, a defect composite image of each of the sample images is obtained; the defect composite image has a defect supervision label; interactive attention processing is performed on the feature sequence of each of the defect composite images and the corresponding auxiliary query feature sequence to obtain each new feature sequence; image reconstruction processing is performed on each of the new feature sequences to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label; based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until the training stop condition is reached to obtain a trained defect detection model, which solves the problem of difficulty and high cost in collecting sample images with defective products in the prior art. In addition, the auxiliary query feature sequence is used to guide and strengthen the attention mechanism for real defects, thereby improving the detection capability and robustness of the defect detection model in complex industrial anomaly detection scenarios.
[0053] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0055] Figure 1 This is one of the flow charts of the method for training a defect detection model provided in an embodiment of the present application;
[0056] Figure 2 is a schematic diagram of a training process of a SAM-based defect detection model provided in an embodiment of the present application;
[0057] Figure 3 This is one of the flow charts of the defect detection method provided in the embodiment of the present application;
[0058] Figure 4 It is a structural schematic diagram of a training device for a defect detection model provided in an embodiment of the present application;
[0059] Figure 5 is a schematic diagram of the structure of a defect detection device provided in an embodiment of the present application;
[0060] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0061] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0062] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0063] In the following, in combination with the accompanying drawings, the training method of the defect detection model, the defect detection method, the training device of the defect detection model, the defect detection device, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.
[0064] Among them, the training method of the defect detection model and / or the defect detection method can be applied to the terminal, and can be specifically executed by hardware or software in the terminal.
[0065] The training method of the defect detection model and / or the defect detection method provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the training method of the defect detection model and / or the defect detection method. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The training method of the defect detection model and the defect detection method provided in the embodiments of the present application are described below using the electronic device as an example of the execution subject.
[0066] like Figure 1 As shown, the training method of the defect detection model includes: step 110, step 111, step 112 and step 113.
[0067] Step 110: Input each sample image into the initial defect detection model, where the sample image is an OK image without defects; perform the following steps 111 to 114 through the defect detection model:
[0068] It is understandable that in the industrial production process, defective products are small in number, and the defective images (also called NG images) corresponding to the defective products are also difficult to collect, and the number of defective images accounts for a relatively small proportion of the total number of collected images; while the number of qualified products without defects is large, and the defect-free images (also called qualified images or "OK" images) corresponding to qualified products are relatively easy to collect. Normally, the number of defect-free images accounts for a relatively large proportion of the total number of collected images. In order to reduce the difficulty of data collection, the embodiment of the present application trains the defect detection model through OK images.
[0069] The defect detection model of the embodiment of the present application can refer to SAM (Segment Anything Model). An important feature of SAM is that it can perform interactive segmentation, that is, the user only needs to simply mark a certain area or object in the image, and the model can accurately segment the object or area. This means that the user does not need to finely mark all the data in actual use, but only needs to provide simple prompts (such as mouse clicks, box selections, etc.).
[0070] During the training phase, each sample image is input into the initial defect detection model. Each sample image is an OK image without defects. Each training sample is processed by each layer of the network structure of the defect detection model.
[0071] Step 111 , simulate and synthesize defects on each sample image to obtain a defect synthesized image of each sample image; the defect synthesized image has a defect supervision label.
[0072] The defect detection model includes a defect simulation and synthesis network, which is used to simulate and synthesize defects on each sample image to obtain a defect synthesis image of each sample image. Any type of image can be synthesized on the sample image, such as scratches, cracks, stains, etc., without limitation.
[0073] When a synthetic defect is simulated on a sample image, the sample image becomes a defect synthetic image, and the image type becomes an NG type for an image with defects.
[0074] In addition, the location and type of the defect can be determined, and one or more of the image type, defect location, defect label image and defect type can be used as a defect supervision label (also called a true defect label) of the defect composite image.
[0075] In some scenarios, the defect composite image can be augmented, such as deforming the defect composite image (scaling, cropping, rotating), adjusting the background, adjusting the lighting, etc., thereby increasing the diversity of defect simulation and the generalization of defect segmentation.
[0076] Step 112: Perform interactive attention processing on the feature sequence of each defect composite image and the corresponding auxiliary query feature sequence to obtain each new feature sequence.
[0077] After obtaining the defect composite images, the embodiment of the present application can perform feature sequence extraction processing on each defect composite image through the image encoder Encoder in the defect detection model to obtain the feature sequence of each defect composite image.
[0078] The image encoder can use the tansforer structure of the lightweight SAM metamodel, focusing on training and deployment efficiency. In some scenarios with higher performance requirements, it can be replaced by the VIT (Vision Transformer) backbone of the SAM large model.
[0079] Auxiliary Query Feature Sequence refers to the process of extracting and processing the features of one or more auxiliary query images to form a feature sequence, and using this sequence as auxiliary information to improve the accuracy and robustness of defect detection.
[0080] Auxiliary query images refer to additional images that serve as auxiliary information to help improve the accuracy and efficiency of the main tasks (such as defect detection, target recognition, image retrieval, etc.) in image processing and computer vision tasks. Auxiliary query images refer to images taken from business scenarios, which can provide the system with richer background information.
[0081] The embodiment of the present application can perform interactive attention processing on the feature sequence of the defect synthesis image and the corresponding auxiliary query feature sequence to obtain each new feature sequence.
[0082] Interactive Attention processing is a mechanism used in neural network models to enhance information interaction between different inputs. The core idea is to calculate the attention weights between inputs so that the model can effectively extract relevant features from different inputs and adjust and weight the information flow according to the relationship between these features, thereby improving the performance of the task.
[0083] The process of interactive attention processing of the feature sequence of the defect synthetic image and the corresponding auxiliary query feature sequence in the embodiment of the present application can guide and strengthen the encoder's learning of the query features, which can be decomposed into: calculating the similarity between the feature sequence of the defect synthetic image and the auxiliary query feature sequence, when the similarity between the two is high, giving a larger attention weight, and when the similarity between the two is low, giving a smaller attention weight, using the calculated attention weights, weighted summing the features, and fusing the important information of the feature sequence of the defect synthetic image and the auxiliary query feature sequence, thereby obtaining a feature representation guided by the auxiliary query feature sequence, that is, obtaining a new feature sequence.
[0084] Step 113: perform image reconstruction processing on each new feature sequence to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label.
[0085] The defect detection model in the embodiment of the present application also includes an image decoder Decoder. After obtaining a new feature sequence, the image decoder Decoder performs image reconstruction processing on each new feature sequence to obtain a reconstructed image of each sample image, and the reconstructed image has a defect prediction label.
[0086] Step 114: Based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until a training stop condition is reached to obtain a trained defect detection model.
[0087] After obtaining the defect supervision label and the corresponding defect prediction label of the sample image, the embodiment of the present application can determine the loss values of the target loss function of the defect detection model based on the defect supervision label and the corresponding defect prediction label of the sample image. The loss value can adopt L1 (sum of squares of errors) and focal loss (weighted cross entropy loss) consistent with the SAM model.
[0088] The initial defect detection model can be trained by reverse error propagation based on each loss value to update the target model parameters of the initial defect detection model until the training stop condition is reached. The training stop condition can be that the target loss function converges or the number of training times reaches the target number. After the training is completed, the trained defect detection model is obtained.
[0089] The embodiments of the present application obtain defect composite images of each sample image by simulating synthetic defects on each sample image; the defect composite images have defect supervision labels; interactive attention processing is performed on the feature sequence of each defect composite image and the corresponding auxiliary query feature sequence to obtain each new feature sequence; image reconstruction processing is performed on each new feature sequence to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label; based on the defect supervision label and the corresponding defect prediction label of each sample image, a defect detection model is trained until the training stop condition is reached to obtain a trained defect detection model, which solves the problem of difficulty and high cost in collecting sample images of defective products in the prior art. In addition, the auxiliary query feature sequence is used to guide and strengthen the attention mechanism for real defects, thereby improving the detection capability and robustness of the defect detection model in complex industrial anomaly detection scenarios.
[0090] In some embodiments, simulating synthetic defects on each sample image to obtain a defect synthetic image of each sample image includes:
[0091] Select a target region from a sample image;
[0092] generating a binary image for indicating that a target area in the sample image is a defect area;
[0093] The pixel value of the binary image is multiplied by the pixel value of the defect texture image in the defect texture library to fill the defects in the target area and obtain a defect composite image.
[0094] The detailed process of simulating synthetic defects on a sample image in an embodiment of the present application may be: selecting a target area from the sample image, where the target area refers to an area of interest to the user, which may be randomly selected based on an algorithm, or may be determined based on a user input operation (click, select, etc.).
[0095] Then, various noise images of the sample image are generated through the noise algorithm, and noise that deviates slightly from the OK sample feature space can be added. In theory, a compact positive sample feature space boundary can be established to determine abnormal defects that are biased towards the boundary.
[0096] Then, the noise image is binarized to obtain a binarized image. The pixel value of the target area in the binarized image is the target value, for example, the pixel value of the target area is 1, and the pixel values of other areas are 0.
[0097] Next, the pixel value of the binarized image can be multiplied by the pixel value of the defect texture image in the defect texture library. The size of the defect texture image can be the same as the size of the binarized image, that is, the pixel value 0 or 1 of the binarized image is multiplied by the pixel value of the defect texture image to fill the defect in the target area.
[0098] In some embodiments, before performing interactive attention processing on the feature sequence of each defect synthetic image and the corresponding auxiliary query feature sequence, the method further includes:
[0099] The auxiliary query feature sequence is obtained based on the following method:
[0100] Extracting and combining the feature maps of each target OK image in the target OK image group to obtain a feature memory library;
[0101] Match the sample image and the feature memory library to obtain a difference feature sequence;
[0102] Perform at least one of illumination processing and deformation disturbance processing on a group of NG images with defects collected from business scenarios to obtain a real defect sequence;
[0103] A feature sequence is selected from the difference feature sequence and the true defect sequence as an auxiliary query feature sequence.
[0104] The above-mentioned embodiments have illustrated that more abundant background information can be provided by using auxiliary query sequences, thereby improving the detection capability and robustness of the defect detection model. The auxiliary query sequence can be obtained based on the target OK image group or the NG image group. Here, the target OK image group and the NG image group can both be regarded as auxiliary query images. Both the target OK image group and the NG image group can contain multiple images, such as 30 images.
[0105] For the target OK image group, the feature map of each target OK image in the target OK image group can be extracted and combined to obtain a feature memory bank. Furthermore, for each sample image, the sample image and the feature memory bank can be matched to obtain a difference image (anomaly map) feature sequence, which can be used as an auxiliary query feature sequence.
[0106] For a group of NG images with defects collected from business scenarios, at least one of light processing and deformation perturbation processing may be performed on the NG image group. Light processing aims to modify or enhance the lighting conditions in the image. Deformation perturbation processing refers to changing the appearance of the image by making nonlinear or random adjustments to the pixel position, geometric shape or structure in the image to help the model learn more robust features. At least one of light processing and deformation perturbation processing is performed on the NG image group to obtain a true defect sequence, which can also be used as an auxiliary query feature sequence.
[0107] In practical applications, a feature sequence can be randomly selected from the difference feature sequence and the real defect sequence as an auxiliary query feature sequence.
[0108] In some embodiments, based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until a training stop condition is reached, including:
[0109] Based on the defect supervision label and the corresponding defect prediction label of each sample image, the loss values of the initial defect detection model are obtained;
[0110] The initial defect detection model is trained by back-propagation error propagation based on each loss value to update the target model parameters of the initial defect detection model until the training stop condition is reached.
[0111] This has been explained in the above-mentioned embodiment and will not be described again here.
[0112] In some embodiments, the defect detection model includes an encoder; the encoder is a tansforer structure that divides all SAM models; the encoder is used for a feature sequence of a defect synthesis image;
[0113] Before performing reverse error propagation training on the initial defect detection model based on each loss value to update the target model parameters of the initial defect detection model, the method further includes:
[0114] The encoder weights are frozen so that the encoder weights are not updated.
[0115] The defect detection model in the embodiment of the present application includes an encoder. The encoder adopts the tansforer structure of the lightweight model. Before performing reverse error propagation training on the initial defect detection model based on each loss value, the weight of the encoder is frozen so that the weight of the encoder is not updated, thereby maintaining the powerful feature extraction capability of the Sam model.
[0116] Of course, in some embodiments, the encoder of the defect detection model is replaced with a larger VIT to improve the feature extraction capability of the model, which can balance the efficiency requirements of different projects in the industrial quality inspection site and can more flexibly adapt to the multi-scenario requirements of the industrial quality inspection site.
[0117] In some embodiments, the defect detection model includes a decoder and a plurality of deconvolutions; the decoder and the plurality of deconvolutions are used to perform interactive attention processing on a feature sequence of each defect synthetic image and a corresponding auxiliary query feature sequence to obtain each new feature sequence, and perform image reconstruction processing on each new feature sequence;
[0118] The target model parameters include the decoder weights and multiple deconvolution weights.
[0119] The Decoder encoder and two Conv trans (deconvolution) also use the tansforer structure of Sam's original model, but the initial weights are the weights of Sam's original model, which will be updated during the training process, and fine-tuning and enhancement will improve the ability to reconstruct new abnormal training images.
[0120] See also Figure 2 , the embodiment of the present application provides a schematic diagram of the training process of a defect detection model based on SAM, which includes multiple stages, namely:
[0121] The simulated abnomal samples stage, in which defects are simulated on each sample image (OK image) to obtain a defect composite image of each sample image (i.e., a simulated NG image). The defect composite image has a true defect (GT) label, which can be a true defect annotation.
[0122] Freeze stage (Freeze), in which the weights of the SAM Encoder are frozen.
[0123] The feature map generation stage is to extract and combine the feature maps of each target OK image in the target OK image group to obtain a feature memory bank.
[0124] In the matching stage, the sample image and the feature memory library are matched to obtain a difference image (anomaly map), and then the difference feature sequence is subjected to feature sequence extraction processing to obtain a difference feature sequence Embedding anomaly.
[0125] A true defect (GT) sequence acquisition stage, in which at least one of light processing and deformation perturbation processing is performed on a group of defective NG images collected from a business scene to obtain a true defect sequence;
[0126] In the interactive attention processing stage, a feature sequence is selected from the difference feature sequence and the true defect sequence through the Decoder as the auxiliary query feature sequence, and the feature sequence of each defect synthetic image and the corresponding auxiliary query feature sequence are interactively processed to obtain each new feature sequence.
[0127] In the reconstruction stage, each new feature sequence is reconstructed through a decoder and two deconvolutions (Conv trans) to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label (Predict), which can be a predicted defect annotation.
[0128] In the training phase, based on the defect supervision labels and corresponding defect prediction labels of each sample image, the loss values of the initial defect detection model are obtained, and each loss value includes the loss of the sum of squared errors (L1 loss) and the weighted cross entropy loss (focal loss); based on each loss value, the initial defect detection model is trained by reverse error propagation to update the target model parameters of the initial defect detection model until the training stop condition is reached.
[0129] See also Figure 3 , the present application embodiment provides a flow chart of a defect detection method, comprising the following steps:
[0130] Step 301: Acquire an image to be detected;
[0131] Step 302: perform defect detection processing on the image to be detected using the defect detection model trained by the aforementioned defect detection model training method to obtain a detection result output by the defect detection model; the detection result is a first detection result or a second detection result;
[0132] The first detection result indicates that the image to be detected is an OK image without defects;
[0133] The second detection result indicates that the image to be detected is at least one of a defective NG image and a defect label.
[0134] After the defect detection model is trained, the embodiment of the present application can deploy the trained defect detection model online. In the industrial production line, the trained defect detection model can be used to perform defect detection on the image to be detected, and the image to be detected includes an image of industrial products.
[0135] The trained defect detection model will output the detection result of each image to be detected, and the detection result is a first detection result or a second detection result;
[0136] The first detection result indicates that the image to be detected is an OK image without defects;
[0137] The second detection result represents that the image to be detected is at least one of a defective NG image and a defect label. Of course, in some embodiments, the second detection result may also include a binary image of the image to be detected and a defect type, etc., which is not limited.
[0138] The embodiments of the present application solve the problems of difficulty and high cost in collecting defective NG samples by introducing a defect detection model trained using synthetic defect technology, significantly improve the ability of anomaly detection, and enhance the industrial quality inspection level of deep learning models.
[0139] The defect detection model training method provided in the embodiment of the present application can be executed by a defect detection model training device. In the embodiment of the present application, the defect detection model training method performed by the defect detection model training device is taken as an example to illustrate the defect detection model training device provided in the embodiment of the present application.
[0140] An embodiment of the present application also provides a training device for a defect detection model.
[0141] like Figure 4 As shown, the training device of the defect detection model includes: an input module 410, a synthesis module 420, a first processing module 430, a second processing module 440 and a training module 450.
[0142] The input module 410 is used to input each sample image into the initial defect detection model, where the sample image is an OK image without defects; the following operations are performed through the defect detection model:
[0143] The synthesis module 420 is used to simulate and synthesize defects on each sample image to obtain a defect synthesis image of each sample image; the defect synthesis image has a defect supervision label;
[0144] The first processing module 430 is used to perform interactive attention processing on the feature sequence of each defect composite image and the corresponding auxiliary query feature sequence to obtain each new feature sequence;
[0145] The second processing module 440 is used to perform image reconstruction processing on each new feature sequence to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label;
[0146] The training module 450 is used to train the defect detection model based on the defect supervision label and the corresponding defect prediction label of each sample image until the training stop condition is reached to obtain a trained defect detection model.
[0147] According to the training device of the defect detection model provided in the embodiment of the present application, by simulating synthetic defects on each sample image, a defect synthetic image of each sample image is obtained; the defect synthetic image has a defect supervision label; interactive attention processing is performed on the feature sequence of each defect synthetic image and the corresponding auxiliary query feature sequence to obtain each new feature sequence; image reconstruction processing is performed on each new feature sequence to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label; based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until the training stop condition is reached to obtain a trained defect detection model, which solves the problem of difficulty and high cost in collecting sample images with defective products in the prior art. In addition, the auxiliary query feature sequence is used to guide and strengthen the attention mechanism for real defects, thereby improving the detection capability and robustness of the defect detection model in complex industrial anomaly detection scenarios.
[0148] In some embodiments, the synthesis module 420 is used to:
[0149] Select a target region from a sample image;
[0150] generating a binary image for indicating that a target area in the sample image is a defect area;
[0151] The pixel value of the binary image is multiplied by the pixel value of the defect texture image in the defect texture library to fill the defects in the target area and obtain a defect composite image.
[0152] In some embodiments, the training device for the defect detection model further includes a third processing module for:
[0153] The auxiliary query feature sequence is obtained based on the following method:
[0154] Extracting and combining the feature maps of each target OK image in the target OK image group to obtain a feature memory library;
[0155] Match the sample image and the feature memory library to obtain a difference feature sequence;
[0156] Perform at least one of illumination processing and deformation disturbance processing on a group of NG images with defects collected from business scenarios to obtain a real defect sequence;
[0157] A feature sequence is selected from the difference feature sequence and the true defect sequence as an auxiliary query feature sequence.
[0158] In some embodiments, the training module is used to:
[0159] Based on the defect supervision label and the corresponding defect prediction label of each sample image, the loss values of the initial defect detection model are obtained;
[0160] The initial defect detection model is trained by back-propagation error propagation based on each loss value to update the target model parameters of the initial defect detection model until the training stop condition is reached.
[0161] In some embodiments, the defect detection model includes an encoder; the encoder is a tansforer structure that divides all SAM models; the encoder is used for a feature sequence of a defect synthesis image;
[0162] The training module includes a freezing submodule;
[0163] The freezing submodule is used to freeze the weights of the encoder so as not to update the weights of the encoder.
[0164] In some embodiments, the defect detection model includes a decoder and a plurality of deconvolutions; the decoder and the plurality of deconvolutions are used to perform interactive attention processing on a feature sequence of each defect synthetic image and a corresponding auxiliary query feature sequence to obtain each new feature sequence, and perform image reconstruction processing on each new feature sequence;
[0165] The target model parameters include the decoder weights and multiple deconvolution weights.
[0166] The defect detection method provided in the embodiment of the present application can be executed by a defect detection device. In the embodiment of the present application, the defect detection device provided in the embodiment of the present application is described by taking the defect detection method executed by the defect detection device as an example.
[0167] The embodiment of the present application also provides a defect detection device.
[0168] like Figure 5 As shown, the defect detection device includes: an acquisition module 510 and a detection module 520.
[0169] An acquisition module 510 is used to acquire an image to be detected;
[0170] The detection module 520 is used to perform defect detection processing on the image to be detected using the defect detection model trained by the aforementioned defect detection model training device to obtain a detection result output by the defect detection model; the detection result is a first detection result or a second detection result;
[0171] The first detection result indicates that the image to be detected is an OK image without defects;
[0172] The second detection result indicates that the image to be detected is at least one of a defective NG image and a defect label.
[0173] According to the defect detection device provided in the embodiment of the present application, the synthetic defect technology is introduced through the trained defect detection model, which significantly improves the ability of anomaly detection and enhances the industrial quality inspection level of the deep learning model.
[0174] The training device of the defect detection model and the defect detection device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0175] The defect detection model training device and defect detection device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0176] The training device of the defect detection model provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0177] The defect detection device provided in the embodiment of the present application can achieve Figure 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0178] In some embodiments, Figure 6As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, the various processes of the embodiments of at least one of the training method of the defect detection model and the defect detection method mentioned above are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0179] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0180] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the embodiments of at least one of the above-mentioned defect detection model training method and defect detection method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0181] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0182] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements at least one of the above-mentioned defect detection model training method and defect detection method.
[0183] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0184] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the embodiments of at least one of the above-mentioned defect detection model training method and defect detection method, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0185] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0186] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0188] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0189] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0190] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.
Claims
1. A method for training a defect detection model, characterized in that: include: Each sample image is input into the initial defect detection model, wherein the sample image is an OK image without defects; the following operations are performed by the defect detection model: Simulating synthetic defects on each of the sample images to obtain a defect synthetic image of each of the sample images; the defect synthetic image has a defect supervision label; Performing interactive attention processing on the feature sequence of each defect synthetic image and the corresponding auxiliary query feature sequence to obtain each new feature sequence; Performing image reconstruction processing on each of the new feature sequences to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label; Based on the defect supervision label and the corresponding defect prediction label of each sample image, the defect detection model is trained until a training stop condition is reached to obtain a trained defect detection model.
2. The defect detection model training method according to claim 1, characterized in that: The simulating and synthesizing defects on each of the sample images to obtain a defect synthesized image of each of the sample images includes: Selecting a target area from the sample image; generating a binary image for indicating that the target area in the sample image is a defect area; The pixel value of the binary image is multiplied by the pixel value of the defect texture image in the defect texture library to fill the defects in the target area to obtain the defect composite image.
3. The defect detection model training method according to claim 1, characterized in that: Before performing interactive attention processing on the feature sequence of each defect composite image and the corresponding auxiliary query feature sequence, the method further includes: The auxiliary query feature sequence is obtained based on the following method: Extracting and combining the feature maps of each target OK image in the target OK image group to obtain a feature memory library; Matching the sample image with the feature memory library to obtain a difference feature sequence; Perform at least one of illumination processing and deformation disturbance processing on a group of NG images with defects collected from business scenarios to obtain a real defect sequence; A feature sequence is selected from the difference feature sequence and the real defect sequence as the auxiliary query feature sequence.
4. The defect detection model training method according to claim 1, characterized in that: The defect detection model is trained based on the defect supervision label and the corresponding defect prediction label of each sample image until a training stop condition is reached, including: Based on the defect supervision label and the corresponding defect prediction label of each sample image, obtaining each loss value of the initial defect detection model; The initial defect detection model is trained by back-propagation error propagation based on each loss value to update the target model parameters of the initial defect detection model until a training stop condition is reached.
5. The method for training a defect detection model according to claim 4, characterized in that: The defect detection model includes an encoder; the encoder is a tansforer structure that divides all SAM models; the encoder is used for the feature sequence of the defect synthesis image; Before performing reverse error propagation training on the initial defect detection model based on each loss value to update the target model parameters of the initial defect detection model, the method further includes: The weight of the encoder is frozen so as not to update the weight of the encoder.
6. The defect detection model training method according to claim 4, characterized in that: The defect detection model includes a decoder and a plurality of deconvolutions; the decoder and the plurality of deconvolutions are used to perform interactive attention processing on the feature sequence of each defect synthetic image and the corresponding auxiliary query feature sequence to obtain each new feature sequence, and perform image reconstruction processing on each new feature sequence; The target model parameters include the weight of the decoder and the weights of the multiple deconvolutions.
7. A defect detection method, characterized in that: include: Acquire the image to be detected; The defect detection model trained by the defect detection model training method according to any one of claims 1 to 6 performs defect detection processing on the image to be detected to obtain a detection result output by the defect detection model; the detection result is a first detection result or a second detection result; The first detection result indicates that the image to be detected is an OK image without defects; The second detection result indicates that the image to be detected is at least one of a defective NG image and a defect label.
8. A training device for a defect detection model, characterized in that: include: An input module, used for inputting each sample image into an initial defect detection model, wherein the sample image is an OK image without defects; The defect detection model performs the following operations: A synthesis module, used for simulating and synthesizing defects on each of the sample images to obtain a defect synthesis image of each of the sample images; the defect synthesis image has a defect supervision label; A first processing module is used to perform interactive attention processing on the feature sequence of each defect composite image and the corresponding auxiliary query feature sequence to obtain each new feature sequence; A second processing module is used to perform image reconstruction processing on each of the new feature sequences to obtain a reconstructed image of each sample image; the reconstructed image has a defect prediction label; A training module is used to train the defect detection model based on the defect supervision label and the corresponding defect prediction label of each sample image until a training stop condition is reached to obtain a trained defect detection model.
9. A defect detection device, characterized in that: include: An acquisition module, used for acquiring an image to be detected; A detection module, which performs defect detection processing on the image to be detected using the defect detection model trained by the defect detection model training device according to claim 8 to obtain a detection result output by the defect detection model; the detection result is a first detection result or a second detection result; The first detection result indicates that the image to be detected is an OK image without defects; The second detection result indicates that the image to be detected is at least one of a defective NG image and a defect label.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the training method of the defect detection model as described in any one of claims 1 to 6, or the defect detection method as described in claim 7 is implemented.