Product defect detection method based on adaptive complementary channel and full-span enhancement

Abstract

The invention discloses a product defect detection method based on a self-adaptive complementary channel and full-span enhancement, and relates to the technical field of industrial defect detection. According to the novel method for synthesizing the anomalies, conventional structural anomalies can be synthesized, special logic anomalies, known anomalies and unknown anomalies can also be synthesized, and the synthesized anomalies are very close to real anomalies; the invention provides an asymmetric teacher-student network model, a self-adaptive channel selection module is designed in a student network, the self-adaptive channel selection module comprises secondary de-noising and self-adaptive channel weight generation, and after preliminary de-noising in a frequency domain and further de-noising by using a proposed LGC-LSTM, a comprehensive de-noising channel weight is obtained. Different levels of channel mapping are respectively carried out to obtain complementary channel information, so that normal features of the student network can be kept, and abnormal features can be repaired; a new maximum diffusion loss is proposed to facilitate the model to further utilize synthetic anomalies.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial defect detection, and in particular to a product defect detection method based on adaptive complementary channels and full-span enhancement. Background Art

[0002] Industrial anomaly detection technology plays a vital role in industrial production quality inspection. With the development of deep learning models, an increasing number of anomaly detection models are being applied to production lines. However, due to the difficulty in obtaining large amounts of labeled defect data, supervised learning methods cannot fully leverage their high accuracy in practical applications. Consequently, unsupervised learning methods have seen rapid development in recent years. Unsupervised anomaly detection is a very effective solution for situations where sample data is scarce. However, when training a network using only normal samples, the network will inevitably focus on the normal sample manifold. This can lead to overgeneralization errors for abnormal samples that are similar to normal samples. Furthermore, abnormal samples are often diverse and unknown, making unsupervised methods difficult to detect. One solution to alleviate this problem is to perform data augmentation on normal samples, artificially creating various defects in them. This can improve model performance to a certain extent. Currently, there are many data enhancement methods in the field of image anomaly detection, such as NSA, CutPaste, and Perlin noise binarization. However, the anomalies generated by these methods are often far from the real anomalies. Although they can generate unknown anomalies, they have little effect on the actual application of the model.

[0003] Knowledge distillation methods based on teacher-student networks have seen rapid growth in recent years, driven by the emergence of pre-trained models, many of which are trained on the ImageNet dataset and are highly effective at extracting image information. Teachers typically use pre-trained networks to guide students in learning knowledge. Traditional unsupervised methods train only on normal samples, with the goal of ensuring that students reach the teacher's level of performance. When anomalies are input into the model, the teacher's output features contain anomalies, while the student, trained only on normal images and never having seen anomalies, cannot regress the teacher's output within the anomaly region during inference. Because the corresponding descriptors were not observed during training, the student network scores the anomaly region highly, allowing it to locate the anomaly. However, the high generalization of CNNs can lead to overgeneralization. The student may encode anomalies in the same way as the teacher, resulting in uncertainty in the model's anomaly detection. Furthermore, most current teacher-student network structures are symmetrical. While symmetric structures offer advantages in feature alignment, they can also blur the boundaries between the model's defect features, leading to overgeneralization. Therefore, industrial defect detection research is still in its early stages of methodological exploration, with neither a complete theoretical framework nor a comprehensive technical framework established. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention provides a product defect detection method based on adaptive complementary channels and full span enhancement to realize industrial defect detection.

[0005] The technical solution of the present invention is:

[0006] The product defect detection method based on adaptive complementary channels and full span enhancement includes the following steps:

[0007] Obtain normal image data from industrial image datasets;

[0008] Acquire a real abnormal image dataset and its corresponding label dataset from an industrial image dataset; the real abnormal image dataset includes a plurality of real abnormal image data; the label dataset includes labels of the real abnormal image data in the real abnormal image dataset;

[0009] Based on the real abnormal image dataset and its corresponding label dataset, the acquired normal image data is enhanced to obtain synthetic abnormal images;

[0010] Constructing an industrial image defect detection model and training the industrial image defect detection model using the synthesized abnormal image to obtain a trained industrial image defect detection model; the industrial image defect detection model is used to perform abnormality detection on input image data;

[0011] During the inference phase, the parameters of the trained industrial image defect detection model are kept unchanged, and the image data to be detected is input into the trained industrial image defect detection model to locate the defect position.

[0012] Furthermore, the industrial image dataset includes M different categories of image datasets, where m categories are object categories, n categories are texture categories, and the condition m+n=M is ​​satisfied; each category of image dataset is divided into a training set, a test set, and a label set, where the training set only includes normal image data, and the test set includes both normal image data and real abnormal image data; the label set includes the label of each real abnormal image data in the test set; the normal image data I refers to the normal image data in the training set.

[0013] Furthermore, the method of obtaining a real abnormal image dataset and its corresponding label dataset from the industrial image dataset is as follows:

[0014] A1: Obtain real abnormal image dataset AI from industrial image datasets;

[0015] Specifically, first find the test set of each category of image dataset, and use the set of real abnormal images in all test sets as the real abnormal image dataset AI;

[0016] A2: Obtain the label dataset GT corresponding to the real abnormal image dataset AI from the industrial image dataset;

[0017] Specifically, first find the label set in each category of image dataset, and use the set of labels of the real abnormal image data in each label set as the label dataset GT.

[0018] Furthermore, the method for obtaining the synthesized abnormal image is specifically as follows:

[0019] B1: Perform background detection on normal image data I belonging to m object categories, and generate an initial mask I with the object part being black and the background part being white B , for normal image data I belonging to n texture categories, directly generate a completely black initial mask I B ;

[0020] B2: Determine the abnormality category of the normal image data I, and use a container to select the abnormality level of the normal image data I based on the abnormality category; the abnormality category includes logical abnormality and structural abnormality; the logical abnormality is the presence of an area of ​​an object in the image data that does not conform to its logical rules; the structural abnormality is the presence of an area of ​​an object or texture in the image data that does not conform to its structural characteristics; the abnormality level includes several logical abnormality levels and several structural abnormality levels;

[0021] The method for selecting the abnormality level of normal image data I using a container is:

[0022] For normal image data I of object categories involving logical anomalies, the main container S main Randomly select an abnormality level from several logical abnormality levels and several structural abnormality levels; for normal image data I of texture category or normal image data I of object category not involving logical abnormality, use auxiliary container S aid Select an abnormality level from several structural abnormality levels, each abnormality level corresponds to a defect rate range; the main container S main and auxiliary container S aid Control the probability of each abnormal level;

[0023] B3: Based on the selected anomaly level, based on the initial mask I B Generate an abnormal mask corresponding to each normal image data I;

[0024] For normal image data I whose abnormality level is one of several structural abnormality levels, the process of generating the corresponding abnormality mask is as follows:

[0025] First, for the normal image data I belonging to the object category, calculate its corresponding initial mask I B The number of pixels of the object part; for the normal image data I belonging to the texture category, the corresponding initial mask I B The total number of pixels in the image is taken as the number of pixels of the object; then, an abnormal mask M with all pixel values ​​​​set to zero is initialized i , from the initialized exception mask M i Randomly select a pixel point (x, y) and determine whether the pixel point is on the object part. If not, continue searching until a pixel point on the object part is found, assign the value of the pixel point to 1, and assign the values ​​of the pixels in the eight directions of the upper, lower, left, right, upper left, lower left, upper right and lower right to 1. Then randomly select a pixel point (x', y') from these eight pixels, and assign the values ​​of the pixels in the eight directions of the pixel point to 1 in the same way. Repeat the process of random selection and assignment of pixels until the number of pixels with a value of 1 meets the point number requirement, and obtain the final abnormal mask M. i The number of points is required to be the number of pixels of the object part in the normal image data I multiplied by a random value within the defect rate range corresponding to the abnormal level of the normal image data I. (x, y) and (x', y') represent different pixels, and i represents the number of the abnormal level.

[0026] For normal image data I whose abnormality level is one of several logical abnormality levels, the process of generating the corresponding abnormality mask is as follows: according to the characteristics of the object, the normal image data I is subjected to frequency domain conversion, filtering, HSV color extraction, threshold segmentation, and then image morphological processing to determine the pixels in the normal image data I that may have logical abnormalities. The pixels that may have logical abnormalities are assigned a value of 1, and the remaining pixels are assigned a value of 0 to generate the abnormality mask;

[0027] B4: According to the abnormality level, the abnormal position is assigned to the abnormality mask to obtain the defective block;

[0028] Several structural abnormality classification levels are divided into two parts. For a part of the structural abnormality classification levels with a low defect rate, the abnormal pixel points in the real abnormal image are first determined according to the label of the real abnormal image, and then the value of the abnormal pixel point is randomly selected and multiplied by the transparency factor α, and assigned to the pixel points in the abnormal area of ​​the abnormal mask. The assignment process is repeated until all the pixel points in the abnormal area of ​​the abnormal mask are assigned values. The abnormal area is the pixel points with a median value of 1 in the abnormal mask; for a part of the structural abnormality classification levels with a high defect rate, the abnormal pixel points in the real abnormal image are determined according to the label of the real abnormal image, and then the value of the abnormal pixel point is randomly selected and Gaussian noise is added, and then multiplied by the transparency factor α and assigned to the pixel points in the abnormal area of ​​the abnormal mask. The assignment process is repeated until all the pixel points in the abnormal area of ​​the abnormal mask are assigned values;

[0029] For logical anomalies, the pixels in the abnormal area of ​​the anomaly mask are assigned values ​​according to the category and characteristics of the object;

[0030] B5: Obtain a synthesized abnormal image based on the defect block, the abnormal mask and the normal image data I;

[0031] Specifically, for normal image data I that does not involve logical anomalies, repeat B2 to B4 for each normal image data I, synthesize the several defective blocks corresponding to the normal image data I, and obtain the abnormal block map M'. i After synthesis, the data is ORed with the normal image data I, and the result of the operation is synthesized with the abnormal block graph M' to obtain a synthesized abnormal image;

[0032] For the normal image data I of the category involving logical anomalies, only one defective block is obtained. The normal image data I is compared with the corresponding abnormal mask M. i Perform an OR operation, and then combine the operation result with the defective block to obtain a synthesized abnormal image.

[0033] Furthermore, the industrial image defect detection model includes a teacher network, a student network and a maximum diffusion loss module;

[0034] The teacher network is used to extract features from the input image data and obtain features

[0035] The student network is used to extract features from the input image data and obtain features

[0036] The maximum diffusion loss module is used to adjust the parameters of the student network and is only used during training.

[0037] Furthermore, the teacher network is fed with synthetic abnormal images during training and image data to be detected during inference. The teacher network first extracts three-level features from the input image data using a pre-trained WideResnet50-2 network and fuses the three-level features to obtain the fused feature f t , the fused feature f t Then, the features are fine-tuned and normalized through an RFCA attention fine-tuning module to obtain the normalized features.

[0038] The specific method of fusing the three-level features is as follows: convolution and pooling operations are performed on the first-layer features and the second-layer features output by the WideResnet50-2 network respectively to align them with the third-layer features, and then the aligned first-layer features, second-layer features and third-layer features are fused by element-by-element addition to obtain the fused feature f t .

[0039] Furthermore, the student network is fed with synthetic abnormal images during training and image data to be detected during inference. First, the WideResnet50-2 network is used to extract three-level features from the input image data and fuse the three-level features to obtain the fused feature f s , and then generate the normal feature f' through the adaptive complementary channel selection strategy s , then fine-tuned the features through an RFCA attention fine-tuning module and normalized them into features

[0040] The adaptive complementary channel selection strategy includes frequency domain de-noising, local-global cascade LSTM and complementary channel weight generation;

[0041] The frequency domain de-noising will fusion feature f sThe time domain is transformed into the frequency domain through fast Fourier transform to obtain the amplitude spectrum and phase spectrum. The amplitude spectrum is adaptively removed using UNet. The removed amplitude spectrum and phase spectrum are then synthesized and inversely transformed to obtain the time domain features and input them into the local-global cascade LSTM.

[0042] The local-global cascade LSTM uses a one-dimensional convolutional LSTM in the first stage and a fully connected LSTM in the second stage. The one-dimensional convolutional LSTM adds a one-dimensional convolution before the activation of the traditional LSTM, and the fully connected LSTM adds a fully connected layer before the activation of the traditional LSTM. The local-global cascade LSTM outputs a comprehensive elimination of channel weights as input for generating complementary channel weights.

[0043] The complementary channel weight generation uses three fully connected layers to map the comprehensive channel weight to three weights, respectively represented as w a ,w b and w c , and correspond to the three hierarchical features generated by the WideResnet50-2 network respectively, then multiply the three weights and their corresponding hierarchical features respectively, and then adjust the three multiplied results to the same size through convolution and pooling operations, and then fuse them to obtain the normal feature f' s .

[0044] Furthermore, the maximum diffusion loss module will feature and features The features after bilinear interpolation is adjusted to the size of the first layer features output by the Wide Resnet50-2 network and As input, first calculate the bilinear interpolation feature and The feature difference map between the two images is obtained, and the loss function of the normal pixel and the abnormal pixel is calculated; the normal pixel is the pixel in the synthesized abnormal image corresponding to the pixel at the median value of the abnormal mask of 0; the abnormal pixel is the pixel in the synthesized abnormal image corresponding to the pixel at the median value of the abnormal mask of 1;

[0045] The feature difference map is calculated by and The two norm between them is:

[0046]

[0047] Among them, fd is the feature after bilinear interpolation and Feature difference map between ;

[0048] The calculation formula of the loss function of normal pixels and abnormal pixels is as follows:

[0049]

[0050] Among them, L oj is the loss function of normal pixels and abnormal pixels, The features of normal pixels extracted by the teacher network, The features of normal pixels extracted by the student network, fd oj is the component in the feature difference map, μ is the mean of the feature difference map, mask(o,j) represents the value of the pixel in the abnormal mask, mask(i,j)=0 and mask(i,j)=1 are the judgment conditions for normal pixels and abnormal pixels, respectively;

[0051] Among them, the specific calculation method of the mean μ of the feature difference map is:

[0052]

[0053] Among them, H k and W k Representative characteristics size;

[0054] The maximum diffusion loss MDLM is calculated as follows:

[0055]

[0056] Where L is the maximum diffusion loss MDLM.

[0057] Furthermore, the image data to be detected is input into the trained industrial image defect detection model to locate the defect position, specifically by calculating the difference in output features between the teacher network and the student network, and the position where the difference is greater than a set threshold is determined as the defect position.

[0058] The beneficial effects of adopting the above technical solution are:

[0059] (1) A new scheme for generating anomalies - full-span structure and logical image enhancement strategy is proposed. To address the problem that the synthesized anomalies are quite different from the real anomalies, the new method of synthesizing anomalies proposed in this paper can synthesize both conventional structural anomalies and special logical anomalies, known anomalies and unknown anomalies. The synthesized anomalies are very close to the real anomalies, providing a new enhancement scheme for unsupervised learning and improving the model's ability to identify anomalies.

[0060] (2) An asymmetric teacher-student network model is proposed. The knowledge distillation architecture is used to fine-tune the pre-trained network parameters off-network, so that the network can better cope with different industrial images, enhance practicality, and improve the generalization performance of the model. An adaptive channel selection module is designed in the student network, which includes secondary de-anomaly removal and adaptive channel weight generation. After preliminary de-anomaly removal in the frequency domain and further de-anomaly removal using the proposed LGC-LSTM, the comprehensive de-anomaly channel weight is obtained. Then, channel mapping at different levels is performed to obtain complementary channel information, thereby ensuring that the student network retains normal features and repairs abnormal features.

[0061] (3) A new maximum diffusion loss (MDLM) is proposed to promote the model to further utilize synthetic anomalies. MDLM uses the embedding of the teacher network projected into the latent space as prior knowledge, explicitly diffusing normal features towards the direction of small differences and abnormal features towards the direction of large differences, obtaining a clear boundary to improve the performance of the model in locating anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A flowchart of a product defect detection method based on adaptive complementary channels and full span enhancement provided by an embodiment of the present invention;

[0063] Figure 2 This is a diagram showing the effect of defect detection on data from 15 types of data sets in MVTecAD provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following is a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings and examples. The following examples are intended to illustrate the present invention but are not intended to limit its scope. This example uses the MVTecAD dataset as an example to implement defect detection using the product defect detection method based on adaptive complementary channels and full span enhancement of the present invention.

[0065] In this embodiment, a product defect detection method based on adaptive complementary channels and full span enhancement is provided. Figure 1 As shown, the following steps are included:

[0066] Step 1: Obtain normal image data I from the industrial image dataset;

[0067] The industrial image dataset includes M different categories of image datasets, where m categories are object categories and n categories are texture categories, and the condition m+n=M is ​​satisfied; each category of image dataset is divided into a training set, a test set, and a label set, where the training set only includes normal image data, and the test set includes both normal image data and real abnormal image data; the label set includes a label for each real abnormal image data in the test set, and the label is a black and white mask, where the white area is the abnormal position and the black part is the normal position; the normal image data I refers to the normal image data in the training set;

[0068] The MVTec AD dataset used in this embodiment is a dataset proposed at the 2019 CVPR conference. It is specifically used to evaluate and test the performance of unsupervised image abnormal pattern detection and segmentation. It contains 15 different categories of image datasets, 10 of which are object categories (Objects) and 5 are texture categories (Textures), with a total of 5354 high-resolution images. Each category dataset is divided into a training set, a test set, and a labeled dataset. The training set only includes normal image data, while the test set includes both normal image data and real abnormal image data. The normal image data obtained from the MVTec AD dataset refers to the normal image data obtained from the training set.

[0069] The MVTecAD dataset has 10 object categories: Bottle, Cable, Capsule, Hazelnut, Screw, Toothbrush, Pill, Zipper, Transistor, and Metal Nut; and 5 texture categories: Carpet, Grid, Leather, Tile, and Wood.

[0070] Step 2: Obtain a real abnormal image dataset AI and its corresponding label dataset GT from the industrial image dataset; the real abnormal image dataset AI includes a number of real abnormal image data; the label dataset GT contains the labels of the real abnormal image data in the real abnormal image dataset AI;

[0071] Step 2.1: Obtain a real abnormal image dataset AI from the industrial image dataset;

[0072] Specifically, first find the test set of each category of image dataset, and use the set of real abnormal images in all test sets as the real abnormal image dataset AI;

[0073] Step 2.2: Obtain the label dataset GT corresponding to the real abnormal image dataset AI from the industrial image dataset;

[0074] Specifically, first find the label set in each category of image dataset, and use the set of labels of the real abnormal image data in each label set as the label dataset GT;

[0075] Step 3: Based on the real abnormal image dataset AI and its corresponding label dataset GT, the acquired normal image data I is enhanced to obtain a synthesized abnormal image;

[0076] Step 3.1: Perform background detection on normal image data I belonging to m object categories, and generate an initial mask I with the object part being black and the background part being white B , for normal image data I belonging to n texture categories, directly generate a completely black initial mask I B ;

[0077] The specific method for background detection in this embodiment is as follows: first, the color of the normal image data I is adjusted by adaptive histogram equalization and global histogram equalization, then Gaussian filtering is performed to remove some noise and then thresholding is performed. Then, image morphological operations are performed to perform opening, closing, erosion, and dilation operations to obtain a black and white image, and the object part is set to black.

[0078] Step 3.2: Determine the abnormality category of the normal image data I, and use a container to select the abnormality level of the normal image data I based on the abnormality category; the abnormality category includes logical abnormality and structural abnormality; the logical abnormality is the presence of an area of ​​an object in the image data that does not conform to its logical rules; the structural abnormality is the presence of an area of ​​an object or texture in the image data that does not conform to its structural characteristics; the abnormality level includes several logical abnormality levels and several structural abnormality levels;

[0079] The method for selecting the abnormality level of normal image data I using a container is:

[0080] For normal image data I of object categories involving logical anomalies (cable, transistor, screw), the main container S main Randomly select an abnormality level from several logical abnormality levels and several structural abnormality levels; for normal image data I of texture category or normal image data I of object category not involving logical abnormality, use auxiliary container S aid Select an abnormality level from several structural abnormality levels, each abnormality level corresponds to a defect rate range; the main container S main and auxiliary container S aidThe probability of each abnormal level can be controlled;

[0081] In this embodiment, the main container is used to select the abnormality level, and the specific formula is expressed as follows:

[0082] ls=Random.choice(S main )

[0083] Among them, S main is the main container, ls is the exception level, and Random.choic(·) indicates random selection;

[0084] In this embodiment, the structural abnormality level is expressed as:

[0085]

[0086] Among them, γ ls represents the defect rate, a and b represent hyperparameters, which are set to 5% and 15% in this embodiment, and Level I, Level II, and Level III are three structural abnormality levels;

[0087] The main container contains structural anomalies and logical anomalies. When the main container cannot select the abnormality level, the auxiliary container will be used to select the abnormality level, which is expressed as:

[0088] ls=Random.choice(S aid )

[0089] Among them, S aid Represented as auxiliary container;

[0090] In this embodiment, the main container contains three structural abnormality levels and one logical abnormality level. When the main container conditions are met (cable, transistor, screw type), that is, the object type will cause logical abnormalities, four abnormality levels are selected. When the main container conditions are not met, that is, no logical abnormalities will occur, only three structural abnormality levels are selected, and no logical abnormality level is involved.

[0091] Step 3.3: Based on the selected anomaly level, the initial mask I B Generate an abnormal mask corresponding to each normal image data I;

[0092] For normal image data I whose abnormality level is one of several structural abnormality levels (i.e., there is structural abnormality), the process of generating the corresponding abnormality mask is as follows:

[0093] First, for the normal image data I belonging to the object category, calculate its corresponding initial mask I BThe number of pixels of the object part; for the normal image data I belonging to the texture category, the corresponding initial mask I B The total number of pixels in the image is taken as the number of pixels of the object;

[0094] I B_num =count(I b =0)

[0095] Among them, I B_num Indicates the number of pixels of the object, I b Represents the initial mask I B Pixel points in, count represents the number of calculated pixels;

[0096] Then, initialize an abnormal mask M with all pixel values ​​​​set to zero i , from the initialized exception mask M i Randomly select a pixel point (x, y) and determine whether the pixel point is on the object part. If not, continue searching until a pixel point on the object part is found, assign the value of the pixel point to 1, and assign the values ​​of the pixels in the eight directions of the upper, lower, left, right, upper left, lower left, upper right and lower right to 1. Then randomly select a pixel point (x', y') from these eight pixels, and assign the values ​​of the pixels in the eight directions of the pixel point to 1 in the same way. Repeat the process of random selection and assignment of pixels until the number of pixels with a value of 1 meets the point number requirement, and the final abnormality mask M can be obtained. i The number of points is required to be the number of pixels of the object part in the normal image data I multiplied by a random value within the defect rate range corresponding to the abnormal level of the normal image data I. (x, y) and (x', y') represent different pixels, and i represents the number of the abnormal level.

[0097] When selecting a pixel point arbitrarily, you can use the method of selecting a patch instead of selecting a pixel point, such as 1X1, 2X2, 3X3, etc. As the patch size increases, the generated abnormal block becomes thicker and larger, so a more realistic abnormal mask can be generated according to the defect type of different objects;

[0098] For normal image data I whose abnormality level is one of several logical abnormality levels (i.e., logical abnormality), the process of generating the corresponding abnormality mask is as follows: based on the characteristics of the object, the normal image data I is subjected to frequency domain conversion, filtering, HSV color extraction, threshold segmentation, and then image morphological processing to determine the pixels in the normal image data I that may have logical abnormalities. The pixels that may have logical abnormalities are assigned a value of 1, and the remaining pixels are assigned a value of 0 to generate the abnormality mask;

[0099] Specifically: For the cable class, it is considered a logical anomaly when the coil colors do not correspond, even if there is no abnormality from a structural point of view, and there is also a type of missing wire in the middle. Therefore, four logical anomalies are set for the cable class. Three are anomalies when the coil colors do not correspond, and the other is the case of missing coils. The position of the wires is filtered out through filtering, and this abnormal mask is retained. For the screw class, an additional abnormality type is added, that is, there is an extra part at the edge, which is common in actual production. First, the edge position is found according to the background detection mask, and the first point is selected from the edge position. Then, if it is not on the object, the next point is selected, so that a new abnormal mask can be generated at the edge. Three logical anomalies are set for the transistor class. One is to rotate the object part based on the existing background detection initial mask to produce an abnormal type of dysposition. The second is to segment the legs of the transistor and obtain an abnormal mask for segmenting the legs. The third is to directly use the mask segmented by the entire transistor position, that is, the initial mask I obtained by the background detection in step 3.1. B ;

[0100] Step 3.4: According to the abnormality level, assign the abnormal position to the abnormality mask to obtain the defective block;

[0101] For primary and secondary structural anomalies, first determine the abnormal pixel points in the real abnormal image according to the label of the real abnormal image, then randomly select the value of the abnormal pixel point and multiply it by the transparency factor α, and assign it to the pixel points in the abnormal area of ​​the abnormal mask. Repeat the assignment process until all the pixel points in the abnormal area of ​​the abnormal mask are assigned values. The abnormal area is the pixel points with a median value of 1 in the abnormal mask. For tertiary structural anomalies, determine the abnormal pixel points in the real abnormal image according to the label of the real abnormal image, then randomly select the value of the abnormal pixel point and add Gaussian noise, multiply it by the transparency factor α and assign it to the pixel points in the abnormal area of ​​the abnormal mask. Repeat the assignment process until all the pixel points in the abnormal area of ​​the abnormal mask are assigned values.

[0102] For logical anomalies, the pixels in the abnormal area of ​​the anomaly mask are assigned values ​​according to the category and characteristics of the object. The assignment method varies depending on the object category.

[0103] Specifically, four logical exceptions are set for the cable class: three for mismatched coil colors and one for missing coils. Based on the color ranges of different coils, the corresponding locations are filtered out, generating an exception mask that is then assigned the color of the other coil. By filtering the wire locations and assigning them a value of 0, a missing wire exception type is generated. Due to the varying characteristics of different images, the same range can produce different results, resulting in situations where one, two, or three wires are missing, which is more realistic. For the screw class, an additional exception type is added for edges, which is common in production. First, the edge location is found based on the initial mask detected by background detection, and then the value assignment is performed in the same manner as for structural exceptions, creating a new exception at the edge. For the transistor class, three logical exceptions are set. One out-of-position exception type directly assigns values ​​to pixels in the original image. A second type assigns the background color to the background by segmenting the transistor legs, generating a missing leg exception. A third type assigns the background color to the entire transistor location, generating a missing transistor exception. Here, assigning the background color refers to averaging the background color from a number of points outside the object.

[0104] Step 3.5: Obtain a synthesized abnormal image based on the defect block, the abnormal mask, and the normal image data I;

[0105] Specifically, for normal image data I that does not involve logical anomalies, repeat steps 3.2 to 3.4 for each normal image data I, synthesize the several defective blocks corresponding to the obtained normal image data I to obtain the abnormal block map M', and then synthesize the several abnormal masks M corresponding to the normal image data I. i After synthesis, the data is ORed with the normal image data I, and the result of the operation is synthesized with the abnormal block graph M' to obtain a synthesized abnormal image;

[0106] For the normal image data I of the category involving logical anomalies, only one defective block is obtained. The normal image data I is compared with the corresponding abnormal mask M. i Perform an OR operation, and then combine the operation result with the defective block to obtain a composite abnormal image;

[0107] Step 4: Construct an industrial image defect detection model and train the industrial image defect detection model using the synthesized abnormal image to obtain a trained industrial image defect detection model; the industrial image defect detection model is used to perform abnormality detection on the input image data;

[0108] The industrial image defect detection model includes a teacher network, a student network and a maximum diffusion loss module;

[0109] The teacher network is used to extract features from the input image data and obtain features

[0110] The student network is used to extract features from the input image data and obtain features

[0111] The maximum diffusion loss module is used to adjust the parameters of the student network and is only used during the training process;

[0112] The teacher network is fed with synthetic abnormal images during training and image data to be detected during inference. The teacher network first extracts three-level features from the input image data using the pre-trained WideResnet50-2 network and fuses the three-level features to obtain the fused feature f t , the fused feature f t Then, the features are fine-tuned and normalized through an RFCA attention fine-tuning module to obtain the normalized features.

[0113] The specific method of fusing the three-level features is as follows: convolution and pooling operations are performed on the first-layer features and the second-layer features output by the WideResnet50-2 network respectively to align them with the third-layer features, and then the aligned first-layer features, second-layer features and third-layer features are fused by element-by-element addition to obtain the fused feature f t ;

[0114] The pre-trained WideResnet50-2 network was pre-trained on ImageNet, and its parameters remained unchanged during training and testing;

[0115] In this embodiment, the number of channels of the first-layer feature, second-layer feature, and third-layer feature output by the WideResnet50-2 network are 256, 512, and 1024 respectively, and the feature map sizes are 64*64, 32*32, and 16*16 respectively;

[0116] The specific method of the normalization is:

[0117]

[0118] Among them, (o,j) represents the feature f t The position in f t (o,j) represents the feature at (o,j), w represents the feature f t width;

[0119] The principle of the RFCA attention mechanism in this example is to guide the attention of the existing spatial attention mechanism to the receptive field spatial features, dynamically generate receptive field spatial features and calculate attention weights, thus solving the limitations caused by parameter sharing in traditional convolution operations.

[0120] The student network is input as a synthetic abnormal image during the training process and as image data to be detected during the inference process. First, the WideResnet50-2 network is used to extract three-level features from the input image data and fuse the three-level features to obtain the fused feature f s , and then generate the normal feature f' through the adaptive complementary channel selection strategy s , then fine-tuned the features through an RFCA attention fine-tuning module and normalized them into features

[0121] The WideResnet50-2 network parameters in the student network are adjustable;

[0122] The adaptive complementary channel selection strategy includes frequency domain de-escalation (FDEA), local global cascade LSTM (LGC-LSTM) and complementary channel weight generation;

[0123] The frequency domain de-noising will fusion feature f s After the fast Fourier transform (FFT) is used to transform the time domain into the frequency domain, the amplitude spectrum and phase spectrum are obtained. Because the phase spectrum contains the phase relationship of each frequency component and is used for inverse transformation, the amplitude is adaptively removed using the UNet only for the amplitude spectrum. The removed amplitude spectrum and phase spectrum are then synthesized and inversely transformed to obtain time domain features, which are then input into the local-global cascade LSTM.

[0124] The local-global cascade LSTM achieves the function of removing and repairing anomalies through the collaborative work of one-dimensional convolution and fully connected layers. The first level adopts a one-dimensional convolutional LSTM and the second level adopts a fully connected LSTM, so that the model can learn more complex sequence dependencies. The one-dimensional convolutional LSTM adds a one-dimensional convolution before the activation of the traditional LSTM, and the fully connected LSTM adds a fully connected layer before the activation of the traditional LSTM. The local-global cascade LSTM outputs a comprehensive outlier channel weight as the input for generating the complementary channel weight.

[0125] The complementary channel weight generation uses three fully connected layers to map the comprehensive channel weight to three weights, respectively represented as w a ,w b and w c, and correspond to the three-level features generated by the WideResnet50-2 network respectively. Then, the three weights and their corresponding level features are multiplied respectively, and the three multiplied results are adjusted to the same size through convolution and pooling operations, and then fused to obtain the normal feature f' that removes the abnormalities by fusing the three layers. s ;

[0126] The maximum diffusion loss module will feature and features The features after bilinear interpolation is adjusted to the size of the first layer features output by the WideResnet50-2 network and As input, the normal pixels are made to be in the feature and The difference distance of the corresponding features in is reduced to 0 as much as possible, and the abnormal pixel point is in the feature and The difference distance of the corresponding features is diffused beyond the mean to the maximum extent, and the differences of other features that meet the requirements are not calculated to avoid adding unnecessary calculations; the normal pixel point is the pixel point in the synthesized abnormal image corresponding to the pixel point at the median value of the abnormal mask of 0; the abnormal pixel point is the pixel point in the synthesized abnormal image corresponding to the pixel point at the median value of the abnormal mask of 1;

[0127] Specifically: First calculate the bilinear interpolation feature and The feature difference map between them; obtain the positions of the corresponding normal pixels and abnormal pixels in the feature difference map according to the abnormal mask, and calculate the loss function of the normal pixels and the abnormal pixels;

[0128] The feature difference map is calculated by and The two norm between them is:

[0129]

[0130] Among them, fd is the feature after bilinear interpolation and Feature difference map between ;

[0131] The calculation formula of the loss function of normal pixels and abnormal pixels is as follows:

[0132]

[0133] Among them, L oj is the loss function of normal pixels and abnormal pixels, The features of normal pixels extracted by the teacher network, The features of normal pixels extracted by the student network, fd oj is the component in the feature difference map, μ is the mean of the feature difference map, mask(o,j) represents the value of the pixel in the abnormal mask, mask(i,j)=0 and mask(i,j)=1 are the judgment conditions for normal pixels and abnormal pixels, respectively. The cosine similarity loss function is used to calculate the loss for normal pixel positions. The abnormal pixels are judged based on the mean of the feature difference map. It is considered that the point features with a difference greater than the mean of the feature difference map meet the conditions. These points will not be used to calculate the loss, and the loss generated for the network is zero. They do not need to be focused on in the next training. For those pixels with a difference less than the mean, the features do not meet the model requirements and need to be readjusted according to the size of the difference. That is, the smaller the difference is than the mean, the greater the loss generated.

[0134] Among them, the specific calculation method of the mean μ of the feature difference map is:

[0135]

[0136] Among them, H k and W k Representative characteristics size;

[0137] During the training process, the Adam optimizer minimizes the maximum diffusion loss (MDLM) to train the industrial image defect detection model. The maximum diffusion loss (MDLM) calculation formula is as follows:

[0138]

[0139] Where, L is the maximum diffusion loss (MDLM);

[0140] Step 5: During the inference phase, the parameters of the trained industrial image defect detection model remain unchanged. The image data to be detected is input into the trained industrial image defect detection model to locate the defect location.

[0141] Specifically: Calculate the difference in output features between the teacher network and the student network, and determine the location where the difference is greater than the set threshold as the defect location.

[0142] In this embodiment, the method of the present invention is also compared with existing methods on the MVTecAD dataset. Table 1 shows the pixel-level AUC-ROC scores of different methods, and Table 2 shows the pixel-level PRO scores of different methods. It can be seen from Table 1 that the method of the present invention and FAIR are significantly superior to existing methods in the AU-ROC and PRO indicators of the texture class. FAIR is more outstanding in AU-ROC performance, while the method is relatively outstanding in PRO performance compared to other methods. Among the 10 data given in the texture class, FAIR has five data that reach the highest, and the method has four that reach the highest. Among the five PRO data, FAIR has two categories of data that reach the highest, and the method has three categories of data that reach the highest. This shows that the method of frequency domain processing can be more conducive to texture detection, because texture detection detects all pixels in the image, while object detection will detect both the object and the background, and the ground truth will never appear on the background, because the model detects the object class to detect the object itself, rather than all pixels in the entire image. In addition, in FAIR, the analysis concluded that the frequency recovery task can effectively improve the overall anomaly detection performance. This method first removes anomalies in the frequency domain, which can also be considered a repair task, so the expected results are unsurprising. Furthermore, the use of a UNet for frequency domain de-anomaly removal is reasonable, as its skip connections significantly reduce normal high-frequency reconstruction errors. Typically, in the frequency domain, an object image contains more high-frequency information, while the background is more low-frequency. The loss function design of this method does not impose requirements on features that already meet the requirements, which facilitates the recovery of these features by minimizing information loss and increasing computational overhead. However, in object detection tasks, defects are relatively rare throughout the object, meaning that most objects still contain high-frequency, normal information. While skip connections in the frequency domain reduce this high-frequency reconstruction error, they are not conducive to image de-anomaly removal, resulting in relatively poor results for transistors, which have larger defects. Therefore, an LGC-LSTM module is added to perform secondary de-anomaly removal, discarding anomalies and updating only the defect locations for repair, further improving object detection performance. In terms of average indicators, this method achieved the best positioning effect with an AU-ROC score of 98.25% and a PRO score of 95.56%. It is 0.57% higher than the second highest RD++. RD++ and CDO based on knowledge distillation achieved good pixel-level segmentation, but the effect can be further improved. This method is also based on the knowledge distillation framework. On the MVTecAD dataset, it achieved better pixel-level positioning effect through secondary frequency domain processing, adaptive complementary channel weight generation, and maximum diffusion loss. This proves that the method of the present invention is superior to most existing methods. The defect detection effect of the method of the present invention is as follows: Figure 2 As shown in the figure, the first row shows the abnormal samples, the second row shows the detection results of the proposed method, and the third row shows the labels corresponding to the abnormal images. It can be seen that the proposed method can effectively detect defects of all sizes, whether large or small, whether linear or dotted, and whether semantic or logical errors, fully demonstrating the effectiveness of the model detection.

[0143] Table 1. Comparison of pixel-level AUC-ROC anomaly detection performance on the MVTecAD dataset

[0144]

[0145]

[0146] Table 2. Comparison of pixel-level AUC-PRO anomaly detection performance on the MVTecAD dataset

[0147]

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A product defect detection method based on adaptive complementary channels and full span enhancement, characterized in that: The following steps are involved: Obtain normal image data from industrial image datasets; Obtaining a real abnormal image dataset and its corresponding label dataset from an industrial image dataset; the real abnormal image dataset includes a plurality of real abnormal image data; the label dataset includes labels of the real abnormal image data in the real abnormal image dataset; Based on the real abnormal image dataset and its corresponding label dataset, the acquired normal image data is enhanced to obtain synthetic abnormal images; Constructing an industrial image defect detection model and training the industrial image defect detection model using the synthesized abnormal images to obtain a trained industrial image defect detection model; the industrial image defect detection model is used to perform abnormality detection on input image data; During the inference phase, the parameters of the trained industrial image defect detection model are kept unchanged, and the image data to be detected is input into the trained industrial image defect detection model to locate the defect position.

2. The product defect detection method based on adaptive complementary channels and full span enhancement according to claim 1 is characterized in that: The industrial image dataset includes M different categories of image datasets, where m categories are object categories and n categories are texture categories, and the condition m+n=M is ​​satisfied; The image dataset of each category is divided into a training set, a test set, and a label set, wherein the training set includes only normal image data, and the test set includes both normal image data and real abnormal image data; the label set includes the label of each real abnormal image data in the test set; The normal image data I refers to the normal image data in the training set.

3. The product defect detection method based on adaptive complementary channels and full span enhancement according to claim 2, characterized in that: The method of obtaining a real abnormal image dataset and its corresponding label dataset from the industrial image dataset is as follows: A1: Obtain real abnormal image dataset AI from industrial image datasets; Specifically, first find the test set of each category of image dataset, and use the set of real abnormal images in all test sets as the real abnormal image dataset AI; A2: Obtain the label dataset GT corresponding to the real abnormal image dataset AI from the industrial image dataset; Specifically, first find the label set in each category of image dataset, and use the set of labels of the real abnormal image data in each label set as the label dataset GT.

4. The product defect detection method based on adaptive complementary channels and full span enhancement according to claim 2, characterized in that: The method for obtaining the synthesized abnormal image is specifically as follows: B1: Perform background detection on normal image data I belonging to m object categories, and generate an initial mask I with the object part being black and the background part being white B , for normal image data I belonging to n texture categories, directly generate a completely black initial mask I B ; B2: Determine the abnormality category of the normal image data I, and use a container to select the abnormality level of the normal image data I based on the abnormality category; The anomaly categories include logical anomalies and structural anomalies; the logical anomaly refers to the presence of regions in the image data that do not conform to its logical rules; the structural anomaly refers to the presence of regions in the image data that do not conform to its structural characteristics; the anomaly levels include several logical anomaly levels and several structural anomaly levels; The method for selecting the abnormality level of normal image data I using a container is: For normal image data I of object categories involving logical anomalies, the main container S main Randomly select an abnormality level from several logical abnormality levels and several structural abnormality levels; for normal image data I of texture category or normal image data I of object category not involving logical abnormality, use auxiliary container S aid Select an abnormality level from several structural abnormality levels, each abnormality level corresponds to a defect rate range; the main container S main and auxiliary container S aid Control the probability of each abnormal level; B3: Based on the selected anomaly level, based on the initial mask I B Generate an abnormal mask corresponding to each normal image data I; For normal image data I whose abnormality level is one of several structural abnormality levels, the process of generating the corresponding abnormality mask is as follows: First, for the normal image data I belonging to the object category, calculate its corresponding initial mask I B The number of pixels in the object part; For normal image data I belonging to the texture category, its corresponding initial mask I B The total number of pixels in the image is taken as the number of pixels of the object; Then, initialize an abnormal mask M with all pixel values ​​​​set to zero i , from the initialized exception mask M i Randomly select a pixel point (x, y) and determine whether the pixel point is on the object part. If not, continue searching until a pixel point on the object part is found, assign the value of the pixel point to 1, and assign the values ​​of the pixels in the eight directions of the upper, lower, left, right, upper left, lower left, upper right and lower right to 1. Then randomly select a pixel point (x', y') from these eight pixels, and assign the values ​​of the pixels in the eight directions of the pixel point to 1 in the same way. Repeat the process of random selection and assignment of pixels until the number of pixels with a value of 1 meets the point number requirement, and obtain the final abnormal mask M. i The number of points is required to be the number of pixels of the object part in the normal image data I multiplied by a random value within the defect rate range corresponding to the abnormal level of the normal image data I. (x, y) and (x', y') represent different pixels, and i represents the number of the abnormal level. For normal image data I whose abnormality level is one of several logical abnormality levels, the process of generating the corresponding abnormality mask is as follows: according to the characteristics of the object, the normal image data I is subjected to frequency domain conversion, filtering, HSV color extraction, threshold segmentation, and then image morphological processing to determine the pixels in the normal image data I that may have logical abnormalities. The pixels that may have logical abnormalities are assigned a value of 1, and the remaining pixels are assigned a value of 0 to generate the abnormality mask; B4: According to the abnormality level, the abnormal position is assigned to the abnormality mask to obtain the defective block; Several structural abnormality classification levels are divided into two parts. For a part of the structural abnormality classification levels with a low defect rate, the abnormal pixel points in the real abnormal image are first determined according to the label of the real abnormal image, and then the value of the abnormal pixel point is randomly selected and multiplied by the transparency factor α, and assigned to the pixel points in the abnormal area of ​​the abnormal mask. The assignment process is repeated until all the pixel points in the abnormal area of ​​the abnormal mask are assigned values. The abnormal area is the pixel points with a median value of 1 in the abnormal mask; for a part of the structural abnormality classification levels with a high defect rate, the abnormal pixel points in the real abnormal image are determined according to the label of the real abnormal image, and then the value of the abnormal pixel point is randomly selected and Gaussian noise is added, and then multiplied by the transparency factor α and assigned to the pixel points in the abnormal area of ​​the abnormal mask. The assignment process is repeated until all the pixel points in the abnormal area of ​​the abnormal mask are assigned values; For logical anomalies, the pixels in the abnormal area of ​​the anomaly mask are assigned values ​​according to the category and characteristics of the object; B5: Obtain a synthesized abnormal image based on the defect block, the abnormal mask and the normal image data I; Specifically, for normal image data I that does not involve logical anomalies, repeat B2 to B4 for each normal image data I, synthesize the several defective blocks corresponding to the obtained normal image data I, and obtain the abnormal block map M. ' , the normal image data I corresponds to several abnormal masks M i After synthesis, perform OR operation with normal image data I, and then compare the operation result with abnormal block image M ' Synthesize to obtain a synthesized abnormal image; For the normal image data I of the category involving logical anomalies, only one defective block is obtained. The normal image data I is compared with the corresponding abnormal mask M. i Perform an OR operation, and then combine the operation result with the defective block to obtain a synthesized abnormal image.

5. The product defect detection method based on adaptive complementary channels and full span enhancement according to claim 1, characterized in that: The industrial image defect detection model includes a teacher network, a student network and a maximum diffusion loss module; The teacher network is used to extract features from the input image data and obtain features The student network is used to extract features from the input image data and obtain features The maximum diffusion loss module is used to adjust the parameters of the student network and is only used during training.

6. The product defect detection method based on adaptive complementary channels and full span enhancement according to claim 5, characterized in that: The teacher network is fed with synthetic abnormal images during training and image data to be detected during inference. The teacher network first extracts three-level features from the input image data using the pre-trained Wide Resnet50-2 network and fuses the three-level features to obtain the fused feature f t , the fused feature f t Then, the features are fine-tuned and normalized through an RFCA attention fine-tuning module to obtain the normalized features. The specific method of fusing the three-level features is as follows: convolution and pooling operations are performed on the first-layer features and the second-layer features output by the Wide Resnet50-2 network respectively to align them with the third-layer features, and then the aligned first-layer features, second-layer features and third-layer features are fused by element-by-element addition to obtain the fused feature f t .

7. The product defect detection method based on adaptive complementary channels and full span enhancement according to claim 5, characterized in that: The student network is input as a synthetic abnormal image during the training process and as image data to be detected during the inference process. First, the Wide Resnet50-2 network is used to extract three-level features from the input image data and fuse the three-level features to obtain the fused feature f s , and then generate the normal feature f' through the adaptive complementary channel selection strategy s , then fine-tuned the features through an RFCA attention fine-tuning module and normalized them into features The adaptive complementary channel selection strategy includes frequency domain de-noising, local-global cascade LSTM and complementary channel weight generation; The frequency domain de-noising will fusion feature f s The time domain is transformed into the frequency domain through fast Fourier transform to obtain the amplitude spectrum and phase spectrum. The amplitude spectrum is adaptively removed using UNet. The removed amplitude spectrum and phase spectrum are then synthesized and inversely transformed to obtain the time domain features and input them into the local-global cascade LSTM. The local-global cascade LSTM uses a one-dimensional convolutional LSTM in the first stage and a fully connected LSTM in the second stage. The one-dimensional convolutional LSTM adds a one-dimensional convolution before the activation of the traditional LSTM, and the fully connected LSTM adds a fully connected layer before the activation of the traditional LSTM. The local-global cascade LSTM outputs a comprehensive elimination of channel weights as input for generating complementary channel weights. The complementary channel weight generation uses three fully connected layers to map the comprehensive channel weight to three weights, respectively represented as w a ,w b and w c , and correspond to the three hierarchical features generated by the Wide Resnet50-2 network respectively, then multiply the three weights and the features of the corresponding hierarchical layers respectively, and then adjust the three multiplied results to the same size through convolution and pooling operations, and then fuse them to obtain the normal feature f' s .

8. The product defect detection method based on adaptive complementary channels and full span enhancement according to claim 5, characterized in that: The maximum diffusion loss module will feature and features The features after bilinear interpolation is adjusted to the size of the first layer features output by the WideResnet50-2 network and As input, first calculate the bilinear interpolation feature and The feature difference map between the two images is obtained, and the loss function of the normal pixel and the abnormal pixel is calculated; the normal pixel is the pixel in the synthesized abnormal image corresponding to the pixel at the median value of the abnormal mask of 0; the abnormal pixel is the pixel in the synthesized abnormal image corresponding to the pixel at the median value of the abnormal mask of 1; The feature difference map is calculated by and The two norm between them is: Among them, fd is the feature after bilinear interpolation and Feature difference map between ; The calculation formula of the loss function of normal pixels and abnormal pixels is as follows: Among them, L oj is the loss function of normal pixels and abnormal pixels, The features of normal pixels extracted by the teacher network, The features of normal pixels extracted by the student network, fd oj is the component in the feature difference map, μ is the mean of the feature difference map, mask(o,j) represents the value of the pixel in the abnormal mask, mask(i,j)=0 and mask(i,j)=1 are the judgment conditions for normal pixels and abnormal pixels, respectively; Among them, the specific calculation method of the mean μ of the feature difference map is: Among them, H k and W k Representative characteristics size; The maximum diffusion loss MDLM is calculated as follows: Where L is the maximum diffusion loss MDLM.

9. The product defect detection method based on adaptive complementary channels and full span enhancement according to claim 5, characterized in that: The image data to be detected is input into the trained industrial image defect detection model to locate the defect position. Specifically, the difference in output features between the teacher network and the student network is calculated. The position where the difference is greater than the set threshold is determined as the defect position.

Images