An industrial flaw detection method based on a frequency domain mask strategy

Abstract

The application discloses a kind of self-supervised industrial defect detection methods based on frequency domain mask strategy, based on Vision Transformer Shared encoder is constructed, feature extraction is carried out to the industrial product surface image after masking and the industrial product surface spectrum diagram after masking, and spatial domain shared encoding feature and frequency domain shared encoding feature are obtained;Two independent encoder-decoder branches are constructed, including spatial domain branch and frequency domain branch, each branch is encoded by Transformer block as encoder, and is decoded by convolution layer as decoder, to obtain spatial domain and frequency domain reconstruction image;Based on pre-training shared encoder and encoder, reconnect detection network, build self-supervised industrial defect detection model, the encoder output of spatial domain branch and frequency domain branch is spliced, input detection network identifies flaw area, and uses labeled sample to fine-tune model;Industrial defect detection is carried out using the adjusted self-supervised industrial defect detection model.The application greatly reduces the labeling cost while improving the performance of industrial defect detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial defect detection, and in particular to an industrial defect detection method based on a frequency domain mask strategy. BACKGROUND

[0002] In industrial production, the detection and control of product surface quality is of great importance. Defect detection is a key link to ensure product quality, improve production efficiency and reduce cost. Traditional defect detection mainly relies on manual inspection, which has the problems of low efficiency, high missed detection rate and great interference from subjective factors. To overcome the above limitations, automatic defect detection methods based on machine vision have attracted more and more attention.

[0003] In recent years, deep learning technology represented by convolutional neural networks has made significant progress in the field of industrial visual detection. However, existing deep learning methods are mostly trained under the supervision of a large number of labeled data, which requires a lot of manpower and time. In industrial scenarios, there is often a lack of large-scale defect labeling samples, which seriously restricts the application of these methods. At the same time, the material, texture, lighting and other factors on the surface of industrial products are complex and changeable, which puts higher requirements on the robustness and generalization ability of the detection algorithm.

[0004] In the face of these challenges, a new method is needed that can make full use of unlabeled data to automatically learn robust feature representation from surface images. In recent years, self-supervised learning methods have provided a new way to solve the problem of lack of labeled samples. Self-supervised learning enables the model to learn general visual representations from a large amount of unlabeled data by designing prediction tasks. On this basis, fine-tuning with a small amount of labeled data can achieve good results on downstream tasks.

[0005] However, existing self-supervised learning methods mainly focus on constructing prediction tasks through various transformations in the spatial domain, such as image rotation, jigsaw puzzle restoration, etc. The features learned by this kind of method lack robustness to changes in lighting, viewing angle, etc., and are difficult to directly apply to industrial detection scenarios. At the same time, existing methods pay insufficient attention to image details and textures, and the sensitivity of the extracted features to small surface defects is limited. SUMMARY

[0006] The purpose of the present application is to provide an industrial defect detection method based on a frequency domain mask strategy.

[0007] The technical solution for achieving the purpose of the present application is as follows: a self-supervised industrial defect detection method based on a frequency domain mask strategy, comprising the following steps:

[0008] Step 1: Obtain an industrial product surface image, mask the industrial product surface image in the spatial domain and the frequency domain, and obtain a masked industrial product surface image and a masked industrial product surface spectrum image.

[0009] Step 2, based on the Vision Transformer, a shared encoder is constructed to extract features from the masked industrial product surface image and the masked industrial product surface spectrum image, obtaining spatial domain shared encoding features and frequency domain shared encoding features;

[0010] Step 3, two independent encoder-decoder branches are constructed, including a spatial domain branch and a frequency domain branch, each branch having a Transformer block as an encoder and a convolution layer as a decoder, wherein the spatial domain branch reconstructs the spatial domain shared encoding features to obtain a spatial domain reconstructed image, and the frequency domain branch reconstructs the frequency domain shared encoding features to obtain a frequency domain reconstructed image;

[0011] Step 4, based on the cosine similarity loss, the frequency domain recovery loss, and the spatial domain recovery loss, a time-frequency consistency loss function is constructed to complete the pre-training of the shared encoder and the encoder-decoder branches;

[0012] Step 5, based on the pre-trained shared encoder and the encoder, a detection network is connected to construct a self-supervised industrial defect detection model, the outputs of the encoders of the spatial domain branch and the frequency domain branch are spliced and input into the detection network to identify the defect area, and the model is fine-tuned using labeled samples;

[0013] Step 6, in the spatial domain and the frequency domain, collect the industrial product surface image, and use the adjusted self-supervised industrial defect detection model to detect industrial defects.

[0014] Further, in step 1, the industrial product surface image is obtained, and the industrial product surface image is masked in the spatial domain and the frequency domain to obtain a masked industrial product surface image and a masked industrial product surface spectrum image, the specific method being:

[0015] Step 1.1, obtain the industrial product surface image X space ;

[0016] Step 1.2, process the industrial product surface image X space by masking in the spatial domain and the frequency domain to more efficiently learn feature expression, wherein:

[0017] In the spatial domain, by blocking part of the pixel area of the image, the situation of surface defects is simulated to obtain the masked industrial product surface image;

[0018] In the frequency domain, first perform discrete cosine transform on the industrial product surface image X space :

[0019] X freq = DCT(Xspace )

[0020] wherein X freq is a frequency domain representation of the industrial product surface image;

[0021] Then, the frequency domain representation of the industrial product surface image is subjected to a masking process:

[0022] X masked = X freq ⊙M

[0023] wherein X masked is the masked industrial product surface frequency spectrum, M represents a dynamically generated masking mask, and is a point-by-point multiplication operation.

[0024] Further, the mask M is generated based on low-high frequency separation masking, random frequency masking, or energy-based adaptive masking.

[0025] Further, in step 2, a shared encoder is constructed based on a Vision Transformer, and feature extraction is performed on the masked industrial product surface image and the masked industrial product surface frequency spectrum to obtain spatial domain shared encoding features and frequency domain shared encoding features. The specific method is as follows:

[0026] The masked industrial product surface image and the masked industrial product surface frequency spectrum are both divided into N token fixed-size image blocks, and all the image blocks are input into a linear projection layer. The corresponding output of each image block is a tensor with a length of D. Then, the N token outputs with a length of D of the two images are spliced to generate an initial feature representation H1 of the masked industrial product surface image and an initial feature representation H2 of the masked industrial product surface frequency spectrum, respectively. The specifications of H1 and H2 are both N token ×D. Then, the two feature representations H1 and H2 are further added with positional encoding to obtain the positional information of the masked industrial product surface image in the spatial domain and the positional information of the masked industrial product surface frequency spectrum in the frequency domain, respectively. Then, the feature representations H1 and H2 are input into the shared encoder.

[0027] In the shared encoder, the feature representations H1 and H2 are sequentially passed through N shareSpecifically, the feature representation H1 is input to the first Transformer block of the shared encoder to obtain its output. This output is then input to the second Transformer block of the shared encoder to obtain its output. The feature representations H1 and H2 are then used in the Nth Transformer block. share The outputs of the last Transformer block are H. space and H freq The specification is N token ×D.

[0028] Further, in step 3, two independent encoder-decoder branches are constructed, including a spatial domain branch and a frequency domain branch. Each branch uses a Transformer block as the encoder and a convolutional layer as the decoder. The specific method is as follows:

[0029] Spatial domain branching is used for the output H of the shared encoder. space In this branch, H is used for image reconstruction. space Input to N branch After an encoder of one Transformer block and a decoder of one convolutional layer, the output is a reconstructed image of the surface of an industrial product.

[0030] Frequency domain branching is used for the output H of the shared encoder. freq In this branch, frequency domain reconstruction is performed, and H is... freq Input to N branch After the encoder of each Transformer block and the final decoder, the output is a reconstructed spectrum of the surface spectrum of the industrial product.

[0031] Further, in step 4, a time-frequency consistency loss function is constructed based on cosine similarity loss, frequency domain recovery loss, and spatial domain recovery loss to complete the training of the encoder-decoder branches. The specific method is as follows:

[0032] A spatial domain recovery loss function is introduced to minimize the reconstructed image of the industrial product surface image. X-ray image of industrial product surface space The goal is to reconstruct an image in the spatial domain that more closely approximates the original surface by taking the pixel differences between them as the objective, where the spatial domain recovery loss function is:

[0033]

[0034] Where M space This is a set of image patch indices in the spatial domain that are masked, which can be obtained when masking images of industrial product surfaces. Let X be the tensor corresponding to the i-th image patch of the reconstructed image of the industrial product surface image. space,i Let N be the tensor corresponding to the i-th image patch of the industrial product surface image. space The number of masking blocks;

[0035] A frequency domain recovery loss function is introduced to minimize the reconstructed spectrum of the surface spectrum of industrial products. X, a spectrum diagram of industrial product surface freq The objective is to reconstruct an image in the frequency domain that more closely approximates the original surface by addressing the difference in DCT coefficients between the two surfaces. The frequency domain restoration loss function is:

[0036]

[0037] Output H in the spatial domain branch space and the output H of the frequency domain branch freq A cosine similarity loss is introduced to enhance the consistency between spatial and frequency domain feature representations, where the cosine similarity loss is:

[0038]

[0039] Where, N token H represents the total number of image patches, as explained above. space,i and H freq,i These represent the spatial domain branch output H, respectively. space and frequency domain branch output H freq The tensor corresponding to the i-th image patch;

[0040] Construct the overall loss function, expressed as:

[0041] L total =λ1L space +λ2L freq +λ3L con

[0042] Where λ1, λ2, and λ3 are weight coefficients, the pre-training process of the shared encoder, spatial domain branch, and frequency domain branch is completed by minimizing the overall loss function.

[0043] Further, in step 5, based on the pre-trained shared encoder and encoder, a detection network is reconnected to construct a self-supervised industrial defect detection model. The encoder outputs from the spatial and frequency domain branches are concatenated and input into the detection network to identify defect regions. Labeled samples are then used to fine-tune the model, where:

[0044] The detection network uses either Faster R-CNN or SSD.

[0045] The application discloses a self-supervised industrial defect detection system based on a frequency domain mask strategy.

[0046] The application discloses a computer device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the self-supervised industrial defect detection method based on the frequency domain mask strategy is implemented, and the self-supervised industrial defect detection based on the frequency domain mask strategy is realized.

[0047] The application discloses a computer device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the self-supervised industrial defect detection method based on the frequency domain mask strategy is implemented, and the self-supervised industrial defect detection based on the frequency domain mask strategy is realized.

[0048] Compared with the prior art, the application has the following advantages:

[0049] (1) effectively reducing the labeling cost. The self-supervised pre-training task is constructed through spatial domain and frequency domain masking, and robust feature representation is learned from a large number of unlabeled product surface images. On this basis, only a small amount of labeled data is needed for fine-tuning, so that an excellent defect detection model can be obtained. Compared with the traditional method which needs large-scale artificial labeling, the application can significantly save the labeling time and labor cost.

[0050] (2) improving the sensitivity to surface defects. The model pays more attention to texture, edge and other fine-grained surface features by randomly masking the image in the frequency domain. At the same time, the time-frequency consistency constraint further enhances the complementarity of spatial domain and frequency domain features, which enables the model to more accurately capture the subtle defects and abnormalities on the product surface, greatly improving the precision and recall rate of detection.

[0051] (3) enhancing the robustness and generalization ability of the model. The Vision Transformer architecture is used to model the global context, so that the model can better understand the correlation between defects and surrounding areas. The pre-training based on frequency domain masking makes the model more robust to changes such as illumination and noise. In addition, the surface features learned by unsupervised pre-training have better transferability, so that the model can better generalize to different kinds of industrial product detection tasks. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The application discloses a computer device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the self-supervised industrial defect detection method based on the frequency domain mask strategy is implemented, and the self-supervised industrial defect detection based on the frequency domain mask strategy is realized.

[0053] Figure 2 The application discloses a computer device comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, the self-supervised industrial defect detection method based on the frequency domain mask strategy is implemented, and the self-supervised industrial defect detection based on the frequency domain mask strategy is realized.

[0054] Figure 3 This is a schematic diagram of the shared encoder and dual-branch structure proposed in the embodiments of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] An industrial defect detection method based on a frequency domain masking strategy is proposed. By randomly occluding the image spectrum, the model learns more fine-grained and global surface features. At the same time, time-frequency consistency constraints are introduced to enhance the complementarity of spatial and frequency domain features. Based on this, the Transformer architecture is used to model the global context, further improving the semantic expressive power of the features.

[0057] like Figure 1 As shown, the self-supervised industrial defect detection method based on frequency domain masking strategy mainly includes the following steps: 1. Masking processing; 2. Input shared encoder; 3. Dual-branch feature recovery; 4. Time-frequency consistency loss training; 5. Model fine-tuning.

[0058] Step 1, masking process:

[0059] An image of the industrial product surface is acquired, and masking processing is performed in both the spatial and frequency domains to obtain the masked industrial product surface image and the masked industrial product surface spectrum. The specific process is as follows:

[0060] Step 1.1: Acquire the surface image X of the industrial product. space These image data can be surface images of various types of products on the production line, such as metal parts, circuit boards, fabrics, glass products, etc.

[0061] Step 1.2: By masking the surface image x of the industrial product in both the spatial and frequency domains. space The features are processed to learn the feature representations more efficiently, where:

[0062] (1) Spatial domain

[0063] In the spatial domain, by occluding a portion of the pixel area of ​​the image, the surface defects are simulated to obtain an image of the surface of the industrial product after occlusion.

[0064] (2) Frequency domain

[0065] In the frequency domain, first analyze the surface image x of the industrial product. space Perform discrete cosine transform:

[0066] x freq = DCT(x space )

[0067] where x freq is the frequency domain representation of the industrial product surface image;

[0068] Then, the frequency domain representation of the industrial product surface image is subjected to a masking process:

[0069] X masked = X freq ⊙M

[0070] where X masked is the masked industrial product surface frequency spectrum image, M represents a dynamically generated masking mask, and is a point-by-point multiplication operation;

[0071] In the above description, the mask M can be designed based on low-high frequency separation masking, random frequency masking, or energy-based adaptive masking, wherein:

[0072] A. Low-high frequency separation masking, different masking strategies are implemented for the low frequency and high frequency parts of the frequency domain representation of the industrial product surface image, respectively focusing on the macro-texture features and micro-defect features of the surface; Figure 2 An example of applying low-high frequency separation masking to the DCT frequency spectrum image is shown.

[0073] The left image is the original frequency spectrum image, and the right image is the masked frequency spectrum image. Figure 2 In the right image, the upper left area of the frequency spectrum image (i.e. the area enclosed by a square with a side length of 28 pixels) is defined as the low frequency area, and the outside is defined as the high frequency area, wherein 90% and 80% of the low frequency area and the high frequency area are randomly masked, respectively, and a mask of size 2x2 is applied to the former and a mask of size 14x14 is applied to the latter.

[0074] B. Random frequency masking, in the frequency domain representation X freq of the industrial product surface image, set the masking ratio to 75%, and use a 14x14 masking block to randomly mask the frequency domain representation X freq ;

[0075] C. Energy-based adaptive masking, calculate the energy distribution map E of the frequency domain representation X freq of the industrial product surface image, where E(i,j) represents the energy value at frequency component (i,j); calculate the total energy E total of the frequency domain representation X freq :

[0076]

[0077] where H,W represent the frequency domain representation X freqheight and width; according to the energy distribution map E and the total energy E total , the occlusion probability P mask (i,j) of each frequency component (i,j) is calculated

[0078]

[0079] According to the occlusion probability P mask (i,j), a mask is generated for the frequency domain representation X freq , and the occlusion ratio is set to 30%.

[0080] Step 2, input the shared encoder:

[0081] Based on the Vision Transformer, a shared encoder is constructed to extract features from the occluded industrial product surface image and the occluded industrial product surface spectrum image, obtaining spatial domain shared encoding features and frequency domain shared encoding features; the specific method is as follows:

[0082] The shared encoder is composed of N share transformer blocks, and its main function is to uniformly map the occluded image data in the spatial domain and the frequency domain to a high-dimensional feature space, thereby providing consistent input representation for subsequent double-branch processing.

[0083] Suppose the occluded industrial product surface image and the occluded industrial product surface spectrum image are both divided into N token fixed-size image blocks, then all these image blocks are input into a linear projection layer (Linear Projection Layer), and the corresponding output of each image block is a tensor with length D, then the N token outputs with length D of the two images are spliced respectively to generate the initial feature representation H1 of the occluded industrial product surface image and the initial feature representation H2 of the occluded industrial product surface spectrum image, and their specifications are both N token ×D, then the two feature representations H1 and H2 are further added with positional encoding (Positional Encoding) to obtain the position information of the occluded industrial product surface image in the spatial domain and the position information of the occluded industrial product surface spectrum image in the frequency domain respectively, then the feature representations H1 and H2 are input into the shared encoder respectively.

[0084] In the shared encoder, the feature representations H1 and H2 are sequentially passed through N shareThere are N Transformer blocks. Specifically, the feature representation H1 is input to the first Transformer block of the shared encoder to obtain its output. This output is then input to the second Transformer block of the shared encoder to obtain its output, and so on. The operation for feature representation H2 is similar. Let feature representations H1 and H2 be in the Nth... share The outputs of the last Transformer block are H. space and H freq Their specifications are also N token ×D.

[0085] Step 3, Two-branch feature recovery:

[0086] Following the shared encoder, this invention designs two independent encoder-decoder branches, used to model features in the spatial and frequency domains respectively. Each branch contains N branch A Transformer block acts as the encoder, and a convolutional layer acts as the decoder.

[0087] Dual-branch architecture, such as Figure 3 As shown, in the dual-branch architecture, the spatial domain branch is used to process the output H of the shared encoder. space In this branch, H is used for image reconstruction. space Input to N branch After an encoder of one Transformer block and a decoder of one convolutional layer, the output is a reconstructed image of the surface of an industrial product. This invention introduces a spatial domain recovery loss function into the spatial domain branch, which minimizes the reconstructed image of an industrial product surface image. X-ray image of industrial product surface space The loss function, defined as the loss between pixels, aims to enable the model to reconstruct an image in the spatial domain that more closely approximates the original surface.

[0088]

[0089] Where M space This is a set of image patch indices in the spatial domain that are masked, which can be obtained when masking images of industrial product surfaces. Let X be the tensor corresponding to the i-th image patch of the reconstructed image of the industrial product surface image. space,i Let N be the tensor corresponding to the i-th image patch of the industrial product surface image. space The number of masking blocks.

[0090] In the specific program implementation, x represents the original unmasked image, xspace represents the spatial domain reconstructed image, and mask represents the spatial masking mask. First, the interpolation of x and xspace is calculated pixel by pixel, and then the interpolation is squared to obtain the pixel-level reconstruction error (x-xspace)^2. Next, mask is multiplied with the pixel-level reconstruction error element by element, that is, mask*(x-xspace)^2, so as to zero the error of the unmasked position and retain the error value of the masked position. Finally, the mean of all elements in the weighted reconstruction error tensor is calculated, that is, mean(mask*(x-xspace)^2), to obtain a scalar loss function value.

[0091] On the other hand, the frequency domain branch is used to reconstruct the output H freq of the shared encoder in the frequency domain. In this branch, H freq is input into the encoder of N branch transformer blocks and the final decoder, and the output is the reconstructed frequency spectrum of the industrial product surface frequency spectrum The present application designs a frequency domain recovery loss function, which takes the minimization of the DCT coefficient difference between the reconstructed frequency spectrum of the industrial product surface frequency spectrum X freq and the industrial product surface frequency spectrum X

[0092]

[0093] In the specific program implementation, the calculation of the frequency domain reconstruction loss is similar to that of the spatial domain. First, the DCT transforms dct(x) and dct(xfreq) of the original image x and the frequency domain reconstructed image xfreq are performed respectively, which map them to the frequency domain representation, denoted as x_dct and xfreq_dct. Then the difference x_dct-xfreq_dct between x_dct and xfreq_dct is calculated frequency component by frequency component. Then the difference is squared and the mean is calculated, that is, mean((x_dct-xfreq_dct)^2), to obtain the frequency domain reconstruction loss.

[0094] Step 4, time-frequency consistency loss training:

[0095] In order to enhance the consistency of the spatial domain and frequency domain feature representations, a cosine similarity loss is imposed between the output H space of the spatial domain branch and the output H freq of the frequency domain branch, which is defined as:

[0096]

[0097] Where N tokenH represents the total number of image patches, as explained above. space,i and H freq,i These represent the spatial domain branch output H, respectively. space and frequency domain branch output H freq The tensor corresponding to the i-th image block.

[0098] In the specific program implementation, the spatial features H_space and H_freq are first subjected to L2 normalization, that is, the operations H_space / norm(H_space) and H_freq / norm(H_freq) are performed to obtain the results H_space_norm and H_freq_norm. Then, the inner product of the two normalized features is calculated, that is, H_space_norm'*H_freq_norm. The average of all inner product results in the batch is then added with a negative sign, that is, -mean(H_space_norm'*H_freq_norm), to obtain the final cosine similarity loss.

[0099] In summary, this method names the overall loss function as the time-frequency consistency loss, which is defined as the weighted sum of the three losses:

[0100] L total =λ1L space +λ2L freq +λ3L con

[0101] Where λ1, λ2, and λ3 are weight coefficients, the pre-training process of the backbone model (i.e., the shared encoder and the spatial and frequency domain branches) is completed by minimizing the above formula.

[0102] Step 5, fine-tune the model:

[0103] During the fine-tuning phase, the decoder portion (i.e., the convolutional layer) of the dual-branch structure is removed from the program definition, retaining only the dual-branch encoder. The output H of the pre-trained spatial domain branch is then used. space and the output H of the frequency domain branch freq The data is then stitched together, resulting in a 2×N dimension. token ×D, Next, based on the specific detection task objective, select a suitable neural network (such as Faster R-CNN, SSD, etc.), input the above splicing results into this neural network, and use a small number of labeled samples for end-to-end supervised training, so that the backbone model and the neural network suitable for this detection task can be fine-tuned and the weights updated according to the needs of this defect detection data.

[0104] Example

[0105] To verify the effectiveness of the method of the present application in the task of industrial defect detection, experiments were conducted on the NEU-DET surface defect dataset. The dataset contains defect images of 6 types of material surfaces: metal surface, ceramic tile surface, glass surface, leather surface, fabric surface and wood surface, each of which has multiple types of defects commonly seen in actual production. A 5-way 1-shot defect detection experiment was conducted on the dataset, i.e. 5 defect categories were randomly selected, and 1 labeled sample from each category was selected for fine-tuning training.

[0106] The method of the present application first pre-trains in the spatial domain and the frequency domain on the 5000 unlabeled surface images of NEU-DET. The frequency domain masking includes three strategies: low-high frequency separation masking, random frequency masking and energy-based adaptive masking. The hyperparameters are determined through multiple ablation experiments, wherein the total number of layers of the encoder Transformer is 12, the first 8 layers are reserved as a shared encoder in the pre-training stage, the frequency domain and spatial domain branches are each composed of 4 Transformer layers, the total number of training rounds in the pre-training stage is 300, and the weights of the loss function are set as λ1=0.25, λ2=0.25, λ3=0.50.

[0107] After pre-training is completed, the detector fine-tuning stage is entered, and SSD is selected as the backbone network of the multi-scale surface defect detector. The model is fine-tuned for 10 rounds using 5-way 1-shot few-shot labeled data, compared with fully supervised SSD, SSD pre-trained directly on ImageNet, and MAE pre-training model without frequency domain masking strategy, and the results are shown in Table 1. The frequency domain masking self-supervised pre-training model of the present application achieves the best result in the MAP index of defect detection of 6 types of materials, fully demonstrating the effectiveness and generalization ability of the method.

[0108] Table 1 Comparison of MAP performance of different surface defect detection methods in NEU-DET 5-way 1-shot detection task

[0109] Method Metal surface Ceramic tile surface Glass surface Leather surface Fabric surface Wood surface Average MAP SSD 42.5 25.1 37.8 31.6 48.5 37.3 37.1 SSD-ImgNet 62.7 33.9 51.4 41.9 57.8 45.6 48.9 SSD-MAE 75.4 44.8 67.9 50.3 70.6 60.1 61.5 SSD-Ours 80.1 49.5 73.6 55.8 76.3 66.4 66.9

[0110] In summary, the self-supervised pre-training method based on frequency domain masking of the present application provides a new idea for industrial product surface defect detection. Through random masking operations on spatial domain and frequency domain images, the model learns more fine-grained surface texture features. At the same time, time-frequency consistency learning further improves the complementarity and consistency of spatial domain and frequency domain features. A large number of experiments show that the present application has achieved excellent performance in defect detection tasks on various material surfaces, significantly improving detection accuracy while significantly reducing dependence on large-scale labeled data, and has high industrial application value. The method can also be applied to other fine-grained surface analysis fields.

[0111] The technical features of the above embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features described above are described, however, as long as the combinations of the technical features do not contradict each other, they should be considered to be within the scope of the present specification.

[0112] The above-described embodiments are merely illustrative of several embodiments of the present application and do not limit the scope of the present application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present application, and such modifications and improvements should be considered to be within the scope of the present application. Therefore, the scope of the present application should be determined by the appended claims.

Claims

1. A self-supervised industrial defect detection method based on a frequency domain mask strategy, characterized in that, The method comprises the following steps: Step 1, obtaining an industrial product surface image, performing masking processing on the industrial product surface image in two dimensions of a spatial domain and a frequency domain, and obtaining a masked industrial product surface image and a masked industrial product surface spectrum image; Step 2, constructing a shared encoder based on a Vision Transformer, performing feature extraction on the masked industrial product surface image and the masked industrial product surface spectrum image, and obtaining spatial domain shared encoding features and frequency domain shared encoding features; Step 3, constructing two independent encoder-decoder branches, including a spatial domain branch and a frequency domain branch, each branch being composed of a Transformer block as an encoder and a convolutional layer as a decoder, wherein the spatial domain branch reconstructs the spatial domain shared encoding features to obtain a spatial domain reconstructed image, and the frequency domain branch reconstructs the frequency domain shared encoding features to obtain a frequency domain reconstructed image; Step 4, constructing a time-frequency consistency loss function based on a cosine similarity loss, a frequency domain recovery loss and a spatial domain recovery loss, and completing pre-training of the shared encoder and the encoder-decoder branches; Step 5, based on the pre-trained shared encoder and the encoder, connecting a detection network again to construct a self-supervised industrial defect detection model, splicing the encoder outputs of the spatial domain branch and the frequency domain branch, inputting the detection network to identify a defect region, and using a labeled sample to fine-tune the model; Step 6, in two dimensions of a spatial domain and a frequency domain, collecting an industrial product surface image to be detected, and using the adjusted self-supervised industrial defect detection model to detect an industrial defect.

2. The self-supervised industrial flaw detection method based on the frequency domain mask strategy according to claim 1, characterized in that, Step 1, obtaining an industrial product surface image, performing masking processing on the industrial product surface image in two dimensions of a spatial domain and a frequency domain, and obtaining a masked industrial product surface image and a masked industrial product surface spectrum image, and the specific method is as follows: Step 1.1, acquiring an image X of the surface of the industrial product space ; Step 1.2: By masking the surface image X of the industrial product in both the spatial and frequency domains. space The features are processed to learn the feature representations more efficiently, where: In the spatial domain, a part of pixel regions of the image is shielded to simulate the situation that defects appear on the surface, and the masked industrial product surface image is obtained. In the frequency domain, first the discrete cosine transform is performed on the industrial product surface image X space : X freq = DCT(X space ) wherein X freq is a frequency domain representation of an image of a surface of an industrial product; Then, the frequency domain representation of the industrial product surface image is masked: X masked = X freq O M where X masked is the masked industrial product surface spectrum, M represents the dynamically generated masking mask, and is a point-by-point multiplication operation.

3. The self-supervised industrial flaw detection method based on the frequency domain mask strategy according to claim 2, characterized in that, The mask M is generated based on low-frequency high-frequency separation masking, random frequency masking or energy-based adaptive masking.

4. The self-supervised industrial defect detection method based on frequency domain mask strategy according to claim 1, characterized in that, Step 2, constructing a shared encoder based on a Vision Transformer, performing feature extraction on the masked industrial product surface image and the masked industrial product surface spectrum image, and obtaining spatial domain shared encoding features and frequency domain shared encoding features, and the specific method is as follows: The masked industrial product surface image and the masked industrial product surface spectrum image are both divided into N token fixed-size image blocks, all of which are input into a linear projection layer, and the corresponding output of each image block is a tensor with a length of D. Then, the N token outputs with a length of D of the two images are spliced respectively to generate an initial feature representation H1 of the masked industrial product surface image and an initial feature representation H2 of the masked industrial product surface spectrum image, both of which have a specification of N token ×D. Then, the two feature representations H1 and H2 are further added with position encoding to obtain the position information of the masked industrial product surface image in the spatial domain and the position information of the masked industrial product surface spectrum image in the frequency domain respectively. Then, the feature representations H1 and H2 are input into a shared encoder respectively; In the shared encoder, the feature representations H1 and H2 are both sequentially passed through N share Transformer blocks, specifically, the feature representation H1 is input to the 1st Transformer block of the shared encoder, to obtain the output of the Transformer block, and then the output is input to the 2nd Transformer block of the shared encoder, to obtain the output of the Transformer block; the outputs of the feature representations H1 and H2 at the N share th Transformer block are H space and H freq , respectively, and the specifications are N token xD.

5. The self-supervised industrial flaw detection method based on the frequency domain mask strategy according to claim 4, characterized in that, Step 3, constructing two independent encoder-decoder branches, including a spatial domain branch and a frequency domain branch, each branch being composed of a Transformer block as an encoder and a convolutional layer as a decoder, and the specific method is as follows: The spatial domain branch is used to reconstruct the output H of the shared encoder space An image is reconstructed, in this branch, H space is input to an encoder of N branch transformer blocks and a one-layer convolutional layer decoder, and the output is a reconstructed image of the industrial product surface image The frequency domain branch is used to reconstruct the output H of the shared encoder freq in the frequency domain. In this branch, H freq is input to an encoder of N branch transformer blocks and a final decoder, and the output is a reconstructed spectrum of the industrial product surface spectrum 6. The self-supervised industrial defect detection method based on frequency domain mask strategy according to claim 4, characterized in that, Step 4, constructing a time-frequency consistency loss function based on a cosine similarity loss, a frequency domain recovery loss and a spatial domain recovery loss, and completing training of the encoder-decoder branches, and the specific method is as follows: A spatial domain recovery loss function is introduced to minimize the pixel difference between the reconstructed image of the industrial product surface image X and the original surface image X space , so that an image closer to the original surface is reconstructed in the spatial domain, where the spatial domain recovery loss function is: where M space is a set of indices of image blocks that are occluded in the spatial domain, which is obtained when occluding the industrial product surface image, is the i-th image block corresponding tensor of the reconstructed image of the industrial product surface image, X space,i is the i-th image block corresponding tensor of the industrial product surface image, N space is the number of occluded blocks; A frequency domain recovery loss function is introduced to minimize the difference between the reconstructed frequency spectrum of the industrial product surface spectrum X and the DCT coefficients of the original surface spectrum X freq The frequency domain recovery loss function is as follows: Output H in the spatial domain branch space and the output H of the frequency domain branch freq A cosine similarity loss is introduced to enhance the consistency between spatial and frequency domain feature representations, where the cosine similarity loss is: where N token is the total number of image blocks, H space,i and H freq,i denote the i-th image block corresponding tensors of the spatial domain branch output H space and the frequency domain branch output H freq respectively. An overall loss function is constructed, which is represented as: L total = λ1L space + λ2L freq + λ3L con Wherein λ1, λ2, λ3 are weight coefficients, by minimizing the overall loss function, the pre-training process of the shared encoder and the spatial domain branch and the frequency domain branch is completed.

7. The self-supervised industrial defect detection method based on frequency domain mask strategy according to claim 4, characterized in that, Step 5, based on the pre-trained shared encoder and encoder, reconnection detection network, build a self-supervised industrial defect detection model, the encoder output of the spatial domain branch and the frequency domain branch is spliced, input into the detection network to identify the defect area, and use the labeled samples to fine-tune the model, wherein: The detection network adopts Faster R-CNN or SSD.

8. A self-supervised industrial defect detection system based on a frequency domain mask strategy, implementing the self-supervised industrial defect detection method based on the frequency domain mask strategy according to any one of claims 1-7, to realize the self-supervised industrial defect detection based on the frequency domain mask strategy.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the self-supervised industrial defect detection method based on the frequency domain mask strategy according to any one of claims 1-7 is implemented, and the self-supervised industrial defect detection based on the frequency domain mask strategy is realized.

10. A computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the self-supervised industrial defect detection method based on the frequency domain mask strategy according to any one of claims 1-7 is implemented, and the self-supervised industrial defect detection based on the frequency domain mask strategy is realized.

Images