Industrial defect detection and positioning method and system based on frequency domain enhancement

Abstract

The invention discloses an industrial defect detection and positioning method and system based on frequency domain enhancement, relates to the field of computer vision and image processing, can discover abnormal modes in an industrial image from a plurality of perspectives, and also has a relatively high reasoning speed. The method comprises the following steps: processing an input image through a high-pass filter to obtain a high-frequency image; inputting the original image and the high-frequency image into an improved U-Net network integrated with a double-domain feature selection module; image reconstruction is carried out in a spatial domain and a frequency domain at the same time; an anomaly score and a localization map are calculated based on the reconstruction result. According to the invention, by combining spatial domain and frequency domain analysis, comprehensive characterization of local pixel change and global frequency mode of the image is realized, and the accuracy of anomaly detection and positioning is improved. The method achieves the optimal performance on a plurality of real industrial scene data sets. The method is suitable for abnormal detection and abnormal area positioning of industrial product images.

Description

Technical Field

[0001] The present invention relates to the field of image anomaly detection, and in particular to a method and system for industrial defect detection and positioning based on frequency domain enhancement. Background Art

[0002] Anomaly detection in industrial images refers to the use of image analysis technology to identify defects in products or components during the manufacturing process. In the manufacturing industry, it is crucial to maintain consistent product quality, and anomaly detection plays a key role in this process. This technology can detect problems at an early stage, thereby reducing costs, improving efficiency, and ensuring end-user satisfaction of the product.

[0003] In industrial production, since normal situations dominate the data, abnormal situations are relatively rare. Therefore, self-supervised image reconstruction methods are widely used in the field of anomaly detection. They use the difference between the original input image and the reconstructed image to detect and locate anomalies. However, reconstruction methods usually have a trade-off between the fidelity of normal reconstruction and the distinguishability of abnormal reconstruction. One way to alleviate the trade-off is to improve the quality of the reconstructed image by adding jump connections or adversarial training, so as to reduce the impact of blurred reconstructed images on the detection results. However, in the process of learning image reconstruction, existing schemes usually only focus on spatial domain information, and do not give full play to the potential of frequency domain information in discovering abnormal patterns. Ultimately, if the local details and global frequency characteristics of the image are to be retained, the accuracy of image abnormality positioning cannot be further improved, thereby failing to further reduce the false detection rate. Summary of the invention

[0004] The embodiments of the present invention provide a method and system for industrial defect detection and positioning based on frequency domain enhancement, which can further improve the accuracy of abnormal positioning of images while retaining local details and global frequency characteristics of the images, thereby further reducing the false detection rate.

[0005] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a method comprising:

[0007] S1. After graying the image to be processed, a high-frequency image is obtained through a high-pass filter, wherein the image to be processed includes: a normal industrial product image without label information; specifically, the sample image to be processed is grayed, and the brightness and texture information of the image are retained after graying, and a high-frequency image is obtained through a Butterworth high-pass filter, and then, Perlin noise and an abnormal texture source data set DTD are used to synthesize anomalies on the high-frequency image, wherein the sample image to be processed includes: a normal industrial product sample image without label information. Note that the operation of synthesizing anomalies is only performed in the training phase, and no anomalies are synthesized in the test phase. It should be noted that the operation of synthesizing anomalies is performed in the training phase. The training phase in this embodiment includes S1-S4, which refers to training the model, and the model is composed of an encoder and a decoder, so the encoder and decoder (the entire model) are trained in the training phase. After the training phase is completed, the model is trained, and the model processes the inference image of the terminal device.

[0008] S2. Input the high-frequency image into the encoder for dual-domain feature selection; specifically, the synthetic abnormal high-frequency image is input into the improved U-Net encoding network with a dual-domain feature selection module to obtain dual-domain features, wherein the dual-domain feature selection module selects features in both the spatial domain and the frequency domain; wherein, after the obtained high-frequency image is subjected to operations such as convolution activation, it is respectively input into the spatial domain selection branch and the frequency domain selection branch of the dual-domain feature selection module. In order to maintain the consistency of feature selection, the dual-domain branches select the same activation function. This dual-domain feature selection method allows the network to focus on the local details and global frequency features of the image at the same time. The feature selection in the spatial domain is used to capture the local structural information such as the shape and texture of the object, while the feature selection in the frequency domain is used to capture the overall frequency distribution of the image. After the feature selection operation of the two branches, the dual-domain features are obtained.

[0009] S3, input the dual-domain features obtained by S2 into the encoder for dual-domain feature fusion to obtain a fused image representation, wherein the dual-domain features include: spatial domain selection features and frequency domain selection features, the spatial domain selection features include the local structure of the image, the frequency domain selection features include the global frequency distribution of the image, and the fused image representation includes spatial domain information and frequency domain information; specifically, input the dual-domain features into a dual-domain feature fusion module to obtain an image representation containing local details of the image and global frequency information, wherein the dual-domain feature fusion module performs concatenation and 1×1 convolution on the dual-domain features; the dual-domain feature fusion module accepts dual-domain features, and the dual-domain features include: spatial domain selection features and frequency domain selection features, which capture the local structure and global frequency distribution of the image respectively. At the beginning of the module, these features will be preliminarily processed, such as standardization or dimensionality adjustment, to ensure that features from different domains can be effectively fused. Next, the model uses feature concatenation and 1×1 convolution to fuse dual-domain features. After the above dual-domain feature selection and dual-domain feature fusion operations at multiple scales, an image representation that integrates spatial domain and frequency domain information is obtained. S2-S3 in this embodiment is an unsupervised method that uses both spatial domain information and frequency domain information to discover abnormal patterns in the image, and this embodiment provides a simple and efficient way to fuse dual-domain features. In the prior art, either a supervised approach is used to train the model, or only spatial domain information is mined without fully utilizing frequency domain information, or the model is relatively complex and difficult to deploy. In this embodiment, an unsupervised model training method is implemented, without the need for manual labeling, and image analysis is performed in both spatial and frequency domains. The model is simple, which is conducive to actual deployment.

[0010] S4, using the fused image representation to reconstruct the image; specifically, constraining the image reconstruction process through spatial domain and frequency domain information; wherein the decoder simply selects a decoder of the UNet network, which includes a series of convolution upsampling operations, and finally decodes the image into a reconstructed image, and the reconstructed image does not contain any abnormal area; the spatial domain and frequency domain information include: l in the spatial domain 2 loss, SSIM loss in the spatial domain, and FFL loss in the frequency domain.

[0011] S5. Anomaly detection is performed using the image reconstructed in S4 and the image to be processed, and the anomaly detection result includes a pixel-level anomaly score; specifically, the anomaly score calculation method includes: anomaly evaluation functions of gradient similarity and color similarity; wherein the gradient similarity anomaly evaluation function applies multi-scale gradient magnitude similarity (MSGMS); the color similarity anomaly evaluation function converts the image into a CIELAB color space that is more in line with human perception of color for comparison; the final anomaly score is composed of a combination of gradient similarity and color similarity.

[0012] S6. After receiving the inference image sent by the terminal device, perform the above steps S1 to S5 on the inference image, and obtain the abnormality detection result corresponding to the inference image, and then determine whether the inference image is abnormal according to the abnormality detection result of the inference image. In practical applications, after receiving the query image sent by the terminal device, the query image is grayed and high-pass filtered, and then converted into a reconstructed image through the model. After that, the image-level and pixel-level abnormality scores are determined according to the difference between the reconstructed image and the original image, and then the image is determined according to the threshold. Whether the image is abnormal and the possible abnormal area, and the query result is fed back to the terminal device.

[0013] In this embodiment, in S1, the high-frequency image is obtained by the high-pass filter, including: after the image to be processed is subjected to fast Fourier transform, high-frequency information is obtained by using a Butterworth high-pass filter, and then the high-frequency information is subjected to fast Fourier inverse transform to obtain the high-frequency image. Specifically, the Butterworth high-pass filter attenuates low-frequency components and passes high-frequency components by setting the center point of the spectrum and the cutoff frequency. Specifically, the sample image to be processed is first grayed, and then the frequency domain information is obtained by fast Fourier transform, and the frequency domain information is high-pass filtered by using a Butterworth high-pass filter, and finally the high-frequency image is obtained by fast Fourier inverse transform. Adjusting the parameters of the Butterworth high-pass filter can obtain a more suitable high-frequency image.

[0014] In this embodiment, in S1, it also includes: if it is in the training stage, the image to be processed is a sample image, and an anomaly is synthesized on the high-frequency image of the sample image, and the synthesized anomaly includes: using Perlin noise to generate an anomaly mask, and then using the abnormal texture source data set DTD and the mask generated by Perlin noise to generate an abnormal area in the high-frequency image.

[0015] In this embodiment, in S2, it includes: in the spatial domain selection branch, spatial domain feature selection is performed through convolution 3×3 convolution-BatchNorm-ReLU-3×3 convolution-BatchNorm; in the frequency domain selection branch, the convolved image is converted into frequency domain information through fast Fourier transform, and then frequency domain feature selection is performed through 1×1 convolution-ReLU-1×1 convolution, and finally the frequency domain information is converted into spatial domain information through inverse fast Fourier transform. Specifically, the reconstruction network includes: an improved U-Net encoder and a UNet decoder with a dual-domain feature selection module; the dual-domain feature selection module includes a spatial domain feature selection branch and a frequency domain feature selection branch; in the feature selection process, the feature map obtained after convolution is input into the spatial domain feature selection branch and the frequency domain feature selection branch respectively, and the spatial domain feature is obtained through spatial domain feature selection, and the frequency domain feature is obtained through frequency domain feature selection. The spatial domain feature and the frequency domain feature are called dual-domain features; in the spatial domain feature selection branch, 3×3 convolution-BatchNorm-ReLU-1×1 convolution is used. eLU-3×3 convolution-BatchNorm is used to perform spatial domain feature selection. Through the superposition of two layers of convolution and the intermediate activation function, the network can effectively extract useful features from the input data and maintain the stability of training through batch normalization; in the frequency domain feature selection branch, the convolved image samples are converted from the spatial domain to the frequency domain through fast Fourier transform, and then 1×1 convolution-ReLU-1×1 convolution is used to achieve cross-channel information aggregation and adjust and optimize the frequency domain features. Finally, the processed frequency domain image is converted back to the spatial domain through inverse fast Fourier transform.

[0016] In this embodiment, S3 includes: splicing the dual-domain features in the dual-domain feature fusion module and then performing 1×1 convolution, wherein each encoding layer of the encoder performs dual-domain feature selection and dual-domain feature fusion; in each encoding layer, the features output by the dual-domain feature fusion module are spliced ​​with the features input to the dual-domain feature selection module, and then down-sampled to obtain the output of one encoding layer, the output of this encoding layer is used as the input of the next encoding layer, and the features obtained by the last encoding layer are used as image representation. Specifically, S3 includes: inputting the obtained dual-domain features into the dual-domain feature fusion module; wherein the dual-domain feature fusion module splices and performs 1×1 convolution on the dual-domain features. After the above-mentioned dual-domain feature selection and dual-domain feature fusion operations at multiple scales, an image representation that integrates spatial domain and frequency domain information is obtained; specifically, each encoding layer performs dual-domain feature selection and dual-domain feature fusion. The features output by the dual-domain feature fusion module are spliced ​​with the features input to the dual-domain feature selection module, and then down-sampled to obtain the output of each encoding layer, the output is used as the input of the next encoding layer, and the features obtained by the last encoding layer are called image representation.

[0017] In this embodiment, in S4, the obtained image representation is decoded into a reconstructed image by a decoder, and the image reconstruction process is constrained by spatial domain and frequency domain information; wherein the decoder is composed of a convolution upsampling operation, the decoder layer corresponds to the encoder layer one by one, and each layer output of the encoder is spliced ​​with the decoder layer through a jump connection. The spatial domain information used to constrain the image reconstruction process includes: l in the spatial domain 2 loss and the structural similarity (SSIM) loss in the spatial domain, where l in the spatial domain 2 The loss is: l 2 The representation loss emphasizes pixel-level accuracy, I and I r are the quantized representations of the original image and the reconstructed image respectively; the SSIM loss is: μ x and μ y I and I respectively r The average brightness, and is I and I r The variance, σ xy It's two I and I r The covariance, C 1 , C 2 is a small constant added to avoid the denominator being zero. H and W are the height and width of the image, respectively. N p is equal to the number of pixels in the image, i and j represent the row and column index of the upper left corner pixel of the current image window, respectively. SSIM(I,I r ) i，j Represents the structural similarity index, which is designed to evaluate the similarity of two images in terms of structure, brightness and contrast. Specifically, SSIM loss is the structural similarity index loss, which is designed to evaluate the similarity of two images in terms of structure, brightness and contrast. To achieve this, instead of calculating a single SSIM value for the entire image, SSIM is calculated separately on multiple small windows of the image. These windows are usually of fixed size, such as a square of 8x8 or 11x11 pixels. Here i and j refer to the row and column indices of the upper left pixel of the current window. SSIM(I, I_r)_{i, j} represents the structural similarity index of the original image I and the reconstructed image I_r; the frequency domain information used to constrain the image reconstruction process includes the frequency domain reconstruction focus loss FFL of the reconstructed image and the original input image. Where M represents the height of the spectrum, N represents the width of the spectrum, and F r (u, v) is the frequency value at the real image spectrum coordinate (u, v), and the corresponding F f(u, v) represents the corresponding value of the spectrum of the reconstructed image, w(u, v) is the element of the spectrum weight matrix, which represents the weight assigned to the frequency component at the position (u, v), w(u, v) = |F r (u,v)-F f (u, v)| α , where α is the scaling factor of flexibility, and u, v represent the indices of horizontal frequency and vertical frequency in the frequency domain, respectively. Specifically, they represent the indices of horizontal frequency and vertical frequency in the frequency space after Fourier transformation of the image.

[0018] Furthermore, a reconstruction model for image reconstruction is established, and the reconstruction model is optimized by the Adam algorithm. During the optimization process, the learning rate is adjusted by the MultiStepLR learning rate scheduler, and the MultiStepLR learning rate scheduler reduces the learning rate according to the preset milestones. The optimization goal is: minL rec , L rec Represents the loss information used to constrain the image reconstruction process, L rec = l 2 (I,I r )+L SSIM (I,I r )+L FFL .

[0019] In this embodiment, in S5, the process of performing anomaly detection includes: Among them, MSGMS(x, y) represents the multi-scale gradient amplitude similarity function, which is used to calculate the multi-scale gradient amplitude similarity between two images x and y, S represents the number of scales to be considered, and GMS s (x, y) is the gradient magnitude similarity at scale i,

[0020] Anomaly score A for gradient similarity anomaly evaluation gradient =1 M×N -MSGMS(I,I r ), anomaly score for color similarity anomaly evaluation L represents brightness, a and b represent color; the abnormality score A of the pixels in the image score =(kA color +A gradient )*f mean , f mean is an average smoothing filter, and k is a scaling factor used to make A color and A gradient are of the same order of magnitude.

[0021] In a second aspect, an embodiment of the present invention provides a system comprising:

[0022] A preprocessing module, used to grayscale the image to be processed and obtain a high-frequency image through a high-pass filter, wherein the image to be processed includes: a normal industrial product image without label information;

[0023] A processing module is used to input the high-frequency image into an encoder for dual-domain feature selection, wherein the encoder includes a dual-domain feature selection module and a dual-domain feature fusion module, and the dual-domain feature selection module includes a spatial domain selection branch and a frequency domain selection branch; input the obtained dual-domain features into the encoder for dual-domain feature fusion to obtain a fused image representation, wherein the dual-domain features include: spatial domain selection features and frequency domain selection features, the spatial domain selection features include the local structure of the image, the frequency domain selection features include the global frequency distribution of the image, and the fused image representation includes spatial domain information and frequency domain information; use the fused image representation to reconstruct the image; use the reconstructed image and the image to be processed to perform anomaly detection, and the anomaly detection result includes a pixel-level anomaly score;

[0024] The query feedback module is used to obtain the reasoning image and obtain the corresponding abnormality detection result after receiving the reasoning image sent by the terminal device, and then determine whether the reasoning image is abnormal according to the abnormality detection result of the reasoning image, and feedback the query result to the terminal device.

[0025] The method and system for industrial defect detection and positioning based on frequency domain enhancement provided by an embodiment of the present invention inputs the sample image to be processed into a preprocessing module and performs grayscale processing and Butterworth high-pass filtering to obtain a high-frequency image. The high-frequency image is passed through an abnormal texture synthesis module to obtain a high-frequency image containing anomalies. The high-frequency image containing anomalies is input into an improved U-Net encoding network with a dual-domain feature selection module to obtain dual-domain features. The dual-domain features are passed through a dual-domain feature fusion module to obtain an image representation that integrates spatial domain and frequency domain information. The image representation is decoded into a reconstructed image by a decoder, and the difference between the reconstructed image and the original image is used as an anomaly score to detect and locate anomalies. The present invention is applicable to the field of industrial image anomaly detection and positioning, and can accurately locate image anomalies while retaining the local details and global frequency characteristics of the image, thereby reducing the false detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a possible implementation of the industrial defect detection and location model provided in an embodiment of the present invention.

[0028] Figure 2 A schematic diagram of a possible implementation of a preprocessing module provided in an embodiment of the present invention.

[0029] Figure 3 A schematic diagram of a possible implementation of a dual-domain feature selection module provided in an embodiment of the present invention.

[0030] Figure 4 A schematic diagram of a method flow chart provided by an embodiment of the present invention.

[0031] Figure 5 A schematic diagram of a system architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention will be described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. It can be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "one", "said" and "the" used herein may also include plural forms. It should be further understood that the term "including" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used here may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0033] The embodiment of the present invention provides an industrial defect detection and positioning method using spatial domain and frequency domain analysis. Specifically, it is an improvement on the anomaly detection technology and a variant of the autoencoder network. The main design idea is: through the combination of spatial domain feature selection and frequency domain feature selection, normal and abnormal modes are considered from a dual-domain perspective, and the features selected by the dual domains are fused to generate an image representation containing local image details and global frequency information. In the pixel reconstruction process, spatial domain loss and frequency domain loss are used simultaneously to constrain the reconstruction process to ensure that normal areas and abnormal areas can be correctly reconstructed, which is beneficial to subsequent anomaly detection and positioning. Specifically, Figure 1 As shown in Figure 2, this method uses dual-domain feature selection and fusion to obtain better image representation and uses dual-domain loss to guide image reconstruction, more comprehensively identify abnormal patterns and reconstruct high-quality normal images. The main method flow is as follows: Figure 1 , 4 As shown, including:

[0034] The sample image to be processed is input into the preprocessing module for preprocessing, including: graying the image, retaining the brightness and texture information of the image, and obtaining a high-frequency image through a high-pass filter. Then, Perlin noise and the abnormal texture source data set DTD are used to synthesize anomalies on the high-frequency image.

[0035] The preprocessing module mentioned in this embodiment can be implemented as a code program in actual application, as shown in the attached Figure 2 As shown, we first grayscale the input RGB image, then perform fast Fourier transform on the grayscale image to convert it from the spatial domain to the frequency domain, and then obtain a high-frequency image through a Butterworth high-pass filter. By setting the parameters of the Butterworth high-pass filter, a more suitable high-frequency image can be obtained. After high-pass filtering, an inverse fast Fourier transform is performed to convert the image from the frequency domain to the spatial domain, and finally, abnormal areas are synthesized in the high-frequency image, which includes: experimental Perlin noise generates random mask areas, and uses the abnormal texture source data set DTD to synthesize anomalies in these mask areas.

[0036] The obtained high-frequency image containing synthetic anomalies is input into the dual-domain feature selection module after convolution.

[0037] The dual-domain feature selection module refers to a module that performs feature selection in both the spatial domain and the frequency domain. It includes a spatial domain branch and a frequency domain branch. The convolved image is input into these two branches for processing. Figure 3The spatial domain branch comes from the encoding layer Block of ResNet, a neural network specifically used to extract feature information from images. The frequency domain branch is similar to the spatial domain branch. In order to maintain the consistency of dual-domain feature selection, the frequency domain branch adds Fourier transform and inverse Fourier transform, and replaces the 3×3 convolution of the spatial domain branch with 1×1 convolution. After dual-domain feature selection, dual-domain features are obtained.

[0038] The obtained dual-domain features are input into a dual-domain feature fusion module to obtain an image representation that contains local image details and global frequency information.

[0039] Among them, the dual-domain feature fusion module refers to a module that splices and fuses the obtained spatial domain features and frequency domain features. The dual-domain feature fusion module directly splices the dual-domain features and then performs 1×1 convolution to fuse the dual-domain features. The fused dual-domain features are simply spliced ​​with the input of the dual-domain feature selection module to obtain the output of the coding layer. It should be noted that each coding layer will perform dual-domain feature selection and dual-domain feature fusion operations. Specifically, the feature map output from the previous coding layer will be used as the input of the dual-domain feature selection module, and the dual-domain features will be obtained after being processed by the dual-domain feature selection module. The dual-domain features are then fused through the dual-domain feature fusion module. Finally, the fused dual-domain features are simply spliced ​​with the feature map output from a coding layer and pooled to obtain the output of the coding layer. The output of the last coding layer is used as an image representation containing local details and global frequency information of the image. The encoder has 5 coding layers, which means that 5 dual-domain feature selection and dual-domain feature fusion operations will be performed during the image coding process.

[0040] The obtained image representation containing local details and global frequency information of the image is decoded into a reconstructed image through a decoder.

[0041] Among them, the decoder uses the decoder of UNet, a popular image segmentation network in deep learning, which is composed of convolution operations and upsampling operations. The image representation is decoded into a reconstructed image by the decoder, and jump connections are applied in the encoding and decoding process; specifically, the input of each decoding layer of the decoder includes: the output of the previous decoding layer, the output of the corresponding encoder layer; specifically, the decoding layer of the first layer receives the output of the last encoding layer as input, the decoding layer of the second layer receives the output of the penultimate encoding layer and the output of the decoding layer of the first layer as input, and so on. In the reconstruction process in which the sample image is encoded into an image representation by the encoder and the image representation is decoded into a reconstructed image by the decoder, the reconstruction process is constrained using spatial domain and frequency domain information; the spatial domain information of the constrained image reconstruction process is the spatial domain reconstruction loss between the reconstructed image and the original input image, including the spatial domain l 2loss and structural similarity (SSIM) loss in the spatial domain; the frequency domain information of the constrained image reconstruction process is the frequency domain reconstruction loss of the reconstructed image and the original input image.

[0042] The reconstructed image and the original input image are compared with each other in terms of gradient and color differences by using anomaly evaluation functions of gradient similarity and color similarity, and anomaly scores are calculated.

[0043] Among them, the gradient similarity applies multi-scale gradient magnitude similarity (MSGMS). MSGMS uses the gradient information of the image at different scales and can provide a more comprehensive image feature description than a single-scale gradient. The color similarity abnormal evaluation function converts the image into a CIELAB color space that is more in line with human perception of color for comparison. The CIELAB space consists of three components: L represents brightness, a and b represent color, and the color similarity abnormal evaluation function ignores L to reduce the impact of illumination.

[0044] The working mode of this embodiment is similar to that of a commonly used image recognition engine. When a user inputs an appearance image of an industrial product into the system, the system will convert the model into a high-frequency image through a preprocessing module, and then the high-frequency image will be input into a reconstruction model to obtain a reconstructed image. The system will calculate the abnormal area determined by the abnormal score by comparing the difference between the input product image and the reconstructed image, and then return the corresponding query result to the user. For example, when a user inputs a picture of a nail, the system will return the result to the user: whether the nail in this image has production defects (abnormalities), and if so, the location of the defect will be given. That is, when a user inputs a picture, the system will tell the user whether the item in the picture has industrial defects and the location of the industrial defects. Among them, the model for detecting abnormalities simultaneously considers the ability of spatial domain information and frequency domain information to discover abnormal patterns, and increases the attention to frequency domain information at both the pixel level (image reconstruction) and the feature level (feature selection). Spatial domain information and frequency domain information complement each other and jointly guide model training. The present invention adopts dual-domain feature selection and dual-domain reconstruction to obtain better image representation and higher quality reconstructed images, which increases the distinction between normal and abnormal.

[0045] In this embodiment, step S1 includes: performing high-pass filtering on the grayscale image data through a Butterworth high-pass filter, wherein the cutoff frequency of the Butterworth high-pass filter is 30, which means that the frequency components less than or equal to 30 from the center frequency in the frequency domain will be significantly attenuated, and the order n of the filter is set to 2. The order determines the speed of filter attenuation and the width of the transition band, and a stability factor is also used to avoid zero division errors in calculations and ensure numerical stability. These three parameters together define the behavior and effect of the Butterworth high-pass filter. By adjusting these parameters, the degree of influence and characteristics of the filter on the image can be controlled, such as edge enhancement, detail extraction or noise suppression.

[0046] Subsequently, Perlin noise and the abnormal texture source dataset DTD are used to synthesize anomalies. Specifically, a random noise image is first generated by Perlin noise to capture various abnormal shapes, and then binarized into an abnormal mask image M by a randomly uniformly sampled threshold. a , the abnormal texture source image A is sampled from an abnormal source image dataset that is independent of the input image distribution, and then data enhancement (such as: degree change, color change, automatic contrast, etc.) is performed on image A. The enhanced texture image A is then masked by the abnormal mask image M a Mask and compare with the high frequency image I h Mixing is done to create anomalies that are close to the normal distribution, which helps tighten the decision boundary in the trained network. Therefore, the synthesized anomaly image I a Defined as:

[0047]

[0048] in, It is M a The inverse of , ⊙ is the element-wise multiplication operation, and β is the opacity parameter in the blending process. This parameter is uniformly sampled from an interval [0.1, 1.0]. Random blending and enhancement can generate different abnormal images from a single texture.

[0049] In this embodiment, step S2 includes:

[0050] Specifically, in each encoding layer of the encoder, the input feature map can be input into the two branches of the dual-domain feature fusion module after operations such as convolution activation to obtain spatial domain features and frequency domain features. Specifically, the spatial domain feature selection branch includes: 3×3 convolution-BatchNorm-ReLU-3×3 convolution-BatchNorm. The frequency domain feature selection branch includes: FFT-1×1 convolution-ReLU-1×1 convolution-IFFT. Here, in order to maintain the consistency of the two selection branches, ReLU is selected as the activation function. FFT and IFFT represent fast Fourier transform and the corresponding inverse transform respectively. 1×1 convolution is mainly to achieve cross-channel information aggregation. The features obtained by the dual-domain feature selection module are called dual-domain features.

[0051] In this embodiment, step S3 includes:

[0052] The obtained dual-domain features are concatenated and fused through a simple dual-domain feature fusion module to obtain image features containing local details and global frequency information of the image. Among them, feature selection in the spatial domain helps to capture local structural information such as the shape and texture of the object, while feature selection in the frequency domain is conducive to capturing the overall frequency distribution of the image. In addition, dual-domain feature selection and dual-domain feature fusion are performed in each coding layer. Specifically, the input of each layer is the feature map output by the previous coding layer. These feature maps first enter the dual-domain feature selection module and generate dual-domain features through processing. Then, these dual-domain features are fused through the dual-domain feature fusion module. The fused dual-domain features are simply concatenated with the feature map of the layer, and then pooling operations are performed to form the output of the coding layer. Finally, the output of the last coding layer will represent the local details and global frequency information of the entire image, also known as image representation. The entire encoder contains 5 such coding layers, so dual-domain feature selection and dual-domain feature fusion will be performed 5 times during the entire image coding process.

[0053] In this embodiment, in step S4, the obtained image representation is decoded into a reconstructed image by a decoder, and the image reconstruction process is constrained by spatial domain and frequency domain information;

[0054] The decoder is composed of convolutional upsampling operations, the decoder layer corresponds to the encoder layer one by one, and the output of each layer of the encoder is spliced ​​with the decoder layer through a jump connection.

[0055] Specifically, based on the traditional image reconstruction method, the system adopts an innovative twin reconstruction strategy, which performs image reconstruction in both the spatial domain and the frequency domain simultaneously. This method not only captures the local details and global structure of the image, but also effectively utilizes the frequency domain information, thereby achieving higher quality image reconstruction. The spatial domain information of the constrained image reconstruction process is the spatial domain reconstruction loss of the reconstructed image and the original input image, including the spatial domain l 2 loss and the structural similarity (SSIM) loss in the spatial domain. 2 The loss is: l 2 The loss emphasizes pixel-level accuracy, where I and I r They are the original image and the reconstructed image respectively. The SSIM loss in the spatial domain focuses more on the overall structure of the image. The definition of SSIM is as follows:

[0056]

[0057] Among them, μ x and μ y are images I and I r The average brightness, and is the variance of the two images, σ xy is the covariance of the two images, C 1 , C 2 is a small constant added to avoid the denominator being zero. The SSIM loss is defined as:

[0058]

[0059] Where H and W are the height and width of the image respectively. p is equal to the number of pixels in the image. The frequency domain information of the constrained image reconstruction process is the focal frequency loss (FFL) of the reconstructed image and the original input image. FFL is a frequency domain-based loss function that aims to address some detail problems that may be overlooked by traditional spatial domain loss functions (such as mean square error or cross entropy). Specifically, it defines the scaled Euclidean distance of these vectors by reducing the weights of simple frequencies using a dynamic spectral weight matrix. Intuitively, the matrix is ​​dynamically updated during training according to the non-uniform distribution of the current loss of each frequency. The model will then quickly focus on hard frequencies and gradually refine the generated frequencies to improve image quality. FFL is defined as:

[0060]

[0061] Where M represents the height of the spectrum, N represents the width of the spectrum, and F r (u, v) is the frequency value at the real image spectrum coordinate (u, v), and the corresponding Ff (u, v) represents the corresponding value of the spectrum of the reconstructed image. w(u, v) is the element of the spectrum weight matrix, which represents the weight assigned to the frequency component at the position (u, v), defined as:

[0062] w(u,v)=|F r (u,v)-F f (u, v)| α

[0063] where α is the scaling factor for flexibility, L FFL It can be regarded as a weighted average of the frequency distance between the real image and the fake image. It focuses on synthesizing hard frequencies by reducing the weights of simple frequencies. Therefore, the spatial domain and frequency domain information used to constrain the image reconstruction process is:

[0064] L rec = l 2 (I, Ir)+L SSIM (I,I r )+L FFL

[0065] Use loss information L rec Guide the model to reconstruct images, and treat the losses in the spatial domain and frequency domain equally during training.

[0066] In this embodiment, in step S5, the difference between the reconstructed image and the input original image is used as an anomaly score to detect and locate anomalies, wherein the gradients and colors of the two images are compared. In order to accurately evaluate these differences, the system uses anomaly evaluation functions of gradient similarity and color similarity. The gradient similarity uses scale gradient magnitude similarity (MSGMS). The MSGMS method uses image pyramid technology to calculate the gradient magnitude similarity (GMS) between the original image and the reconstructed image at multiple scales. The specific operation is to first create multiple versions of the image at different scales, and then calculate the gradients of the two corresponding images at each scale respectively. By comparing these gradient magnitudes, the similarity score of each scale can be obtained, and finally these scores are averaged to obtain the final MSGMS value. The advantage of this multi-scale method is that it can capture and evaluate the details and structural differences of the image at different resolutions. MSGMS is defined as:

[0067]

[0068] Among them, MSGMS(x, y) is the multi-scale gradient amplitude similarity between two images x and y, S represents the number of scales to be considered, GMS s (x, y) is the gradient magnitude similarity at scale i, and the gradient similarity anomaly evaluation function can be defined as:

[0069] A gradien t=1 M×N -MSGMS(I,I r ),

[0070] The color similarity anomaly evaluation function converts the image into the CIELAB color space that is more consistent with human perception of color for comparison. The CIELAB space consists of three components: L represents brightness, a and b represent color. In order to reduce the impact of illumination changes in anomaly detection, the evaluation function deliberately ignores the L channel and only uses the a and b color channels for comparison. This is because the brightness L component is very sensitive to illumination conditions and may mask or distort important anomaly signals caused by color differences. The color similarity anomaly evaluation function is defined as:

[0071]

[0072] The superscripts a and b represent the color components in the CIELAB space, and I and I r Denote the original image and the reconstructed image respectively. The difference score calculation formula between the images is defined as:

[0073] A score =(kA color +A gradient )*f mean ,

[0074] Among them, f mean is an average smoothing filter. k is a scaling factor that makes A color and A gradient are of the same order of magnitude. score The anomaly score of each pixel in the image is calculated, and for the image-level detection score, the maximum value of all pixel-level anomaly scores is selected. In addition, the color similarity anomaly evaluation function using the CIELAB color space and ignoring the brightness component can provide a more stable and accurate way to analyze and compare color differences between images. This method strengthens the ability of the anomaly detection model to more effectively identify and locate color-related anomalies in complex visual environments.

[0075] The advantage of this embodiment is that by using spatial domain information and frequency domain information to jointly guide the training of the model, the dual-domain information complements each other and jointly discovers abnormal patterns in the image. The image preprocessing module highlights the high-frequency information that is easily lost in image reconstruction to greatly avoid the model from seeing abnormalities during reasoning. The dual-domain feature selection module considers both the spatial domain and the frequency domain when modeling the normal image distribution. Dual-domain reconstruction not only captures the local details and global structure of the image, but also effectively utilizes the frequency domain information, thereby achieving higher quality image reconstruction, which directly determines the detection performance of the model.

[0076] It should be noted that this embodiment is not a simple calculation method, but can be applied to industrial production and assist in improving the production line of industrial products. For example, in practical applications, the method of this embodiment can be applied to a Figure 5 The system shown comprises:

[0077] The preprocessing module is used to perform fast Fourier transform on the image data to be processed, obtain high-frequency information using a Butterworth high-pass filter, and perform inverse fast Fourier transform on the high-frequency information to obtain a high-frequency image;

[0078] A processing module is used to input the high-frequency image into an improved U-Net encoding network with a dual-domain feature selection module to obtain dual-domain features, wherein the dual-domain feature selection module selects features in the spatial domain and the frequency domain at the same time; input the dual-domain features into a dual-domain feature fusion module to obtain an image representation containing local details of the image and global frequency information, wherein the dual-domain feature fusion module performs splicing and 1×1 convolution on the dual-domain features; decode the image representation into a reconstructed image through a UNet decoding network, wherein the reconstructed image is constrained by a reconstruction loss in the spatial domain and a reconstruction loss in the frequency domain; use the difference between the reconstructed image and the original image as an anomaly score to detect and locate anomalies, wherein the difference includes a color difference and a gradient difference of the image, the anomaly score is an anomaly score of each pixel, and the image-level anomaly score selects the maximum value of all pixel-level anomaly scores;

[0079] The query feedback module is used to receive the query image sent by the terminal device, convert the query image into a reconstructed image generated by the model, and then determine the image-level and pixel-level anomaly scores based on the difference between the reconstructed image and the original image, and feed back the query result to the terminal device.

[0080] Specifically, this embodiment is applicable to defect detection and positioning of industrial images, that is, the product image is converted into a reconstructed image through the pre-training module and the trained model. The model will further compare the gradient and color differences between the input product image and the output reconstructed image to detect and locate the anomaly, and then return the corresponding detection and positioning results. For example: the working method is similar to the currently commonly used image recognition. When the assembly line produces industrial parts, the machine on the assembly line takes a picture of the part, obtains the image and inputs it into the model. Then the model reconstructs the image and uses the gradient and color anomaly evaluation function to evaluate whether the image is abnormal. Finally, the result is returned. This result can be used to determine whether the assembly line continues to produce, because when an industrial product with defects is produced, it means that there is a problem with the assembly line, and it may be necessary to stop production and repair the machine, otherwise it will cause a waste of resources. It can also be used to guide the subsequent repair of faulty machines. The location where the defect occurs may also contain information about the fault.

[0081] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for industrial defect detection and location based on frequency domain enhancement, characterized in that: include: S1. After graying the image to be processed, a high-frequency image is obtained through a high-pass filter, wherein the image to be processed includes: a normal industrial product image without label information; S2. Inputting the high-frequency image into an encoder for dual-domain feature selection, wherein the encoder includes a dual-domain feature selection module and a dual-domain feature fusion module, and the dual-domain feature selection module includes a spatial domain selection branch and a frequency domain selection branch; S3, inputting the dual-domain features obtained by S2 into the encoder for dual-domain feature fusion to obtain a fused image representation, wherein the dual-domain features include: spatial domain selection features and frequency domain selection features, the spatial domain selection features include the local structure of the image, the frequency domain selection features include the global frequency distribution of the image, and the fused image representation includes spatial domain information and frequency domain information; S4, performing image reconstruction using the fused image representation; S5, performing anomaly detection using the image reconstructed in S4 and the image to be processed, wherein the anomaly detection result includes an anomaly score at the pixel level; S6. After receiving the inference image sent by the terminal device, execute the above steps S1 to S5 on the inference image, and obtain an abnormality detection result corresponding to the inference image, and then determine whether the inference image is abnormal according to the abnormality detection result of the inference image.

2. The method according to claim 1, characterized in that: In S1, obtaining a high-frequency image through a high-pass filter includes: After performing fast Fourier transform on the image to be processed, a Butterworth high-pass filter is used to obtain high-frequency information, and then a fast Fourier inverse transform is performed on the high-frequency information to obtain the high-frequency image.

3. The method according to claim 1, characterized in that In S1, it also includes: If it is in the training stage, the image to be processed is a sample image, and an anomaly is synthesized on the high-frequency image of the sample image, wherein the synthesized anomaly includes: generating an anomaly mask using Perlin noise, and then generating an abnormal area in the high-frequency image using an abnormal texture source data set DTD and a mask generated by Perlin noise.

4. The method according to claim 1, characterized in that: In S2, it includes: In the spatial domain selection branch, spatial domain feature selection is performed through convolution 3×3 convolution-BatchNorm-ReLU-3×3 convolution-BatchNorm; In the frequency domain selection branch, the convolved image is converted into frequency domain information through fast Fourier transform, and then frequency domain feature selection is performed through 1×1 convolution-ReLU-1×1 convolution, and finally the frequency domain information is converted into spatial domain information through inverse fast Fourier transform.

5. The method according to claim 1, characterized in that In S3, this includes: In the dual-domain feature fusion module, the dual-domain features are concatenated and then subjected to 1×1 convolution, wherein each encoding layer of the encoder performs dual-domain feature selection and dual-domain feature fusion; In each coding layer, the features output by the dual-domain feature fusion module are concatenated with the features input to the dual-domain feature selection module, and then downsampled to obtain the output of a coding layer. The output of this coding layer serves as the input of the next coding layer, and the features obtained by the last coding layer are used as image representation.

6. The method according to claim 1, characterized in that In S4, the spatial domain information used to constrain the image reconstruction process includes: The l2 loss in the spatial domain and the structural similarity (SSIM) loss in the spatial domain, where the l2 loss in the spatial domain is: l2 indicates that the loss emphasizes pixel-level accuracy, I and I r are the quantized representations of the original image and the reconstructed image, respectively; The SSIM loss is: μ x and μ y I and I r The average brightness, and is I and I r The variance, σ xy It's two I and I r covariance, C1, C2 are small constants added to avoid the denominator being zero, H and W are the height and width of the image respectively, N p is equal to the number of pixels in the image, i and j represent the row and column index of the upper left corner pixel of the current image window, respectively. SSIM(I,I r ) i,j stands for structural similarity index, which is designed to evaluate the similarity of two images in terms of structure, brightness, and contrast; The frequency domain information used to constrain the image reconstruction process includes the frequency domain reconstruction focus loss FFL of the reconstructed image and the original input image. Where M represents the height of the spectrum, N represents the width of the spectrum, and F r (u,v) is the frequency value at the real image spectrum coordinate (u,v), and the corresponding F f (u, v) represents the corresponding value of the spectrum of the reconstructed image, w(u, v) is the element of the spectrum weight matrix, which represents the weight assigned to the frequency component at the position (u, v), w(u, v) = |F r (u,v)-F f (u,v)| α , where α is the scaling factor of the flexibility, and u, v represent the indices of the horizontal and vertical frequencies in the frequency domain, respectively.

7. The method according to claim 6, characterized in that Also includes: A reconstruction model for image reconstruction is established, and the reconstruction model is optimized by the Adam algorithm. During the optimization process, the learning rate is adjusted by the MultiStepLR learning rate scheduler, and the MultiStepLR learning rate scheduler reduces the learning rate according to the preset milestones. The optimization goal is: min L rec , L rec Represents the loss information used to constrain the image reconstruction process, L rec =l2(I,I r )+L SSIM (I,I r )+L FFL .

8. The method according to claim 1 or 6, characterized in that: In S5, the process of anomaly detection includes: Among them, MSGMS(x,y) represents the multi-scale gradient amplitude similarity function, which is used to calculate the multi-scale gradient amplitude similarity between two images x and y, S represents the number of scales to be considered, and GMS s (x,y) is the gradient magnitude similarity at scale i, Anomaly score A for gradient similarity anomaly evaluation gradient =1 M×N -MSGMS(I,I r ), anomaly score for color similarity anomaly evaluation L represents brightness, a and b represent color; The anomaly score A of the pixels in the image score =(kA color +A gradient )*f mean , f mean is an average smoothing filter, and k is a scaling factor used to make A color and A gradient are of the same order of magnitude.

9. A system for industrial defect detection and positioning based on frequency domain enhancement, characterized in that: include: A preprocessing module, used to grayscale the image to be processed and obtain a high-frequency image through a high-pass filter, wherein the image to be processed includes: a normal industrial product image without label information; A processing module is used to input the high-frequency image into an encoder for dual-domain feature selection, wherein the encoder includes a dual-domain feature selection module and a dual-domain feature fusion module, and the dual-domain feature selection module includes a spatial domain selection branch and a frequency domain selection branch; input the obtained dual-domain features into the encoder for dual-domain feature fusion to obtain a fused image representation, wherein the dual-domain features include: spatial domain selection features and frequency domain selection features, the spatial domain selection features include the local structure of the image, the frequency domain selection features include the global frequency distribution of the image, and the fused image representation includes spatial domain information and frequency domain information; use the fused image representation to reconstruct the image; use the reconstructed image and the image to be processed to perform anomaly detection, and the anomaly detection result includes a pixel-level anomaly score; The query feedback module is used to obtain the reasoning image and obtain the corresponding abnormality detection result after receiving the reasoning image sent by the terminal device, and then determine whether the reasoning image is abnormal according to the abnormality detection result of the reasoning image, and feedback the query result to the terminal device.

Images