A method and apparatus for detecting denim defects based on Transformer and Discrete Fourier Transform

Abstract

This invention discloses a method and apparatus for denim defect detection based on Transformer and Discrete Fourier Transform. The method includes the following steps: the input image is first subjected to Discrete Fourier Transform, and the amplitude spectrum and phase spectrum images of the Fourier Transform are calculated; then, the amplitude spectrum and phase spectrum are respectively passed through a neural network consisting of an image segmentation module, a fully connected layer module, a Transformer module, and a deconvolution module, to obtain new amplitude spectrum and new phase spectrum; finally, these two new spectra are used to generate the defect detection result through inverse Discrete Fourier Transform. This invention is the first to propose a method for denim defect detection that combines Transformer and Discrete Fourier Transform. This method has a lower computational cost than existing Transformer-based defect detection methods, but achieves better denim defect detection results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and more specifically, relates to a method for detecting defects in denim fabric based on Transformer and Discrete Fourier Transform. Background Technology

[0002] Defect detection in denim is a crucial technology and method for ensuring the quality of denim production. Currently, denim defect detection methods fall into two main categories: The first category uses traditional manual feature design or simple empirical methods for defect detection. These methods can only handle a limited number of denim fabrics under very specific working conditions. As the variety of denim fabrics and the complexity of defect types increase, these methods are increasingly unable to meet the actual needs of industrial production. The second category uses deep neural networks for defect detection. While these methods can generate better expressive features and achieve better detection results, they generally suffer from excessive model parameters, large model sizes, and slow computation speeds. These issues affect the practicality and deployability of these methods. Furthermore, the second category of methods often only detects denim fabrics and defect types seen during the training process, making it difficult to handle new denim fabrics and defect types that have not been encountered before. Summary of the Invention

[0003] To address the aforementioned problems and shortcomings of current methods and technologies, this invention proposes for the first time a denim defect detection method based on Transformer and Discrete Fourier Transform. The method specifically includes the following steps:

[0004] Step 1: Take the denim image as input and perform a Discrete Fourier Transform to calculate the corresponding amplitude spectrum and phase spectrum images. Then, process the amplitude spectrum and phase spectrum through a first neural network and a second neural network respectively to obtain new amplitude spectrum and new phase spectrum. The first neural network includes an image segmentation module, a first fully connected layer module, a first Transformer module, and a first deconvolution module. The second neural network includes an image segmentation module, a second fully connected layer module, a second Transformer module, and a second deconvolution module.

[0005] Step 2: After the amplitude spectrum and phase spectrum are each processed through a neural network with the same structure, new amplitude spectrum and new phase spectrum will be generated. Finally, these two new spectra are used to generate the defect detection results of the denim image through inverse discrete Fourier transform.

[0006] Furthermore, the discrete Fourier transform used is the standard two-dimensional discrete Fourier transform, and the amplitude spectrum and phase spectrum are obtained by standard calculation through this transform. The length and width of the amplitude spectrum and phase spectrum are the same as those of the input image.

[0007] Furthermore, the image segmentation module segments the input image into non-overlapping sub-blocks of size P×P, and stretches the features of each sub-block into an image block feature vector. Let the spatial resolution of the amplitude spectrum image be H×W, where H and W represent the height and width of the image, respectively. Then the overall dimension of the image block features obtained after image segmentation and feature stretching is ((H / P)×(W / P))×C1, where C1 is the length of the image block feature.

[0008] Furthermore, the first fully connected layer module adjusts the dimensions of the image patch features obtained by the image segmentation module. This fully connected layer module contains a fully connected layer.

[0009] Furthermore, the first Transformer module consists of two normalization layers, a first feature converter, a second feature converter, a convergence layer, two fusion layers, and a third fully connected layer module.

[0010] The specific calculations within the first Transformer module are as follows: First, the input image patch features are sequentially passed through a normalization layer, a first feature converter, a convergence layer, a second feature converter, and a fusion layer. The normalization layer is implemented through layer normalization calculations. The first feature converter transforms the image patch features into a feature map style, adjusting the dimensions from two to three dimensions. Then, an N×N convergence calculation is performed in the convergence layer, where N is the neighborhood range of the convergence, and the converged feature map is enlarged to the size before convergence. Next, in the second feature converter, the features are further transformed into a two-dimensional image patch feature style. The fusion layer's calculations are performed through the image... The fusion layer is implemented by adding block features. The computational object of the fusion layer is the input image block features and the output image block features of the second feature converter. Then, the added image block features are fed into the normalization layer and then pass through the third fully connected layer module and the fusion layer in sequence. The input of the fusion layer is the output of the aforementioned fusion layer and the output of the third fully connected layer module, which contains a fully connected layer. Finally, the added image block features are the output of the first Transformer module. When the input image block features pass through two first Transformer modules in sequence, the dimensions of the output image block features generated by each first Transformer module are completely consistent.

[0011] Furthermore, the first deconvolution module consists of a first feature converter, a 5×5 deconvolution layer, a GELU activation function layer, a batch normalization layer, and a Sigmoid function layer connected in series; the feature dimension of the image patch features calculated by the first feature converter is (H / P)×(W / P)×C4, where C4 is the feature length of the feature map; the output of the first deconvolution module is a new amplitude spectrum with a spatial size of H×W.

[0012] Furthermore, the phase spectrum is sequentially processed through an image segmentation module, a second fully connected layer module, two second Transformer modules, and a first deconvolution module. The image segmentation module segments the input phase spectrum image into non-overlapping sub-blocks of size P×P, and stretches the features of each sub-block into an image block feature vector. Let the spatial resolution of the amplitude spectrum image be H×W, where H and W represent the height and width of the image, respectively. The phase spectrum calculation is performed through the second fully connected layer module, the second Transformer module, and the second deconvolution module. Block 2 has the same structure as the first fully connected layer module, the first Transformer module, and the first deconvolution module, only with different parameters; the feature dimension output by the second fully connected layer module is ((H / P)×(W / P))×C5, where C5 is the length of the image block feature output by the second fully connected layer module; the feature dimension output by the second Transformer module is ((H / P)×(W / P))×C6, where C6 is the length of the image block feature output by the second Transformer module; the result output by the second deconvolution module is a new phase spectrum with a spatial size of H×W.

[0013] Furthermore, the newly generated amplitude spectrum and phase spectrum are used to generate defect detection results through inverse discrete Fourier transform, namely a grayscale image of size H×W. The higher the grayscale value in the image, the greater the probability that the pixel location contains a defect.

[0014] Furthermore, the entire neural network is trained end-to-end, with the objective function being the cross-entropy loss function, used to measure the difference between the generated defect detection map and the real manually labeled map. The calculation method of the cross-entropy loss function is shown in the following formula:

[0015]

[0016] Among them, Y - and Y + b represents the set of non-defect pixels and defect pixels in the manually marked result, where b = |Y + | / (|Y - |+|Y + |),|Y - | and |Y + | represents the number of non-defect pixels and defect pixels, respectively; b is used to balance the imbalance of sample numbers during defect detection; y i This represents the probability value of each pixel belonging to a defect, which is the value of each pixel in the defect detection map.

[0017] The present invention also provides a denim defect detection device based on Transformer and Discrete Fourier Transform, comprising:

[0018] One or more processors;

[0019] A storage device is provided for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a denim defect detection method based on Transformer and Discrete Fourier Transform as described above.

[0020] In summary, the technical solutions designed by this invention have the following advantages compared with the prior art:

[0021] Thanks to the creative integration of frequency domain processing and Transformer network structure, as well as the corresponding design of a simple and efficient network structure, this invention can generate more robust features, has a simpler deep neural network structure, requires less computation, and has better generalization ability compared to other existing defect detection methods. It can effectively detect defect types that have not been seen during the training process. Attached Figure Description

[0022] Figure 1 The overall flowchart of a denim defect detection method based on Transformer and Discrete Fourier Transform provided by the present invention;

[0023] Figure 2 This is a schematic diagram of the structure of the first Transformer module and the second Transformer module. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] This invention provides a method for detecting defects in denim fabric based on Transformer and Discrete Fourier Transform, such as... Figure 1 As shown, it includes the following steps:

[0026] Step 1: Take the denim image as input and perform a Discrete Fourier Transform to calculate the corresponding amplitude spectrum and phase spectrum images; then pass the amplitude spectrum through a neural network consisting of an image segmentation module, a first fully connected layer module, two second Transformer modules, and a first deconvolution module connected in series; and simultaneously pass the phase spectrum through another neural network consisting of an image segmentation module, a second fully connected layer module, two second Transformer modules, and a second deconvolution module connected in series.

[0027] Step 2: After the amplitude spectrum and phase spectrum are each processed through a neural network with the same structure, new amplitude spectrum and new phase spectrum will be generated. Finally, these two new spectra are used to generate the defect detection results of the denim image through inverse discrete Fourier transform.

[0028] This invention mainly consists of two parts: (1) optimizing the amplitude spectrum and phase spectrum, and (2) outputting the defect detection results. The steps in this invention will be described in detail below.

[0029] 1. Optimize amplitude and phase spectra

[0030] The Discrete Fourier Transform used in this invention is the standard two-dimensional Discrete Fourier Transform. The amplitude spectrum and phase spectrum are obtained by standard calculation using this transform. The length and width of these two images are the same as the input image. The specific function of the image segmentation module is to segment the input image into non-overlapping sub-blocks of size P×P. In this embodiment, P is 4. The features of each sub-block are stretched into an image block feature vector. Let the spatial resolution of the amplitude spectrum image be H×W, where H and W represent the height and width of the image, respectively. Then the overall dimension of the image block features obtained after image segmentation and feature stretching is ((H / P)×(W / P))×C1, where C1 is the length of the image block feature. In this embodiment, C1 is 32.

[0031] The specific function of the first fully connected layer module in this invention is to adjust the dimension of image patch features with a dimension of ((H / P)×(W / P))×C1. This fully connected layer module contains a fully connected layer, and the dimension of the image patch features after adjustment is ((H / P)×(W / P))×C2, where C2 is the length of the image patch features after processing by the fully connected layer. In this embodiment, C2 is 64. Each first Transformer module consists of two normalization layers, a first feature converter, a second feature converter, a convergence layer, two fusion layers, and a third fully connected layer module, as shown below. Figure 2 As shown.

[0032] The specific calculations within each first Transformer module in this invention are as follows: Figure 2As shown: First, the input image patch features are sequentially passed through a normalization layer, a first feature converter, a convergence layer, a second feature converter, and a fusion layer. The normalization layer is implemented through layer normalization calculations. The first feature converter transforms the image patch features into a feature map style, adjusting the features from two dimensions (number of image patches × length of image patches) to three dimensions (length × width × length of image patches). Then, an N×N convergence calculation is performed in the convergence layer, where N is the neighborhood range of the convergence; in this embodiment, N is 3, and the converged feature map is enlarged to the size before convergence. Next, in the second feature converter, the features are converted back into two-dimensional image patch feature styles. The fusion layer is calculated by adding the image patch features; the fusion layer's calculation object is the input image. The first image block feature is the output image block feature of the second feature converter. Then, the summed image block features are fed into the normalization layer and then sequentially through the third fully connected layer module and the fusion layer. The input of the fusion layer is the output of the aforementioned fusion layer and the output of the third fully connected layer module. The third fully connected layer module contains a fully connected layer. Finally, the summed output image block features are the output results of the first Transformer module. When the input image block features sequentially pass through two first Transformer modules, the dimension of the output image block features generated by each first Transformer module is completely consistent, which is ((H / P)×(W / P))×C3. C3 is the length of the image block features output by each first Transformer module. In this embodiment, the value of C3 is 128.

[0033] The first deconvolution module in this invention consists of a first feature converter, a 5×5 deconvolution layer, a GELU activation function layer, a BN (batch normalization) layer, and a Sigmoid function layer connected in series. The feature dimension of the image patch features calculated by the feature converter 1 is (H / P)×(W / P)×C4, where C4 is the feature length of the feature map. In this embodiment, C4 is 128. The output of the first deconvolution module is a new amplitude spectrum with a spatial size of H×W.

[0034] In this invention, the phase spectrum and amplitude spectrum generated after the input image undergoes a discrete Fourier transform are the same, such as... Figure 1As shown, the image is processed sequentially through an image segmentation module, a second fully connected layer module, two second Transformer modules, and a second deconvolution module. The structure and parameters of the image segmentation module are exactly the same for both the phase spectrum and the amplitude spectrum. The second fully connected layer module, the second Transformer module, and the second deconvolution module used for phase spectrum calculation have the same structure as the first fully connected layer module, the first Transformer module, and the first deconvolution module, only with different parameters. The feature dimension output by the second fully connected layer module is ((H / P)×(W / P))×C5, where C5 is the length of the image block feature output by the second fully connected layer module. In this embodiment, C5 is 32. The feature dimension output by the second Transformer module is ((H / P)×(W / P))×C6, where C6 is the length of the image block feature output by each second Transformer module. In this embodiment, C6 is 128. The result output by the second deconvolution module is a new phase spectrum with a spatial size of H×W.

[0035] 2. Output defect detection results

[0036] In this invention, the newly generated amplitude spectrum and phase spectrum are used to generate defect detection results through inverse discrete Fourier transform, which is a grayscale image of size H×W. The higher the grayscale value in the image, the greater the possibility that the pixel position contains a defect.

[0037] In this invention, the entire neural network is trained end-to-end. The objective function for optimization is the cross-entropy loss function, which measures the difference between the generated defect detection map and the real manually labeled map. The calculation method of the cross-entropy loss function is shown in the following formula:

[0038]

[0039] Among them, Y - and Y + b represents the set of non-defect pixels and defect pixels in the manually marked result, where b = |Y + | / (|Y - |+|Y + |),|Y - | and |Y + | represents the number of non-defect pixels and defect pixels, respectively; b is used to balance the imbalance of sample numbers during defect detection; y i This represents the probability value of each pixel belonging to a defect, which is the value of each pixel in the defect detection map.

[0040] This invention also provides a denim defect detection device based on Transformer and Discrete Fourier Transform, comprising:

[0041] One or more processors;

[0042] A storage device is provided for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a denim defect detection method based on Transformer and Discrete Fourier Transform as described above.

[0043] The table below shows the results of comparing the present invention with three mainstream methods, STPM, DRAEM and PatchCore, using common IAP metrics on the mainstream MVTecAD database. As can be seen from the table, the present invention outperforms all the comparison methods, demonstrating the effectiveness and advantages of the present invention.

[0044] STPM DREAM PatchCore This invention IAP 83.2 90.3 81.8 93.4

[0045] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for denim defect detection based on Transformer and Discrete Fourier Transform, characterized in that, Comprising the following steps: Step 1, the denim image is inputted into discrete Fourier transform, the corresponding amplitude spectrum and phase spectrum images are calculated, then the amplitude spectrum and phase spectrum are processed by the first neural network and the second neural network respectively to obtain new amplitude spectrum and new phase spectrum; the first neural network comprises an image cutting module, a first full connection layer module, a first Transformer module and a first deconvolution module, and the second neural network comprises an image cutting module, a second full connection layer module, a second Transformer module and a second deconvolution module; The first full connection layer module is used for dimension adjustment of image block features; The first Transformer module is internally composed of two normalization layers, a first feature converter, a second feature converter, a convergence layer, two fusion layers and a third full connection layer module; The first deconvolution module is composed of a first feature converter, a 5x5 deconvolution layer, a GELU activation function layer, a batch normalization layer and a Sigmoid function layer in series; The phase spectrum is sequentially processed by the image cutting module, a second full connection layer module, two second Transformer modules and a first deconvolution module; the image cutting module cuts the input phase spectrum image into non-overlapping sub-blocks with a spatial size of PxP, and stretches each sub-block feature into an image block feature vector; the spatial resolution of the amplitude spectrum image is HxW, wherein H and W represent the height and width of the image respectively; the second full connection layer module, the second Transformer module and the second deconvolution module have the same structure as the first full connection layer module, the first Transformer module and the first deconvolution module when calculating the phase spectrum, except that the parameters are different; the feature dimension output by the second full connection layer module is ((H / P)x(W / P))xC5, and C5 is the length of the image block feature output by the second full connection layer module; the feature dimension output by the second Transformer module is ((H / P)x(W / P))xC6, and C6 is the length of the image block feature output by the second Transformer module; the result output by the second deconvolution module is a new phase spectrum image with a spatial size of HxW; Step 2, after the amplitude spectrum and the phase spectrum are respectively processed by the neural network with the same structure, new amplitude spectrum and new phase spectrum are generated, and finally the two new spectrum images are processed by inverse discrete Fourier transform to generate the defect detection result of the denim image; The newly generated amplitude spectrum and phase spectrum are processed by inverse discrete Fourier transform to generate the defect detection result, that is, a gray-scale image with a size of HxW, wherein the higher the gray value, the more likely the pixel position contains defects.

2. The method of claim 1, wherein the method is a method of denim defect detection based on a Transformer and a discrete Fourier transform. The discrete Fourier transform adopted is a standard two-dimensional discrete Fourier transform, from which the amplitude spectrum image and the phase spectrum image are obtained by standard calculation, and the length and width of the amplitude spectrum image and the phase spectrum image are the same as those of the input image.

3. The method of claim 1, wherein the method is based on a Transformer and a discrete Fourier transform for denim defect detection. The image segmentation module segments the input image into sub-blocks with a spatial size of P×P that do not overlap with each other, and stretches the features of each sub-block into an image block feature vector; assuming that the spatial resolution of the amplitude spectrum image is H×W, where H and W represent the height and width of the image respectively, the overall dimension of the image block features obtained after image segmentation and feature stretching is ((H / P)×(W / P))×C1, and C1 is the length of the image block features.

4. The method of claim 1, wherein the method is a method of denim defect detection based on a Transformer and a discrete Fourier transform. The first fully connected layer module adjusts the dimension of the image block features processed by the image segmentation module, and the fully connected layer module includes one fully connected layer.

5. The method of claim 1, wherein the method is based on a Transformer and a discrete Fourier transform for denim defect detection. The specific calculation inside the first Transformer module is as follows: first, the input image block features are sequentially passed through a normalization layer, a first feature converter, a pooling layer, a second feature converter and a fusion layer; the normalization layer is realized by layer normalization calculation; the first feature converter converts the image block features into the style of a feature map, i.e., adjusts the dimension from two-dimensional features to three-dimensional features; then, N×N pooling calculation is performed in the pooling layer, N is the neighborhood range of the pooling, and the pooled feature map is enlarged to the size before the pooling; then, in the second feature converter, the features are converted into the style of two-dimensional image block features again; the calculation of the fusion layer is realized by adding the image block features, and the calculation object of the fusion layer is the input image block features and the output image block features of the second feature converter; then, the added image block features are sent to the normalization layer, and sequentially passed through the third fully connected layer module and the fusion layer, the input of the fusion layer is the output of the aforementioned fusion layer and the output of the third fully connected layer module, and the third fully connected layer module includes one fully connected layer; finally, the added output image block features are the output results of the first Transformer module; when the input image block features are sequentially passed through two first Transformer modules, the dimension of the output image block features generated by each first Transformer module is completely consistent.

6. The method of claim 1, wherein the method is a method of denim defect detection based on a Transformer and a discrete Fourier transform. The feature dimension obtained by the calculation of the first feature converter is (H / P)×(W / P)×C4, and C4 is the feature length of the feature map; the output result of the first deconvolution module is a new amplitude spectrum image with a spatial size of H×W.

7. The method of claim 1, wherein the method is based on a Transformer and a discrete Fourier transform for denim defect detection. The entire neural network is trained in an end-to-end manner, and the objective function for optimization is a cross-entropy loss function used to measure the difference between the generated defect detection map and the real human-labeled map; the calculation method of the cross-entropy loss function used is as follows: ； wherein, and denote the set of non-defect pixels and defect pixels in the artificial marking result, , and denote the number of non-defect pixels and defect pixels, respectively, for balancing the imbalance of the number of samples in defect detection, denote the probability value of each pixel belonging to a defect, that is, the value of each pixel of the defect detection map.

8. A denim defect detection device based on Transformer and Discrete Fourier Transform, characterized by, The method comprises: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for detecting defects in denim fabric based on a Transformer and a discrete Fourier transform according to any one of claims 1 to 7. The method comprises: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for detecting defects in denim fabric based on a Transformer and a discrete Fourier transform according to any one of claims 1 to 7.

Images