An industrial image anomaly perception system based on multi-frequency fusion

Abstract

The present invention discloses an industrial image anomaly perception system based on multi-frequency fusion, including a multi-frequency fusion module, an image generation module based on a generative adversarial network, and an anomaly detection and classification module; the multi-frequency fusion module can capture feature information of different scales and frequencies, and the image generation module based on a generative adversarial network uses multi-frequency images to generate industrial images, which is more conducive to generating real images; anomaly detection uses important feature information extracted from the image to realize industrial image anomaly detection and classification. The present invention enables workers to perform accurate measurements without contacting the measurement target, solves the dangers and inefficiencies caused by contact measurement in industrial environments, and successfully improves the operating efficiency of the production line and reduces the measurement error rate by realizing non-contact anomaly detection, and lays a solid foundation for the development of automation and intelligence in the industrial production process.

Description

Technical Field

[0001] The present invention relates to the field of target anomaly detection algorithms in industrial machine vision, and in particular to an industrial image anomaly perception system based on multi-frequency fusion. Background Art

[0002] With the continuous improvement of the automation and intelligence level of industrial production, the quality requirements for production products are also getting higher and higher. By analyzing the surface, size, shape and other aspects of the product, defects, cracks, deformation and other problems in the product can be detected, so as to adjust the production process or eliminate defective products in time. Most of the existing product inspection methods are based on contact measurement, using tactile sensors, probes or scanners to touch and sense the surface of the product to obtain texture information. This method is often dangerous and inefficient. We use non-contact measurement technology to inspect products, thereby improving product quality and production efficiency. Abnormal conditions in the production process can be detected early, and corresponding measures can be taken to avoid adverse consequences such as production interruption, quality reduction, and cost increase.

[0003] Traditional image anomaly detection methods are mainly based on statistical models or machine learning algorithms, but in complex industrial environments, these methods often have difficulty accurately capturing abnormal patterns, especially for the scarcity of sample data for industrial images, and may not be able to accurately classify abnormal images. Therefore, generating industrial images based on generative model methods for data enhancement plays an important role in improving the accuracy and robustness of anomaly detection.

[0004] Multi-frequency branch images refer to sub-images extracted from industrial images that contain information of different frequencies, where low-frequency images contain more semantic information and high-frequency images contain more detailed texture information. Multi-frequency branches are used to extract deep features of images, and pseudo images are synthesized through generative adversarial networks. Feature learning of the synthesized images is performed, and combined with advanced anomaly detection algorithms, abnormal areas in real images are identified. Specifically, the model generates images by fusing multiple frequency branches, which can capture feature information of different scales and frequencies in the image, thereby better detecting different types of anomalies. Summary of the invention

[0005] Based on the above ideas, the present invention proposes an industrial image anomaly perception system based on multi-frequency fusion. This system realizes the generation of images containing information of different frequencies, effectively solves the problem of insufficient industrial image data, and thus improves the accuracy of industrial anomaly detection.

[0006] In order to achieve the above purpose, the following technical solutions are adopted:

[0007] An industrial image anomaly perception system based on multi-frequency fusion, including a multi-frequency fusion module, an image generation module based on a generative adversarial network, and an anomaly detection and classification module:

[0008] Multi-frequency fusion module: For industrial image datasets provided in industrial production and manufacturing, multi-frequency branches are used to extract sub-images of different frequencies from industrial images. The frequency energy distribution of the image contains different types of image information. The lower frequency component image contains the overall structure and background information of the image, while the higher frequency component reflects the details and texture information of the image, thereby obtaining richer image feature information;

[0009] The multi-frequency fusion module of the model includes a convolutional neural network and a residual network. First, the extracted industrial image is input into the convolutional neural network for feature extraction, and the industrial image is decomposed into sub-images of different frequencies. The frequency sub-branches are then input into the residual network, and the residual network is used to perform frequency enhancement on the obtained multi-branch sub-images.

[0010] The multi-frequency fusion module includes:

[0011] Convolution operation unit: input industrial images into the convolution layer to perform different convolution operations;

[0012] Upsampling difference unit: upsample the sub-image obtained after convolution to an image of the same size as the previous frequency and make a difference;

[0013] Residual enhancement unit: A residual network is added to compensate for the information loss caused by frequency division;

[0014] Summation unit: concatenates the enhanced frequency-separated image with its adjacent high-frequency image and then performs another enhancement;

[0015] The multi-frequency fusion module of the industrial image anomaly perception system based on multi-frequency fusion performs downsampling operations on the industrial image of the input convolution layer with different steps, and then uses the residual network to perform frequency enhancement to obtain the final frequency-divided sub-image.

[0016] The industrial image anomaly perception system based on multi-frequency fusion, the convolution operation unit: Assume that I is an input industrial image, c = [c1, c2 ... c n ] is the convolution kernel of the convolution operation, c n represents the parameter kernel stride of the nth convolution kernel; the input industrial image I and the convolution kernel c are convolved and downsampled, and the initial frequency-divided image output by the operation is F = [F1, F2, ... F n ], then the preliminary frequency image of the nth branch is expressed as follows:

[0017]

[0018] In the above formula represents the convolution operation, It is a two-dimensional convolution kernel, s represents the number of moving strides of the convolution kernel. The moving stride of each convolution kernel is different. After convolution operations with different strides, down-sampled images of different sizes are obtained.

[0019] The industrial image anomaly perception system based on multi-frequency fusion, the upsampling difference unit: in order to remove relatively low-frequency information in the image and retain high-frequency information, the downsampled image obtained by each branch is subjected to a bilinear upsampling operation to obtain an image of the same size as the adjacent high-frequency image. The formula for the bilinear upsampling operation is as follows:

[0020] x=p*(M-1) / (P-1)y=q*(N-1) / (Q-1)

[0021] f(p,q)=(1-u)*(1-v)*f(x1,y1)+u*(1-v)*f(x2,y2)+(1-u)*v*f(x3,y3)+u*v*f(x4,y4)

[0022] In the above formula, (M, N) represents the size of the original image, (P, Q) represents the size after upsampling, (x, y) represents the position of the pixel in the original image, f(x1, y1), f(x2, y2), f(x3, y3), f(x4, y4) represent the values ​​of the neighboring pixels of the target pixel, f(p, q) represents the value of the target pixel, u = x-x1 and v = y-y1 represent the relative distance between the target pixel and the nearest neighboring pixels, and the upsampled image is formed from f(0, 0) to f(p, q) The obtained upsampled image is subtracted from the adjacent high-frequency image to obtain the difference image f n , the difference formula is as follows:

[0023]

[0024] The industrial image anomaly perception system based on multi-frequency fusion, the residual enhancement unit: in order to compensate for the information loss caused by frequency division, a residual network is integrated to enhance the divided image, and the fusion of residual information and original information helps each branch to selectively fuse the corresponding frequency components at different stages, thereby significantly improving the frequency information representation capability of each branch, and gradually using the enhanced low-frequency image and high-frequency image to splice together to enhance the high-frequency image. The formula for residual enhancement frequency is as follows:

[0025]

[0026]

[0027]

[0028] In the above formula, Res(·) represents the frequency enhancement unit, and the differenced image is subjected to a residual neural network for frequency enhancement. Con(·) represents the concatenation of two enhanced frequency images. This is a multi-frequency branch image after frequency enhancement; since it is difficult to enhance high-frequency branch images, we will use a gradual approach to enhance high-frequency branch images from low to high. We gradually strengthen the representation of low-frequency information and use it as a guide to help enhance high-frequency components. By connecting different frequency components, we can better capture the relationship between them, thereby improving the overall image enhancement effect, and can effectively process the details of high-frequency features, making the enhanced image clearer and richer.

[0029] The industrial image anomaly perception system based on multi-frequency fusion is based on an image generation module of a generative adversarial network: the image generation module includes a generator unit fused with a residual block and a discriminator unit, generates frequency images of different branches, evaluates the authenticity of the image, and outputs a generation result;

[0030] The industrial image anomaly perception system based on multi-frequency fusion, the generator unit of the image generation module based on the generative adversarial network: the generator G has two inputs, one is the image of different frequencies after frequency decomposition, and the other is the random noise vector Z∈R sampled by Gaussian distribution Z ~N(0,1), the noise vector is first sent to a fully connected layer, and then sent to the convolution module and upsampling module together with the branch images of different frequencies to perform upsampling operations on the image features for extracting image features; the network backbone of the generator G adopts an upsampling block, which specifically includes a convolution layer, a residual block, a self-attention layer, and an upsampling layer; the self-attention layer makes the generator G more nonlinear, enabling the generator to generate more realistic images. The formula of the generator is as follows:

[0031]

[0032] The above formula G(·) is the operation corresponding to the backbone of the generator network, I G It is a synthetic image generated by the generator that integrates multi-frequency branch information.

[0033] The industrial image anomaly perception system based on multi-frequency fusion, the discriminator unit of the image generation module based on the generative adversarial network: the discriminator D has two inputs, one is the real image of the multi-frequency branch, and the other is the synthetic image I generated by the generator that fuses the multi-frequency branches G , the discriminator D hopes to distinguish the synthetic image I G and real images To help the generator G generate more realistic images, the discriminator D will generate the generated image I G Converted into a feature map, and downsampled the input feature map through a series of downsampling blocks. The last layer of discriminators usually obtains information from a larger image area and guides the generator to generate images with better global consistency. At the same time, in order to improve the generalization ability of the discriminator, a regularization term is added to make the discriminator perform a zero-centered gradient penalty on the input. The zero-centered gradient penalty function formula is as follows: (loss function formula)

[0034]

[0035] The industrial image anomaly perception system based on multi-frequency fusion, the anomaly detection and classification module: includes an encoder, a decoder including an encoding and decoding feature fusion module (EDFF), a feature fusion module including a weighted feature fusion module (WFF), and a classifier integrating EDFF and WFF; the generated multi-frequency branch image is used as the input of the classification model, the important features of the generated image are extracted through the encoder and decoder, and the output is then sent to the feature fusion module to retain important classification information, and finally sent to the classifier for classification;

[0036] The industrial image anomaly perception system based on multi-frequency fusion, the encoder and decoder: each encoder block consists of a convolution block with residual connection and a maximum pooling layer; the input image is first reduced to 1 / 6 of the original channel after 1×1 convolution, and then restored to its original size after 3×3 convolution and residual connection to retain the previous features; the learning ability of the network is enhanced to capture rich image features, and then batch normalization and ReLU activation function are performed, and then maximum pooling is used to halve the spatial dimension of the feature map;

[0037] The output of the encoder is input into the decoder. Each decoder consists of a transposed convolution layer, an EDFF module, and a residual convolution block. The transposed convolution layer restores the spatial dimension of the feature map to the size corresponding to the spatial dimension of the encoder and unifies the number of channels of the feature map. The EDFF module pre-integrates the features of the encoder and decoder, and the residual convolution block further learns more critical classification features. The output is classified through the softmax activation function.

[0038] The industrial image anomaly perception system based on multi-frequency fusion, the EDFF module: fuses the output of the corresponding layer of the encoder and the output features of the previous layer of the decoder. E and X D After 1×1 convolution concatenation and 1×1 convolution reshaping, it is multiplied with the input feature and then added to obtain the final output feature X F , the formula of EDFF is as follows:

[0039]

[0040] In the above formula, X E and X D Represented as the features of the encoder and decoder, Conv(·) is a 1×1 convolution, CR(·) is a 1×1 convolution operation on the feature map and reshaped to a size of 1×HW, and RS(·) is a feature map reshaped to 1×H×W; the shallow features of the encoder can provide the decoder features with the texture and detail information required by the decoder, and the deep features of the decoder can provide the encoder features with a higher level of semantic abstraction, thereby effectively reducing the semantic gap between features.

[0041] The industrial image anomaly perception system based on multi-frequency fusion, the feature fusion device: the output of the decoder is input into the feature fusion device, the feature fusion device is composed of a convolution layer with a step size of 2 and a WFF module, the convolution layer with a step size of 2 is used to reduce the feature map, retain important information, and the WFF module is used to guide the fusion of classification features at different levels;

[0042] The industrial image anomaly perception system based on multi-frequency fusion, the WFF module: the module fuses the features of the decoder and the features in the feature fusion device according to weighting, so that the features at different levels guide each other for fusion, and splices the features at different levels after summing and convolution to obtain the weight of feature fusion, and fuses the input features according to the weight. The formula of WFF is as follows:

[0043] X weight =Con(Conv(X S ),X S +X C ,Conv(X C ))

[0044]

[0045] The industrial image anomaly perception system based on multi-frequency fusion, the classifier module: consists of an EDFF module, a WFF module and a global average pooling; the shallow features of the decoder and the more advanced fused features processed by the feature fusion device are processed by EDFF and WFF again, and then the fused features and features at different levels are processed by global average pooling respectively, the processed features are subjected to two full connections, and then classified by the SoftMax activation function.

[0046] The beneficial effects of the present invention are:

[0047] The method of the present invention constructs an industrial image anomaly perception system based on multi-frequency fusion, which is used to realize non-contact measurement in a real industrial environment. The invention is based on the improvement of multi-frequency branches and generative models. In view of the problem of scarce industrial image data samples in actual industrial scenarios, a generative model that integrates multi-frequency branches is used to generate new industrial images to train anomaly detection classification models, thereby improving the detection efficiency and production efficiency of factory production. The results of the present invention can be widely used in multiple scenarios such as quality inspection, equipment monitoring, safety monitoring and data analysis in the industrial field, providing effective support and improvement for industrial production and operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the overall process of the industrial image anomaly perception system based on multi-frequency fusion of the present invention;

[0049] Figure 2 It is a specific flow chart of the multi-frequency fusion module of the present invention;

[0050] Figure 3 It is a specific flow chart of the anomaly detection classification module of the present invention;

[0051] Figure 4 It is a specific flow chart of the encoder and decoder of the anomaly detection classification module of the present invention;

[0052] Figure 5 It is a specific flow chart of the encoding and decoding feature fusion module in the anomaly detection classification module of the present invention;

[0053] Figure 6 It is a specific flow chart of the weighted feature fusion module of the anomaly detection classification module of the present invention;

[0054] Figure 7 Partial images of the data set used by the industrial image anomaly perception system based on multi-frequency fusion of the present invention, divided into texture anomalies and structural anomalies;

[0055] Figure 8 It is the visualization of the anomaly detection and classification results of the industrial image anomaly perception system based on multi-frequency fusion of the present invention;

[0056] Fig. 9 This is a comparison chart of the accuracy of the industrial image anomaly perception system based on multi-frequency fusion and the other two methods;

[0057] Fig.10 The ROC curves of the industrial image anomaly perception system based on multi-frequency fusion and the other two methods; DETAILED DESCRIPTION

[0058] The present invention is described in detail below in conjunction with specific embodiments.

[0059] An industrial image anomaly perception system based on multi-frequency fusion includes a multi-frequency fusion module, an image generation module based on a generative adversarial network, and an anomaly detection and classification module; the overall process diagram is as follows Figure 1 As shown, the multi-frequency fusion module is as follows Figure 2 As shown, including the following:

[0060] A1 uses industrial images provided by industrial production and manufacturing to extract sub-images containing different frequency information from the obtained industrial images. Among them, sub-images of different frequencies contain different types of image information. The lower frequency component sub-image corresponds to the overall structure and background information of the image, while the higher frequency component reflects the details and texture information of the image, thereby obtaining richer image feature information;

[0061] A2 first inputs the extracted industrial image into a convolutional neural network for feature extraction, and decomposes the industrial image into sub-images of different frequencies. The frequency sub-branches are then input into a residual network, and the residual network is used to perform frequency enhancement on the obtained multi-branch sub-images.

[0062] Step A2 combines the fusion module of convolutional neural network and residual network. Figure 2 As shown in the figure, the network uses convolution to downsample the input image. In order to obtain images of different frequencies, different step sizes are selected to perform convolution downsampling on the image, and the low-frequency image is upsampled to the size of the adjacent high-frequency image using bilinear upsampling operation, and the difference is made to retain the information of the high-frequency image. The images of different frequencies are frequency enhanced through the residual network, and the enhanced low-frequency and high-frequency images are gradually spliced ​​together to enhance the high-frequency image, and finally a multi-frequency branch image is obtained:

[0063] The specific steps of step A2 are as follows:

[0064] A21. Perform convolution downsampling operations on the input image with different step amplitudes

[0065] like Figure 2 As shown, assume that I is an input industrial image, c = [c1, c2…c n] is the convolution kernel of the convolution operation, c n represents the parameter kernel stride of the nth convolution kernel; the input industrial image I and the convolution kernel c are convolved and downsampled, and the initial frequency-divided image output by the operation is F = [F1, F2, ... F n ], then the preliminary frequency image of the nth branch is expressed as follows:

[0066]

[0067] In the above formula represents the convolution operation, It is a two-dimensional convolution kernel, s represents the number of moving strides of the convolution kernel. The moving stride of each convolution kernel is different. After convolution operations with different strides, down-sampled images of different sizes are obtained.

[0068] A22. After the convolution downsampling operation, it enters the upsampling difference unit

[0069] In order to remove the relatively low-frequency information in the image and retain the high-frequency information, the downsampled image obtained by each branch is subjected to a bilinear upsampling operation to obtain an image of the same size as the adjacent high-frequency image. The formula for the bilinear upsampling operation is as follows:

[0070] x=p*(M-1) / (P-1)y=q*(N-1) / (Q-1)

[0071] f(p,q)=(1-u)*(1-v)*f(x1,y1)+u*(1-v)*f(x2,y2)+(1-u)*v*f(x3,y3)+u*v*f(x4,y4)

[0072] In the above formula, (M, N) represents the size of the original image, (P, Q) represents the size after upsampling, (x, y) represents the position of the pixel in the original image, f(x1, y1), f(x2, y2), f(x3, y3), f(x4, y4) represent the values ​​of the neighboring pixels of the target pixel, f(p, q) represents the value of the target pixel, u = x-x1 and v = y-y1 represent the relative distance between the target pixel and the nearest neighboring pixels, and the upsampled image is formed from f(0, 0) to f(p, q) ; Subtract the upsampled image from the adjacent high-frequency image to obtain the difference image f n , the difference formula is as follows:

[0073]

[0074] A23. Input the differenced image into the frequency enhancement unit

[0075] In order to compensate for the information loss caused by frequency division, a residual network is integrated to enhance the divided image. By fusing the residual information with the original information, it helps each branch to selectively fuse the corresponding frequency components at different stages, thereby significantly improving the frequency information representation ability of each branch, and gradually using the enhanced low-frequency image to splice with the high-frequency image to enhance the high-frequency image. The formula for residual enhancement frequency is as follows:

[0076]

[0077]

[0078]

[0079] In the above formula, Res(·) represents the frequency enhancement unit, and the differenced image is subjected to a residual neural network for frequency enhancement. Con(·) represents the concatenation of two enhanced frequency images. This is a multi-frequency branch image after frequency enhancement; since it is difficult to enhance high-frequency branch images, we will use a gradual approach to enhance high-frequency branch images from low to high. We gradually strengthen the representation of low-frequency information and use it as a guide to help enhance high-frequency components. By connecting different frequency components, we can better capture the relationship between them, thereby improving the overall image enhancement effect, and can effectively process the details of high-frequency features, making the enhanced image clearer and richer.

[0080] A3 inputs the multi-frequency branch image into the image generation module

[0081] The image generation module of step A3 includes a generator fused with a residual block and a discriminator, generates frequency images of different branches, evaluates the authenticity of the image, and outputs a generation result;

[0082] The specific steps of step A3 are as follows:

[0083] A31. Input multi-frequency branch image into the generator unit

[0084] The generator G has two inputs, one is the image of different frequencies after frequency decomposition, and the other is the random noise vector Z∈R sampled from the Gaussian distribution Z~N(0,1), the noise vector is first sent to a fully connected layer, and then sent to the convolution module and upsampling module together with the branch images of different frequencies to perform upsampling operations on the image features for extracting image features; the network backbone of the generator G adopts an upsampling block, which specifically includes a convolution layer, a residual block, a self-attention layer, and an upsampling layer; the self-attention layer makes the generator G more nonlinear, enabling the generator to generate more realistic images. The formula of the generator is as follows:

[0085]

[0086] The above formula G(·) is the operation corresponding to the backbone of the generator network, I G It is a synthetic image generated by the generator that integrates multi-frequency branch information.

[0087] A32. Input generated images and real images into the discriminator unit

[0088] The input of the discriminator D is two: one is the real image of the multi-frequency branch, and the other is the synthetic image I generated by the generator that integrates the multi-frequency branches. G , the discriminator D hopes to distinguish the synthetic image I G and real images To help the generator G generate more realistic images, the discriminator D will generate the generated image I G Converted into a feature map, and downsampled the input feature map through a series of downsampling blocks. The last layer of discriminators usually obtains information from a larger image area and guides the generator to generate images with better global consistency. At the same time, in order to improve the generalization ability of the discriminator, a regularization term is added to make the discriminator perform a zero-centered gradient penalty on the input. The zero-centered gradient penalty function formula is as follows: (loss function formula)

[0089]

[0090] A4 inputs the generated image into the anomaly detection classification module

[0091] The overall flow chart of the anomaly detection and classification module in step A4 is as follows: Figure 3 As shown, it includes an encoder, a decoder including an encoding and decoding feature fusion module (EDFF), a feature fusion module including a weighted feature fusion module (WFF), and a classifier integrating EDFF and WFF; the generated multi-frequency branch image is used as the input of the classification model, the important features of the generated image are extracted through the encoder and decoder, and the output is then sent to the feature fusion module to retain important classification information, and finally sent to the classifier for classification;

[0092] The specific steps of step A4 are as follows:

[0093] A41. Input image into encoder and decoder

[0094] Encoders and decoders such as Figure 4 As shown in the figure, each encoder block consists of a convolution block with residual connection and a maximum pooling layer; the input image is first reduced to 1 / 6 of the original channel after 1×1 convolution, and then restored to its original size after 3×3 convolution and residual connection to retain the previous features; the learning ability of the network is enhanced to capture rich image features, and then batch normalization and ReLU activation function are performed, and then maximum pooling is used to halve the spatial dimension of the feature map;

[0095] The output of the encoder is input into the decoder. Each decoder consists of a transposed convolution layer, an EDFF module, and a residual convolution block. The transposed convolution layer restores the spatial dimension of the feature map to the size corresponding to the spatial dimension of the encoder and unifies the number of channels of the feature map. The EDFF module pre-integrates the features of the encoder and decoder, and the residual convolution block further learns more critical classification features. The output is classified through the softmax activation function.

[0096] EDFF modules such as Figure 5 As shown, the output of the encoder corresponding layer and the output features of the previous layer of the decoder are fused. E and X D After 1×1 convolution concatenation and 1×1 convolution reshaping, it is multiplied with the input feature and then added to obtain the final output feature X F , the formula of EDFF is as follows:

[0097]

[0098] In the above formula, X E and X D Represented as the features of the encoder and decoder, Conv(·) is a 1×1 convolution, CR(·) is a 1×1 convolution operation on the feature map and reshaped to a size of 1×HW, and RS(·) is a feature map reshaped to 1×H×W; the shallow features of the encoder can provide the decoder features with the texture and detail information required by the decoder, and the deep features of the decoder can provide the encoder features with a higher level of semantic abstraction, thereby effectively reducing the semantic gap between features.

[0099] A42. Input the extracted feature map into the feature fusion device

[0100] Feature fuser such as Figure 3As shown in the Feature Fusion module in , the output of the decoder is input into the feature fusion device, which consists of a convolutional layer with a step size of 2 and a WFF module. The convolutional layer with a step size of 2 is used to reduce the feature map and retain important information. The WFF module is used to guide the fusion of classification features at different levels.

[0101] WFF modules such as Figure 6 As shown in the figure, the WFF module fuses the features of the decoder and the features in the feature fuser according to the weights, so that the features at different levels guide each other for fusion. The features at different levels are concatenated after summation and convolution to obtain the weight of feature fusion, and the input features are fused according to the weights. The formula of WFF is as follows:

[0102]

[0103]

[0104] A43. Input the feature map after feature fusion into the classifier module

[0105] Classifier modules such as Figure 3 As shown in the Classifier module in the figure, the classifier consists of an EDFF module, a WFF module and a global average pooling module. The shallow features of the decoder and the more advanced fused features processed by the feature fuser are processed by EDFF and WFF again, and then the fused features and features at different levels are processed by global average pooling respectively. The processed features are subjected to two full connections and then classified by the SoftMax activation function.

[0106] Figure 8 To visualize the anomaly detection and classification results of the industrial image anomaly perception system based on multi-frequency fusion in the present invention, we present the results of several common texture anomaly classifications in the form of a heat map, with the normal part in blue and the red and yellow parts indicating the location of the anomaly.

[0107] Fig. 9 The following is a comparison chart of the accuracy of the industrial image anomaly perception system based on multi-frequency fusion and the other two ablation methods. Method 1 uses an anomaly perception system that does not use the multi-frequency fusion module to generate images and detect and classify anomalies; Method 2 uses an anomaly perception system that does not use the feature fusion device in the anomaly detection and classification module to generate images and detect and classify anomalies.

[0108] Fig.10 The ROC curve diagram of the industrial image anomaly perception system based on multi-frequency fusion and the other two methods.

[0109] Method 1 and Method 2 are described in the same Fig. 9.

[0110] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. An industrial image anomaly perception system based on multi-frequency fusion, characterized in that: Including multi-frequency fusion module, image generation module based on generative adversarial network and anomaly detection and classification module: Multi-frequency fusion module: For industrial image datasets provided in industrial production and manufacturing, multi-frequency branches are used to extract sub-images of different frequencies from industrial images. The frequency energy distribution of the image contains different types of image information. The lower frequency component image contains the overall structure and background information of the image, while the higher frequency component reflects the details and texture information of the image, thereby obtaining richer image feature information; The multi-frequency fusion module includes a convolutional neural network and a residual network. First, the extracted industrial image is input into the convolutional neural network for feature extraction, and the industrial image is decomposed into sub-images of different frequencies. The frequency sub-branches are then input into the residual network, and the residual network is used to perform frequency enhancement on the obtained multi-branch sub-images. The multi-frequency fusion module includes: Convolution operation unit: perform different convolution operations on the industrial image input to the convolution layer, and perform downsampling operations on the industrial image input to the convolution layer with different steps; the convolution operation unit: assume that I is an input industrial image, c = [c1, c2…c n ] is the convolution kernel of the convolution operation, c n represents the parameter kernel stride of the nth convolution kernel; the input industrial image I and the convolution kernel c are convolved and downsampled, and the initial frequency-divided image output by the operation is F = [F1, F2, ... F n ], then the preliminary frequency image of the nth branch is expressed as follows: In the above formula represents the convolution operation, It is a two-dimensional convolution kernel, s represents the number of movement strides of the convolution kernel, and each convolution kernel has a different movement stride. After convolution operations with different strides, downsampled images of different sizes are obtained; Upsampling difference unit: upsamples the sub-image obtained after convolution to an image of the same size as the previous frequency and makes a difference; the upsampling difference unit: in order to remove the relatively low-frequency information in the image and retain the high-frequency information, the downsampled image obtained by each branch is subjected to a bilinear upsampling operation to obtain an image of the same size as the adjacent high-frequency image The formula for the bilinear upsampling operation is as follows: x=p*(M-1) / (P-1)y=q*(N-1) / (Q-1) f(p,q)=(1-u)*(1-v)*f(x1,y1)+u*(1-v)*f(x2,y2) +(1-u)*v*f(x3,y3)+u*v*f(x4,y4) In the above formula, (M, N) represents the size of the original image, (P, Q) represents the size after upsampling, (x, y) represents the position of the pixel in the original image, f(x1, y1), f(x2, y2), f(x3, y3), f(x4, y4) represent the values ​​of the neighboring pixels of the target pixel, f(p, q) represents the value of the target pixel, u = x-x1 and v = y-y1 represent the relative distance between the target pixel and the nearest neighboring pixels, and the upsampled image is formed from f(0, 0) to f(p, q) The obtained upsampled image is subtracted from the adjacent high-frequency image to obtain the difference image f n , the difference formula is as follows: Residual enhancement unit: A residual network is added to compensate for the information loss caused by frequency division; the residual enhancement unit: in order to compensate for the information loss caused by frequency division, a residual network is integrated to enhance the frequency-divided image, and the fusion of residual information and original information helps each branch to selectively fuse the corresponding frequency components at different stages, thereby significantly improving the frequency information representation capability of each branch, and gradually using the enhanced low-frequency image to splice with the high-frequency image to enhance the high-frequency image. The formula of residual enhancement frequency is as follows: In the above formula, Res(·) represents the frequency enhancement unit, and the differenced image is subjected to a residual neural network for frequency enhancement. Con(·) represents the concatenation of two enhanced frequency images. It is a multi-frequency branch image after frequency enhancement. Since it is difficult to enhance high-frequency branch images, a gradual approach is adopted to enhance high-frequency branch images from low to high, gradually strengthening the representation ability of low-frequency information, and using it as a guide to help enhance high-frequency components. By connecting different frequency components, the relationship between them can be better captured, thereby improving the overall image enhancement effect, and the details of high-frequency features can be effectively processed, making the enhanced image clearer and richer. Summation unit: concatenates the enhanced frequency-separated image with its adjacent high-frequency image and then performs another enhancement; The image generation module based on the generative adversarial network includes a generator unit fused with a residual block and a discriminator unit, the generator unit generates frequency images of different branches, and the discriminator unit evaluates the authenticity of the image and outputs a generation result; The generator unit: The generator G has two inputs, one is the image of different frequencies after frequency decomposition, and the other is the random noise vector Z∈R sampled from the Gaussian distribution Z ~N(0,1), the noise vector is first sent to a fully connected layer, and then sent to the convolution module and upsampling module together with the branch images of different frequencies to perform upsampling operations on the image features for extracting image features; the network backbone of the generator G adopts an upsampling block, which specifically includes a convolution layer, a residual block, a self-attention layer, and an upsampling layer; the self-attention layer makes the generator G more nonlinear, enabling the generator to generate more realistic images. The formula of the generator is as follows: The above formula G(·) is the operation corresponding to the backbone of the generator network, I G It is a synthetic image generated by the generator that integrates multi-frequency branch information; The discriminator unit: The discriminator D has two inputs, one is the real image of the multi-frequency branch, and the other is the synthetic image I generated by the generator that integrates the multi-frequency branches. G , the discriminator D hopes to distinguish the synthetic image I G and real images To help the generator G generate more realistic images, the discriminator D will generate the generated image I G Converted into a feature map, and downsampled the input feature map through a series of downsampling blocks. The last layer of the discriminator obtains information from a larger image area and guides the generator to generate an image with better global consistency. At the same time, in order to improve the generalization ability of the discriminator, a regularization term is added to make the discriminator perform a zero-centered gradient penalty on the input. The zero-centered gradient penalty function formula is as follows: The anomaly detection classification module includes an encoder, a decoder including an encoding and decoding feature fusion module EDFF, a feature fusion module including a weighted feature fusion module WFF, and a classifier integrating EDFF and WFF; the generated multi-frequency branch image is used as the input of the classification model, the important features of the generated image are extracted through the encoder and decoder, and the output is then sent to the feature fusion module to retain important classification information, and finally sent to the classifier for classification; The encoder and decoder described: Each encoder block consists of a convolution block with a residual connection and a maximum pooling layer; the input image is first reduced to 1 / 6 of the original channel after a 1×1 convolution, and then restored to its original size after a 3×3 convolution and residual connection to retain the previous features; the learning ability of the network is enhanced to capture rich image features, and then batch normalization and ReLU activation function are performed, and then maximum pooling is used to halve the spatial dimension of the feature map; The output of the encoder is input into the decoder. Each decoder consists of a transposed convolution layer, an EDFF module, and a residual convolution block. The transposed convolution layer restores the spatial dimension of the feature map to the size corresponding to the spatial dimension of the encoder and unifies the number of channels of the feature map. The EDFF module pre-integrates the features of the encoder and decoder, and the residual convolution block further learns more critical classification features. The output is classified through the softmax activation function. The EDFF module described above: combines the output of the corresponding layer of the encoder with the output features of the previous layer of the decoder; combines the feature X of size C×W×H E and X D After 1×1 convolution concatenation and 1×1 convolution reshaping, it is multiplied with the input feature and then added to obtain the final output feature X f , the formula of EDFF is as follows: In the above formula, X E and X D Represents the features of the encoder and decoder, Conv(·) is a 1×1 convolution, CR(·) is a 1×1 convolution operation on the feature map and reshapes it to the size of 1×HW, R S (·) Reshape the feature map into 1×H×W; the shallow features of the encoder provide the texture and detail information required by the decoder, and the deep features of the decoder provide a higher level of semantic abstraction for the encoder features, thereby effectively reducing the semantic gap between features; The feature fusion device: the output of the decoder is input into the feature fusion device, the feature fusion device is composed of a convolution layer with a step size of 2 and a WFF module, the convolution layer with a step size of 2 is used to reduce the feature map and retain important information, and the WFF module is used to guide the fusion of classification features at different levels; The WFF module described above: fuses the features of the decoder and the features in the feature fuser according to weights, so that features at different levels guide each other for fusion, and splices the features at different levels after summing and convolution to obtain the weight of feature fusion, and fuses the input features according to the weights. The formula of WFF is as follows: X weight =Con(Conv(X S ),X S +X C ,Conv(X C ))

Images