Defect image enhancement method based on generative adversarial network
By using the defect image enhancement method of deep transfer learning and attention mechanism in defect detection, the problem of insufficient data sets in defect detection is solved, the detection accuracy and computing efficiency are improved, and data set expansion and model optimization are realized.
Patent Information
- Application Number
- CN202411190366.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art is difficult to construct large-scale defect image data sets in defect detection, which makes it difficult for deep learning methods to improve detection accuracy, and the number of defect detection data sets is limited, so the advantages of deep learning methods cannot be fully utilized.
Using a defect image enhancement method based on deep transfer learning, using an Efficient GAN network pre-trained on the BeanTech industrial defect detection dataset, the residual connection and attention mechanism are introduced, and the full connection layer in the replacement generator and discriminator is 1* 1's convolutional layer, add attention mechanism module, and build defective image enhancement model for data expansion.
Effectively expanding defect image datasets improves the accuracy of defect detection, reduces model complexity and improves computing efficiency, and solves the problem that deep learning networks cannot be trained due to the lack of large datasets.
Smart Images

Figure CN119090768A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of defect image detection, and in particular is a defect image enhancement method based on a generative adversarial network. Background Art
[0002] Defective image detection is a computer vision technology designed to help computers automatically identify and locate defects, anomalies or flaws in images. Defective image detection has a very wide range of application scenarios, such as judging the patient's condition based on medical images in medical fields, and performing quality inspections and defect detection on parts and industrial products in manufacturing.
[0003] Traditional defect detection methods usually include steps such as image preprocessing and feature extraction, defect location and classification. Image preprocessing is used to remove noise, enhance contrast and other operations; feature extraction aims to extract representative features from the image for subsequent location and classification tasks; and defect location and classification is to match the extracted features with known defect types to achieve the location and classification of defects in the image.
[0004] With the development of deep learning, defect detection methods based on deep neural networks have gradually attracted attention. For example, defect detection methods based on convolutional neural networks (CNNs), defect detection methods based on generative adversarial networks, and defect detection methods based on target detection and image segmentation. However, there are still certain difficulties in using deep learning technology to achieve anomaly detection, because the number of abnormal images is usually much smaller than the number of normal images, and it is difficult to build a large-scale defect image dataset in many real-world scenarios. Therefore, it is difficult for traditional deep learning methods to play their advantages, and it is also difficult to further improve the accuracy, resulting in the limited number of defect detection image datasets, which cannot give full play to the advantages of deep learning methods for effective expansion. Summary of the invention
[0005] In order to improve the accuracy of defect detection and give full play to the role of generative adversarial networks in the field of defect detection, this paper proposes a defect image enhancement method based on deep transfer learning. Using the Efficient GAN network pre-trained on the BeanTech industrial defect detection dataset as the basic algorithm, residual connections are introduced in the encoder and discriminator networks respectively, and then the six fully connected layers in the generator and discriminator are replaced with 1*1 convolutional layers. Finally, attention mechanism modules are added between layer3 and layer4 of the generator and discriminator, and between layer4 and last, as a defect image enhancement model to perform data expansion on defect images or defect image datasets.
[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a defect image enhancement method based on a generative adversarial network, which performs the following steps to achieve data expansion and enhancement, including the following steps:
[0007] S1, collect industrial defect images for preprocessing and annotation, and build a defect image dataset;
[0008] S2. Based on the improved Efficient GAN algorithm, a defect image enhancement model is constructed and trained to obtain an ideal model, which can expand and enhance the input image;
[0009] S3. Input the image to be processed into the ideal model of defect image enhancement to obtain an enhanced image.
[0010] The preprocessing includes trimming to a uniform size, removing noise, and enhancing contrast; the labeling is labeling the defective image according to the defect information corresponding to the image to indicate whether there are defects.
[0011] The step S2 comprises the following steps:
[0012] S2.1. Construct defect image enhancement model based on Efficient GAN network;
[0013] S2.2, use the BeanTech industrial defect detection dataset to first train the network until it converges;
[0014] S2.3. Use the constructed defect detection image dataset as the input data of the model and the label data as the output data of the model to perform a second learning iteration training on the defect image enhancement model, and reversely adjust the network model parameters to obtain an ideal defect image enhancement model, so that the ideal model can expand and enhance the input image.
[0015] The defect image enhancement model includes an encoder, a generator, and a discriminator:
[0016] The encoder receives the image data x and extracts its latent variable features E(x); it also generates noise z;
[0017] Generator, image G(z) generated based on noise z;
[0018] The discriminator receives the concatenated tensors (x, E(x) and (z, G(z)) and determines whether the pixel error between G(z) and E(x) meets the threshold range requirement. If not, it calls back the network parameters of the encoder and generator to continue iterative learning until the threshold requirement is met, and an ideal model is obtained, and a picture that meets the requirements is output as the enhanced image data.
[0019] The encoder includes a modified inverted bottleneck residual structure and convolutional layers connected in sequence.
[0020] The generator includes two convolutional layers and three transposed convolutional layers connected in series, and an attention mechanism module is inserted between three adjacent transposed convolutional layers.
[0021] The discriminator includes five inverted bottleneck residual structures and three convolutional layers connected in series, and an attention mechanism module is inserted between the third, fourth and fifth inverted bottleneck residual structures.
[0022] The inverted bottleneck residual structure is:
[0023] The feature map f is passed through the first convolutional layer and the first BatchNorm2d layer to obtain the feature map f1;
[0024] The feature map f1 is processed by the second convolutional layer and the second BatchNorm2d layer to obtain the feature map f2; the number of channels in the second convolutional layer is four times that of the first convolutional layer;
[0025] The feature map f2 is processed by the third convolutional layer and the third BatchNorm2d layer to obtain the feature map f3; the number of channels in the third convolutional layer is equal to that in the first convolutional layer;
[0026] The feature maps f and f3 are concatenated and passed into the next residual structure after being transformed by the GeLU function.
[0027] The added attention mechanism module is the ParNet-Attention module.
[0028] A defect image enhancement system based on a generative adversarial network includes a control background and a front-end interface; the control background is provided with a memory and a processor; the memory is used to store a computer program; the processor is used to implement the method as described above when executing the computer program to achieve data expansion and enhancement; the front-end interface is used to input instructions, image data and model parameters, and visually display the enhanced image data.
[0029] The present invention has the following beneficial effects and advantages:
[0030] 1. Based on transfer learning technology, the present invention uses the Efficient GAN pre-trained on the BeanTech industrial defect detection dataset as the basic algorithm, and uses the channel attention mechanism to retrain the defect detection dataset. It can effectively expand the defect images that need to be enhanced or the defect image dataset, and solves the problem that the deep learning network cannot be trained and the advantages of the deep learning network cannot be brought into play due to the lack of large defect detection datasets in actual scene applications, thereby improving the accuracy of defect detection.
[0031] 2. The present invention replaces the six fully connected layers of the Efficient GAN generator, discriminator and encoder with 1*1 convolutional layers, introduces residual connections in the generator and encoder network structures, and adds attention mechanism modules between layer3 and layer4 of the generator and discriminator, and between layer4 and last layer, thereby reducing the model complexity and improving the overall computational efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a flow chart of the method of the present invention;
[0033] Figure 2 Schematic diagram of the channel attention mechanism module structure;
[0034] Figure 3 Schematic diagram of the improved inverted bottleneck residual connection structure;
[0035] Figure 4 Schematic diagram of the defect image enhancement model structure. DETAILED DESCRIPTION
[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.
[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0038] like Figure 1 As shown, a defect image enhancement method based on a generative adversarial network includes the following steps:
[0039] Step 1: Collect defect image data of the product.
[0040] The data of the present invention comes from a workpiece production line, and the data includes the collected defect image data and defect information of a product, such as whether the current image has defects.
[0041] Step 2: Data preprocessing: Clean the collected defect image data and defect information, delete the data with serious missing or contaminated images, and then preprocess the data.
[0042] The specific operations are: trimming to a uniform size, removing noise, enhancing contrast, etc. The uniform size of the acquired defect image information is scaled to 224*224 pixels.
[0043] Step 3: Data labeling: Label the image data processed in step 2, that is, use the defect information to manually label the defect image as the label of the defect image to create a defect image enhancement dataset.
[0044] Step 4: Make the labeled data into a data set. Divide the data into training set, validation set and test set in a ratio of 3:1:1, and put them into three folders: train, val and test respectively. Put these three folders into the same directory. The training set is used for training, the validation set is used to verify the model accuracy and the test set is used for testing.
[0045] Step 5: Build a defect image enhancement model based on the Efficient GAN network; replace the Efficient GAN encoder and discriminator network with a residual connection structure using an improved inverted bottleneck, replace the six fully connected layers of the Efficient GAN network with 1*1 convolutional layers, and add an attention mechanism module between the layer3 and layer4 layers of the generator and the discriminator, and between the layer4 and the last layer.
[0046] Specifically, the defect image enhancement model includes an encoder, a generator, and a discriminator: the encoder receives image data x and extracts its latent variable feature E(x); generates noise z at the same time; the generator generates a picture G(z) according to the noise z; the discriminator receives the spliced tensor (x, E(x) and (z, G(z)), and determines whether the pixel error between G(z) and E(x) meets the threshold range requirement; if not, the network parameters of the encoder and the generator are reversely called back to continue iterative learning until the threshold requirement is met, an ideal model is obtained, and a picture that meets the requirements is output as enhanced image data. Among them, the encoder includes an improved inverted bottleneck residual structure and a convolution layer connected in sequence. The generator includes two convolution layers and three transposed convolution layers connected in series in sequence, and an attention mechanism module is inserted between three adjacent transposed convolution layers. The discriminator includes five inverted bottleneck residual structures and three full convolution layers connected in series in sequence, and an attention mechanism module is inserted between the third, fourth, and fifth inverted bottleneck residual structures.
[0047] Step 6: Use the defect image enhancement model trained on the BeanTech dataset.
[0048] Step 7: Retrain using the defect image dataset and use the best performing model on the validation set as the image enhancement model.
[0049] Step 8: For the defect image that needs to be enhanced or the test image data in the defect image dataset sent by the user, first resize it to 224*224, and then pass it into the image enhancement model. After processing, the image enhancement model outputs the enhanced defect image data.
[0050] like Figure 2 As shown, the channel attention mechanism module is the ParNet-Attention module, and the specific process is as follows:
[0051] Step 1: The input feature map x undergoes a 1*1 convolution operation and performs a BatchNorm2d normalization operation to obtain a new feature map x1.
[0052] Step 2: The feature map x passes through an SSE attention mechanism layer to obtain a new feature map x2.
[0053] Step 3: The input feature map x is subjected to a 3*3 convolution operation and a BatchNorm2d normalization operation to obtain a new feature map x3.
[0054] Step 4: Steps 1, 2, and 3 obtain feature maps x1, x2, and x3, perform arithmetic addition operations, and then map them through the SiLU function to obtain a new feature map Same size as x, both are H*W*C.
[0055] like Figure 3 As shown, the specific process of the improved inverted bottleneck residual connection is:
[0056] Step 1: The input feature map f undergoes a 3*3 convolution operation and performs BatchNorm2d normalization operation to obtain a new feature map f1.
[0057] Step 2: After the feature map f1 undergoes a 1*1 convolution operation and performs a BatchNorm2d normalization operation, a new feature map f2 is obtained. The number of channels in this 1*1 convolution layer is 4 times that of the previous 3*3 convolution layer.
[0058] Step 3: The feature map f2 is subjected to a 1*1 convolution operation again and a BatchNorm2d normalization operation is performed to obtain a new feature map f3. The number of channels in the 1*1 convolution layer is equal to that in the first 3*3 convolution layer.
[0059] Step 4: Concatenate the feature map f and the new feature map f3 and transform them through the GeLU function before passing them into the calculation module.
[0060] like Figure 4 As shown, the specific process of the defect image enhancement model is as follows:
[0061] Step 1: The input image data x is passed into the encoder, and the encoder obtains the hidden variable E(x) of the input data x.
[0062] Step 2: The encoder generates noise z and passes it into the generator to obtain the generated image G(z).
[0063] Step 3: Concatenate x with E(x) and z with G(z) and pass them into the discriminator.
[0064] Step 4: The discriminator determines whether the concatenated tensor (x, E(x) and (z, G(z)) comes from the encoder or the generator. The discriminator determines this by comparing the error between E(x) and G(z) based on the pixel threshold range.
[0065] Step 5: The image that meets the requirements output by the discriminator is used as the enhanced image data.
[0066] Step 4: Build an image generation network and retrain it on the defect image dataset. The model parameters that perform best on the test set are used as the final ideal image generation model.
[0067] Step 5: Pass the collected defect image data into the generative adversarial network model to obtain an enhanced defect image dataset.
[0068] The above is a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A defect image enhancement method based on generative adversarial network, characterized in that: Perform the following steps to implement data augmentation and enhancement, including the following steps: S1, collect industrial defect images for preprocessing and annotation, and build a defect image dataset; S2. Build and train a defect image enhancement model based on the improved Efficient GAN algorithm to obtain an ideal model that can expand and enhance the input image; S3. Input the image to be processed into the ideal model of defect image enhancement to obtain an enhanced image.
2. The defect image enhancement method based on generative adversarial network according to claim 1, characterized in that: The preprocessing includes trimming to a uniform size, removing noise, and enhancing contrast; The labeling is to label the defect image according to the defect information corresponding to the image to indicate whether there is a defect.
3. The defect image enhancement method based on generative adversarial network according to claim 1, characterized in that: The step S2 comprises the following steps: S2.
1. Construct defect image enhancement model based on Efficient GAN network; S2.2, use the BeanTech industrial defect detection dataset to first train the network until it converges; S2.
3. Use the constructed defect detection image dataset as the input data of the model and the label data as the output data of the model to perform a second learning iteration training on the defect image enhancement model, and reversely adjust the network model parameters to obtain an ideal defect image enhancement model, so that the ideal model can expand and enhance the input image.
4. A defect image enhancement method based on generative adversarial network according to claim 1 or 3, characterized in that: The defect image enhancement model includes an encoder, a generator, and a discriminator: The encoder receives the image data x and extracts its latent variable features E(x); it also generates noise z; Generator, image G(z) generated based on noise z; The discriminator receives the concatenated tensors (x, E(x) and (z, G(z)) and determines whether the pixel error between G(z) and E(x) meets the threshold range requirement. If not, it calls back the network parameters of the encoder and generator to continue iterative learning until the threshold requirement is met, and an ideal model is obtained, and a picture that meets the requirements is output as the enhanced image data.
5. The defect image enhancement method based on generative adversarial network according to claim 4, characterized in that: The encoder includes a modified inverted bottleneck residual structure and convolutional layers connected in sequence.
6. The defect image enhancement method based on generative adversarial network according to claim 4, characterized in that: The generator includes two convolutional layers and three transposed convolutional layers connected in series, and an attention mechanism module is inserted between three adjacent transposed convolutional layers.
7. The defect image enhancement method based on generative adversarial network according to claim 4, characterized in that: The discriminator includes five inverted bottleneck residual structures and three convolutional layers connected in series, and an attention mechanism module is inserted between the third, fourth and fifth inverted bottleneck residual structures.
8. The defect image enhancement method based on generative adversarial network according to claim 5 or 7, characterized in that: The inverted bottleneck residual structure is: The feature map f is passed through the first convolutional layer and the first BatchNorm2d layer to obtain the feature map f1; The feature map f1 is processed by the second convolutional layer and the second BatchNorm2d layer to obtain the feature map f2; the number of channels in the second convolutional layer is four times that of the first convolutional layer; The feature map f2 is processed by the third convolutional layer and the third BatchNorm2d layer to obtain the feature map f3; the number of channels in the third convolutional layer is equal to that in the first convolutional layer; The feature maps f and f3 are concatenated and passed into the next residual structure after being transformed by the GeLU function.
9. The defect image enhancement method based on generative adversarial network according to claim 6 or 7, characterized in that: The added attention mechanism module is the ParNet-Attention module.
10. A defect image enhancement system based on generative adversarial network, characterized in that: It includes a control background and a front-end interface; the control background is provided with a memory and a processor; the memory is used to store a computer program; the processor is used to implement the method as described in any one of claims 1 to 9 when executing the computer program, so as to achieve data expansion and enhancement; the front-end interface is used to input instructions, image data and model parameters, and to visually display the enhanced image data.
Citation Information
Cited By
Adversarial generative network enhanced recognition method for electrical equipment defect image
CN120580445A
