Defect image generation method and system fusing attention mechanism and perception loss

Abstract

The application discloses a defect image generation method fusing attention mechanism and perception loss, and belongs to the technical field of screen detection. The method comprises the following steps: S1, acquiring a real image; S2, constructing a SWG-VGG model based on a WGAN-GP model and training the SWG-VGG model, wherein the SWG-VGG model comprises a generator, a discriminator and a VGG-19 network; the VGG-19 network is used to extract high-level features of a generated image and a real image during training, so as to calculate the difference between the features, finally, the feature differences of different layers are weighted and summed to obtain a final perception loss, and the final perception loss is fed back to the generator; and S3, inputting the real image into the trained SWG-VGG model to obtain a generated image. The data set generated by the SWG-VGG model reaches 0.976 on mAP (50), and is improved by 10.5%; the PSNR, SSIM, RMSE and CS values of the generated image also show high image quality.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of screen detection, and particularly relates to a defect image generation method and system fusing an attention mechanism and a perception loss. BACKGROUND

[0002] With the increasingly comprehensive penetration of intelligent technology in modern production and life, mobile devices, wearable devices and various smart home products are being used more and more widely. Small-size liquid crystal display screens, as an indispensable interactive interface, are in increasing demand. Due to the inevitable production of various defects in the manufacturing process of liquid crystal screens, such as Figure 1 point defects, line defects, liquid leakage and scratches, etc. Therefore, liquid crystal screens must be strictly quality inspected before being packaged and marketed. At present, artificial quality inspection is mainly relied on, which is not only time-consuming, labor-intensive and costly, but also cannot meet the real-time detection needs of the continuously upgraded intelligent production line, which has become a key bottleneck restricting production capacity. With the rapid development of machine vision, target detection technology based on machine vision has become a key technology for intelligent upgrading in the defect detection field, such as intelligent detection methods based on YOLO and Faster R-CNN. A diversified balanced defect dataset is a necessary data basis to ensure the detection accuracy, robustness and real-time performance of the model. As shown in Figure 2 , point defects and scratch defects are very common, and the number of line defects and liquid leakage samples is very small, and the dataset is extremely unbalanced. The reasons for this situation are very complex, and optimization of data collection methods cannot improve this situation. Therefore, we try to obtain an effective balanced defect image dataset through intelligent data enhancement technology to create an effective data basis for intelligent detection of liquid crystal screens.

[0003] A large number of studies have shown that, in order to solve the problem of unbalanced defect dataset, using data enhancement technology can significantly improve the performance of the defect detection model. There are currently two types of data enhancement methods:

[0004] (1) Traditional data enhancement, i.e. geometric transformation, adjusting brightness and contrast, adding noise, etc. to the original defect image, which retains the basic features and details of the original image, but the generated image is limited by the existing image and can only overcome the problem of unbalanced sample quantity, and cannot solve the imbalance of the sample set in randomness and diversity.

[0005] (2) Intelligent image generation. The core task of image generation is to build an image generation model and adjust the parameters to minimize the divergence of the probability distribution of the generated data and the real data. In the field of industrial defect image generation, the most widely used generators include GANs, VAEs, or DDPMs. GANs can effectively generate high-quality defect images through the adversarial training mechanism of the generator and the discriminator. VAEs combine the advantages of variational inference and deep learning, and can also achieve the purpose of generating defect images through the encoder and decoder reconstruction of the original data. The study generates defect samples through DDPMs and optimizes the generated data through feature editing. The image generation process of the GANs model is simpler, and the model training is faster; and GANs allows the generation of diverse samples through significant space operations to ensure the diversity of the data set.

[0006] However, Figure 2 The data distribution graph of various defects in the display screen manufacturing plant 2024.4.18-2024.4.26 for 8 days directly generates liquid crystal screen defect images using the GANs model, which still has the following challenges: (1) The balance between the generator and the discriminator is difficult to maintain due to the scarcity of defect data set samples, resulting in unstable training. (2) The shape of the liquid leakage defect area is complex and irregular, which leads to insufficient details and accuracy in the generated images by GANs. (3) The original network architecture cannot fully capture the deeper features of the defects, resulting in partial distortion of the generated images.

[0007] GAN has been widely applied in the field of machine vision since its inception in 2014. However, when generating defect images, the gap between the generator and the discriminator is too large, leading to unstable training. To address this issue, researchers have made various improvements: CGAN introduces label information during the generation process, guides image generation through input conditions and labels, improves the quality of generated samples, reduces the gap between the generator and the discriminator, and optimizes the problem of unstable GAN training. However, the generated images by CGAN in complex scenarios still have the problem of blurring. Alec et al. proposed a deep convolutional generative adversarial network (DCGAN) that replaces the traditional GAN's fully connected layer with a fully convolutional structure, significantly improving the model's feature extraction ability for images, laying the foundation for GAN's application in image generation. However, the effect of generating complex textures or structures is still not ideal. Arjovsky et al. constructed WGAN by introducing Wasserstein distance instead of traditional JS divergence or KL divergence, solving the gradient vanishing problem in the GAN training process. However, WGAN uses weight clipping mechanism during training, limiting the model's expression ability. Gulrajani et al. proposed WGAN-GP, which avoids weight clipping by introducing a gradient penalty mechanism, further improving the model's training stability.

[0008] The liquid leakage defect image often has irregular morphology and complex local details, and the WGAN-GP cannot perfectly capture these detail features, and the local details of the generated image lack sufficient precision. In recent years, attention mechanisms have been widely used in various fields and have shown significant effects in machine vision. Vaswani et al. first proposed a self-attention mechanism, which has shown excellent performance in natural language processing and image generation tasks; Hu et al. constructed an SE module, which can help the model focus on the key areas of the image by modeling and adjusting the importance of different channels, and help accurately restore key information. Therefore, we attempt to embed the SE attention module into the generator, so that the model can adaptively adjust the attention to the defect part, increase the weight of the defect part, and ensure that the details of the generated image are more realistic.

[0009] The traditional pixel-level loss function cannot fully capture the high-level semantic information of the image, resulting in a difference between the generated image and the target image in vision, and causing regional distortion. Johnson et al. proposed a perception loss, which measures the similarity between the generated image and the target image at the perception level by pre-training the feature extraction layer of the neural network, and improves the distortion of the generated image. Li et al. combined the perception loss with the image generation model, which significantly improved the visual quality of the image after adding the watermark. Therefore, we also attempt to extract deeper features in the defect image by using a deep convolutional neural network VGG-19 to construct a perception loss, and improve the fidelity of the generated defect image in the overall structure. SUMMARY

[0010] To solve the above problems, the present application constructs a defect image generation model SWG-VGG based on WGAN-GP, which realizes the accurate generation and detail restoration of line defect and liquid leakage defect images. The model combines the SE attention mechanism, increases the weight of the model in the defect part, and enhances the ability of the model to represent the defect morphology; the depth and width of the generator are expanded, which can better capture the detail features of the defect image; the VGG-19 is used to extract deep features in the defect image to construct a perception loss to improve the loss function, and overcome the distortion problem in image generation. The technical scheme is as follows:

[0011] In one aspect, the embodiment of the present application provides a defect image generation method fusing attention mechanism and perception loss, the method comprising: S1: acquiring a real image, the real image being a liquid crystal display screen surface defect image; S2: constructing a SWG-VGG model based on a WGAN-GP model and training the SWG-VGG model, the SWG-VGG model comprising a generator, a discriminator and a VGG-19 network; wherein the generator is adjusted relative to the generator of the WGAN-GP model as follows: two up-sampling units are added before an output layer as high-resolution feature reconstruction modules, an SE attention layer is added before the high-resolution feature reconstruction modules as an SE attention module, and an SE attention layer is added after an input layer and each up-sampling unit; correspondingly, the discriminator is adjusted relative to the discriminator of the WGAN-GP model by adding two convolution blocks; the VGG-19 network is used to extract high-level features of a generated image and a real image during training to calculate the difference between the features, and finally the feature differences of different layers are weighted and summed to obtain a final perception loss and feedback to the generator; wherein a loss function of the SWG-VGG model is a weighted sum of a loss function of the WGAN-GP model and a perception loss function of the VGG-19 network; S3: inputting the real image into the trained SWG-VGG model to obtain a generated image.

[0012] In another aspect, the embodiment of the present application further provides a defect image generation system fusing attention mechanism and perception loss, comprising: a real image acquisition module: used for acquiring a real image, the real image being a liquid crystal display screen surface defect image; a SWG-VGG model comprising a generator, a discriminator and a VGG-19 network; wherein the generator is adjusted relative to the generator of the WGAN-GP model as follows: two up-sampling units are added before an output layer as high-resolution feature reconstruction modules, an SE attention layer is added before the high-resolution feature reconstruction modules as an SE attention module, and an SE attention layer is added after an input layer and each up-sampling unit; correspondingly, the discriminator is adjusted relative to the discriminator of the WGAN-GP model by adding two convolution blocks; the VGG-19 network is used to extract high-level features of a generated image and a real image during training to calculate the difference between the features, and finally the feature differences of different layers are weighted and summed to obtain a final perception loss and feedback to the generator; wherein a loss function of the SWG-VGG model is a weighted sum of a loss function of the WGAN-GP model and a perception loss function of the VGG-19 network; a training module: used for training the SWG-VGG model through the loss function; and a generated image module: used for inputting the real image into the trained SWG-VGG model to obtain a generated image.

[0013] In another aspect, the embodiments of the present application also provide an application of the generated image generated by the method in a target detection model YOLOv8, and the YOLOv8 model trained using the generated data reaches 94.6%, 93.8% and 97.6% in precision, recall and mAP(50) respectively.

[0014] The patent proposes an intelligent defect image generation model SWG-VGG based on deep learning, introduces an SE channel attention module into WGAN-GP, and introduces a perception loss into the model through VGG-19. The generated defect image is fused with the original data set. The generated defect image data set is compared with the original defect image data set using the YOLOv8 defect detection model. The experimental results show that the data set generated by the SWG-VGG model reaches 0.976 in mAP(50), which is improved by 10.5%; the PSNR, SSIM, RMSE and CS values of the generated image also show high image quality. The model provides a high-quality data set for display defect detection, and has important engineering application value. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a defect sample collected by a factory; Figure 2 is a distribution diagram of the defect sample collected by the factory; Figure 3 is a principle diagram of the basic structure of GAN; Figure 4 is a framework diagram of the SWG-VGG model of the present application; Figure 5 is a structural schematic diagram of the generator of the SWG-VGG model; Figure 6 is a structural schematic diagram of the SE attention module; Figure 7 is a structural schematic diagram of the discriminator of the SWG-VGG model; Figure 8 is a flowchart of perception loss calculation; Figure 9 is a comparison diagram of the generator loss curves of different models of different modules; Figure 10 is a comparison diagram of the scores of 200 defect images randomly generated by three generation models; Figure 11 is a comparison diagram of the scores of defect images of different background colors generated by three generation models; Figure 12 is a comparison of images of line defects and liquid leakage defects actually generated by three generation models; Figure 13 is a diagram of the values of different indicators in the training process of YOLOv8 using different data sets; Figure 14 is a comparison diagram of the PR curves of each type of defect in the training process of YOLOv8 using different data sets; Figure 15 is a comparison diagram of the confusion matrices of different types of defects in the training process of YOLOv8 using two data sets. DETAILED DESCRIPTION

[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0017] I. Relevant Theoretical Basis

[0018] 1.1 Generative Adversarial Networks

[0019] Generative Adversarial Networks (GANs) were proposed by Ian Goodfellow et al. in 2014. The core idea of ​​GANs is to enable the generator to produce more realistic data through adversarial training between the generator and the discriminator. The basic structure of a GAN is as follows: Figure 3 As shown. Specifically, GAN mainly consists of a generator and a discriminator. The generator is responsible for generating data. It receives a random noise vector that follows a Gaussian or uniform distribution and uses deconvolution to gradually map the low-dimensional vector to the target data space. The generator's goal is to generate fake data that resembles real data to deceive the discriminator. The discriminator is a binary classifier used to determine whether the input data is real data or fake data generated by the generator. The discriminator generally uses a convolutional neural network to extract features from the input real data and generated data, and then outputs a probability to represent the authenticity of the input data through a fully connected layer. The training process of GAN is a game of minimization and maximization. The generator's goal is to minimize the discriminator's correctness, making it unable to distinguish between generated and real data, while the discriminator's goal is to maximize its own correctness and accurately judge the authenticity of the input data. The objective function of GAN is defined as:

[0020] (1)

[0021] In equation (1), D(x) is the output of the discriminator to the real data x, and D(G(z)) is the output of the discriminator to the generated data G(z). For the distribution of real data, The input noise distribution is given. In this optimization process, the generator and discriminator are trained alternately. First, the generator is fixed to train the discriminator, and then the discriminator is fixed to train the generator, eventually reaching Nash equilibrium, where the samples generated by the generator are so similar to the real samples that they cannot be accurately distinguished.

[0022] 1.2 Channel Attention Mechanism

[0023] Channel attention is an attention mechanism in deep learning that weights different channels in the channel dimension of a feature map, allowing the model to adaptively focus on those channels that are more important in the current task. In a convolutional neural network, a feature map contains multiple channels, each of which may represent a certain type of feature (edge, texture, etc.). However, not all of these channels are useful for the task, and the goal of the channel attention mechanism is to highlight important features and ignore irrelevant features by learning to assign weights to each channel. The implementation process of the channel attention mechanism mainly consists of three steps. First, feature aggregation is performed using a global average pooling operation to aggregate information for each channel in order to obtain global information for each channel. The information calculation for the cth channel can be represented as:

[0024] (2)

[0025] where X is the feature map, H and W are the height and width of the feature map respectively, and c is the number of channels. Then, weight generation is performed using a Sigmoid nonlinear activation function to map the aggregated features to channel weights, calculating the attention weight of each channel , the calculation process can be represented as:

[0026] (3)

[0027] where is the Sigmoid activation function. Finally, the channels are reweighted, and the learned weights are applied to the corresponding channels to achieve the effect of channel attention, and the calculation process can be represented as:

[0028] (4)

[0029] where is the feature map after channel weighting.

[0030] 2.3 Perception loss

[0031] Perceptual loss is a loss function used in deep learning for image generation and image translation tasks, aiming to improve the visual quality of images by measuring the similarity of generated images and target images in high-level feature space. Compared with traditional pixel-level loss, perceptual loss pays more attention to the high-level semantic information and perceptual effect of images, so it has been widely used in style transfer, image denoising, super-resolution and other tasks. Perceptual loss calculates the difference between the high-level features of generated images and real images extracted by a pre-trained convolutional neural network, and finally sums the weighted differences of different layers to get the final perceptual loss. Perceptual loss is composed of content loss, which compares the difference between generated images and target images in a certain layer feature space, and style loss, which compares the similarity between generated images and target images in feature distribution. Among them, style loss is used to ensure that the generated image and the target image are consistent in style, focusing on the color, texture, structure and other features of the image. Content loss is used to ensure that the generated image and the target image are consistent in content. It calculates the difference between the feature representations of the two images in some feature layers, ensuring that the generated image retains the main structure and layout of the target image. The calculation of style loss is defined as:

[0032] (5)

[0033] wherein, in formula (5), and are the input generated image and real image respectively, is the Gram matrix of the lth layer, is a set of selected multiple feature layers. The calculation of content loss can be defined as:

[0034] (6)

[0035] wherein, in formula (6), is the feature map extracted in the lth layer. The final perceptual loss can be represented as:

[0036] (7)

[0037] wherein, in formula (7), and are the weight hyperparameters of style loss and content loss respectively.

[0038] II. Structure Overview

[0039] 2.1 General Description

[0040] The model SWG-VGG proposed in this patent is based on the improved generative adversarial network WGAN-GP, which is improved for small size display screen defect complex morphology and generated image distortion problem, and the overall framework is as follows: Figure 4As shown, the model consists of a generator, a discriminator and a VGG-19. In the training phase, the generator first receives a 100 (preferably) or 108-dimensional randomly sampled noise vector Z from a standard normal distribution, which represents a random point in the latent space. The task of the generator is to map this noise through the network to an image with structural and textural features consistent with the defect sample. This process is achieved through a deconvolution operation, in which the generator gradually upsamples the input noise Z to generate higher-resolution intermediate feature maps, and finally generates a color image with a size of 256x256. The discriminator learns the distribution difference between the generated image and the real image, and calculates the loss to help the generator optimize. VGG-19 extracts high-level features of the generated image and the real image, and calculates their perceptual loss in these feature maps and feeds it back to the generator, so that the generator not only generates realistic images at the pixel level, but also maintains consistency with real defect images at a high level of structure. In the defect image generation phase, the trained generator receives the noise vector Z, which is gradually mapped to the target defect image through upsampling.

[0041] 2.2 Generator and discriminator design

[0042] Traditional WGAN architecture often has problems such as low image quality and missing details when generating high-resolution images. In order to solve these problems and improve the quality of generated images, the ability to capture key features is enhanced. The generator structure is redesigned in this patent, as shown in Figure 5 The generator structure mainly includes 5 parts:

[0043] (1) Input layer, i.e. noise mapping layer, is responsible for accepting high-dimensional random noise vector Z and mapping it to a high-dimensional feature space to provide initial features for generating images. Z is 100 or 108 dimensions, etc. Specifically, it is composed of an upsampling unit. Compared with the existing WGAN-GP model, an SE attention layer is added after the upsampling unit. (2) Upsampling module, which gradually expands the resolution of the feature map to the target size. Specifically, it is composed of three upsampling units. Compared with the existing WGAN-GP model, an SE attention layer is added after the existing upsampling unit. (3) SE attention module, which is an additional added module, is responsible for adjusting the channel weight of the generated feature map, so that the model can more accurately capture the key features of the image. It is located between the third upsampling unit and the fourth upsampling unit. (4) High-resolution feature reconstruction module, which is an additional added module, further upsamples the feature map and gradually reduces the number of channels, so that the high-resolution features match the target output size. Specifically, it is composed of two upsampling units. Compared with the existing WGAN-GP model, an SE attention layer is added after the upsampling unit. (5) Output layer, mainly using transpose convolution layer to convert the feature map to the target image channel number of RGB three channels and limiting the pixel value to the range of [-1, 1] through the Tanh activation function. Compared with the existing WGAN-GP model, two upsampling units are added. That is, the generator of the present patent has six upsampling units and seven SE attention layers.

[0044] The generator structure is composed of several 4x4 deconvolution layers and SE attention layers (SELayer). The present patent embeds SELayer into the generator and expands the structure of the generator to increase the resolution of the output image from 64x64 to 256x256, greatly enhancing the detail expression ability of the defect image. Each upsampling unit (block) contains a deconvolution layer, a batch normalization layer, a ReLU activation function layer, and an SELayer. The role of these layers is to gradually enlarge the low-resolution features and optimize the channel feature expression using the attention mechanism. The workflow of the generator is first to use the deconvolution operation to upsample the input feature map and extract important features. The calculation process can be represented as:

[0045] (8)

[0046] wherein in formula (8), and are the height of the output and input feature map respectively, is the stride, is the padding size, is the convolution kernel size. Then the output feature map is subjected to batch normalization operation to standardize each batch of feature maps, reduce the internal covariance bias in training, and thus speed up convergence and stabilize training. For each feature channel, the batch normalization process is:

[0047] (9)

[0048] (10)

[0049] wherein in formula (9) and (10), x1 is a feature map, is a small constant, which is to prevent division by zero; and are learnable parameters and are scaling coefficients and translation coefficients, respectively, , . Then, a ReLU activation function is introduced to introduce nonlinearity and enhance feature expression capability. Finally, an SE layer is used to enhance the expression capability of channel features.

[0050] wherein the SE attention layer is a channel attention module for improving the performance of a convolutional neural network. In the display defect image generation task, the SE attention layer can weight the importance of each channel, thereby enhancing the attention of the generator to the defect morphology to improve the clarity of the generated defect image. The structure of the SE attention layer is shown in Figure 6 .

[0051] First, the input feature Figure X is transformed into a feature map U by a standard convolution operator Ftr, i.e., the input feature Figure X is transformed into a feature map U by a standard convolution operator Ftr, i.e., the input feature

[0052] (11)

[0053] wherein in formula (11), x1 is a feature map, is the value of channel c at spatial position is the value of channel c at spatial position is the value of channel c at spatial position The resulting channel descriptors are input into the weight generation network for weight generation after feature compression, to learn the weight of each channel. Weight generation is divided into two steps. First, the channel descriptors are passed through the first fully connected layer and the feature dimension is reduced to C / r of the original, where r is the scaling factor, using the ReLU activation function. Then the dimension is restored to C through the second fully connected layer, and the weight is limited to the range of [0, 1] using the Sigmoid activation function. Each channel will get a corresponding weight to represent its importance in the current task. The weight generation process can be represented as:

[0054] (12)

[0055] In formula (12), z is the channel descriptor vector, W1 and W2 are the weight matrix of the first fully connected layer and the weight matrix of the second fully connected layer respectively, and are the ReLU and Sigmoid activation functions respectively, s is the channel weight vector and it is ; finally, the generated channel weight s is multiplied with the feature map U channel by channel to realize feature weighting and the final output. Feature weighting can be represented as:

[0056] (13)

[0057] In formula (13), is the weight of the cth channel, is the value of the feature map U in channel c, and the final output is the weighted feature map.

[0058] In WGAN-GP, the objective of the discriminator is different from that of the traditional GAN. The discriminator of WGAN-GP uses the Wasserstein distance instead of the cross-entropy loss. WGAN-GP optimizes the adversarial between the generator and the discriminator by minimizing the Wasserstein distance. In order to guarantee the Lipschitz continuity of the discriminator, WGAN-GP also introduces gradient penalty. The structure of the discriminator is shown in Figure 7 The input is a 3x256x256 RGB image containing fake images generated by the generator and images in the original dataset. In order to correspond to the generator, the discriminator structure contains 5 convolutional blocks, each of which consists of a convolutional layer, a batch normalization layer and a LeakyReLU activation function, gradually downsampling the image features, and finally outputting a real score value D(x) through a 4x4 convolutional layer. The loss calculation part of the discriminator is the weighted sum of the Wasserstein loss and the gradient penalty. The calculation process of the Wasserstein loss can be represented as:

[0059] (14)

[0060] where, in formula (14), D(x) is a real score value of the discriminator output to represent the probability that the sample x comes from the real data; E is expectation; and are real image samples and generated image samples respectively, and are the distributions of generated data and real data respectively. The goal of the discriminator loss function is to maximize the score of real data and minimize the score of generated samples. In order to ensure the 1-Lipschitz continuity of the discriminator, the WGAN-GP introduces a gradient penalty term. This penalty term forces the gradient of the discriminator to satisfy the Lipschitz continuity condition on the interpolation samples of real images and generated images. The calculation process of the gradient penalty term can be represented as:

[0061] (15)

[0062] (16)

[0063] where, in formula (9) and (10), is a linear interpolation between the real image sample and the generated image sample is a random number sampled from a uniform distribution, is the data distribution of . So the total loss of the discriminator is the weighted sum of the Wasserstein loss and the gradient penalty, which can be represented as:

[0064] (17)

[0065] 2.3 Loss function improvement

[0066] The loss function of the original WGAN-GP model cannot fully capture the high-level semantic information of the image, so the model introduces a perceptual loss to improve the image distortion problem generated by the generator. The perceptual loss calculates the difference between the high-level features of the generated image and the real image by extracting the features of the two images through VGG-19, and finally the feature differences of different layers are weighted and summed to obtain the final perceptual loss. The calculation process of the perceptual loss is shown in the figure. Figure 8 First, the input image X is output through a generation network (ImageTransformNet in the figure) , and the VGG network extracts the feature map of the input image, then calculates the style image, content image and generated image ​The mean square error of the generated image and the target image is used to obtain a loss function 、 , and finally the weight of the generated network is adjusted according to the loss value to output an image . The model extracts image features by using five convolutional layers of VGG-19 (specifically, the first layer of the five blocks), so that Conv1_1, Conv1_12_1, Conv3_1, Conv4_1 and Conv5_1 calculate the style loss at different levels , the content loss is calculated using Conv3_1 , and the perceptual loss is calculated by integrating the above , and the calculation process can be represented as:

[0067] (18)

[0068] (19)

[0069] (20)

[0070] In formula (18), (19) and (20), L is a set of feature layers, , , are the number of channels, height and width of the feature map respectively, is the generated image, is the style image, is the content image, is the Gram matrix calculated for the target image, indicates the feature mapping of the image at the third layer, l indicates the five feature layers, i is the channel index, and j is the height index and k is the width index. The total loss function of the present application combines the WGAN-GP loss and the perceptual loss, and is defined as follows:

[0071] (21)

[0072] In formula (21), is the weight coefficient of the perceptual loss, which is used to adjust the weight of the perceptual loss in the overall loss, and through experimental analysis, the value is set to 0.001 when the model loss convergence process is best.

[0073] Correspondingly, the embodiment of the present application also provides a defect image generation system combining attention mechanism and perceptual loss, comprising:

[0074] A real image acquisition module is used to acquire a real image, which is a liquid crystal display screen surface defect image.

[0075] The SWG-VGG model comprises a generator, a discriminator and a VGG-19 network. The generator is adjusted relative to the generator of the WGAN-GP model as follows: two up-sampling units are added before the output layer as a high-resolution feature reconstruction module, an SE attention layer is added before the high-resolution feature reconstruction module as an SE attention module, and an SE attention layer is added between the third up-sampling unit and the fourth up-sampling unit. An SE attention layer is added after the input layer and each up-sampling unit. The generator comprises an input layer, an up-sampling module, an SE attention module, a high-resolution feature reconstruction module and an output layer arranged in sequence. The input layer comprises one up-sampling unit and is used to obtain a high-dimensional random noise vector Z and map it to a high-dimensional feature space to provide initial features for generating an image. The up-sampling module comprises three up-sampling units arranged in sequence. The SE attention module comprises one SE attention layer. The high-resolution feature reconstruction module comprises two up-sampling units arranged in sequence and is used to up-sample feature maps and gradually reduce the number of channels so that high-resolution features match the target output size. The output layer is used to convert the feature map to the target image channel number of three RGB channels through a transposed convolution layer and limit the pixel value to the range of [-1, 1] through a Tanh activation function.

[0076] The up-sampling unit comprises a deconvolution layer, a batch normalization layer, a ReLU activation function layer and an SE attention layer arranged in sequence. The deconvolution layer is specifically a 4x4 deconvolution layer. The SE attention layers of the input layer, the up-sampling module and the high-resolution feature reconstruction module are used to weight the importance of each channel to enhance the attention of the generator to the defect morphology to improve the clarity of the generated image. The SE attention layer of the SE attention module is used to first transform the input feature map U into a feature map U through a standard convolution operator Ftr, then perform a compression operation to compress the global spatial information into a channel descriptor, then input the obtained channel descriptor into the weight generation network to generate weights s to learn the weights of each channel, and finally multiply the generated channel weights s with the feature map U channel by channel to realize feature weighting. Figure X The discriminator comprises five convolution blocks and a 4x4 convolution layer arranged in sequence. The convolution block comprises a convolution layer, a batch normalization layer and a LeakyReLU activation function arranged in sequence, and is used to gradually down-sample image features of the generated image generated by the generator and the real image in the original data set; the 4x4 convolution layer is used to output a real score value D(x). The VGG-19 network is used to extract high-level features of the generated image and the real image during training to calculate the difference between these features, and finally the feature differences of different layers are weighted and summed to obtain the final perceptual loss and feedback to the generator.

[0077] The training module is used to train the SWG-VGG model through a loss function.

[0078] Image generation module: for inputting real images into the trained SWG-VGG model to obtain generated images.

[0079] III. Effect verification

[0080] 3.1 Experimental environment and data set

[0081] To verify the effectiveness of the method, three groups of experiments are designed to verify the effectiveness of the method. All experiments in this patent are implemented under the Pytorch framework, the operating system is Windows 11, the CPU is 12th Gen Intel(R) Core(TM) i5-12400F 2.50 GHz, the GPU is NVIDIA GeForce RTX 3080 Ti, and the running memory is 12G. The liquid crystal display surface defect image data used in this patent is collected on the spot in the factory. To ensure the authenticity and diversity of the data, image collection was performed on liquid crystal displays of different batches in the actual production environment. Table 1 shows the distribution of defect sample data. The defect detection process requires the display to be turned on first. Since the appearance and state of the defect under different background colors are different, the display is required to have no defects under the five background colors of white, blue, green, red, and color. The black in the line defect represents the situation that the display cannot be normally turned on and a bright line appears on the screen. Since the color of the liquid leakage defect itself is black, it cannot be displayed when the display is not turned on or in the black screen state. Therefore, there are no liquid leakage defects under the black background in the sample data. This patent collects each type of defect under different background colors. Each defect is collected under different background colors, and 30 different defect images are collected for each defect under each background color, forming a multi-background and multi-angle defect image data set. A high-resolution industrial camera was used during the collection process. The data set contains defect images from multiple production lines and different process conditions to ensure the representativeness of the data.

[0082] Table 1 Distribution of line defect and liquid leakage defect sample data

[0083]

[0084] 3.2 Evaluation index

[0085] In this patent, multiple commonly used image quality evaluation indexes are selected to comprehensively evaluate the quality of the generated images, including peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), root mean square error (RMSE), and similarity coefficient (CS).

[0086] 3.3 Experimental results and analysis

[0087] 3.3.1 Ablation analysis

[0088] To verify the effectiveness of the proposed model, a series of ablation experiments were conducted to evaluate the contribution of each module to the quality of generated images by gradually adding SE modules and perceptual loss. The experiments were based on a small-size liquid crystal screen surface defect image dataset for training and testing, with a uniform image resolution of 256x256. PSNR, SSIM, MI, and CS were used as image quality evaluation indicators. The following four comparison models were designed:

[0089] Model 1 is the traditional WGAN-GP (baseline model);

[0090] Model 2 adds an SE module to the baseline model (an SE attention layer is added before the high-resolution feature reconstruction module as an SE attention module, and an SE attention layer is added after the input layer and each upsampling unit);

[0091] Model 3 introduces perceptual loss based on the baseline model;

[0092] Model 4 uses the complete model of the present patent.

[0093] Under the same experimental environment, using the same computer and the same original dataset, four groups of defect image generation experiments were conducted using the four generation models, and the average indicator scores of each group of experiment generated liquid leakage defect and line defect images were as shown in Table 2, and the loss curves of each model generator were as shown in Figure 9 To better demonstrate the impact of each module on the model, the present patent respectively statistics the score comparison of each model at different training stages, Table 3 is the score comparison of liquid leakage defect at different training stages, Table 4 is the score comparison of line defect at different training stages; Figure 10 , 11 The generated liquid leakage defect and line defect images at different stages of model training are compared.

[0094] Table 2: Comparison of PSNR, SSIM, RMSE, and CS scores of four models

[0095]

[0096] From the above experimental data, it can be seen that:

[0097] ​(1) Compared with model 1, the scores of each index of model 2 are better than those of model 1, wherein the PSNR, SSIM and CS values are increased by 0.57, 0.0104 and 0.014 respectively, and the RMSE is reduced by 1.1001; Tables 3 and 4 show that the indicators of the defect images generated by model 2 after 1900 epochs are all better than those of model 1. This shows that the introduction of the SE module does improve the overall quality of the generated images to some extent, especially the details in the perception level of brightness, contrast and structural features. In addition, by Figure 9 It can be seen that the loss curve convergence speed of the generator of model 2 is obviously faster than that of model 1, and the loss value is slightly lower than that of model 1. It can be seen that the SE module does help us to better capture the inter-channel dependency of the image and improve the local detail quality of the generated image.

[0098] (2) Compared with model 1, the PSNR, SSIM and CS of model 3 are increased by 0.39, 0.0212 and 0.0097 respectively, and the RMSE value is reduced by 0.7345, and the overall improvement is slightly lower than that of model 2; Tables 3 and 4 show that the indicators of the defect images generated by model 3 in each training stage are mostly better than those of model 1; Figure 9 It can be seen that the loss convergence speed of the generator of model 3 is roughly the same as that of model 1, but the loss value is significantly lower than that of model 1. This shows that the introduction of the perception loss improves the overall quality of the generated image, and in terms of overall quality and perception quality of the image, the loss of the generator with the introduction of the perception loss is significantly lower than that of model 1 and model 2.

[0099] (3) Compared with model 1, the PSNR, SSIM and CS of model 4 are increased by 1.6, 0.0417 and 0.0442 respectively, and the RMSE value is reduced by 1.7238; Tables 3 and 4 show that among the four evaluation indicators, model 4 achieves the best results in most training stages among the four models. And the generator loss convergence speed is slightly faster than that of model 1, and the loss value is the same as that of model 3. This shows that the introduction of the SE module and the perception loss at the same time has made a major breakthrough in the overall similarity and perception level of the reconstructed image quality. By Figure 10 and Figure 11 It can also be seen that the images of the liquid leakage defects and line defects generated by model 4 are better than those of the other three models in quality and details. Therefore, the introduction of the SE module or the perception loss alone can improve the image quality, and the introduction of both can greatly enhance the performance of the model. It can be seen that the combination of the SWG-VGG architecture design and the perception loss proposed in the present patent can greatly improve the quality of the generated defect images of the small-size liquid crystal screen.

[0100] Table 3 Score table of the liquid leakage defect images generated by the four models in different training stages

[0101]

[0102] 4 Four models generate line defect image score table at different training stages

[0103]

[0104] 3.3.2 Comparative analysis

[0105] In order to comprehensively evaluate the performance of the SWG-VGG model proposed in this patent, comparative experiments were conducted with the classic WGAN-GP and DCGAN models. The performance of different models in generating two typical defect (line defect and liquid leakage defect) images was tested, and the experimental data set still used the data set shown in Table 1. In the same experimental environment, using the same computer, using the same original data set, three groups of defect image generation experiments were conducted using the three generation models respectively, and the PSNR, SSIM, RMSE and CS of the generated images in each group of experiments were counted, and the values of each index are shown in Table 5. And the average value of the images generated by the three models at different training stages is shown in Table 6. The score values of 200 randomly generated line defect and liquid leakage defect images of the three models and the scores of each index under various backgrounds are compared as shown in Table 5 and Table 6. Figure 12 、 13 The line defect and liquid leakage defect images actually generated by the three models are compared as shown in Table 5 and Table 6. Figure 14

[0106] Table 5 Score table of WGAN-GP, DCGAN and SWG-VGG model generated defect images in each index

[0107]

[0108] Table 5 and Figure 12 It is shown that: the PSNR, SSIM, RMSE and CS scores of the line defect and liquid leakage defect images generated by the SWG-VGG model proposed in this patent are better than those of the other two models, among which the liquid leakage defect image scores are 23.7, 0.83, 19.13 and 0.84 respectively; the line defect image scores are 23.41, 0.78, 22.92 and 0.81 respectively, and the average scores are 23.56, 0.80, 21.02 and 0.83 respectively. Compared with the defect images generated by the WGAN-GP model, the PSNR, SSIM and CS values are improved by 1.6, 0.04 and 0.05 respectively; the RMSE value is reduced by 1.73. From Figure 12 ​It can also be seen in the scatter plot that the scores of the liquid leakage defect images generated by the SWG-VGG model are significantly better than those of the WGAN-GP model in the four indicators. Compared with the DCGAN model, the PSNR, SSIM and CS values of the defect images generated by the model proposed in the patent are improved by 1.13, 0.03 and 0.04 respectively; and the RMSE value is reduced by 0.81. Figure 12 It can be seen in the scatter plot that the scores of the liquid leakage defect images generated by the model proposed in the patent are significantly better than those of the DCGAN model in the RMSE value, and the score distribution of the other indicators is slightly higher than that of the defect images generated by the DCGAN model.

[0109] Table 6 WGAN-GP, DCGAN and SWG-VGG model in different training stage each index score table

[0110]

[0111] Figure 13 With Figure 14 It is shown that for the liquid leakage defect, the PSNR of SWG-VGG is the highest in all background colors. The PSNR of WGAN-GP and DCGAN is lower, especially in white and red background, and the performance of WGAN-GP is the worst. The SSIM shows similar trend among the three models, and the structural similarity of SWG-VGG is the best, which is always higher than the other two models. The SSIM of DCGAN is better than WGAN-GP, especially in color and white background. The RMSE value is still the lowest in SWG-VGG, which proves that it has the smallest error when generating images. In contrast, the RMSE value of WGAN-GP is always the highest, indicating that it has the largest error in generating images. The CS index performs best in SWG-VGG, further proving that the generated images have the highest correlation with the reference images. For line defects, SWG-VGG performs best in PSNR, but unlike liquid leakage defects, the influence of background color is greater, especially in white and red background, with a slight decline in performance. SSIM and CS are also higher in SWG-VGG, especially in black and blue background, showing its stable performance. WGAN-GP performs the worst, especially in RMSE and CS, which verifies its shortcomings in generating line defect images. Figure 14 It is shown that the defect images generated by SWG-VGG highly restore the original image in color, texture and structure, accurately and clearly capturing the defect target in all examples, except for a slight color fading problem of the target in some backgrounds; the color and white background line defect images generated by the WGAN-GP model have image distortion problems, and the defect images generated by the DCGAN model have fewer distortion problems, but the generated defect images are poor in quality.

[0112] 3.3.3, application analysis

[0113] To verify the effectiveness of the defect images generated by the SWG-VGG model in practical applications, the current high-performance object detection model YOLOv8 is used for testing and analysis. YOLOv8 is a single-stage object detection algorithm that can achieve a good balance between detection speed and accuracy, and is widely used in the field of defect detection. In the experiment, the training set uses defect images generated by the SWG-VGG model, and the verification set uses original defect images. The input image size of YOLOv8 is uniformly adjusted to 416 × 416 to adapt to its network structure. In the training process, the hyperparameters used by the model are set as follows: the number of training rounds is 50, the batch size is 16, and the optimizer is Adam. In the training process, the loss function is used to calculate the error between the predicted results and the true labels, and the model parameters are continuously optimized through backpropagation. The precision, recall, and average precision mean mAP(50) are used to evaluate the performance of the model on the verification set. The training results are shown in Table 6 and Table 7. Figure 14 and Table 7.

[0114] Table 7 mAP(50) values of each type of defect in the original data set and the data set after adding generated images

[0115]

[0116] From Figure 15As can be clearly seen, the YOLOv8 model trained using generated data achieves 94.6%, 93.8% and 97.6% in precision, recall and mAP(50), respectively, while the model trained only using original images performs significantly worse on the same test set, with precision, recall and mAP(50) of only 89.9%, 74.5% and 87.1%, respectively. The YOLOv8 model trained using generated data improves by 4.7%, 19.3% and 10.5%, respectively. This difference shows that the introduction of the generated data set, especially in terms of defect sample diversity and quantity, significantly improves the performance of YOLOv8. The number of defect samples in the original data set is small, and the sample types are single, which leads to overfitting of the model during training, especially in the over-reliance on specific types of defect samples. Overfitting phenomenon makes the model's recall rate and detection accuracy significantly decrease when facing new samples. As can be seen from the average detection accuracy mAP(50) values of each type of defect in Table 7, the average mAP(50) value of all types using the original data set is only 0.871, among which the leakage defect is 0.941, the point defect is 0.962, the scratch defect is 0.972, and the line defect is only 0.609, which shows obvious shortcomings. This shows that the variety of line defect shapes and colors further magnifies the lack of diversity in the original data set, and the recall rate decreases significantly. Compared with the detection results of the original data set, the overall performance of the model is significantly improved after adding the generated data set, with mAP(50) reaching 0.976, an increase of 10.5% in detection accuracy. The detection performance of each category is significantly improved, with mAP(50) of leakage and scratch defects reaching 0.995, point defects reaching 0.923, and line defects reaching 0.990, close to the optimal state. It can be seen that the generated data plays a key role in improving the detection ability of the model for weak categories in the original data set, such as line defects. This improvement can be attributed to the fact that the generated data effectively expands the diversity and balance of the original data, enhancing the model's ability to learn features of each category.

[0117] To further verify the influence of the number of data sets on the stability of the defect detection model, the present patent designs a series of experiments, using the generated defect image data set to enhance to 2000, 4000, 6000 and 8000 data for model training and testing. By comparing the performance of the model under different data set sizes, we aim to explore the specific influence of data set size on model training effect, precision and stability. In the experiment, we mainly focus on the precision of the model in detecting the four types of defects and the stability of the training process, especially under the condition of small data set, whether the model is prone to overfitting or unstable training, etc. Table 8 is the data distribution table of the data set, and Table 9 is the detection accuracy statistics table of each type of defect using different data sets.

[0118] Table 8 Distribution of each type of defect in different data sets

[0119]

[0120] Table 9 Statistics table of accuracy of each defect and average accuracy in different data sets

[0121]

[0122] Table 9 shows that the average accuracy of data set 2 is 97.8%, which is improved by 0.2% compared with data set 1, and the improvement is small. The accuracy of line defects is lower than that of data set 1, and the accuracy of point defects is significantly higher than that of data set 1, indicating that the increase of data volume is beneficial to the detection of small size defects such as point defects, although the overall accuracy is improved slightly. The average accuracy of data set 3 is 97.5%, which is roughly the same as data set 1, and the detection accuracy of each type of defect is reduced. The accuracy of point defects is higher than that of data set 1, indicating that the increase of data volume increases the stability of the detection model. The average accuracy of data set 4 reaches the highest, and the accuracy of point defects reaches 95.4, which is significantly higher than the accuracy of point defects of the previous three data sets. The overall improvement is relatively stable, indicating that the increase of data set size helps to improve the detection accuracy of the model, especially when dealing with complex and difficult to identify defect types, the expansion of data volume is particularly important.

[0123] IV. Conclusion

[0124] Aiming at the problem that the data of small size LCD defect sample set is rare and the data set is extremely unbalanced, which seriously affects the precision of intelligent detection model, this patent proposes a defect image generation model SWG-VGG based on WGAN-GP, which realizes the accurate generation and detail restoration of line defect and leakage defect images, and creates an effective and balanced defect sample data set for intelligent defect detection of LCD. In this model, we introduce the Wasserstein distance and gradient penalty mechanism to effectively improve the stability of model training; fuse the SE attention mechanism to improve the weight of the defect part of the model and enhance the ability of the model to represent the defect morphology; expand the depth and width of the generator to better capture the details of the texture, color and shape of the defect image; use VGG-19 to extract deeper features of the defect image to construct the perception loss and improve the loss function, which overcomes the distortion problem in image generation. The ablation comparison experiment results show that each improvement design of this model improves the quality of the generated image, and the overall improved model ensures more stable training and faster convergence, and significantly improves the four evaluation indicators PSNR, SSIM, MI and CS of the generated image. The comparison experiment with DCGAN and WGAN-GP shows that the generated image of SWG-VGG has absolute advantages in quality under any background. The application implementation shows that the defect detection model trained by the defect sample set generated by SWG-VGG improves the detection accuracy from 87.1% to 97.6%, especially the detection accuracy of line defects is significantly improved. Therefore, the SWG-VGG intelligent image generation model can indeed realize the accurate generation and detail restoration of small size LCD line defect and leakage defect images, and can create an effective data set for intelligent defect detection of LCD, which is a kind of LCD defect data enhancement method with universal engineering application value.

Claims

1. A defect image generation method integrating attention mechanism and perceptual loss, characterized in that, The method includes: S1: Obtain a real image, wherein the real image is an image of surface defects of the liquid crystal display screen; S2: Construct and train the SWG-VGG model based on the WGAN-GP model. The SWG-VGG model includes a generator, a discriminator, and a VGG-19 network. The generator comprises an input layer, an upsampling module, an SE attention module, a high-resolution feature reconstruction module, and an output layer arranged sequentially. The input layer includes an upsampling unit that acquires a high-dimensional random noise vector Z and maps it to a high-dimensional feature space, providing initial features for the generated image. The upsampling module includes three upsampling units arranged sequentially. The SE attention module includes an SE attention layer. The high-resolution feature reconstruction module includes two upsampling units arranged sequentially, used to upsample the feature map and gradually reduce the number of channels, ensuring that the high-resolution features match the target output size. The output layer converts the feature map into the target image's RGB three-channel number using a transposed convolutional layer and restricts pixel values ​​to [-1, 1] using a Tanh activation function. Within the range of ]; wherein, the upsampling unit includes a deconvolution layer, a batch normalization layer, a ReLU activation function layer and an SE attention layer arranged sequentially; wherein, the SE attention layer of the input layer, the upsampling module and the high-resolution feature reconstruction module is used to weight the importance of each channel to enhance the generator's attention to defect morphology and improve the clarity of the generated image; the SE attention layer of the SE attention module is used to first transform the input feature map X into a feature map U through a standard convolution operator Ftr, then perform a compression operation to compress the global spatial information into a channel descriptor, then input the obtained channel descriptor into the weight generation network to generate weights to learn the weights s of each channel, and finally multiply the generated channel weights s with the feature map U channel by channel to achieve feature weighting; The discriminator includes five convolutional blocks arranged in sequence and a 4×4 convolutional layer; the convolutional block includes a convolutional layer, a batch normalization layer and a LeakyReLU activation function arranged in sequence, which are used to progressively downsample image features of the generated image generated by the generator and the real image in the original dataset; the 4×4 convolutional layer is used to output a real number score value D(x); The loss function of the SWG-VGG model is a weighted sum of the loss function of the WGAN-GP model and the perceptual loss function of the VGG-19 network. The loss function of the SWG-VGG model for: ； in, The loss function of the WGAN-GP model; To extract high-level features between generated and real images using VGG-19 and calculate the differences between these features, the perceptual loss function is obtained by weighted summation of the feature differences from different layers. These are the weighting coefficients for perceived loss; S3: Input the real image into the trained SWG-VGG model to obtain the generated image; The real images include images where the display screen cannot be lit normally and bright lines are produced on the screen, images with line defects under five background colors (white, blue, green, red, and colored), and images with leakage defects under five background colors (white, blue, green, red, and colored). The real images come from multiple production lines and different process conditions; the resolution of the generated image is 256×256; the generator takes a noise vector Z from a 100-dimensional random sample from a standard normal distribution and finally generates a color image with a size of 256×256. It is 0.

001.

2. The method according to claim 1, characterized in that, The deconvolutional layer is a 4×4 deconvolutional layer, and the working process of the deconvolutional layer is as follows: The input feature map is upsampled and important features are extracted through deconvolution. The calculation process can be represented as follows: ； in, and These represent the heights of the output and input feature maps, respectively. stride, For fill size, The kernel size; For each feature channel, the batch normalization process is as follows: ； ； Where x1 is the feature map, It is a constant. and Let be learnable parameters, and let them be scaling and translation coefficients, respectively. , .

3. The method according to claim 1, characterized in that, In the SE attention module, the compression process is as follows: the feature map U containing global information is compressed into a 1×1×C channel descriptor through global average pooling of the channels. The calculation process is as follows: ； in, For the spatial position of channel c The value on, To use global average pooling of channels, ; Weight generation consists of two steps. First, the channel descriptors are passed through a first fully connected layer and the ReLU activation function is used to reduce the feature dimension to C / r, where r is the scaling factor and C represents the dimension. Then, the dimension is restored to C through a second fully connected layer, and the Sigmoid activation function is used to constrain the weights to the range [0, 1]. Each channel will receive a corresponding weight to represent its importance in the current task. This weight generation process is as follows: ； Where z is the channel descriptor vector, and W1 and W2 are the weight matrices of the first and second fully connected layers, respectively. and ReLU and Sigmoid activation functions are respectively, where s is the channel weight vector and it is... Finally, the generated channel weights s are multiplied channel by channel with the feature map U to achieve the final weighted output. The feature weighting process is as follows: ； in, Let c be the weight of the c-th channel. Let U be the value of feature map U in channel c.

4. The method according to claim 1, characterized in that, The The calculation process is as follows: ； in, Loss to Wasserstein Gradient penalty; in, ; Where D(x) is the real score output by the discriminator, representing the probability that sample x comes from real data; E is the expectation. and These are real image samples and generated image samples, respectively. and The distributions of generated data and real data are respectively; in, ; ； in, Real image samples With generated image samples Linear interpolation between them For random numbers sampled from a uniform distribution, , for Data distribution.

5. The method according to claim 4, characterized in that, The process of generating the perceptual loss is as follows: Input image X is processed by a generator network and outputs an image. The VGG network extracts feature maps from the input image through five convolutional layers, and then calculates the style image, content image, and generated image respectively. The mean square error yields the loss function. , Finally, the generated image is adjusted based on the loss value. The five convolutional layers are Conv1_1, Conv2_1, Conv3_1, Conv4_1, and Conv5_1. Style loss at different levels is calculated through Conv1_1, Conv2_1, Conv3_1, Conv4_1, and Conv5_1. Content loss is calculated using Conv3_1. ; ； ； ； Where L is the set of feature layers. , , These represent the number of channels, height, and width of the feature map, respectively. To generate an image, For style images, For content images, To calculate the Gram matrix for the target image, Let l be the feature map of the image at layer 3, l be the five feature layers, i be the channel index, j be the height index, and k be the width index.

Images