An unsupervised data enhancement method for hot-rolled coil surface defect images

Abstract

A method for unsupervised enhancement of hot-rolled coil surface defect image data belongs to the technical field of metallurgical hot-rolled coil defect detection. The generative adversarial network consists of two deep neural networks that compete with each other. It is a probabilistic generation model. The way it generates samples is to forward propagate through the generator, and then use gradient backpropagation to optimize the calculation after the discriminator discriminates. It does not rely on any prior assumptions. In the process of training the generative adversarial network, the two networks continuously iterate and optimize in the mutual game. The generator learns to generate more realistic samples, while the discriminator network tries to discriminate whether the data samples are real samples or generated fake samples. The two sides continue to compete until neither side can get better. Finally, the two networks reach a dynamic equilibrium, that is, the image distribution generated by the generator is close to the real image distribution, and the discriminator cannot recognize true and false images. The advantage is that it solves the problems of small and difficult to obtain image data of surface defects of hot-rolled coils in the field of metallurgical steel rolling, and low recognition accuracy.

Description

Technical Field

[0001] The invention belongs to the technical field of metallurgical hot-rolled coil defect detection and is a method for generating surface defect image data of a hot-rolled coil based on an artificial intelligence network model. Background Art

[0002] In the era of continuous industrial development, steel has become the "food" of industry, and it is one of the most important materials in modern times. However, in the actual production process of steel plates and coils in steel mills, hot-rolled coils with surface defects are often produced due to factors such as equipment problems and processing technology. If the surface defects of the coils are not effectively detected and identified, it will seriously affect its quality, external appearance, economic benefits, performance, etc. as a steel product. Therefore, the research on defect recognition on the surface of the coils has great practical significance. Early defect detection required quality inspectors to identify and judge with their eyes, which was labor-intensive, inefficient, time-consuming and labor-intensive. Later, machine vision based on traditional machine learning was used for defect detection. Although the recognition rate was improved, it still required manual auxiliary judgment, and feature design had to be performed manually through feature engineering in machine learning, making this method more cumbersome. In recent years, the rise of deep learning technology has brought a qualitative leap in image recognition. Deep learning originated from the research of artificial neural networks, which is a deep neural network containing multiple hidden layers. Deep learning combines low-level features to form more abstract high-level representation attribute categories or features to discover the distributed pattern feature representation of data. Deep learning does not require manual design of features and has a simple process. It starts by inputting images from one end of the network and outputting prediction results from the other end. It automatically learns the image texture features of a large amount of data and combines and abstracts them. It can better characterize the rich intrinsic information of the data and has improved learning capabilities compared to traditional networks, thereby improving the accuracy of image recognition of surface defects of hot-rolled coils.

[0003] Using deep learning technology to detect surface defects of hot-rolled coils first requires training the corresponding neural network model, but it usually relies on a large amount of labeled data, and the demand for training samples is very large. The larger the amount of data, the better the generalization performance of the trained model, and the higher the accuracy of identifying surface defects of hot-rolled coils. However, it is very difficult to collect data on surface defects of hot-rolled coils in the field of metallurgical steel rolling, making this kind of industrial data a very precious resource. The appearance of various surface defects of hot-rolled coils is inherently random. After collection, steel rolling experts still need to spend a lot of time to annotate defect data, which is time-consuming and laborious. Data is the input of deep neural networks. The image data of coils and their characteristics often determine the upper limit of the final model recognition results, and the selection and optimization of model algorithms are only gradually approaching this upper limit. The quality and quantity of data play a vital role in the final training performance of the model. Data enhancement technology came into being in this context. Data enhancement, also known as data amplification, is to generate more new data by making some series of processing changes to the existing original data without adding external data, so that limited data can generate more value while also improving the recognition accuracy and generalization ability of the classification model.

[0004] Unsupervised learning model is a kind of artificial intelligence model. Its characteristic is that for its input examples, it will automatically find its potential rules from these examples. Unsupervised learning characterizes the distribution characteristics of sample data by learning the essential characteristics of real data, and generates new data similar to training samples. The parameters of the model are much smaller than the amount of training data, so the model can discover and effectively internalize the essence of the data, so that it can generate these data. The idea of ​​generative adversarial network (GAN, Generative Adversarial Networks) originated from game theory. It belongs to unsupervised learning network and was proposed by Ian Goodfellow in 2014. The GAN model learns through the mutual game of two modules in the framework: generator (Generator) and discriminator (Discriminator) to finally produce good output. The generator randomly generates observation data by giving some implicit information. In the present invention, the input of the generator network is a random noise variable, and the output is the surface defect image data of hot-rolled coil generated by computer. The generator mainly learns to fit the real high-dimensional data distribution from a low-dimensional data distribution, while the discriminator is mainly to distinguish whether the data comes from the real image or the image generated by the generator. The discriminator model needs to input images and predict judgments through the model's loss function. During training, the generator and discriminator are trained alternately. The discriminator is trained first, and then both real data and generated data are input into the discriminator to train a discriminator with good results; then the generator is trained with this trained discriminator, with the goal of making the data generated by the generator deceive the discriminator as much as possible. They continue to confront and learn until the discriminator can no longer distinguish between generated images and real images, and finally reach Nash equilibrium (a balance in game theory, which means that improvements by either party will not lead to an increase in overall benefits). Summary of the invention

[0005] The purpose of the present invention is to provide a method for stably and effectively generating hot-rolled coil surface defect image data, so as to solve the problem that the hot-rolled coil surface defect image data in the field of metallurgical steel rolling is scarce and difficult to obtain, resulting in low accuracy in identifying defects using artificial intelligence machine vision technology, thus seriously affecting its product quality. The present invention is different from the previous supervised data enhancement. Traditional data enhancement uses fixed preset rules to perform simple geometric transformations (flipping, rotation, cropping, deformation, scaling, etc.) and color transformations (noise, blur, brightness, erasing, jittering, filling, etc.) on existing images. Traditional methods do not take into account the differences between different tasks and the richness and diversity of samples after data enhancement and the quality of images, and are still not efficient and stable enough.

[0006] The original GAN ​​theory does not require that both the generator and the discriminator are neural networks, but only needs to be able to fit the corresponding generation and discrimination functions. In the present invention, the generative adversarial network is composed of two deep neural networks that compete with each other. It is a probabilistic generation model. The way it generates samples is forward propagated through the generator, and after the discriminator judges, it uses gradient backpropagation for optimization calculation. It does not rely on any prior assumptions. Many traditional data enhancement methods generally assume that the data obeys a certain distribution, and then use maximum likelihood to estimate the data distribution, which is very complicated. In GAN, the generator takes random noise as input and generates samples, and the discriminator is used to judge whether the data generated by the generating network is true or false. In the production adversarial network training process, the two networks are continuously iterated and optimized in the mutual game. The generator learns to generate more realistic samples, while the discriminant network tries to judge whether the data sample is a real sample or a generated fake sample. The two sides continue to compete until the end, neither side can get better, and finally the two networks reach a dynamic equilibrium, that is, the image distribution generated by the generator is close to the real image distribution, and the discriminator cannot recognize true and false images. The earliest generator and discriminator did not use deep neural networks but perceptrons, and their optimization objective function is shown as follows:

[0007]

[0008] Where x represents a real sample, z represents the random noise input to the generator G, and G(z) represents the image generated by the generator G. D(x) represents the probability that the discriminator D judges whether the image is a real image, and D(G(z)) is the probability that the discriminator D judges whether the image generated by G is real. p(x) represents the distribution of real data, and p(z) represents the distribution of generated data. This formula requires maximizing the probability estimate of the discriminant model for real data and minimizing the probability estimate of the discriminant model for generated data. This generative adversarial network has training instability during the training process, and can only grasp the local variance characteristics of the data distribution, which is prone to model collapse problems. There are also problems such as training difficulties, the loss of the generator and discriminator cannot indicate the training process, and the lack of diversity in the generated samples, which makes the generated images blurry and of poor quality.

[0009] In view of the above problems, the present invention improves and optimizes the original generative adversarial network. The generative adversarial network in the present invention consists of two deep neural networks (generator sequence network and discriminator sequence network) that compete with each other.

[0010] Step 1: The present invention defines a generator function as a generator sequence GEN(z) consisting of generator modules, where the initial module is defined as:

[0011]

[0012] Where A is the image space generated by mapping the noise space Z, and Z is a Gaussian noise latent vector that follows a normal distribution ( Figure 3 The space composed of the latent vector input on the left side. N(0,1) is the standard normal (Gaussian) distribution, R represents the spatial dimension, is a notation for mapping, A0 represents the first image generated by noise space mapping.

[0013] Step 2: Assume that the generator sequence has k layers (k is set by the model designer, and k is set to 7 in this invention), let G k As a general function:

[0014]

[0015] It represents the basic generator module. In the implementation, the function includes an upsampling operation so that the image resolution of each layer is larger than the original one.

[0016] Step 3: Define k such generator modules to synthesize the final generator sequence:

[0017]

[0018] Step 4: Define a function r (in Figure 3 The function has the function of generating images of different resolutions at different stages of different modules in the generator sequence. r is modeled as a module composed of 1×1 convolution and activation function (a function that increases the nonlinearity of the neural network model). It can greatly increase the nonlinear characteristics of the module while keeping the feature map scale unchanged (that is, without losing resolution) (using the subsequent nonlinear activation function to convert the intermediate convolution value activation into an image, and its output O i Corresponding to different downsampled images of the final output image), improve the expression ability of the neural network model. The formula for defining r is as follows:

[0019]

[0020]

[0021] Among them, r i The function acts as a regularizer (a regular function that improves the generalization ability of the model), projects the learned feature map into the RGB image space, and generates an intermediate layer image.

[0022] Step 5: From step 1, we can get g i (z) = a i , so we have: i (g i (z)) = r i(a i )=o i

[0023] Among them, i is the image generated by the middle layer of the i-th layer of the generator. These images of different sizes and resolutions are generated. i All of them must be sent to the discriminator, so that the gradient information flows into the current layer during back propagation to enhance information interaction and prevent gradient disappearance.

[0024] Step 6: Define the discriminator sequence. We use the letter D to represent the discriminator sequence consisting of all modules of the discriminator. We define the last layer of the discriminator as d(z') and the first layer of the discriminator as d 0 , use y to represent the real sample, y' to represent the generated sample image, and define d similarly j is the intermediate layer function of the discriminator. When j=ki, i and j are always related to each other. Therefore, the output of the jth intermediate layer function of the discriminator is defined as:

[0025] a' j =d j (combine(o k-j ,a' j-1 ))

[0026] =d j (combine(o i ,a' j-1 ))

[0027] Where combine is the output o of the i-th intermediate layer of the generator i Perform 1×1 convolution (channel cascade combination operation) with the corresponding output of the j-1th intermediate layer of the discriminator to achieve feature information integration. Figure 3 The final discriminant loss function of the discriminator is not only a function of the final output of the generator, but also a function of the output of the intermediate layer. The connection of the intermediate layer allows the gradient information to flow from the intermediate layer of the discriminator to the intermediate layer of the generator during the model training process, making the network training more stable.

[0028] Step 7: Finally, the formula for the entire discriminator sequence is recorded as:

[0029]

[0030] Step 8: The present invention improves the structure of the traditional generative adversarial network and defines a new loss function as shown below:

[0031]

[0032]

[0033]

[0034]

[0035] Where L D represents the loss function of the discriminator D, L G represents the loss function of the generator G, x f Image data generated by the generator, x t is the real image data, and E represents the function of finding the mean. The new loss function evaluates the probability that the average given real data is more real than the randomly sampled fake data. This enables the entire network to generate stable and high-quality data images (with finer edges and richer textures) from relatively small samples, and also reduces the time required for model training;

[0036] Step 9: In deep learning, the loss function measures the gap between prediction and reality and narrows this gap through optimization strategies. The model optimization rule adopted by this patent is a gradient descent method that calculates the first-order moment estimate and second-order moment estimate of the gradient and the adaptive learning rate of the parameters, which is different from the strategy of stochastic gradient descent that only calculates the gradient to correct the error. It can adjust the learning rate according to the historical information of the gradient, so that the memory required for training data to update the neural network weights is reduced and the training speed is faster.

[0037] The advantage of the present invention is that a novel unsupervised data enhancement method is proposed to generate surface defect image data of hot-rolled coils with different resolutions (such as Figure 4 As shown). The generative adversarial network in the present invention is different from other artificial intelligence networks. It is a generative adversarial network that can stably generate images of multiple resolutions of surface defects of hot-rolled coils. It generates high-quality image sample data while solving the instability of the original generative adversarial network training. The difference in the structure of this network lies in the connection between the intermediate layers of the generator and the discriminator. The role of this operation is to introduce a regular term between the input vector and the generated result to ensure that similar input vectors do not always fall into a mapping area. The connected layers are trained at the same time, saving a lot of time for network training, instead of the traditional way of training the network by gradually adding layers, which improves the efficiency of generating images and is suitable for rapid engineering implementation. The advantage of the present invention is that the discriminator result is jointly determined by the output of the generator and the output of the intermediate layer, and the gradient information features are transferred to images of different resolutions. Compared with the existing technology, the present invention further alleviates the problem of difficulty in obtaining training sets of surface defect images of hot-rolled coils, and lays a better data foundation for AI models to improve recognition accuracy and improve model generalization ability. The data generated based on the new deep production adversarial network is a good AI model training resource, which breaks the data barrier to a certain extent and provides a reference for more basic research and practical project applications to obtain the desired image data samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The distribution diagram of the generated data fitting the real data. Line b is the Gaussian distribution of the real data, and line c is the data distribution learned by the generative adversarial network. At the beginning, it is randomly initialized with noise. Line a is the probability that the discriminator network determines that the image is real data. The goal of the generative adversarial network is to make the c curve gradually fit the b curve by continuously training the generator and discriminator networks, that is, Figure 1 In the process from left to right in the figure, when the distribution of the generated sample and the real sample completely overlap, the sample data is difficult to distinguish between true and false, and the discriminator judges them as true or false with the same probability. The horizontal line marked with x represents the sampling space that obeys the Gaussian distribution x, and the horizontal line marked with z represents the sampling space that obeys the distribution z. The arrow with the z axis pointing to the x axis indicates that the generative adversarial network has learned the mapping relationship from the z space to the x space;

[0039] Figure 2 Concept diagram for generative adversarial networks. The generator network takes a random noise vector z as input and generates an image x; the discriminator is trained as a binary classification network based on two types of data, one is real data samples, all marked as 1; the other is data from the generator marked as 0. The real data and the generated data are input as mixed samples, and the output is the probability that the input image is a real image. The discriminator is used to determine whether the input image is a real image or a generated image. Under ideal conditions, the generator network can generate a picture x that is "real" enough, and the discriminator network has difficulty determining whether the picture generated by the generator network is real or generated. Once such a generative adversarial model is obtained, it can be used to generate target images for unsupervised data enhancement;

[0040] Figure 3 This is a diagram of the model network architecture. This architecture connects the middle layer of the generator sequence with the middle layer of the discriminator sequence (this is different from the traditional production adversarial network), transmits the multi-size images of the middle layer of the generator sequence to the discriminator and connects them with the corresponding activation function obtained from the previous convolutional layer in the discriminator. The model in the present invention allows the discriminator to obtain not only the final output (highest resolution) of the generator, but also the output of the middle layer. This connection operation allows the production adversarial network to better adjust its parameters and improve the stability of network training. The horizontal rectangular module at the far left of the figure that is input to the generator sequence is the potential noise vector, which then passes through the two convolution modules of the vertical rectangular box (one 4×4 convolution and one 3×3 convolution), and then continuously upsamples through the middle layer convolution, as shown in g in the figure. 1 ,g 2..., until the specified (highest resolution samples) image samples are generated ( Figure 3 The generated image and training image are then input into the discriminator sequence network on the right and continuously downsampled through the deconvolution module, as shown in the d k-2 ,d k-1 …, during which the generator middle layer is connected, and multiple images of different resolutions and the corresponding activation values ​​obtained from the previous convolutional layer are connected together through the combination module (the vertical module Combine Function on the right side of the figure), and then pass through the MinibatchStd module (the MinibatchStd module calculates the difference information between the feature maps of a small batch of samples in a certain layer of the discriminator network as the additional output of the next layer in the discriminator network to achieve information interaction between each sample to improve sample diversity, thereby achieving our multi-resolution setting and detecting whether the generator mode collapses), then connected to the fully connected layer, and finally input into the discriminator's loss function Critic-loss function for sample discrimination;

[0041] Figure 4 This is an example of the generated surface cracking defect of hot-rolled coil. From left to right and from top to bottom, the resolution sizes of the images are 4×4, 8×8, 16×16, 32×32, 64×64, and 128×128. Theoretically, the generated image resolution can also be 256×256, 512×512, or 1024×1024, which is related to the resolution of the image dataset used for training, the performance of the training machine, the number of model training rounds (number of iterations), the training time, the neural network hyperparameter settings, and the number of generator sequence layers k. In the present invention, the generator and the discriminator use the same hyperparameter settings, the model learning rate is 0.003, and the batch size is 120, which is simpler than other neural network training parameters and is easy to implement. The resolution of the training dataset used in the present invention is 128×128, the number of training rounds is 57290, and the training time is one week. The performance of the machine is introduced in the specific implementation method;

[0042] Figure 5 Another example of the generated hot rolled coil surface rolling defect. From left to right, from top to bottom, the resolution of the image is 4×4, 8×8, 16×16, 32×32, 64×64, 128×128. The principle of generation is similar to Figure 4 same, Figure 5 , Figure 4They are all hot-rolled coil surface defect data images with various resolutions randomly generated by a trained (i.e., fitting the true distribution of real data) production adversarial network model. DETAILED DESCRIPTION

[0043] 1. The implementation of this method requires a deep learning computer server, 2 Tesla (version V100) GPU graphics cards, each with a memory size of 32G, 128 RAM, and a 1TB solid-state drive as the hardware platform;

[0044] 2. Install the Linux 64-bit (Ubuntu 16.04) operating system on the computer, install the Anaconda software library, install the Pytorch deep learning framework, install the CUDA Toolkit version 10.2, the graphics driver version NVIDI-440.33.01, and the cudnn acceleration package version 7.6.5 as the software environment, and use the Python programming language;

[0045] 3. Use the command source axtivate pytorch to activate the deep learning development environment;

[0046] 4. Prepare the data set. The image training set in the present invention comes from the actual production line of hot-rolled coils in a steel plant. It is captured by a surface defect detector and is a labeled data set sorted and marked by workers and experts. Image cleaning is performed to remove some wrong format images that do not conform to scene recognition, noise and damaged images, and to unify the image size. After preprocessing, the data set contains 325 rolling defect images with a resolution of 128×128 and an image format of jpg.

[0047] 5. Define the structure of the generative adversarial network, define the dimension shape of the input data, define the shape and initialization method of each layer, and define the loss function and Gaussian noise distribution of the two networks: the discriminator D and the generator G;

[0048] 6. Initialize the parameters of the two networks, generator G and discriminator D. Train the generator, generate image samples using its defined noise distribution, and read data from the training set;

[0049] 7. Fix the parameters of the generator G, train the discriminator, put both the generated pictures and the real pictures into the discriminator, and use the loss function to distinguish the real and the fake as much as possible;

[0050] 8. After the discriminator distinguishes the real data from the generated data, fix the discriminator parameters, train the generator, and let it generate image data again;

[0051] 9. Select an optimization strategy to optimize the model. After updating k times in a loop, the discriminator cannot distinguish between true and false samples. When the goal of Nash equilibrium is achieved, it can be considered that the generator captures the true distribution of real data.

[0052] 10. Examples of images generated using a trained generative adversarial model Figure 4 , 5 As shown, please refer to the accompanying drawings for description.

Claims

1. A method for unsupervised data enhancement of hot-rolled coil surface defect images, characterized in that: It consists of two deep neural networks that compete with each other: the generator sequence network and the discriminator sequence network, including the following steps: Step 1: Define the generator function as a generator sequence GEN(z) consisting of generator modules, where the initial module is defined as: Z=R 512 ,Z~N(0,1),A0=R 4×4×512 Where A is the image space generated by mapping the noise space Z, Z is the space composed of Gaussian noise latent vectors that obey the normal distribution; N(0,1) is the standard normal Gaussian distribution, R represents the spatial dimension, To represent the symbols of mapping, A0 represents the image generated by the noise space mapping for the first time; Step 2: Assume that the generator sequence has k layers, k is set by the model designer, k is set to 7, let G k As a general function: Among them A i →A i+1 Indicates the generation process of implementing upsampling operation in the generator module so that the resolution of each layer of image is larger than the original one. i+1 Indicated by A i In the image generated by the upsampling operation in the generator module, N is a natural number set {0, 1, 2, 3...}; Step 3: Define k such generator modules to synthesize the final generator sequence: Step 4: Define a function r, which has the function of generating images of different resolutions at different stages in different modules in the generator sequence. Model r as a module composed of 1×1 convolution and activation function, which can greatly increase the nonlinear characteristics of the module while keeping the scale of the feature map unchanged. Use the subsequent nonlinear activation function to convert the intermediate convolution value activation into an image; its output O i Corresponding to different downsampled images of the final output image, the expression ability of the neural network model is improved. The formula for defining r is as follows: Among them, r i The function is equivalent to a regularizer, which is a regular function that improves the generalization ability of the model. It projects the learned feature map into the RGB image space to generate an intermediate layer image. Step 5: Get g from step 1 i (z) = a i , so we have: i (g i (z)) = r i (a i )=o i Among them, i It is the image generated by the i-th intermediate layer of the generator. These images of different sizes and resolutions are generated o i All of them must be sent to the discriminator, so that the gradient information flows into the current layer during back propagation to enhance information interaction and prevent gradient disappearance; Step 6: Define the discriminator sequence. Use the letter D to represent the discriminator sequence composed of all modules of the discriminator. Define the last layer of the discriminator as d(z'), and define the first layer output of the discriminator as d 0 , use y to represent the real sample, y' to represent the generated sample image, and define d similarly j is the intermediate layer function of the discriminator; when j = ki, i and j are always related to each other; therefore, the output of the jth intermediate layer function of the discriminator is defined as: a' j =d j (combine(o k-j ,a' j-1 )) =d j (combine(o i ,a' j-1 )) Where combine is the output o of the i-th intermediate layer of the generator i Perform 1×1 convolution with the corresponding output of the j-1th intermediate layer of the discriminator to integrate feature information; since the final discriminant loss function of the discriminator is not only a function of the final output of the generator, but also a function of the output of the intermediate layer, the connection of the intermediate layer allows the gradient information to flow from the intermediate layer of the discriminator to the intermediate layer of the generator during model training; Step 7: Finally, the formula for the entire discriminator sequence is recorded as: Step 8: Improve the structure of the traditional generative adversarial network and define a new loss function. The new loss function is as follows: Where L D represents the loss function of the discriminator D, L G represents the loss function of the generator G, x f Image data generated by the generator, x t is the real image data, E represents the function of finding the expected mean; the new loss function evaluates the probability that the average given real data is more real than the randomly sampled fake data This enables the entire network to generate stable, high-quality images and reduce model training time; Step 9: Optimize the model by calculating the first-order moment estimate and second-order moment estimate of the gradient and the gradient descent method with parameter adaptive learning rate.

Images