Photovoltaic cell defect detection dataset augmentation method based on generative adversarial network

Abstract

The application provides a photovoltaic cell defect detection dataset augmentation method based on a generative adversarial network, comprising: screening pictures in an original dataset according to categories, performing edge detection on the pictures according to the categories, and obtaining a binary mask corresponding to each image; constructing an improved DCGAN model, inputting the mask of each category and the corresponding image into the improved DCGAN model for training, and saving the generator network weight; using the generator network to generate augmented pictures and label, and obtaining an augmented photovoltaic cell defect detection dataset. The discriminator and the generator structure of the DCGAN network are deepened, especially the up-sampling module is added to the discriminator network to improve the feature extraction capability, and the generative adversarial network is used to augment the photovoltaic cell defect detection dataset, avoiding the problems of cutting, unreasonable distortion and color transformation of the defect patterns caused by the augmentation mode matching problem in the traditional augmentation, and further ensuring the correctness of the semantic information of the target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular, especially relates to a photovoltaic cell defect detection data set augmentation method based on a generative adversarial network. BACKGROUND

[0002] As a clean and renewable power generation method, photovoltaic power generation makes up for the defect of traditional fossil fuels that are not renewable. In the face of growing energy consumption, photovoltaic power generation has become one of the fastest growing renewable energy sources. Photovoltaic cells are the core components of photovoltaic power generation systems, and the quality of their products will directly affect the power conversion efficiency and service life. Some defects of photovoltaic cells are hidden inside, and some problems can only be observed externally after a period of operation. Therefore, it is necessary to detect defects on the photovoltaic cell panel in the industrial production line.

[0003] Electroluminescence, English name Electroluminescence, abbreviated as EL, is a common defect detection imaging technology in photovoltaic cell production lines. When the PN junction of the photovoltaic cell panel is forward powered, the panel surface will emit near-infrared light, which can be observed using a CCD camera. Since damaged areas will have obvious dark spots, visual defect detection can be used. In the traditional industrial scene, photovoltaic cell defect detection still relies on manual detection. The accuracy of manual detection depends largely on the experience, fatigue level and production line speed of workers, and is greatly affected by human factors. With the gradual maturity of AI intelligent technology, the defect detection mode based on deep learning algorithm has been widely applied, which greatly improves the efficiency and accuracy of defect detection. At the same time, since the defect texture and background texture of EL images are complex to distinguish, the detection method based on deep learning is more desirable.

[0004] Although the defect detection method based on deep learning has many advantages, it is still a data-intensive detection method. When training a deep learning model-based detection method, sufficient data is required, otherwise the model may overfit, have low generalization ability, and even not converge when the data is extremely insufficient. Therefore, ensuring sufficient training data is a basic requirement for training a deep learning model.

[0005] In actual industrial production processes, photovoltaic cell defect detection image data has the problem of insufficient number of pictures. Limited by the production environment and lack of data collection awareness in factories, defective workpieces with practical significance cannot be deliberately manufactured, and even if defects occur in the normal production process, the samples are not effectively saved. Therefore, it is absolutely necessary to augment the photovoltaic cell EL defect data set.

[0006] The main way of data set augmentation can be divided into two kinds: traditional augmentation and augmentation based on generative adversarial network. The traditional augmentation includes geometric transformation methods such as translation, flip, rotation and distortion, color channel adjustment of brightness value, local position erasure and noise addition. The traditional augmentation can generate new samples using a single or a few pictures, has the advantages of low requirement for original data set and convenient implementation of augmentation operation, and is often used in the preprocessing link of defect detection field. However, the steps such as distortion and erasure in the traditional augmentation method have the probability of damaging the semantic information of the original image, causing confusion in the training of the deep learning model, so that the rationality of the augmented picture needs to be verified manually in practical application. The augmentation method based on generative adversarial network relies on training a deep learning model to learn the latent distribution of the data set, so as to generate new data according to the feature distribution of the training data set. Unlike the method of using a single or a few pictures in the traditional augmentation method, the augmentation method based on generative adversarial network generates new samples according to the distribution of the whole data set as prior knowledge, so as to obtain augmented data with greater difference. The generative adversarial network is divided into generator and discriminator network two parts: in the training, the generator continuously optimizes and adjusts its own weight to fit the distribution of the real data set, the discriminator learns the real picture and the false picture generated by the generator, and determines the authenticity of the input picture as a binary classification task, and the network output is the probability value of the authenticity of the current picture. The core idea of training of the discriminator and the generator is derived from the zero-sum game between the two, that is, the total loss of the discriminator and the generator is zero, and the stronger one of the two networks is the loss of the other, so the two networks will continuously improve their ability to resist the other network in the training. The prior art still has the following defects:

[0007] The photovoltaic cell defect detection data set has the problem of insufficient data quantity, and when used for training a deep learning model, the model has insufficient detection capability.

[0008] The existing traditional image augmentation method has limited help for training of a deep learning model.

[0009] The existing DCGAN network is difficult to distinguish the background and the photovoltaic cell defect texture, and the network tends to generate augmented pictures containing only the background, which is not helpful for data set expansion. SUMMARY

[0010] According to the above technical problems, a photovoltaic cell defect detection data set augmentation method based on a generative adversarial network is provided.

[0011] The technical means adopted by the application are as follows:

[0012] A photovoltaic cell defect detection data set augmentation method based on a generative adversarial network, comprising:

[0013] Obtain the original dataset, filter the images in the original dataset according to category, and perform edge detection on the images according to category to obtain the binary mask corresponding to each image;

[0014] An improved DCGAN model structure is constructed, and the mask and corresponding image of each type are input into the improved DCGAN model structure for training, while saving the generator network weights.

[0015] An augmented image dataset for photovoltaic cell defect detection is obtained by using a generator network to generate and label augmented images.

[0016] Further, the step of obtaining the original dataset, filtering the images in the original dataset according to category, and performing edge detection on the images according to category to obtain a binary mask corresponding to each image includes:

[0017] Read an image and its corresponding JSON annotation file from the photovoltaic cell defect detection dataset, and extract the image at the corresponding defect location according to the target coordinates given in the annotation file.

[0018] The brightness of the image at the defect location is determined and adjusted.

[0019] Use a Gaussian filter Perform convolution operations on the pixels of the image after brightness assessment and adjustment, where σ 2 The standard deviation is a Gaussian distribution, and the kernel size is 3×3.

[0020] Calculate the gradient value and direction at each point in the image obtained after convolution;

[0021] The obtained gradient matrix is ​​subjected to non-maximum suppression. If one of the pixels is the maximum value in the same gradient direction, it is retained; otherwise, the pixel value is cleared to zero.

[0022] Double threshold filtering is applied to the image matrix obtained after nonmaximum suppression.

[0023] The image obtained after double threshold filtering is binarized. If a pixel has a non-zero pixel value, its pixel value is changed to "1". After that, the image only contains pixel values ​​"0" and "1", which constitutes a mask for the defective image.

[0024] The mask of the obtained defect image is subjected to dilation processing to improve the occasional missed detection problem in the edge detection process while supplementing the mask information in the middle part of the defect.

[0025] Furthermore, the step of determining and adjusting the brightness of the image at the defect location specifically includes:

[0026] Adjust the picture color space from RGB mode to HSV mode, extract the brightness channel to calculate the mean value;

[0027] When the picture brightness mean value is lower than 120, adjust the brightness of the picture, and scale the pixel value in the image to 0-255;

[0028] The value calculation formula of each pixel point after adjustment is as follows:

[0029]

[0030] Wherein, x is the current pixel value, x max , x min is the maximum and minimum value of the pixel point in the picture, x max′ and x min′ are the maximum and minimum values of the pixel scaling interval, that is, 255 and 0.

[0031] Further, the value and direction of the gradient of each point in the image obtained after the calculation through convolution operation are calculated, which specifically includes:

[0032] The gradient amplitude G x in x direction and the gradient amplitude G y in y direction are calculated using the Sobel operator in x direction and y direction respectively, and the calculation formula is as follows:

[0033]

[0034]

[0035] Wherein, I represents the matrix formed by the image, is the convolution operation;

[0036] The gradient G and the gradient direction θ of the pixel point are calculated, and the calculation formula is as follows:

[0037]

[0038] Further, the image matrix obtained after the non-maximum suppression is filtered by double threshold, which specifically includes:

[0039] Set double threshold to distinguish edge pixels, if one of the pixels is higher than the upper threshold, it is recorded as strong edge; When the pixel value is less than the lower threshold, it is discarded; When the pixel value is between the lower and upper threshold, if it is connected with strong edge, it is retained, if not, it is discarded.

[0040] Further, the improved DCGAN model structure is constructed, which specifically includes:

[0041] The improved generator network structure is constructed:

[0042] To improve the feature extraction ability and training stability of the generator network, a 3x3 convolution layer with a step of 1 is added after the first three transpose convolution layers to extract more features. The number of convolution kernels in each convolution layer is the same as that in the adjacent previous layer. To maintain the stability of the training, a spectral normalization layer is used after the third deconvolution layer to promote model convergence.

[0043] Improved discriminator network structure is constructed:

[0044] To adapt to the mask shape, the discriminator network needs to have a feature layer consistent with the size of the input picture. The discriminator network is mirrored and expanded with the middle output head as the midpoint. Through four deconvolution layers, the feature layer is restored to 128x128 size. The specific steps are as follows: the network input layer size is defined as 128x128. After the input layer, three convolution layers are connected in succession, and the feature map is down-sampled to 16x16x512, denoted as M1. To prevent overfitting of the network and improve stability, a spectral normalization layer and a Dropout layer are connected after the M1 layer, and the output is denoted as M2. After the M2 layer, a convolution layer is connected to down-sample the feature map M3 to 8x8x1024. M3 has two output channels. One outputs a 1x1 value through a fully connected layer and then outputs a probability value D mid with a Sigmoid function, with an interval of 0-1. The other channel is connected in series with three deconvolution layers to up-sample the feature map M3 to obtain a 64x64x128 feature map M4. Corresponding to the down-sampling link, a spectral normalization layer and a Dropout layer are connected after M4 to output a feature map M5. Finally, a deconvolution layer is connected after M5 to output a feature map with a size of 128x128x3. After passing through a Sigmoid function, it is output as the probability map at the end of the network, denoted as D out ;

[0045] Improved loss function is designed:

[0046] To highlight the difference between the defect part and the background, the edge detection is used to obtain the corresponding position of the defect part pixels in the picture. After generating the corresponding mask, the elements of the discriminator output probability matrix are processed according to the binary classification task. The designed loss function is based on the binary cross-entropy loss function, and the labels are adjusted according to the different positions of the output layer. The overall loss function of the network is:

[0047]

[0048] In the formula, s is the number of samples, and the loss is divided into three parts: the discriminator real picture loss D T , the discriminator fake picture loss D F , and the generator real picture loss G T .

[0049] The discriminator real picture loss D T is calculated as follows:

[0050]

[0051] Where λ1, λ2, and λ3 are weighting coefficients, m and n are the width and height of the image, respectively, and D appears in the formula. mid (x), D out (x) all correspond to the discriminator network structure, D mid (x) refers to the output probability value in the middle of the network, D out (x) refers to the probability graph of the final output of the network; L mask (i,j) represents the mask loss, and the specific formula is as follows: L DTout (i,j) represents the overall pixel loss of the discriminator in the real image, and the specific formula is: L DTout (i,j)=logD out (x), the overall pixel loss of the discriminator's true image L DTout (i,j) are also processed using the binary cross-entropy loss function on the output probability map D. out Each element in (x) is calculated, and the label is "1";

[0052] Discriminator Pseudoimage Loss D F The calculation formula is:

[0053]

[0054] The discriminator then inputs the image generated by the generator and sets D... mid (x), D out In (x), x is rewritten as the image G(z) randomly generated by the generator; for the probabilistic graph D in the middle of the network mid (G(z)) and terminal probability plot D out Each element of (G(z)) is used to calculate its loss using the binary cross-entropy loss function, and the average of the two is taken. The discriminator needs to classify the entire image as a pseudo, labeled "0", therefore no additional mask loss is required. The discriminator's pseudo-image loss D... F L DFout (i,j) represents the overall pixel loss of the discriminator pseudo-image:

[0055] L DFout (i,j)=log(1-D out (G(z)))

[0056] Generator True Graph Loss G T The calculation formula is:

[0057]

[0058] Locking the discriminator parameters when training the generator parameters so that they are not updated, at which time the generator output picture is judged as a real picture to train its ability to produce real pictures, and similarly for the middle part of the network D mid (G(z)) and the end D out (G(z)) Each element of the end D T GTout (i,j) is the generator overall pixel loss:

[0059] L GTout (i,j)=logD out (G(z))。

[0060] Further, the mask of each category and the corresponding image are input into the improved DCGAN model structure for training, and the generator network weight is saved, specifically including:

[0061] Set the network training configuration parameters:

[0062] The generator is trained once for every 5 times to limit the training progress of the discriminator;

[0063] The model saves the weight file of each round of more than 100 rounds of each category defect during training, and randomly selects multiple weights to generate pictures;

[0064] After the model is trained for 50 rounds, a random number is added to the label, so that in the subsequent training rounds, the true label takes a random value of 0.8-1.0, and the pseudo label takes a random value between 0-0.2. Fluctuate, rather than fixed 0 or 1;

[0065] Train the improved DCGAN model:

[0066] Step 1, read the pictures and corresponding masks in the data set, and scale the size to 128x128;

[0067] Step 2, send the scaled picture into the discriminator network, and calculate the loss function of the middle output and the end output and its corresponding label to update the discriminator network weight parameter;

[0068] Step 3, generate a random number with a dimension of 1x100, input into the generator network, and the network outputs a feature map of 128x128x3;

[0069] Step 4, input the obtained feature map into the discriminator network to obtain the loss function calculation of the middle output, the end output and its corresponding label; freeze the discriminator network parameters and only update the generator network weight parameters;

[0070] ​Step 5, after repeating steps 3 and 4 five times, save the generator network weight file of the current round;

[0071] Step 6, repeat steps 1 to 5 until the specified training rounds are met.

[0072] Further, the augmented image is generated and labeled using the generator network to obtain an augmented photovoltaic cell defect detection dataset, specifically comprising:

[0073] Select the average precision mean of the target detection field evaluation index as the evaluation index of the defect detection dataset augmentation algorithm, and the calculation process is as follows:

[0074] Calculate the intersection over union: the calculation formula is as follows:

[0075]

[0076] Where A and B represent the predicted box and the true box respectively;

[0077] Calculate the precision (P) and recall (R):

[0078] Among all the targets predicted by the target detection model, there are four cases, respectively:

[0079] True positive (TP): the number of positive samples predicted as positive samples;

[0080] False positive (FP): the number of negative samples predicted as positive samples;

[0081] False negative (FN): the number of positive samples predicted as negative samples;

[0082] True negative (TN): the number of negative samples predicted as negative samples;

[0083] The precision P represents the proportion of actual positive samples in all positive samples predicted by the model, and the calculation formula is as follows:

[0084]

[0085] The recall R represents the ratio of the number of positive samples predicted by the model to the actual number of positive samples, and the calculation formula is as follows:

[0086]

[0087] The positive and negative samples are determined by setting the intersection over union (IoU) threshold, and the recall rate and the precision rate can be plotted according to the target categories in the data set according to the current threshold, in which case the AP is defined as the area enclosed by the P-R curve:

[0088]

[0089] The mean average precision (mAP) is calculated, which represents the average of AP of all target categories, and the calculation formula is:

[0090]

[0091] Wherein, AP i represents the AP value of the i-th category, and C is the total number of categories.

[0092] Compared with the prior art, the present application has the following advantages:

[0093] 1. Restricted by the actual production environment of the photovoltaic industry, the defective products on the production line have the characteristics of random sample generation time and random defect type, which causes difficulties in establishing a photovoltaic cell defect detection data set. The discriminator and generator structure of the original DCGAN network are deepened in the present application, and an upsampling module is added to the discriminator network to improve its feature extraction capability. The existing generative adversarial network is based on the learning of the overall features of the picture to generate new samples. However, in the field of defect detection, most defect textures only occupy a very small area of the entire picture, and the existing generative adversarial network usually pays excessive attention to the background texture of the picture, while ignoring the defect texture. The present application proposes a mask loss function and a corresponding discriminator network structure to guide the network to focus on the texture of the defect site, thereby generating more pictures with defect texture that are more helpful for expanding the defect detection data set. Compared with the traditional data set expansion method, the present application uses the generative adversarial network to expand the photovoltaic cell defect detection data set, avoiding the problems of cutting, unreasonable distortion and color transformation of defect patterns caused by the expansion method in the traditional expansion method, thereby ensuring the correctness of the semantic information of the target and playing a positive role in the detection model.

[0094] 2. The present application takes the expansion of the defect detection data set as the starting point to improve the accuracy of the defect detection algorithm based on deep learning, which is an offline image expansion operation. Therefore, compared with the improvement of the detection algorithm, the improved DCGAN defect detection data set expansion method proposed in the present application is only used as a preprocessing means during the training of the detection algorithm, and does not need to occupy time during the inference of the detection algorithm, thereby giving a margin for further improvement of the subsequent detection algorithm itself.

[0095] 3、 The improved DCGAN model can effectively improve the detection accuracy of the target detection algorithm, and can be applied to the augmentation task of photovoltaic cell defect detection data set.

[0096] Based on the above reasons, the present application can be widely popularized in the field of computer vision. BRIEF DESCRIPTION OF DRAWINGS

[0097] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0098] Figure 1 The method flowchart of the present application.

[0099] Figure 2 The improved generator structure of the present application.

[0100] Figure 3 The improved discriminator structure of the present application.

[0101] Figure 4 The P-R curve of the original data training of the present application.

[0102] Figure 5 The P-R curve of the traditional augmentation training of the present application.

[0103] Figure 6 The P-R curve of the improved DCGAN training of the present application. DETAILED DESCRIPTION

[0104] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0105] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. The description of the at least one exemplary embodiment is actually only illustrative, not as any limitation on the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0106] It is to be understood that the terms so far as the grammar used herein is concerned are to be interpreted in their dictionary meanings and are not to be interpreted in the context of legal terms unless so explicitly stated. It is also to be understood that the terminology and description provided above are for the purpose of simplifying the present disclosure and the invention, and are not intended to limit the scope of the application of the present invention, and the use of such terminology, and description is understood to also cover any technical equivalents for the subject matter covered. It is also to be understood that the terminology and description provided above are for the purpose of simplifying the present disclosure and the invention, and are not intended to limit the scope of the application of the present invention, and the use of such terminology, and description is understood to also cover any technical equivalents for the subject matter covered.

[0107] The relative arrangement of parts and steps, numerical expressions, and numerical values set forth in the examples are not intended to limit the scope of the application unless specifically so stated. It is also to be understood that the drawings are not necessarily drawn to scale and that the dimensions of the various parts shown in the drawings are intended to be illustrative only and not limiting of the scope of the application. Techniques, methods, and apparatus known to those of ordinary skill are not discussed in detail but can be employed with the systems and methods described herein. Unless otherwise specifically stated, all examples shown and discussed herein are exemplary only and are not intended to be limiting of the scope of the application. Thus, other examples of the exemplary embodiments can have different values. It is to be noted that like numbers and letters used herein refer to like items throughout the several views and the use of "about", "approximately", "substantially" or "near" indicates that the value intended to be used is not to be taken as an absolute quantity or range but is subject to variation as understood by one of ordinary skill in the art, unless specifically so stated. It is also to be understood that the terminology and description provided above are for the purpose of simplifying the present disclosure and the invention, and are not intended to limit the scope of the application of the present invention, and the use of such terminology, and description is understood to also cover any technical equivalents for the subject matter covered.

[0108] In the description of the present application, it is to be understood that the orientation or positional relationships indicated by orientation words such as "front, back, upper, lower, left, right", "horizontal, vertical, perpendicular, horizontal" and "top, bottom" and the like are generally based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate and imply that the devices or elements referred to must have a particular orientation or be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the scope of protection of the present application: the orientation words "inner, outer" refer to the inner and outer relative to the contour of each component itself.

[0109] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0110] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0111] like Figure 1 As shown, this invention provides a method for augmenting photovoltaic cell defect detection datasets based on generative adversarial networks, including:

[0112] S1. Obtain the original dataset, filter the images in the original dataset according to their categories, and perform edge detection on the images according to their categories to obtain the binary mask corresponding to each image.

[0113] S2. Construct an improved DCGAN model structure, input the mask and corresponding image of each type into the improved DCGAN model structure for training, and save the generator network weights.

[0114] S3. Use a generator network to generate augmented images and label them to obtain an augmented photovoltaic cell defect detection dataset.

[0115] In a specific implementation, as a preferred embodiment of the present invention, step S1 involves obtaining the original dataset, filtering the images in the original dataset according to their categories, performing edge detection on the images according to their categories, and obtaining a binary mask corresponding to each image, including:

[0116] S11. Read an image and its corresponding JSON annotation file from the photovoltaic cell defect detection dataset, and extract the image of the corresponding defect location according to the target coordinates given in the annotation file.

[0117] S12. The brightness of the defect location image is determined and adjusted; in this embodiment, this specifically includes:

[0118] S121, adjust the picture color space from RGB mode to HSV mode, extract the luminance channel to calculate the mean value;

[0119] S122, when the picture luminance mean value is lower than 120, adjust the picture luminance, scale the pixel value in the image to 0-255;

[0120] S123, the value of each pixel point after adjustment is calculated as follows:

[0121]

[0122] Wherein, x is the current pixel value, x max , x min is the maximum and minimum value of the pixel point in the picture, x max′ and x min′ are the maximum and minimum values of the pixel scaling interval, i.e. 255 and 0.

[0123] S13, use the Gaussian filter to perform convolution operation on the pixel of the picture after luminance judgment and adjustment, wherein σ 2 is the standard deviation of Gaussian distribution, and the convolution kernel size is 3x3;

[0124] S14, calculate the value and direction of the gradient of each point in the image obtained after convolution operation; in this embodiment, specifically including:

[0125] S141, use the Sobel operator in x direction and y direction to calculate the gradient amplitude G x in x direction and the gradient amplitude G y in y direction respectively, and the calculation formula is as follows:

[0126]

[0127]

[0128] Wherein, I represents the matrix formed by the image, is the convolution operation;

[0129] S142, calculate the gradient G and the gradient direction θ of the pixel point, and the calculation formula is as follows:

[0130]

[0131] S15, perform non-maximum suppression on the obtained gradient matrix, if one of the pixel points is the maximum value point in the same gradient direction, it is retained, otherwise the pixel value is cleared; after non-maximum suppression, the originally blurred edge will be trimmed more finely, and the more real edge information is retained;

[0132] S16, performing double-threshold filtering on the image matrix obtained after the non-maximum suppression; in the embodiment, specifically comprising:

[0133] The double thresholds are set to distinguish edge pixels, if one pixel is higher than the upper threshold, it is recorded as a strong edge; if the pixel value is less than the lower threshold, it is discarded; if the pixel value is between the lower and upper thresholds, if it is connected with a strong edge, it is retained, if not, it is discarded. The defect features in the defect data set are different, and different manifestations exist in a single defect category, therefore, in actual application, the threshold is flexibly adjusted according to the actual edge detection effect.

[0134] S17, performing binaryzation processing on the image obtained after the double-threshold filtering, if one pixel has a non-zero pixel value, the pixel value is changed to “1”, thereafter, the image only contains pixel values “0” and “1”, which constitutes a mask of the defect image;

[0135] S18, applying inflation processing to the mask of the obtained defect image, for supplementing the middle part of the mask information in the defect and improving the occasional missing detection problem in the edge detection link.

[0136] In specific implementation, as a preferred embodiment of the present application, in the step S2, the improved DCGAN model structure is constructed, specifically comprising:

[0137] S21, constructing an improved generator network structure:

[0138] The generator network structure is shown in Figure 2 In order to improve the feature extraction ability and training stability of the generator network, a convolution with a convolution kernel of 3x3 and a step of 1 is additionally added after the first three transpose convolution layers in the network for extracting more features, and the number of convolution kernels in each convolution layer is the same as that of the adjacent previous layer; in the figure, “SN” is a spectral normalization layer, in order to maintain the training stability, a spectral normalization layer is used after the third deconvolution layer in the network to promote model convergence; it should be additionally noted that the purpose of the spectral normalization used in the present application is inconsistent with the original intention, here, it is only for increasing stability of a specific network layer number. Except for the activation function at the end of the network, Leaky ReLU is used as the activation function in the present application.

[0139] S22, constructing an improved discriminator network structure:

[0140] The discriminator network structure is shown in Figure 3As shown, in order to adapt to the mask shape, the discriminator network needs to have a feature layer consistent with the input picture size; inspired by the U-Net network structure, the discriminator network is mirror expanded with the middle output head as the midpoint, and the feature layer is restored to 128x128 size through 4 deconvolution layers, the specific steps are as follows: the network input layer size is defined as 128x128, 3 convolution layers are connected after the input layer, and the feature map is down-sampled to 16x16x512, denoted as M1; in order to prevent network overfitting and improve stability, a spectral normalization layer and a Dropout layer are connected after the M1 layer, and the output is denoted as M2; a convolution layer is connected after the M2 layer to down-sample the feature map M3 to 8x8x1024; M3 has two output channels, one of which outputs a value of 1x1 through a fully connected layer, and then outputs a probability value D mid with a Sigmoid function, the interval is 0-1; the other channel is connected in series with 3 deconvolution layers to up-sample the feature map M3 to obtain a feature map M4 of 64x64x128; corresponding to the down-sampling link, a spectral normalization layer and a Dropout layer are connected after M4 to output a feature map M5; finally, a deconvolution layer is connected after M5 to output a feature map with a size of 128x128x3, which is output as a probability map after passing through a Sigmoid function, denoted as D out ;

[0141] S23, design improved loss function:

[0142] It can be seen from observing the defect image that it has the characteristics of low distinction between defect texture and background and that background pixels occupy most of the image area. Without guiding the network, the original version of DCGAN tends to generate pictures containing only background texture. In order to highlight the difference between the defect part and the background, the edge detection is used to obtain the position corresponding to the defect part pixel in the picture; after generating the corresponding mask, each element of the discriminator output probability matrix is processed according to the binary classification task; the designed loss function takes the binary cross entropy loss function as the basement, and adjusts the label according to the different positions of the output layer, and the overall loss function of the network is:

[0143]

[0144] In the formula, s is the number of samples, and the loss is divided into three parts: discriminator real picture loss D T , discriminator fake picture loss D F , and generator real picture loss G T ;

[0145] The calculation formula of the discriminator real picture loss D T is:

[0146]

[0147] wherein λ1, λ2, λ3 are weight coefficients, m and n are width and height of the image respectively, D mid (x) and D out (x) correspond to the discriminator network structure, D mid (x) refers to the intermediate output probability value in the network, D out (x) refers to the final output probability map of the network; L mask (i,j) represents the mask loss, and the specific formula is: L DTout (i,j) represents the overall pixel loss of the discriminator real image, and the specific formula is: L DTout (i,j) = log D out (x), the overall pixel loss of the discriminator real image L DTout (i,j) is also calculated using a binary cross-entropy loss function on each element of the output probability map D out (x), and the label is "1";

[0148] The discriminator fake image loss D F is calculated as follows:

[0149]

[0150] The discriminator inputs the image generated by the generator at this time, and D mid (x), D out (x) is rewritten as the generator randomly generates an image G(z); the probability map D mid (G(z)) in the middle of the network and the final probability map D out (G(z)) are calculated using a binary cross-entropy loss function, and the average of the two is taken at the end; the discriminator needs to determine the entire image as fake at this time, and the label is "0", so there is no need to calculate the mask loss additionally; the discriminator fake image loss D F is calculated as follows: DFout (i,j) is the overall pixel loss of the discriminator fake image:

[0151] L DFout (i,j) = log(1-D out (G(z))

[0152] The calculation formula of the generator real image loss G T is as follows:

[0153]

[0154] When training the generator parameters, the discriminator parameters are locked and not updated, at this time the generator output image is determined as a real image to train its ability to generate real images, and the same is true for the intermediate image D mid (G(z)) and the final Dout Each element of G(z) uses a binary cross-entropy loss function to calculate its loss, with the label being "1"; the generator loss G T L GTout (i,j) is the generator overall pixel loss:

[0155] L GTout (i,j)=logD out (G(z)).

[0156] In specific implementation, as a preferred embodiment of the present application, in step S2, the mask and the corresponding image of each category are input into the improved DCGAN model structure for training, and the generator network weight is saved, which specifically includes:

[0157] S21, setting network training configuration parameters:

[0158] The generator is trained 1 time after every 5 training times of the discriminator to limit the training progress of the discriminator;

[0159] The model saves the weight file of each round of more than 100 rounds of each category defect during training, and randomly selects multiple weights to generate pictures;

[0160] After 50 rounds of model training, a random number is added to the label, so that in the subsequent training rounds, the true label takes a random value of 0.8-1.0, and the pseudo label takes a random value between 0-0.2, instead of a fixed 0 or 1;

[0161] S22, training the improved DCGAN model:

[0162] S221, reading the pictures and corresponding masks in the data set, and scaling the size of the two to 128x128;

[0163] S222, sending the scaled pictures into the discriminator network, and performing loss function calculation on the middle output and end output of the network and the corresponding label, and updating the discriminator network weight parameters;

[0164] S223, generating a random number with a dimension of 1x100, inputting it into the generator network, and outputting a feature map with a size of 128x128x3 from the network;

[0165] S224, inputting the obtained feature map into the discriminator network to obtain the loss function calculation of the middle output and end output of the network and the corresponding label; freezing the discriminator network parameters and only updating the generator network weight parameters;

[0166] S225, after repeating steps S223 and S224 five times, saving the generator network weight file of the current round;

[0167] S226, repeating steps S221 to S225 until a predetermined training round is met.

[0168] In particular implementation, as a preferred embodiment of the present application, in step S3, the generator network is used to generate augmented pictures and labeling, and an augmented photovoltaic cell defect detection dataset is obtained, specifically including:

[0169] The mean Average Precision (mAP) in the target detection field is selected as the evaluation index of the defect detection dataset augmentation algorithm, and the calculation process is as follows:

[0170] The Intersection over Union (IoU) is calculated, that is, the ratio of the intersection and union of the area of the frame predicted by the target detection network and the real frame. The larger the intersection over union, the higher the coincidence degree of the network prediction result and the actual target position, and the more accurate the prediction. The calculation formula of the intersection over union is as follows:

[0171]

[0172] Wherein, A and B represent the predicted frame and the real frame respectively;

[0173] The precision (P) and recall (R) are calculated:

[0174] Among all the targets predicted by the target detection model, there are four cases, respectively:

[0175] True Positive (TP): the number of positive samples predicted as positive samples;

[0176] False Positive (FP): the number of negative samples predicted as positive samples;

[0177] False Negative (FN): the number of positive samples predicted as negative samples;

[0178] True Negative (TN): the number of negative samples predicted as negative samples;

[0179] The precision P represents the proportion of actual positive samples in all positive samples predicted by the model, and the calculation formula is as follows:

[0180]

[0181] The recall R represents the ratio of the number of positive samples predicted by the model to the actual number of positive samples, and the calculation formula is as follows:

[0182]

[0183] Positive and negative samples are determined by setting the intersection over union (IoU) threshold, and the recall rate and precision rate can be plotted according to the target class in the data set according to the current threshold. In this case, the AP is defined as the area enclosed by the P-R curve:

[0184]

[0185] The mean average precision (mAP) is calculated, which represents the average of AP of all target categories, and the calculation formula is:

[0186]

[0187] where AP i represents the AP value of the i-th category, and C is the total number of categories. In the present application, the IoU threshold is selected as 0.5 with a span of 0.05 to 0.95, and the average is denoted as mAP@0.5 and mAP@0.5:0.95.

[0188] Embodiment:

[0189] To demonstrate the effectiveness and superiority of the model proposed in the present application, the PVEL-AD data set is selected for image augmentation experiments. The data set contains 12 types of defect EL images and good images of photovoltaic cells. The present application selects 7 types of defect images (including broken grid, linear crack, star-shaped crack, black core, thick line, misalignment, and short circuit) to train the target detection model YOLOv5-m to compare the augmentation effect. Among the 7 types of images, 4 types of target detection model detection effect are selected, including broken grid, linear crack, star-shaped crack, and thick line, to perform augmentation. In the experiment, the augmentation method provided by the model is disabled, and only the externally given data set is used for training. In the experiment, the improved DCGAN method proposed in the present application is compared with the traditional augmentation method. The output augmented images using the traditional augmentation method are as follows: horizontal flip, vertical flip, horizontal and vertical translation, random cutting, random cutting of the label box region, and random contrast. A single image can be processed by one or more strategies. To correspond to the number of images generated by M-DCGAN, the traditional augmentation method also generates 2400 images. In the experiment, the training set selects the following number combination scheme, and the real images include 7190 defect-containing images and 5548 good images:

[0190] 1. 12738 real images

[0191] 2. 2400 traditional augmented images + 12738 real images

[0192] 3,240 improved DCGAN augmented pictures + 12738 real pictures

[0193] The verification set is a combination of 1796 defect-containing pictures and 5805 good pictures. The experimental results are shown in Table 1. The addition of the augmented pictures generated by the improved DCGAN of the present application can make the mAP@0.5 of the YOLOv5-m model increase by 1.1% based on the original data, and exceed the traditional augmented algorithm by 0.7%; under the mAP@0.5:0.95 rating index, it exceeds the original data by 2%, and exceeds the traditional augmented algorithm by 1.1%.

[0194] Table 1 Comparison of training effects of improved DCGAN augmented algorithm

[0195]

[0196] The experiment further explores the improvement ability of the present application to each defect category in the data set. According to the YOLOv5-m training results, the P-R curve of each defect category under the IoU threshold of 0.5 is drawn, as shown in Figures 4 to 6 .

[0197] The experimental results show that the four defect pictures augmented by the network of the present application are all improved, among which the broken grid increases by 0.3% relative to the original data and by 0.9% relative to the traditional augmented; the linear crack increases by 2.9% relative to the original data and by 1.5% relative to the traditional augmented; the star crack increases by 3.5% relative to the original data and by 2.6% relative to the traditional augmented; the thick line increases by 0.2% relative to the original data and by 0.3% relative to the traditional augmented. The traditional augmented has a negative effect on the broken grid and thick line categories, which is speculated to be due to the random contrast operation in the traditional augmented method, which has a probability of damaging the image features, indicating that the traditional augmented method cannot adapt to all targets, and the augmented based on the improved DCGAN of the present application does not have a negative effect on the data set.

[0198] In summary, the augmented pictures generated by the improved DCGAN model of the present application can effectively improve the detection accuracy of the target detection algorithm, and can be applied to the augmentation task of the photovoltaic cell defect detection data set.

[0199] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for part or all of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for augmenting photovoltaic cell defect detection datasets based on generative adversarial networks, characterized in that, include: Obtain the original dataset, filter the images in the original dataset according to category, and perform edge detection on the images according to category to obtain the binary mask corresponding to each image; An improved DCGAN model structure is constructed, and the mask and corresponding image of each type are input into the improved DCGAN model structure for training, while saving the generator network weights. The construction of the improved DCGAN model structure includes: Constructing an improved generator network structure: To improve the feature extraction capability and training stability of the generator network, an additional convolutional layer with a kernel of 3 is added after the first three transposed convolutional layers of this network.

3. Convolutions with a stride of 1 are used to extract more features. The number of kernels in each convolutional layer is the same as that in the previous neighboring layer. To maintain training stability, a spectral normalization layer is used after the third deconvolutional layer of the network to promote model convergence. Constructing an improved discriminator network structure: To adapt to the mask shape, the discriminator network needs to have feature layers of the same size as the input image; the discriminator network is mirror-expanded with the middle output head as the midpoint, and the feature layers are restored to 128 using 4 deconvolution layers. The specific steps are as follows: Specify the network input layer size as 128. 128, with three consecutive convolutional layers following the input layer, downsampling the feature map to 16. 16 512, denoted as M1; to prevent network overfitting and improve stability, a spectral normalization layer and a Dropout layer are connected after layer M1, and the output is denoted as M2; a convolutional layer is then added after layer M2 for downsampling to obtain 8. 8 Feature map M3 with 1024; M3 has two output channels, one of which outputs 1 through a fully connected layer. A value of size 1, followed by the Sigmoid function to output the probability value. The range is 0~1; another channel connects three deconvolutional layers in series to upsample the feature map M3 to obtain 64. 64 Feature map M4 is 128; corresponding to the downsampling stage, a spectral normalization layer and a Dropout layer are followed after M4 to output feature map M5; finally, a deconvolution layer is followed after M5 to output feature map M5 of size 128. 128 The feature map of 3, after passing through the Sigmoid function, becomes the output probability map at the end of the network, denoted as . ; Design an improved loss function: To highlight the difference between the defect and the background, edge detection is used to obtain the pixel positions corresponding to the defect in the image. After generating the corresponding mask, each element of the discriminator's output probability map matrix is ​​processed as a binary classification task. The designed loss function is based on the binary cross-entropy loss function, and its label is adjusted according to different positions in the output layer. The overall network loss function is: In the formula, Given the number of samples, the loss is divided into three parts: discriminator ground truth loss. Discriminator Pseudoimage Loss Generator True Image Loss ; Discriminator True Image Loss The calculation formula is: in, These are the weighting coefficients. and These represent the width and height of the image, respectively, and appear in the formula. , All correspond to the discriminator network structure. Refers to the output probability value in the middle of the network. Refers to the probability graph of the final output of the network; Representing mask loss, the specific formula is: ; The formula represents the overall pixel loss of the discriminator's real image, specifically: The overall pixel loss of the discriminator in the true image Similarly, the binary cross-entropy loss function is used to analyze the output probability map. Calculate for each element and assign the label "1"; Discriminator Pseudoimage Loss The calculation formula is: The discriminator then inputs the image generated by the generator and... , middle Rewrite as a generator to randomly generate images ; Probability graph of the middle part of the network With terminal probability plot Each element is processed using the binary cross-entropy loss function, and the average of the two losses is taken. The discriminator then needs to classify the entire image as a fake, labeled "0", therefore no additional mask loss is required. The discriminator's fake image loss... middle Overall pixel loss for the discriminator pseudoimage: Generator True Graph Loss The calculation formula is: During generator parameter training, the discriminator parameters are locked and not updated. At this point, the generator's output images are treated as real images to train its ability to generate real images. The same applies to the network's mid-image... With end Each element has its loss calculated using the binary cross-entropy loss function, labeled "1"; generator loss middle Overall pixel loss for the generator: An augmented image dataset for photovoltaic cell defect detection is obtained by using a generator network to generate and label augmented images.

2. The method for augmenting photovoltaic cell defect detection datasets based on generative adversarial networks according to claim 1, characterized in that, The process of obtaining the original dataset, filtering the images in the original dataset according to category, performing edge detection on the images according to category, and obtaining a binary mask corresponding to each image includes: Read an image and its corresponding JSON annotation file from the photovoltaic cell defect detection dataset, and extract the image of the corresponding defect location according to the target coordinates given in the annotation file; The brightness of the image at the defect location is determined and adjusted. Use a Gaussian filter The image pixels after brightness assessment and adjustment are convolved, where... The standard deviation is Gaussian, and the kernel size is 3. 3; Calculate the gradient value and direction at each point in the image obtained after convolution; The obtained gradient matrix is ​​subjected to non-maximum suppression. If one of the pixels is the maximum value in the same gradient direction, it is retained; otherwise, the pixel value is cleared to zero. Double threshold filtering is applied to the image matrix obtained after nonmaximum suppression. The image obtained after double threshold filtering is binarized. If a pixel has a non-zero pixel value, its pixel value is changed to "1". After that, the image only contains pixel values ​​"0" and "1", which constitutes a mask for the defective image. The mask of the obtained defect image is subjected to dilation processing to improve the occasional missed detection problem in the edge detection process while supplementing the mask information in the middle part of the defect.

3. The method for augmenting photovoltaic cell defect detection datasets based on generative adversarial networks according to claim 2, characterized in that, The step of determining and adjusting the brightness of the defect location image specifically includes: Adjust the image color space from RGB mode to HSV mode, and extract the brightness channel to calculate the mean. When the average brightness of the image is below 120, adjust the brightness of the image and scale the pixel values ​​in the image to between 0 and 255. The formula for calculating the value of each pixel after adjustment is as follows: ， in, The current pixel value. These are the maximum and minimum values ​​of the pixels in the image. These are the maximum and minimum values ​​of the pixel scaling range, namely 255 and 0.

4. The method for augmenting photovoltaic cell defect detection datasets based on generative adversarial networks according to claim 2, characterized in that, The calculation of the gradient values ​​and directions at each point in the image obtained after convolution specifically includes: use x direction and y The Sobel operators for the directions are calculated respectively. x gradient magnitude in direction and y gradient magnitude in direction The calculation formula is as follows: in, Refers to the matrix formed by the image. This is a convolution operation; Calculate the gradient of this pixel. With gradient direction The calculation formula is as follows: 。 5. The method for augmenting photovoltaic cell defect detection datasets based on generative adversarial networks according to claim 2, characterized in that, The process of performing dual-threshold filtering on the image matrix obtained after non-maximum suppression specifically includes: Set two thresholds to distinguish edge pixels. If a pixel is slightly higher than the upper threshold, it is recorded as a strong edge. If the pixel value is less than the lower threshold, it is discarded. If the pixel value is between the lower and upper thresholds, it is retained if it is connected to a strong edge, and discarded if it is not connected.

6. The method for augmenting photovoltaic cell defect detection datasets based on generative adversarial networks according to claim 1, characterized in that, The process of inputting each type of mask and corresponding image into the improved DCGAN model structure for training, and saving the generator network weights, specifically includes: Configure network training parameters: The discriminator training progress is limited by stipulating that the generator is trained once every 5 training iterations; During training, the model saves the weight file for each type of defect for more than 100 rounds, and randomly selects multiple weights from it to generate images. After training the model for 50 rounds, add random numbers to the labels so that in subsequent training rounds, the true labels take random values ​​between 0.8 and 1.0, and the pseudo labels take random values ​​between 0 and 0.2, instead of being fixed at 0 or 1. Training the improved DCGAN model: Step 1: Read the images and corresponding masks from the dataset, and scale them to 128. 128; Step 2: Feed the scaled image into the discriminator network. Calculate the loss function using the network's middle and end outputs and their corresponding labels, and update the discriminator network weight parameters. Step 3: Generate a dimension of 1 A random number of 100 is input into a generator network, and the network outputs 128. 128 3. Feature map; Step 4: Input the obtained feature map into the discriminator network, obtain the middle output and the end output of the network and their corresponding labels, and calculate the loss function; freeze the discriminator network parameters and only update the generator network weight parameters; Step 5: After repeating steps 3 and 4 five times, save the generator network weight file for the current round. Step 6: Repeat steps 1 through 5 until the required number of training rounds is met.

7. The method for augmenting photovoltaic cell defect detection datasets based on generative adversarial networks according to claim 1, characterized in that, The process of generating and labeling augmented images using a generator network to obtain an augmented photovoltaic cell defect detection dataset specifically includes: The mean accuracy, a key evaluation metric in the field of object detection, is selected as the evaluation metric for augmentation algorithms on defect detection datasets. The calculation process is as follows: Calculate the intersection-union ratio: The formula is as follows: in, A and B These represent the predicted bounding box and the ground truth bounding box, respectively. Calculate precision and recall: Of all the targets predicted by the object detection model, there are four categories: True case: The number of positive samples that are predicted as positive samples; False positives: The number of negative samples that are predicted as positive samples; False negatives: The number of positive samples that are predicted as negative samples; True counterexamples: The number of negative samples that are predicted as negative samples; Precision P represents the proportion of actual positive samples out of all positive samples predicted by the model. The calculation formula is as follows: Recall R represents the ratio of the number of positive samples predicted by the model to the actual number of positive samples. The formula is as follows: Positive and negative samples are determined by setting an Intersection over Union (IoU) threshold. Recall and precision can be plotted as PR curves according to the target categories in the dataset based on the current threshold. In this case, AP is defined as the region enclosed by the PR curve. The mean accuracy (AP) is calculated, which represents the mean AP for all target types. The formula is as follows: in, AP i Indicates the first i A variety AP value, C This represents the total number of categories.

Images