Solar screen surface defect detection method based on branch training

Abstract

The invention belongs to the field of image recognition, and provides a solar screen surface defect detection method based on branch training. Aiming at the problem of poor solar screen surface defect detection effect in the prior art, the invention provides the following scheme: obtaining a solar screen image through an image collection platform, then performing feature extraction on the image by utilizing a model, and finally performing prediction by utilizing a model output feature value. And the surface defects of the solar screen printing plate are classified and positioned. According to the invention, a module with multi-input auxiliary training branches and autonomous learning hidden correlation is redesigned, the gradient transmission depth is reduced, and the small target detection capability is improved; a multi-branch decoupling head with a re-parameterization capability is designed, and a DBB module is used to realize the agglomeration of multi-scale semantic information; by using a ShapeIoU loss function, target length and width information is fused into loss calculation. According to the method, the training parameters are added in the training stage, so that the recognition speed is not increased, and the detection precision is relatively high.

Description

Technical Field

[0001] The invention belongs to the field of image recognition and provides a solar panel surface defect detection method based on branch training. Background Art

[0002] With the development of the global economy, the demand for energy continues to increase. In the context of the transformation of the energy structure, solar energy as a clean and renewable resource has attracted much attention and has become the focus of attention of governments and enterprises around the world. Solar photovoltaic power generation is one of the main ways to utilize solar energy, and one of its core components is the solar screen. However, the manufacturing process of the solar screen is complicated and prone to various defects, such as hole blockage, fiber filaments, steel wire abnormalities, etc. Such defects may affect its conversion efficiency and service life. Therefore, the surface defect detection of the solar screen is a very important step in upstream production and downstream incoming material inspection.

[0003] At present, the surface defect detection of solar screens in my country mainly relies on manual inspection, which uses LED light to illuminate the screen, and then observes it directly with the naked eye or with a magnifying glass. The labor intensity is high, and there is no standard, so it can only rely on the subjective judgment of workers. However, this method has problems such as low efficiency and low accuracy. Therefore, it is of great significance to develop an efficient and accurate algorithm for detecting surface defects of solar screens.

[0004] Researchers have widely applied deep learning algorithms to solve the problem of solar panel surface defect detection. However, most scholars only focus on small targets or the compatibility of speed and accuracy, and pay less attention to issues such as the number of sample categories and abnormal aspect ratios of targets in the dataset.

[0005] A new detection method is needed to address the problems of small targets, insufficient training samples for some categories of defects, and unbalanced aspect ratio of targets in existing solar panel surface defects. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings of low solar panel surface detection accuracy and poor real-time performance in the prior art, and to propose a solar panel surface defect detection method based on multi-input auxiliary branch training.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for detecting surface defects of solar panels based on branch training comprises the following steps:

[0009] S1: Use the image collection and detection platform to capture images for neural network training;

[0010] S2: Pack the training images and use the multi-branch backbone for feature extraction and fusion;

[0011] S3: Send the fused features to the multi-branch detection head for classification and positioning;

[0012] S4: Loss calculation between the output value of the detection head and the real labeled data;

[0013] S5: Using loss function to optimize neural network;

[0014] S6: Use the optimized model to process the actual image.

[0015] In S1, the images are taken by using an image collection and detection platform, and the taken photos are used as training data for the neural network.

[0016] In S2, a multi-branch trunk is used for feature extraction and fusion. A side structure with a main branch structure and an auxiliary branch to help the main branch learn is designed. In the auxiliary branch, a multi-input fusion module with autonomous learning implicit correlation is used to realize feature fusion of different branches. The module calculation formula is shown in formula (2):

[0017] Output = Conv(Concat(Input n )+A)×M,n∈[1,+∞] (2)

[0018] Where: Input n represents the nth input feature map of the ICCBFuse module, Output represents the output feature map result of the ICCBFuse module, A represents the addition matrix of implicit training parameters, and M represents the multiplication matrix of implicit training parameters.

[0019] The auxiliary branch in S2 can assist in optimizing the training parameters of the main branch when the loss is obtained, thereby reducing the gradient transfer path of the main branch. The direct loss fusion ratio of the main branch and the auxiliary branch is shown in formula (1):

[0020] L=0.25×L a +L m (1)

[0021] Where: L is the total loss value, L a Predict the head loss value for the auxiliary branch, L m Predict the head loss value for the main branch.

[0022] The detection head in S3 is built based on a multi-branch DBB module, which has 1×1 and K×K convolutions and average pooling operations. These calculations are used to build a module with a variety of internal branches, and are re-parameterized into a K×K convolutional single-path structure in the inference stage to maintain high-speed prediction.

[0023] A common convolution kernel is essentially a tensor whose shape is (number of output channels, number of input channels, convolution kernel height, convolution kernel width). However, the convolution operation is essentially a linear operation. Therefore, in some cases, the convolution operation has a linear property. Its linear formula is shown in equations (1) and (2):

[0024] Additive property: When the two convolution kernels have the same shape, the result of the convolution operation satisfies the additive property.

[0025]

[0026] Homogeneity:

[0027]

[0028] Where: F1 and F2 represent two independent convolution kernels, p is a real number, It is a convolution operation, Input is the input.

[0029] The loss function in S5 uses the ShapeIoU loss function, and the overall formula of the loss function is as shown in the formula:

[0030] L = 1-IOU + L d +0.5×L Ω (3)

[0031] Where: L represents the overall result of the loss function calculation. IOU (Intersection over Union) is a common intersection loss and is also the most commonly used regression coordinate loss in the field of target detection, as shown in formula (4). As shown in formula (8), L d Represents the distance loss of the shape, as shown in formula (9), L Ω Represents the value loss of the shape.

[0032]

[0033] Where: B represents the location area of ​​the prediction box, B gt Represents the location area of ​​the ground-truth box.

[0034]

[0035] Where s represents the scaling hyperparameter related to the task and model. In this paper, the small model uses s=1.0, the large model uses s=1.5, and w gt and h gt Represent the width and height of the real box respectively, W and H represent the weight coefficients in the horizontal and vertical directions respectively, and these two values ​​are related to the shape of the real box.

[0036] Ld =H×(xx gt ) 2 / c 2 +W×(yy gt ) 2 / c 2 (6)

[0037] Where: (x gt ,y gt ) represents the coordinates of the center point of the real box, (x, y) represents the coordinates of the center of the predicted box, and c represents the direct distance between the center points of the real box and the predicted box.

[0038]

[0039] Where: w and h represent the width and height of the prediction box.

[0040] Compared with the prior art, the advantages of the present invention are:

[0041] The defect of deep learning visual algorithms without threshold adaptation when using solar screen surface defect detection is solved. First, through multi-input auxiliary training branches, the impact of image information loss and information bottleneck is reduced, and a module with self-learning hidden correlation is used to enhance the understanding of input feature maps and improve the detection ability of small targets; secondly, a multi-branch decoupling head is proposed, which has multiple branches in the training stage to increase the observation angle, and re-parameterization and merging are performed in the inference stage to improve the accuracy of difficult-to-train samples with a small number of categories; finally, the ShapeIoU loss function is used to solve the coordinate accuracy of samples with imbalanced aspect ratios. The model in the present invention achieves higher detection accuracy while maintaining the original parameter quantity, calculation amount and detection speed, and can quickly and accurately detect surface defects of solar screens. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A structural diagram of the network proposed by the present invention;

[0043] Figure 2 A comparison diagram of the original PGI structure and the multi-input assisted implicit training structure in the present invention;

[0044] Figure 3 The module structure diagram of learning hidden correlation proposed by the present invention;

[0045] Figure 4 This is a structural diagram of the DBB module in the multi-branch decoupling head proposed by the present invention;

[0046] Figure 5 This is a structural diagram of the solar screen surface defect image collection and detection platform proposed by the present invention;

[0047] Figure 6 This is a loss structure diagram of the focusing target aspect ratio proposed by the present invention. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0049] Reference Figure 1 First, use the image collection and detection platform to shoot images for neural network training, then package the training images, and use the multi-branch backbone to extract and fuse features. Then send the fused feature map to the detection head for classification and positioning. The output of the detection head is compared with the real data to calculate the loss function, and the loss function is used to optimize the neural network. Finally, the optimized neural network is used to process the actual image.

[0050] The GELAN-C model is explained as follows: The GELAN-C model is the inference version of the YOLOv9-C model. It is a single-stage target detection model with accuracy and efficiency. It can directly output the category and location information of the target to achieve an end-to-end detection process. The GELAN-C model introduces an ELAN module (RepNCSPELAN / RepNCSPELAN4) with reparameterization capabilities to improve model performance without increasing additional computational effort. The RepNCSPELAN module is optimized based on CPSNet and ELAN, and retains the multi-branch cascade structure of ELAN.

[0051] Among them, the RepNCSP module with internal re-parameterized convolution is used as the branch of ELAN, and the gradient shortest and longest path characteristics of the ELAN module are inherited to obtain different depth information and balance the network learning ability. In addition, the GELAN-C model adopts a decoupled head structure to predict position information and category information respectively, cancel the parameter sharing of the two tasks of classification and regression, thereby reducing redundancy, effectively reducing the number of parameters and computational complexity, and enhancing the generalization ability and robustness of the model. To further optimize the training process, the GELAN-C model adopts an auxiliary supervision framework of programmable gradient information (PGI). In the training stage, a reversible auxiliary training branch is introduced to connect the input with the prediction result, which is used to transfer the gradient and learn to optimize the parameters of the backbone network. In the inference stage, the auxiliary branch is discarded, so as not to increase the additional inference calculation cost. The auxiliary reversible design mainly solves the problems caused by the network depth in the deep learning model. With the increase of the number of model layers and the effect of cross layers, information bottlenecks may occur, resulting in unreliable gradients generated by the loss function.

[0052] The model is improved on the basis of GELAN-C. An auxiliary branch training mechanism of multi-input implicit autonomous learning is added on the basis of the original PGI, and a multi-branch decoupling head module composed of DiverseBranchBlock is used. In addition, the original (CompleteIntersectionoverUnion, CIoU) loss function is replaced by the ShapeIoU loss function. The improved GELAN-C model structure is as follows: Figure 1 In this paper, C refers to the large model with the default number of channels, and CN refers to the small model with the number of channels reduced to 1 / 4 based on the C model.

[0053] Regarding the auxiliary branch training of multi-input implicit autonomous learning, the specific implementation principle is as follows: In order to address the problem of small target information gradients and features decreasing step by step during model depth transmission, multiple inputs can be used to enhance the basic information volume, while using auxiliary branches to reduce the network depth, thereby reducing the small target gradient transmission loss.

[0054] The auxiliary branch structure of multi-input implicit autonomous learning is as follows Figure 1 As shown, it is composed of modules 23 to 37 and an auxiliary head. The specific structure of each module is as follows Figure 1As shown. To ensure the shortest gradient path, the structure uses the same Conv module and Adwon module (with a step size of 2) as the main branch for downsampling operations, and uses the RepNCSPELAN4 module to achieve efficient feature aggregation, and finally outputs the results through the auxiliary head. Among them, the predictions of the auxiliary head are A3, A4, and A5, and the predictions of the main branch head are P3, P4, and P5. These two sizes of output heads are used to predict targets of different sizes.

[0055] In the loss function calculation stage, the prediction results of the two detection heads will be matched with the annotation box for loss calculation respectively. Finally, the auxiliary loss is superimposed on the main branch loss to achieve the aggregation of the two losses. Finally, the gradient is back-propagated. The main branch has the auxiliary loss information and shortens the gradient transmission distance between the main branch trunk and the prediction head. The loss function fusion ratio is shown in formula (1).

[0056] L=0.25×L a +L m (1)

[0057] Where: L is the total loss value, L a Predict the head loss value for the auxiliary branch, L m Predict the head loss value for the main branch.

[0058] In the original PGI auxiliary training branch framework, the main branch and auxiliary branch share the same input, and the input image size is: X∈R H×w×3 , the model structure is as follows Figure 2 As shown in (a). On this basis, a fusion module of grayscale space input and implicit autonomous learning is added, where the auxiliary branch input image size is X∈R H×w×1 , while the main branch input image size is maintained as X∈R H ×w×3 The final model structure is as follows: Figure 2 (b) As shown. The newly added grayscale image input is not sent to the main branch, but is used for low-semantic feature extraction and learning through the auxiliary training branch. This can obtain low-semantic specific information while reducing the gradient transfer depth, which is conducive to maintaining efficient knowledge transfer to the main branch. This design ensures that the main branch has the longest and shortest complete gradient transfer path. Its shortest path reduces the information attenuation of image information due to continuous downsampling of the model and the depth of the model, and reduces the impact of information bottlenecks. The auxiliary training structure can effectively utilize diversified input information and enhance the ability to understand the input feature map through autonomous learning, thereby improving the performance of the model.

[0059] In the baseline model, the auxiliary branch performs feature fusion with the main branch through CBFuse. However, the CBFuse module only uses a simple bitwise addition method to fuse multiple input features, and does not consider the potential hidden correlations between different inputs.

[0060] In view of the above shortcomings, the present invention designs a multi-input fusion module (ImplicitConvCBFuse, ICCBFuse) for autonomous learning of hidden correlations, and its structure is as follows: Figure 3 shown.

[0061] In the ICCBFuse module, the hidden correlations between different inputs are modeled as self-trainable implicit parameters A and M. Parameter A is responsible for performing the channel-by-channel addition operation between the input feature map, and parameter M is responsible for performing the convolution operation between the input feature map. The internal calculation of the module is shown in formula (2).

[0062] Output = Conv(Concat(Input n )+A)×M,n∈[1,+∞] (2)

[0063] Where: Input n represents the nth input feature map of the ICCBFuse module, Output represents the output feature map result of the ICCBFuse module, A represents the addition matrix of implicit training parameters, and M represents the multiplication matrix of implicit training parameters.

[0064] In the auxiliary branch, the CBLinear module obtains the main branch feature maps of different scales from the 5th, 7th, and 9th layers. After that, the auxiliary training uses the information of the main branch and auxiliary branch obtained by the ICCBFuse module and performs autonomous hidden learning. Therefore, the gradient feedback of the auxiliary training branch will help enhance the feature learning ability of the main branch trunk. The multi-input implicit autonomous learning module realizes adaptive attention to information of different depths and scales through parameters A and M, thereby being able to retain more shallow small target information.

[0065] In order to verify the effect of multi-input auxiliary branch training, the present invention conducted an ablation experiment on a self-built solar panel surface defect detection dataset. The specific method of the ablation experiment of multi-input auxiliary branch training is shown in Table 1. Model a adds a new grayscale image input on the basis of the original model, model b replaces the original CBFuse module with the ConvCBFuse module on the basis of a, and model c replaces the CBFuse module with the ImplicitConvCBFuse module on the basis of a.

[0066] Table 1 Ablation experiments of various structures of auxiliary branches with multiple inputs

[0067]

[0068] In the task of solar panel surface defect detection, the number of defect categories is unbalanced due to the scarcity of some categories of defects. The original GELAN-C model detection head only has ordinary convolution operations, and the extracted feature information is relatively limited. To solve this problem, a multi-branch structure (DiverseBranchBlock, DBB) module is used to replace the original ordinary convolution and form a multi-branch decoupling head. Figure 4 As shown in (a), the module includes 11 and KK convolutions and average pooling in the branch, and uses a variety of combined convolution methods to extract richer feature information, thereby enhancing the ability of feature maps to express multi-scale feature semantics at the end, effectively alleviating the problem of multi-scale information loss in deep detection heads. DBB has also upgraded its re-parameterization capabilities based on ACNet and Repvgg. Figure 4 As shown in (b), DBB reparameterizes multiple branch structures into a single-path structure of KK convolution. The reparameterization does not increase the additional computational cost in the inference stage, while maintaining high-speed prediction and reducing the negative impact of the imbalanced number of categories. The DBB module is introduced as follows Figure 1 The multi-branch decoupling head is shown in Figure 2.

[0069] In the solar panel surface defect dataset, there are fewer targets in the fiber, wire anomaly, bubble, notch, pinhole and pressure line categories, which are difficult to train in the long-tail distribution of the dataset. The mAP50 and mAP50-95 experimental results for these categories are shown in Table 2.

[0070] Table 2 Experiments on improving difficult training samples

[0071] Model mAP50 mAP50-95 GELAN-C 71.2 54.9 GELAN-C+ multi-branch decoupling head 72.2 55.7 GELAN-CN 70.7 52.1 GELAN-CN+ multi-branch decoupling head 71.4 54.7

[0072] As can be seen from the table, the use of multi-branch decoupling heads can effectively improve the detection ability of long-tail distribution samples. Compared with the original model, the mAP50 and mAP50-95 of the C model increased by 1.0% and 0.8% respectively, and the mAP50 and mAP50-95 of the CN model increased by 0.7% and 2.6% respectively. Therefore, the multi-branch DBB structure has stronger multi-angle feature extraction capabilities without increasing the inference calculation cost.

[0073] The CIoU loss function is used by default in the GELAN-C model. This loss function takes into account the center point distance, overlapping area, and aspect ratio between the predicted box and the true box, so as to better regress the coordinates and size of the target box. Although CIoU covers multiple important factors, it ignores the shape and scale factors of the target box itself, which will also affect the regression of the border. For the dataset used in this article, the grating category has a serious imbalance in aspect ratio, so the ShapeIoU loss function is introduced to replace the CIoU loss function in the GELAN-C model, such as Figure 5 As shown. ShapeIoU focuses on the shape and size information of the bounding box itself and incorporates this information into the IoU loss function.

[0074] The overall formula of its loss function is shown in formula (5):

[0075] L=1-IOU+L d +0.5×L Ω (8)

[0076] Where: L represents the overall result of the loss function calculation, IOU (Intersection over Union) is a common intersection loss, and is also the most commonly used regression coordinate loss in the field of target detection, as shown in formula (6). d represents the distance loss of the shape, as shown in formula (8). Ω Represents the value loss of the shape, as shown in formula (9).

[0077]

[0078] Where: B represents the location area of ​​the prediction box, B gt Represents the location area of ​​the ground-truth box.

[0079]

[0080] Where s represents the scaling hyperparameter related to the task and model. In this paper, the small model uses s=1.0, the large model uses s=1.5, and w gt and h gt Represent the width and height of the real box respectively, W and H represent the weight coefficients in the horizontal and vertical directions respectively, and these two values ​​are related to the shape of the real box.

[0081] L d =H×(xx gt ) 2 / c 2 +W×(yy gt ) 2 / c 2 (11)

[0082] Where: (x gt ,y gt ) represents the coordinates of the center point of the real box, (x, y) represents the coordinates of the center of the predicted box, and c represents the direct distance between the center points of the real box and the predicted box.

[0083]

[0084] Where: w and h represent the width and height of the prediction box.

[0085] In the solar panel surface defect detection dataset, the grating category objects show a serious imbalance in aspect ratio, with aspect ratios ranging from 2.2 to 53.3. Table 4 shows the experimental results of whether the grating category model uses ShapeIoU loss.

[0086] Table 4: Experiments on improving ShapeIoU

[0087] Model Grating mAP50 Grating mAP50-95 GELAN-C 93.6 66.9 GELAN-C+ShapeIoU 96.8 70.3 GELAN-CN 94.3 66.8 GELAN-CN+ShapeIoU 95.1 69.4

[0088] As can be seen from the table, compared with the default CIoU loss function, after using ShapeIoU, the mAP50 and mAP50-95 of the C model increased by 3.2% and 3.4%, respectively, and the mAP50 and mAP50-95 of the CN model increased by 0.8% and 2.6%, respectively. Therefore, ShapeIoU focuses on the shape and size information of the bounding box itself and integrates this information into the IoU loss function, so that the calculated knowledge gradient contains the information of the target aspect ratio, and the length and width information of the defect can be paid attention to in the prediction of the reasoning stage.

[0089] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A solar screen surface defect detection method based on branch training, characterized in that: The following steps are involved: S1: Use the image collection and detection platform to take images of the solar screen surface for neural network training; S2: Pack the training images and use the multi-branch backbone structure model for feature extraction and fusion; S3: Send the fused features to the multi-branch detection head for classification and positioning; S4: Loss calculation between the output value of the detection head and the real labeled data; S5: Using loss function to optimize neural network; S6: Use the optimized model to process the actual image.

2. The solar panel surface defect detection method based on branch training according to claim 1 is characterized in that: The image collection and detection platform includes a camera, a lens, a guide rail, a host, an operating table, a detection table and a display; the solar screen is placed on the detection table, the camera moves on the detection table through the guide rail, and collects the image information of the solar screen through the lens, and also feeds back the image information to the host and the imager, and the operating table controls the image acquisition of the camera, the operation of the guide rail, and the display of the display.

3. The method for detecting surface defects of solar panels based on branch training according to claim 1, characterized in that: In the multi-branch trunk structure module, five downsampling operations are performed in different branches, and in the step sizes of 8, 16 and 32, the multi-input fusion module with autonomous learning hidden correlation in the auxiliary branch is used to realize the feature fusion of multiple branches, and finally three four-dimensional feature maps with sizes of B×C×80×80, B×C×40×40 and B×C×20×20 are obtained.

4. The method for detecting surface defects of solar panels based on branch training according to claim 3 is characterized in that: The auxiliary branch can perform branch stripping in the inference phase, while restoring the original model structure and parameter quantity to maintain the inference speed.

5. The method for detecting surface defects of solar panels based on branch training according to claim 1, characterized in that: The multi-branch detection head uses a DBB module in the detection head of the main branch to enable the detection head to have multi-angle feature extraction capabilities.

6. The method for detecting surface defects of solar panels based on branch training according to claim 5, characterized in that: The DBB module has 1×1 and K×K convolutions and average pooling operations inside to achieve multi-angle observation in the training phase. In the inference phase, the linear properties of convolution are used to achieve re-parameterized fusion of multiple branches, which can eventually be merged into a K×K convolution.

7. The method for detecting surface defects of solar panels based on branch training according to claim 1, characterized in that: The length and width information of the target is added to the loss calculation, and the length and width information is used to form the weight coefficients in the horizontal and vertical directions. The ShapeIoU loss function is adopted to achieve the attention of targets with imbalanced aspect ratios.

Images