Defect detection method and device, equipment and storage medium

Abstract

The invention provides a defect detection method and device, equipment and a storage medium, and the method comprises the steps: obtaining target picture data of a to-be-detected solar cell panel, inputting the target picture data into a defect detection model, and obtaining the output of the defect detection model, so as to obtain a defect detection result of the to-be-detected solar cell panel. Wherein the defect detection model is obtained by training a pre-training model through a training sample data set; the pre-training model is constructed by adding an attention model based on a preset backbone network and a feature scale extraction model. A pre-training model is constructed on the basis of a preset backbone network, a feature scale extraction model thereof and an attention model, the pre-training model is trained to obtain a lightweight defect detection model for defect detection of the solar cell panel, and the defect detection method has the characteristics of high real-time processing capability, simple deployment and model detection precision, and can be applied to the field of solar cell panel defect detection. And the solar cell panel defect detection requirement in an industrial scene can be met.

Description

Technical Field

[0001] The present application relates to the technical field of solar panel, and in particular to a defect detection method, device, equipment and storage medium. Background Art

[0002] Due to the fragility of their crystal structure, solar panels are very likely to be damaged, cracked or fragmented during transportation and installation. During the actual operation of photovoltaic power stations, some cracked panels will be further damaged and develop into fragmentation defects.

[0003] The presence of cracks will not only seriously reduce the power generation efficiency and service life of the solar panel, but it is even very easy to cause fire accidents due to local overheating at the defective part, resulting in direct economic losses. Therefore, in order to ensure the safe and efficient operation of photovoltaic power stations, defect detection of solar panels has profound practical significance.

[0004] At present, the traditional detection method based on computer vision mainly identifies defects based on artificially designed feature defects, and mainly classifies defect images through regression, classification and clustering methods. The detection speed of the model in this method is slow, and the model accuracy is limited, which cannot meet the actual production needs. It can be seen that a solar panel defect detection solution is urgently needed to overcome the defects in related technologies. Summary of the invention

[0005] The present application provides a defect detection method, device, equipment and storage medium, which are used to overcome the technical problems of the prior art in solar panel defect detection, such as slow detection speed and limited detection model accuracy that cannot meet production requirements.

[0006] In a first aspect, the present application provides a defect detection method, comprising:

[0007] Acquire target image data of the solar panel to be inspected through an image acquisition device;

[0008] Inputting the target image data into a defect detection model, obtaining an output of the defect detection model, so as to obtain a defect detection result of the solar cell panel to be inspected;

[0009] Among them, the defect detection model is obtained by training a pre-trained model through a training sample data set; the pre-trained model is constructed based on a preset backbone network and a feature scale extraction model, and an attention model is added.

[0010] A lightweight defect detection model is obtained by training a pre-trained model based on a preset backbone network, a feature scale extraction model, and an attention model. The defect detection model is used to perform defect detection on the solar panel to be inspected. This defect detection method has the characteristics of strong real-time processing capability, simple deployment, and model detection accuracy, and can meet the needs of solar panel defect detection in industrial scenarios.

[0011] In a possible design, an original sample data set is obtained, and original defect samples in the original sample data set are preprocessed to obtain the training sample data set, wherein the original defect samples include solar panel images classified by characteristic defect categories.

[0012] The obtained original sample data set is preprocessed to obtain a training sample data set, so that the training samples in the training sample data set meet the requirements of model training.

[0013] In a possible design, the preset backbone network includes a pre-trained yolov5 backbone, and the feature scale extraction model includes a small and medium scale feature extraction branch in the yolov5 Neck network.

[0014] Optimize yolov5 to build a pre-trained model so that the defect detection model trained by the pre-trained model can extract small and medium-scale features based on the features extracted by the backbone of yolov5, obtain feature information of different granularities of the target image data of the solar panel to be inspected, and improve the detection accuracy.

[0015] In a possible design, obtaining the original sample data set includes:

[0016] Acquire multiple first solar panel images having the characteristic defects through the EL public dataset;

[0017] Acquire a plurality of second solar panel images having the characteristic defects;

[0018] Generate the original sample data set according to the first solar panel picture and the second solar panel picture;

[0019] The second solar panel picture is a picture of the solar panel having the characteristic defect taken according to different shooting parameters; the shooting parameters include different shooting angles and / or different shooting brightness.

[0020] The original defect samples are obtained from different aspects to construct the original sample data set, enriching the diversity of the original defect samples.

[0021] In a possible design, preprocessing the original defect samples in the original sample dataset to obtain the training sample dataset includes:

[0022] Amplifying the number of samples of the original defect samples in the original sample dataset through data augmentation to obtain the target defect samples corresponding to the original defect samples, and obtaining the training sample dataset according to the target defect samples and the original defect samples.

[0023] The preprocessing method can be to amplify the number of samples through data augmentation so that the number of samples in the training sample dataset reaches the number of samples required for model training.

[0024] In a possible design, the data augmentation method includes at least one of brightness reduction, brightness enhancement, contrast reduction, contrast enhancement, horizontal flipping, or vertical flipping.

[0025] Amplify the number of samples through different data augmentation methods to enrich the number of samples.

[0026] In a possible design, after obtaining the training sample dataset, it further includes:

[0027] Generating corresponding defect label information for the training samples in the training sample dataset, where the defect label information is used to label the category and location information of the characteristic defects of the training samples;

[0028] Adjusting the picture size of the training samples to a target unified size.

[0029] Performing defect annotation on the training samples in the training sample dataset and unifying the picture size of the training samples to facilitate model training.

[0030] In a possible design, there is also a residual connection between the shallow convolutional layer and the deep convolutional layer of the backbone of yolov5, and the attention model is located after the convolutional layer of the backbone of yolov5, and the convolutional layer includes the shallow convolutional layer and the deep convolutional layer;

[0031] Among them, the residual connection is used for feature fusion in defect recognition, and the attention model is used to obtain edge detail features in defect recognition.

[0032] During defect detection, the attention model can focus on the features of small regions, and by adding a residual connection, the trained defect model can perform feature fusion on shallow features and deep features, thereby ensuring the integrity of the extracted picture features.

[0033] In a possible design, the pre-trained model is trained using the training sample dataset to obtain a defect detection model, including:

[0034] The pre-trained model is trained using the training subset in the training sample dataset; the training subset is obtained from the training sample dataset according to a first preset ratio;

[0035] The network parameters of the pre-trained model are optimized using the gradient descent method according to the prediction results obtained during model training;

[0036] When the preset loss function in model training reaches the convergence condition, the optimal network parameters are obtained, the model training is ended, and the pre-trained model containing the optimal network parameters is determined as the defect detection model;

[0037] Among them, the convergence condition is to minimize the error, and the preset loss function includes the intersection over union of the predicted box and the ground truth box.

[0038] Training the pre-trained model using the training subset in the training sample dataset and optimizing the network parameters using the gradient descent method, aiming at the minimum error, and optimizing the network parameters through backpropagation of the minimum error to obtain the optimal network optimization result, which is beneficial to improving the detection accuracy of the trained defect detection model.

[0039] In a possible design, the training of the pre-trained model using the training subset in the training sample dataset includes:

[0040] Freeze the backbone of yolov5 in the pre-trained model and perform preliminary training on the network structure except the backbone of yolov5 using a larger learning rate;

[0041] Unfreeze the backbone of yolov5 and fine-tune the pre-trained model after preliminary training using a smaller learning rate.

[0042] Using a larger learning rate for preliminary training to make the network loss value drop rapidly, and then using a smaller learning rate for training as the number of iterations increases, preventing crossing the optimal network parameters and avoiding overfitting during the training process.

[0043] In a possible design, after obtaining the defect detection model, it further includes:

[0044] Using the test subset in the training sample dataset to evaluate the performance of the defect detection model based on evaluation metrics to obtain the precision rate of the defect detection model;

[0045] Wherein, the test subset is obtained from the training sample data set according to a second preset ratio; the sum of the first preset ratio and the second preset ratio is 1; and the evaluation index includes an average precision mean.

[0046] After the defect detection model is obtained through training, the detection accuracy of the defect detection model is evaluated using evaluation indicators to ensure that the defect detection model has high detection accuracy.

[0047] In a possible design, the characteristic defect includes a pure crack defect or a pure chip defect.

[0048] The characteristic defects may include different types of defects, so that the defect detection model can detect different types of defects that the solar panel to be inspected may have.

[0049] In a second aspect, the present application provides a defect detection device, comprising:

[0050] A detection data acquisition module, used to acquire target image data of the solar panel to be detected through an image acquisition device;

[0051] A detection module, used for inputting the target image data into a defect detection model, obtaining an output of the defect detection model, so as to obtain a defect detection result of the solar cell panel to be detected;

[0052] Among them, the defect detection model is obtained by training a pre-trained model through a training sample data set; the pre-trained model is constructed based on a preset backbone network and a feature scale extraction model, and an attention model is added.

[0053] A lightweight defect detection model is obtained by training a pre-trained model based on a preset backbone network, a feature scale extraction model, and an attention model. The defect detection model is used to perform defect detection on the solar panel to be inspected. This defect detection method has the characteristics of strong real-time processing capability, simple deployment, and model detection accuracy, and can meet the needs of solar panel defect detection in industrial scenarios.

[0054] In a possible design, the apparatus further includes: a sample data acquisition module; the sample data acquisition module is used to:

[0055] An original sample data set is obtained, and original defect samples in the original sample data set are preprocessed to obtain the training sample data set, wherein the original defect samples include solar panel images classified by characteristic defect categories.

[0056] The obtained original sample data set is preprocessed to obtain a training sample data set, so that the training samples in the training sample data set meet the requirements of model training.

[0057] In a possible design, the preset backbone network includes the backbone of the pre-trained YOLOv5, and the feature scale extraction model includes the medium and small scale feature extraction branches in the Neck network of the YOLOv5.

[0058] Optimize YOLOv5 to build a pre-trained model, so that the defect detection model obtained by training the pre-trained model can, on the basis of extracting features through the backbone of YOLOv5, also extract medium and small scale features, obtain feature information of different granularities of the target picture data of the solar panel to be detected, and improve the detection accuracy.

[0059] In a possible design, the sample data acquisition module is further configured to:

[0060] Obtain multiple first solar panel pictures with the feature defects through the EL public dataset;

[0061] Obtain multiple second solar panel pictures with the feature defects;

[0062] Generate the original sample dataset according to the first solar panel pictures and the second solar panel pictures;

[0063] Wherein, the second solar panel pictures are pictures of the solar panels with the feature defects taken according to different shooting parameters; the shooting parameters include different shooting angles and / or different shooting brightnesses.

[0064] Obtain the original defect samples from different aspects to construct the original sample dataset, and enrich the diversity of the original defect samples.

[0065] In a possible design, the sample data acquisition module is further configured to:

[0066] Amplify the number of samples of the original defect samples in the original sample dataset through data augmentation to obtain the target defect samples corresponding to the original defect samples, and obtain the training sample dataset according to the target defect samples and the original defect samples.

[0067] The preprocessing method can be to amplify the number of samples through data augmentation so that the number of samples in the training sample dataset reaches the number of samples required for model training.

[0068] In a possible design, the data augmentation method includes at least one of brightness reduction, brightness enhancement, contrast reduction, contrast enhancement, horizontal flipping or vertical flipping.

[0069] Amplify the number of samples through different data augmentation methods to enrich the number of samples.

[0070] In a possible design, the sample data acquisition module is further configured to:

[0071] Generate corresponding defect label information for the training samples in the training sample dataset, where the defect label information is used to label the category and location information of the feature defects of the training samples;

[0072] Adjust the picture size of the training samples to a target unified size.

[0073] Perform defect annotation on the training samples in the training sample dataset and unify the picture sizes of the training samples to facilitate model training.

[0074] In a possible design, there is also a residual connection between the shallow convolutional layer and the deep convolutional layer of the backbone of the YOLOv5, and the attention model is located after the convolutional layer of the backbone of the YOLOv5, and the convolutional layer includes the shallow convolutional layer and the deep convolutional layer;

[0075] Among them, the residual connection is used for feature fusion in defect recognition, and the attention model is used to obtain edge detail features in defect recognition.

[0076] During defect detection, the attention model can focus on the features of small regions, and by adding a residual connection, the trained defect model can perform feature fusion on shallow features and deep features, thereby ensuring the integrity of the extracted picture features.

[0077] In a possible design, the device further includes: a model training module; the model training module is configured to:

[0078] Perform model training on the pre-trained model through the training subset in the training sample dataset; the training subset is obtained from the training sample dataset according to a first preset ratio;

[0079] Optimize the network parameters of the pre-trained model by using the gradient descent method according to the prediction results obtained during model training;

[0080] When the preset loss function in model training reaches the convergence condition, obtain the optimal network parameters, end the model training, and determine the pre-trained model containing the optimal network parameters as the defect detection model;

[0081] Among them, the convergence condition is to minimize the error, and the preset loss function includes the intersection over union of the predicted bounding box and the ground truth bounding box.

[0082] The pre-trained model is trained using the training subset in the training sample dataset, and the gradient descent method is used to optimize the network parameters. With the aim of minimizing the error, the network parameters are optimized by minimizing the error backpropagation to obtain the optimal network optimization result, which is beneficial to improving the detection accuracy of the trained defect detection model.

[0083] In a possible design, the model training module is further configured to:

[0084] Freeze the backbone of yolov5 in the pre-trained model, and use a relatively large learning rate to preliminarily train the network structure except the backbone of yolov5;

[0085] Unfreeze the backbone of yolov5, and use a relatively small learning rate to fine-tune the pre-trained model after preliminary training.

[0086] Use a relatively large learning rate for preliminary training to rapidly decrease the network loss value, and then use a relatively small learning rate for training as the number of iterations increases, to prevent crossing the optimal network parameters and avoid overfitting during the training process.

[0087] In a possible design, the device further includes an evaluation module; the evaluation model is configured to:

[0088] Use the test subset in the training sample dataset to perform performance evaluation on the defect detection model based on evaluation metrics to obtain the precision rate of the defect detection model;

[0089] Wherein, the test subset is obtained from the training sample dataset according to a second preset ratio; the sum of the first preset ratio and the second preset ratio is 1; the evaluation metrics include the mean average precision.

[0090] After the training is completed to obtain the defect detection model, the detection accuracy of the defect detection model is further evaluated using the evaluation metrics to ensure that the defect detection model has high detection accuracy.

[0091] In a possible design, the characteristic defect includes a pure crack defect or a pure fragment defect.

[0092] The characteristic defect can include different types of defects, enabling the defect detection model to detect different types of defects that the solar panel to be detected may have. Thirdly, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0093] The memory stores computer-executable instructions;

[0094] The processor executes the computer-executable instructions stored in the memory to implement any possible defect detection method provided in the first aspect.

[0095] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any possible defect detection method provided in the first aspect.

[0096] In a fifth aspect, a computer program product includes computer execution instructions, which, when executed by a processor, are used to implement any possible defect detection method provided in the first aspect.

[0097] The present application provides a defect detection method, apparatus, device and storage medium. First, the target image data of the solar panel to be detected is obtained through an image acquisition device, and then the target image data is input into a defect detection model to obtain the output of the defect detection model to obtain the defect detection result of the solar panel to be detected. Among them, the defect detection model is obtained by training a pre-trained model through a training sample data set; the pre-trained model is based on a preset backbone network and a feature scale extraction model, and is constructed by adding an attention model. The pre-trained model is constructed through a preset backbone network and trained to obtain a lightweight defect detection model for defect detection of solar panels. In view of the defect detection model obtained after optimization, the defect detection method provided in the embodiment of the present application has the characteristics of strong real-time processing capability, simple deployment, and both model detection accuracy, which can meet the defect detection needs of solar panels in industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0099] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0100] Figure 2 A schematic diagram of a defect detection method provided in an embodiment of the present application;

[0101] Figure 3 A schematic diagram of the structure of a pre-training model provided in an embodiment of the present application;

[0102] Figure 4A structural schematic diagram of an attention model provided by an embodiment of the present application;

[0103] Figure 5 A flowchart of obtaining an original data set provided by an embodiment of the present application;

[0104] Figure 6 A schematic diagram of an original defect sample provided by an embodiment of the present application;

[0105] Figure 7 A schematic diagram of a training sample provided by an embodiment of the present application;

[0106] Figure 8 A flowchart of model training provided by an embodiment of the present application;

[0107] Figure 9 A structural schematic diagram of a defect detection device provided by an embodiment of the present application;

[0108] Figure 10 A structural schematic diagram of another defect detection device provided by an embodiment of the present application;

[0109] Figure 11 A structural schematic diagram of yet another defect detection device provided by an embodiment of the present application;

[0110] Figure 12 A structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0111] Here, exemplary embodiments will be described in detail, and their examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of methods and devices consistent with some aspects of the present application as detailed in the appended claims.

[0112] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0113] Due to the fragility of its crystal structure, solar panels are very prone to damage during transportation and installation, and cracks or fragments may occur. In the actual operation of photovoltaic power stations, some panels with cracks will be further damaged and develop into fragment defects. The presence of cracks will not only seriously reduce the power generation efficiency and service life of the panels, but it is even very easy to cause fire accidents due to local overheating at the defects, resulting in direct economic losses. Solar panel defect detection mainly predicts the classification and positioning information of multi-scale defect information in the panels. For example, solar panel defects mainly include pure crack defects and pure fragment defects, among which crack defects have random scale size and random texture shape; fragment defects are manifested as randomly distributed blocky black spots, and the main grid lines of solar panels have similar linear characteristics to cracks, so there will be pseudo defects, all of which make solar panel defect detection extremely difficult. At present, the traditional detection method based on computer vision is mainly based on the identification of artificially designed feature defects, which mainly classifies defect images through regression, classification and clustering methods. The detection speed of the model in this method is slow and the accuracy of the model is limited, which cannot meet the actual production needs. It can be seen that a solar panel defect detection solution is urgently needed to overcome the defects in the related technology.

[0114] In view of the above problems in the prior art, the present application provides a defect detection method, apparatus, device, and storage medium. The inventive concept of the defect detection method provided by the present application lies in: constructing a pre-trained model based on a preset backbone network, a feature scale extraction model, and an added attention model, training the pre-trained model through a training sample data set to obtain a defect detection model. Using the defect detection model to detect whether the solar panel to be detected has defects, and obtaining corresponding defect detection results, ensuring the safe and efficient operation of the photovoltaic power station. Among them, the defect detection model obtained by constructing a pre-trained model based on a preset backbone network, a feature scale extraction model, and an added attention model and training the pre-trained model is a lightweight model. On the one hand, it is simple to deploy, and has the characteristic of strong real-time processing ability of the preset backbone network, which can ensure the real-time detection speed. At the same time, it can also have the model detection performance and ensure the detection accuracy, so that the defect detection method provided by the embodiments of the present application can meet the defect detection requirements of solar panels in industrial scenarios.

[0115] Figure 1 FIG. is a schematic diagram of an application scenario provided by an embodiment of the present application. As Figure 1 shown, the electronic device 100 is configured to execute the defect detection method provided by the embodiment of the present application. For example, the electronic device 100 constructs a pre-trained model based on a preset backbone network, a feature scale extraction model, and an attention model, and trains the pre-trained model through a training sample data set to obtain a defect detection model. The image acquisition device 200 acquires target picture data of the solar panel 300 to be detected, inputs the target picture data into the defect detection model constructed in the electronic device 100, and performs defect detection on the solar panel 300 to be detected through the defect detection model, obtaining the defect detection result of the solar panel to be detected. The defect detection result may include information such as the type of defect existing in the solar panel to be detected and the positioning of the defect.

[0116] In some embodiments, the electronic device 100 may be, for example, a device such as a computer, a server, a smart terminal, etc. The present application does not limit the device type of the electronic device 100. Figure 1 The electronic device 100 in is illustrated by taking a computer as an example. The image acquisition device 200 may be a device with a picture shooting function, such as any camera such as a camera, an industrial camera, etc. The present application does not limit the specific type of the image acquisition device 200. Figure 1 The image acquisition device 200 in is illustrated by taking a camera as an example.

[0117] It should be noted that the above application scenario is only illustrative. The defect detection method, apparatus, device, and storage medium provided by the embodiments of the present application include but are not limited to the above application scenario.

[0118] Figure 2 This is a schematic flowchart of a defect detection method provided by an embodiment of the present application. As Figure 2 shown, the defect detection method provided by the embodiment of the present application includes:

[0119] S101: Obtain target picture data of the solar panel to be detected through an image acquisition device.

[0120] S102: Input the target picture data into the defect detection model, and obtain the output of the defect detection model to obtain the defect detection result of the solar panel to be detected.

[0121] Among them, the defect detection model is obtained by training a pre-trained model with a training sample data set. The pre-trained model is based on a preset backbone network and a feature scale extraction model, and is constructed by adding an attention model.

[0122] Based on a preset backbone network and a feature scale extraction model, an attention model is added to construct a pre-trained model. Then, the pre-trained model is trained with a training sample data set to obtain a defect detection model. The picture of the solar panel to be detected obtained through the image acquisition device is determined as the target picture data. The target picture data is used as the input of the defect detection model, so that the defect detection model performs defect recognition and obtains the output of the defect detection model. Its output is the defect detection result of the solar panel to be detected. Thus, based on the defect detection result, the category and location of the defect of the solar panel to be detected are known. The defect location is the location information of the defect on the solar panel to be detected.

[0123] In some embodiments, for example, the target picture data of the solar panel to be detected is obtained through an industrial camera, and the picture size in the target picture data is adjusted to a target unified size. The target picture data is input into the defect detection model to obtain the defect detection result of the output of the defect detection model. If the defect detection result includes defect information, it means that the solar panel to be detected has a defect, and the category and location of the defect of the solar panel to be detected are determined according to the defect information. Among them, the defect information includes the defect category and defect location information, etc. If the defect detection result does not include defect information, it means that the solar panel to be detected does not have a defect.

[0124] The defect detection method provided in the embodiment of the present application can detect defects of solar panels in industrial scenarios through the defect detection model to ensure the safe and efficient operation of photovoltaic power stations. In addition, a pre-trained model is constructed through a preset backbone network and trained to obtain a lightweight defect detection model. In view of the defect detection model obtained after optimization, the defect detection method provided in the embodiment of the present application has the characteristics of strong real-time processing capability, simple deployment, and model detection accuracy, which can meet the needs of solar panel defect detection in industrial scenarios.

[0125] In one possible design, a preset backbone network for constructing a pre-trained model includes a pre-trained backbone of yolov5, and a feature scale extraction model includes a small and medium scale feature extraction branch in a Neck network of yolov5.

[0126] For example, based on the pre-trained yolov5 backbone network, when using the yolov5 backbone network to extract features, by adding an attention model (Convolutional Block Attention ModuleConvolutional Block Attention Module, CBAM), the backbone network can focus on the features of small areas. According to the characteristics of solar panel samples, the large-scale feature extraction branch of the yolov5 Neck network is deleted, and only the small-scale and medium-scale feature extraction branches are retained. The small-scale and medium-scale feature extraction branches are defined as feature scale extraction models, which are used to predict features extracted at different scales in defect recognition.

[0127] The pre-trained model of the embodiment of the present application is constructed based on the backbone network and feature scale extraction model of yolov5 and adding the attention model. The feature scale extraction model can obtain information of different granularities of the image, and adding the attention model to the backbone network of yolov5 can avoid feature loss and ensure the integrity of feature extraction, so as to realize the pre-trained model constructed by optimizing yolov5.

[0128] In one possible design, reference Figure 3 The pre-trained model shown in the figure also includes residual connections between the shallow convolutional layer and the deep convolutional layer of the backbone of yolov5. The residual connections are as follows Figure 3The connection between CBAM and Concat in the Chinese context. The role of the residual connection is that in defect recognition, the shallow features and deep features can be fused through the residual connection to ensure the integrity of the extracted image feature information, so as to make up for the information loss caused by the extraction of the image features through multiple convolutional neural networks. The attention model is located after the convolutional layer of the backbone of yolov5, and the convolutional layer can include, for example, Figure 3 the shallow convolutional layer CSP1-1 and the deep convolutional layer CSP1-3 shown in the figure. The role of the attention model is that in defect recognition, it can dig out the edge detail features of the feature defects, making the extracted features more complete. The structural schematic diagram of the attention model is as shown in Figure 4 the figure.

[0129] By optimizing yolov5 as described above, the pre-trained model of the embodiment of this application is constructed as shown in Figure 3 the figure. By using the feature scale extraction model and the attention model to extract the image features, information with different granularities of the image can be obtained. After extracting the information with different granularities, the features extracted at different scales are respectively predicted to obtain the prediction results corresponding to the scales. Finally, the feature information of all scales is fused to perform defect detection prediction on the total features, so as to ensure that the defect detection model obtained by training the pre-trained model can extract complete image feature information.

[0130] In a possible design, the defect detection method provided by the embodiment of this application further includes: obtaining an original sample data set, and preprocessing the original defect samples in the original sample data set to obtain a training sample data set. Among them, the original defect samples include solar panel pictures classified by the categories of feature defects.

[0131] For example, solar panel pictures with feature defects can be obtained through the EL public data set and the actual acquisition of the image acquisition device. The obtained solar cell pictures with feature defects are classified according to the categories of feature defects. For example, solar panel pictures with pure crack defects as the feature defect are classified as pure crack defect samples, and solar cell pictures with pure fragment defects as the feature defect are classified as pure fragment defect samples. The pure crack defect samples and the pure fragment defect samples are collectively referred to as the original defect samples. Each original defect sample constitutes the original sample data set as an element in the set to obtain the original sample data set.

[0132] In a possible design, a possible implementation manner of obtaining the original data set is as shown in Figure 5 the figure. Figure 5 This is a schematic flowchart of a method for obtaining the original data set provided by the embodiment of this application. As shown in Figure 5 the figure, the embodiment of this application includes:

[0133] S201: Obtain multiple first solar panel images with characteristic defects from the EL public dataset.

[0134] Download the Electroluminescence (EL) public dataset. The EL public dataset includes multiple solar panel images with characteristic defects. For the convenience of description, the solar panel images obtained from the EL public dataset are defined as the first solar panel images.

[0135] S202: Obtain multiple second solar panel images with characteristic defects.

[0136] Among them, the second solar panel images are images of solar panels with characteristic defects taken according to different shooting parameters. The shooting parameters include different shooting angles and / or different shooting brightnesses. For example, an industrial camera is used to shoot a solar panel with pure fragment defects and / or pure crack defects according to the set shooting parameters to obtain the corresponding images. The object to be photographed by the industrial camera is called the second solar panel, and the obtained corresponding images are the second solar panel images.

[0137] Among them, the shooting parameters may include different shooting angles and / or different shooting brightnesses. Different shooting angles can be preset, and different shooting brightnesses can be the preset different degrees of light and darkness of the shooting light. The specific values of different shooting angles and different shooting brightnesses can be set according to the actual working conditions, and the embodiments of the present application do not limit this.

[0138] S203: Generate an original sample dataset based on the first solar panel images and the second solar panel images.

[0139] Manually screen the multiple second solar panel images obtained by shooting. The screened second solar panel images and the multiple first solar panel images obtained from the EL public dataset are collectively referred to as the original defect samples. Each original defect sample is used as an element of the set to form the original sample dataset, thereby obtaining the original sample dataset.

[0140] It can be understood that the original defect samples are classified according to the categories of characteristic defects to obtain pure crack samples and pure fragment samples. A pure crack sample refers to an original defect sample whose characteristic defect is a pure crack, and a pure fragment sample refers to an original defect sample whose characteristic defect is a pure fragment. The original sample dataset is used for subsequent model training. Therefore, the original sample dataset can be divided into a training subset and a test subset according to a ratio, where the ratio includes a first preset ratio and a second preset ratio. For example, the number of samples with a first preset ratio of the original sample dataset is divided into the training subset in the original sample dataset, and the number of samples with a second preset ratio of the original sample dataset is divided into the test subset in the original sample dataset. For example, the first preset ratio is 70% and the second preset ratio is 30%, and the sum of the first preset ratio and the second preset ratio is 1. Table 1 lists the distribution of the original sample dataset.

[0141] Table 1

[0142]

[0143] In the embodiment of the present application, defect samples of different characteristic defect categories are obtained through the EL public dataset and actual shooting, and the original dataset is composed of the original defect samples to prepare data for the training samples of subsequent model training.

[0144] After obtaining the original dataset, since the number of samples in the original dataset obtained in general industrial scenarios is limited and may not be sufficient to support the large number of samples required for model training, therefore, it is necessary to perform preprocessing including data augmentation methods on the original defect samples in the original sample dataset, and form a training sample dataset according to the preprocessed original defect samples, and use the training sample dataset for model training.

[0145] For example, the number of samples of each original defect sample in the original sample dataset is amplified through a data augmentation method, and the picture amplified through the data augmentation method is determined as the target defect sample corresponding to each original defect sample, and the target defect sample and the original defect sample are used as set elements to form a training sample dataset.

[0146] In a possible design, the data augmentation method may include at least one of brightness reduction, brightness enhancement, contrast reduction, contrast enhancement, horizontal flipping, or vertical flipping.

[0147] For example, the original defect sample is as Figure 6 shown, Figure 6 The characteristic defects of the original defect sample shown in are pure crack defects and pure fragment defects, and the pure crack defects can be manifested as single-line cracks and double-line cracks. Figure 6Among (a), (b), and (c), a single-line crack, a double-line crack, and fragments are shown respectively. Taking the single-line crack and the double-line crack as examples, each target defect sample obtained by data augmentation of the original defect sample with this characteristic defect is as Figure 7 shown. Figure 7 In (a) of Figure 7 , it represents the original defect sample with a single-line crack (the original picture is Figure 6 in (a) of Figure 6 ) and the target defect sample obtained by data augmentation of it, Figure 7 in (b) of Figure 7 represents the original defect sample with a double-line crack (the original picture is Figure 6 in (b) of Figure 6 ) and the target defect sample obtained by data augmentation of it.

[0148] As listed in Table 1, the original sample dataset can obtain the distribution of the training sample dataset shown in Table 2 through data augmentation.

[0149] Table 2

[0150]

[0151] It can be understood that the training subset in the training sample dataset is obtained by expanding the number of samples in the training subset of the original sample dataset, and the test subset in the training sample dataset is obtained by expanding the number of samples in the test subset of the original sample dataset. Therefore, the training subset in the training sample dataset is obtained from the training sample dataset according to the first preset ratio, and the side-hand subset in the training sample dataset is obtained from the training sample dataset according to the second preset ratio. The sum of the first preset ratio and the second preset ratio is 1. As shown in Table 2, the first preset ratio is 70% and the second preset ratio is 30%.

[0152] In a possible design, after obtaining the training sample dataset by data augmentation, the preprocessing can further include setting corresponding defect label information for the training samples in the training sample dataset, and annotating the category of the characteristic defect and the positioning information of the characteristic defect of the training sample corresponding picture through the defect label information.

[0153] Optionally, according to the data requirements input by the pre-trained model, the picture size of each training sample is adjusted to the target unified size, for example, uniformly adjusted to 640×640.

[0154] Through the above Figure 5 shown embodiments, the original sample dataset can be obtained. Preprocessing the original sample dataset can obtain the training sample dataset, and then the pre-trained model constructed can be trained using the training sample dataset to obtain the defect detection model.

[0155] In a possible design, the pre-trained model is trained using a training sample data set, and a possible implementation of obtaining a defect detection model is as follows: Figure 8 as shown in Figure 8 FIG. 4 is a schematic diagram of a model training process provided by an embodiment of the present application. As shown in Figure 8 FIG. 4, the embodiment of the present application includes:

[0156] S301: Train the pre-trained model using a training subset in the training sample data set.

[0157] Among them, the training subset in the training sample data set is obtained from the training sample data set according to a first preset ratio.

[0158] For example, the training process can be divided into two steps. First, the first n layers of the backbone of yolov5 in the pre-trained model can be frozen, and a relatively large learning rate can be used to perform preliminary training on the network structure other than the backbone of yolov5 to preferentially train the network structure other than the backbone. Then, an unfreezing operation is performed, that is, the first n layers of the backbone of yolov5 are unfrozen, and a relatively small learning rate is used to fine-tune the entire network of the pre-trained model. In some embodiments, in order to prevent falling into a local optimal solution, the cosine annealing algorithm can also be used to update the learning rate.

[0159] Referring to Figure 3 the schematic diagram of the pre-trained model structure shown in FIG. 4, in model training, solar panel pictures of unified size in the training subset of the training sample data set are input into the pre-trained model. Through the feature extraction of its backbone and attention model, the picture features extracted by the backbone network are obtained. Subsequently, the picture features are transmitted to the Neck network to perform feature extraction on the feature output from the backbone. When performing feature extraction, the intermediate layer features from the backbone network are fused, aiming to make up for the feature loss caused by the extraction of picture features in the CNN network. Through the fusion of the shallow layer features from the backbone and the processing of the attention model, deep picture features for prediction can be obtained. In the prediction stage, the features of small and medium scales and the backbone output features are respectively predicted to obtain corresponding prediction results (Output).

[0160] S302: Optimize the network parameters of the pre-trained model using the gradient descent method according to the prediction results obtained in the model training.

[0161] S303: When the preset loss function in the model training reaches the convergence condition, obtain the optimal network parameters, end the model training, and determine the pre-trained model containing the optimal network parameters as the defect detection model.

[0162] Specifically, the prediction result is compared with the sample label, i.e., the labeled defect label information, to calculate the error between the prediction result and the sample label. Subsequently, with the aim of minimizing the error, i.e., taking the minimization of the error as the convergence condition, the network parameters are optimized through error backpropagation. Among them, the preset loss function can be IOU, where IOU is the intersection over union of the predicted bounding box and the ground truth bounding box. The larger the intersection over union, the more accurate the prediction result. IOU and the function L IOU The expressions are shown in the following formulas (1) and (2):

[0163]

[0164] L IOU = 1 - IOU (2)

[0165] where b and b gt represent the areas of the predicted bounding box and the ground truth bounding box respectively.

[0166] During the training process, the network parameters are optimized by backpropagation through minimizing L IOU to obtain the network result with optimal network parameters, end the model training, and obtain the defect detection model.

[0167] In the model training of the defect detection method provided by the embodiments of the present application, the pre-trained model constructed by optimizing yolov5 is trained with the training subset in the training sample dataset. In the model training, the gradient descent method is used to optimize the network parameters, and the optimal network parameters are obtained with the aim of minimizing the error, so as to obtain the defect detection model that has completed training, which is beneficial to improving the detection accuracy of the trained defect detection model.

[0168] In a possible design, after obtaining the defect detection model in step S303, it further includes:[[]]

[0169] After the model training is completed, the performance of the model is tested with the test subset in the training sample dataset, and the test subset in the training sample dataset is obtained from the training sample dataset according to the second preset ratio.

[0170] For example, evaluation metrics can be used to evaluate the performance of the defect detection model to evaluate the precision of the defect detection model.

[0171] In some embodiments, the evaluation metric used is the mean average precision (mAP), which refers to the average of the average precisions (AP) under different categories, as shown in the following formula (3):

[0172]

[0173] where N represents the category.

[0174] Performance evaluation through a test subset can obtain the precision rate of this defect detection model. In the embodiment of this application, the accuracy rate can reach 96%, and the inference speed is comparable to that of yolov5s. Specifically, as shown in Table 3:

[0175] Table 3

[0176]

[0177] As can be seen from Table 3, by optimizing yolov5 to construct a pre-trained model, the defect detection model obtained by training the pre-trained model has the advantages of high detection precision rate and fast inference time.

[0178] Figure 9 This is a schematic structural diagram of a defect detection device provided by an embodiment of this application. As Figure 9 shown, the defect detection device 400 provided by the embodiment of this application includes:

[0179] A detection data acquisition module 401, configured to acquire target picture data of a solar panel to be detected through an image acquisition device;

[0180] A detection module 402, configured to input the target picture data into a defect detection model, obtain the output of the defect detection model, so as to obtain a defect detection result of the solar panel to be detected;

[0181] Among them, the defect detection model is obtained by training a pre-trained model with a training sample data set; the pre-trained model is constructed based on a preset backbone network and a feature scale extraction model, and an attention model is added.

[0182] In Figure 9 addition, Figure 10 This is a schematic structural diagram of another defect detection device provided by an embodiment of this application. As Figure 10 shown, the defect detection device 400 provided by the embodiment of this application further includes: a sample data acquisition module 403, and the sample data acquisition module 403 is configured to:

[0183] Acquire an original sample data set, and preprocess the original defect samples in the original sample data set to obtain a training sample data set, where the original defect samples include solar panel pictures classified by the category of feature defects.

[0184] In a possible design, the preset backbone network includes the backbone of pre-trained yolov5, and the feature scale extraction model includes the small and medium scale feature extraction branches in the Neck network of yolov5.

[0185] In a possible design, the sample data acquisition module 403 is further configured to:

[0186] Obtain multiple first solar panel pictures with characteristic defects through the EL public dataset;

[0187] Obtain multiple second solar panel pictures with characteristic defects;

[0188] Generate an original sample dataset based on the first solar panel pictures and the second solar panel pictures;

[0189] Among them, the second solar panel pictures are pictures of solar panels with characteristic defects taken according to different shooting parameters; the shooting parameters include different shooting angles and / or different shooting brightnesses.

[0190] In a possible design, the sample data acquisition module 403 is further configured to:

[0191] Amplify the number of samples of the original defect samples in the original sample dataset through data augmentation to obtain target defect samples corresponding to the original defect samples, and obtain a training sample dataset according to the target defect samples and the original defect samples.

[0192] In a possible design, the data augmentation method includes at least one of brightness reduction, brightness enhancement, contrast reduction, contrast enhancement, horizontal flipping, or vertical flipping.

[0193] In a possible design, the sample data acquisition module 403 is further configured to:

[0194] Generate corresponding defect label information for the training samples in the training sample dataset, and the defect label information is used to label the category and location information of the characteristic defects of the training samples;

[0195] Adjust the picture size of the training samples to a target unified size.

[0196] Perform defect annotation on the training samples in the training sample dataset and unify the picture sizes of the training samples to facilitate model training.

[0197] In a possible design, there is also a residual connection between the shallow convolutional layer and the deep convolutional layer of the backbone of yolov5, and the attention model is located after the convolutional layer of the backbone of yolov5, and the convolutional layer includes a shallow convolutional layer and a deep convolutional layer;

[0198] Among them, the residual connection is used for feature fusion in defect recognition, and the attention model is used to obtain edge detail features in defect recognition.

[0199] In Figure 10 Based on Figure 11 This is a schematic structural diagram of another defect detection device provided by the embodiments of the present application. As Figure 11As shown in the figure, the defect detection device 400 provided by the embodiment of the present application further includes: a model training module 404, and the model training module 404 is configured to:

[0200] Perform model training on the pre-trained model through a training subset in the training sample dataset; the training subset is obtained from the training sample dataset according to a first preset ratio;

[0201] Optimize the network parameters of the pre-trained model by using the gradient descent method according to the prediction results obtained during the model training;

[0202] When the preset loss function in the model training reaches the convergence condition, obtain the optimal network parameters, end the model training, and determine the pre-trained model containing the optimal network parameters as the defect detection model;

[0203] Wherein, the convergence condition is to minimize the error, and the preset loss function includes the intersection over union of the predicted box and the true box.

[0204] In a possible design, the model training module 404 is further configured to:

[0205] Freeze the backbone of yolov5 in the pre-trained model, and perform preliminary training on the network structure except the backbone of yolov5 using a larger learning rate;

[0206] Unfreeze the backbone of yolov5, and fine-tune the pre-trained model after preliminary training using a smaller learning rate.

[0207] In a possible design, the defect detection module 400 further includes: an evaluation module, and the evaluation model is configured to:

[0208] Perform performance evaluation on the defect detection model based on evaluation metrics using a test subset in the training sample dataset to obtain the precision rate of the defect detection model;

[0209] Wherein, the test subset is obtained from the training sample dataset according to a second preset ratio; the sum of the first preset ratio and the second preset ratio is 1; the evaluation metrics include the mean average precision.

[0210] In a possible design, the characteristic defects include pure crack defects or pure fragment defects.

[0211] The defect detection device provided by the embodiment of the present application can execute the steps of the defect detection method in the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0212] Figure 12 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 12As shown in the figure, the electronic device 500 provided by the embodiment of the present application may include: a processor 501, and a memory 502 communicatively connected to the processor 501.

[0213] The memory 502 is used to store programs. Specifically, the program may include program code, and the program code includes computer execution instructions.

[0214] The memory 502 may include a high-speed RAM memory, and may also include a non-volatile memory (NoN-volatile memory), such as at least one disk memory.

[0215] The processor 501 is used to execute the computer execution instructions stored in the memory 502 to implement the defect detection method.

[0216] Among them, the processor 501 may be a central processing unit (CPU for short), or a specific integrated circuit (ASIC for short), or one or more integrated circuits configured to implement the embodiment of the present application.

[0217] Optionally, the memory 502 may be either independent or integrated with the processor 501. When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include:

[0218] A bus 503 for connecting the processor 501 and the memory 502. The bus may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0219] Optionally, in a specific implementation, if the memory 502 and the processor 501 are integrated on a chip, the memory 502 and the processor 501 can communicate through an internal interface.

[0220] The present application also provides a computer-readable storage medium, which may include: various media capable of storing program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs. Specifically, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used for the steps of the method in the above embodiments.

[0221] The present application also provides a computer program product, including computer-executable instructions, and when the computer instructions are executed by a processor, the method in the above embodiments is implemented.

[0222] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the claims.

[0223] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A defect detection method, characterized in that, Including: Obtaining target picture data of a solar panel to be detected through an image acquisition device; Inputting the target picture data into a defect detection model, and obtaining the output of the defect detection model to obtain the defect detection result of the solar panel to be detected; Wherein, the defect detection model is obtained by training a pre-trained model with a training sample data set; the pre-trained model is based on a preset backbone network and a feature scale extraction model, and is constructed by adding an attention model.

2. The defect detection method according to claim 1, wherein The method further includes: Obtaining an original sample data set, and preprocessing the original defect samples in the original sample data set to obtain the training sample data set, where the original defect samples include solar panel pictures classified by the categories of characteristic defects.

3. The defect detection method according to claim 1 or 2, characterized in that, The preset backbone network includes the backbone of pre-trained yolov5, and the feature scale extraction model includes the medium and small scale feature extraction branches in the Neck network of yolov5.

4. The defect detection method according to claim 2, wherein The obtaining of the original sample data set includes: Obtaining multiple first solar panel pictures with the characteristic defects through an EL public data set; Obtaining multiple second solar panel pictures with the characteristic defects; Generating the original sample data set according to the first solar panel pictures and the second solar panel pictures; Wherein, the second solar panel pictures are pictures of solar panels with the characteristic defects taken according to different shooting parameters; the shooting parameters include different shooting angles and / or different shooting brightnesses.

5. The defect detection method according to claim 2, wherein, Preprocessing the original defect samples in the original sample data set to obtain the training sample data set includes: Amplifying the number of samples of the original defect samples in the original sample data set through data augmentation to obtain target defect samples corresponding to the original defect samples, and obtaining the training sample data set according to the target defect samples and the original defect samples.

6. The defect detection method according to claim 5, wherein The data augmentation method includes at least one of brightness reduction, brightness enhancement, contrast reduction, contrast enhancement, horizontal flipping or vertical flipping.

7. The defect detection method according to claim 5, characterized in that, After obtaining the training sample data set, it further includes: Generating corresponding defect label information for the training samples in the training sample data set, where the defect label information is used to label the category and positioning information of the characteristic defects of the training samples; Adjusting the picture size of the training samples to a target unified size.

8. The defect detection method according to claim 3, characterized in that There is also a residual connection between the shallow convolutional layer and the deep convolutional layer of the backbone of yolov5, and the attention model is located after the convolutional layer of the backbone of yolov5, and the convolutional layer includes the shallow convolutional layer and the deep convolutional layer; Wherein, the residual connection is used for feature fusion in defect recognition, and the attention model is used for obtaining edge detail features in defect recognition.

9. The defect detection method according to claim 8, characterized in that, Training the pre-trained model with the training sample data set to obtain a defect detection model, including: Training the pre-trained model with a training subset in the training sample dataset; the training subset is obtained from the training sample dataset according to a first preset ratio; Optimizing the network parameters of the pre-trained model by using the gradient descent method according to the prediction results obtained during model training; When the preset loss function in model training reaches the convergence condition, obtaining the optimal network parameters, ending the model training, and determining the pre-trained model containing the optimal network parameters as the defect detection model; Wherein, the convergence condition is to minimize the error, and the preset loss function includes the intersection over union of the predicted box and the ground truth box.

10. The defect detection method according to claim 9, wherein, The training the pre-trained model with a training subset in the training sample dataset includes: Freezing the backbone of yolov5 in the pre-trained model and performing preliminary training on the network structure except the backbone of yolov5 with a larger learning rate; Thawing the backbone of yolov5 and fine-tuning the pre-trained model after preliminary training with a smaller learning rate.

11. The defect detection method according to claim 9, wherein, After obtaining the defect detection model, it further includes: Evaluating the performance of the defect detection model based on evaluation metrics by using a test subset in the training sample dataset to obtain the precision rate of the defect detection model; Wherein, the test subset is obtained from the training sample dataset according to a second preset ratio; the sum of the first preset ratio and the second preset ratio is 1; the evaluation metrics include the mean average precision.

12. The defect detection method according to claim 1, characterized in that, The characteristic defects include pure crack defects or pure fragment defects.

13. A defect detection device, characterized in that, It includes: A detection data acquisition module, configured to acquire target picture data of a solar panel to be detected through an image acquisition device; A detection module, configured to input the target picture data into the defect detection model and obtain the output of the defect detection model to obtain the defect detection result of the solar panel to be detected; Wherein, the defect detection model is obtained by training a pre-trained model with a training sample dataset; the pre-trained model is constructed based on a preset backbone network and a feature scale extraction model and by adding an attention model.

14. An electronic device, characterized in that, It includes: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the defect detection method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the defect detection method according to any one of claims 1 to 12.

16. A computer program product, including computer-executable instructions, which are used to implement the defect detection method according to any one of claims 1 to 12 when executed by a processor.

Images