Photovoltaic module infrared defect detection method and system based on fine-grained classification

Abstract

The invention discloses a photovoltaic module infrared defect detection method and system based on fine-grained classification, and solves the problems in the prior art that unmanned aerial vehicle infrared image recognition is easily interfered by background and shooting height, the recognition accuracy is low, similar defects are easily confused, and junction box heating and point-like hot spots cannot be distinguished. The method comprises the steps of collecting and marking a photovoltaic module infrared image data set, establishing a photovoltaic module pair detection model, extracting a single module unit image from the photovoltaic module pair image, establishing a photovoltaic module defect fine-grained classification model, inputting a to-be-detected image into the detection model, extracting the single module unit image, and inputting the single module unit image into the classification model to obtain a classification result. According to the invention, a method of first component segmentation and then fine-grained classification of infrared defects is adopted, high interference of environment and shooting is avoided, a method based on a fine-grained classification model is beneficial to distinguishing easily-confused defects, extraction of single component images retaining relative position information between the components is provided, and distinguishing of junction box heating and point-like hot spots is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared defect detection of photovoltaic panels, and in particular to a method and system for infrared defect detection of photovoltaic components based on fine-grained classification. Background Art

[0002] Infrared defects are key defects that affect the normal operation of photovoltaic power stations. Strengthening infrared defect detection is crucial to the operation and maintenance of photovoltaic power stations. At present, many technologies use drones equipped with infrared cameras to take infrared images of photovoltaic modules in photovoltaic power stations and identify infrared defects in images, but there are still many defects, including: 1. Some existing technologies directly identify infrared images taken by drones. However, since the infrared images taken by drones include both photovoltaic panels and backgrounds, areas in the background that are similar to photovoltaic defect features will interfere with the detection model.

[0003] 2. Some existing technologies directly recognize infrared images taken by drones, but it is difficult to completely unify the shooting height when drones are shooting, resulting in inconsistent sizes of photovoltaic modules in the image, which reduces the accuracy of model recognition.

[0004] 3. The characteristics of some photovoltaic defects are similar, and the existing technology is prone to confusion in classification.

[0005] 4. Some existing technologies use the method of segmentation before detection. However, since the junction box on the photovoltaic panel will also generate slight heat, its characteristics on the infrared image are very similar to those of point-like hot spots. If the segmented component image is directly identified, the position information will be missing, and it will be impossible to distinguish between the heating of the junction box and the point-like hot spots. Summary of the invention

[0006] The present invention mainly solves the problems that the existing technology is easily disturbed by the background and shooting height when recognizing drone infrared images, has low recognition accuracy, is easily confused in classification, and cannot distinguish between heating of the junction box and point-like hot spots. It provides a photovoltaic module infrared defect detection method and system based on fine-grained classification.

[0007] The above technical problem of the present invention is mainly solved by the following technical solution: A photovoltaic module hot spot detection method based on fine-grained classification comprises the following steps: S1. Photovoltaic module infrared image dataset collection and annotation; S2. Build and train a photovoltaic module pair detection model; S3. Using the photovoltaic module pair detection model to segment and obtain a photovoltaic module pair image set; S4. Based on the determined junction box position information, a single component unit image is extracted from the photovoltaic component image set; S5. Build and train a fine-grained classification model for photovoltaic module defects; S6. The image to be inspected is input into the photovoltaic module pair inspection model, and the single module unit image is extracted and then input into the photovoltaic module defect fine-grained classification model to obtain the classification result.

[0008] The present invention adopts the method of first segmenting components and then classifying hot spots in a fine-grained manner, avoiding interference from the environment and shooting height. During the classification process, it proposes single-component image extraction for photovoltaic component images to determine the position information of the junction box, so that the relative position information between components can be retained, which is conducive to distinguishing between junction box heating and point-like hot spots.

[0009] As a preferred solution, step S1 specifically includes: An infrared camera is used to obtain images of photovoltaic modules, and the outline of the area where each module is located in the image is marked.

[0010] In this solution, the infrared data of photovoltaic modules is collected by the infrared camera carried by the drone. Select some images from the collected images. Due to the shooting characteristics of the drone, adjacent images usually overlap partially. The overlap between the selected images should be kept as small as possible. Label the selected infrared images. It is preferred to use roLabelImg to label the outline of the area where each component pair in the image is located. Generally, a box is used to label the outline of the photovoltaic module pair.

[0011] As a preferred solution, step S2 specifically includes: Establish training set, validation set and test set based on the infrared image dataset of photovoltaic modules; A photovoltaic module pair detection model is established using a rotating target detection network; The training set is input into the model, and multiple rounds of iterative training are carried out until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the trained photovoltaic module pair detection model is obtained through verification on the test set.

[0012] The construction of photovoltaic module detection model uses rotating target detection network R 2 CNN is used as its model. The dataset is divided into training set, validation set and test set in a ratio of 7:1.5:1.5. The PV module pair detection model is trained in an end-to-end manner. When the model is initialized, the pre-trained parameters are loaded first, and then fine-tuned on its own dataset. The model is continuously trained through multiple rounds of iterations until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and further verification is performed on the test set. The trained PV module pair detection model is obtained through verification. The PV module pair detection model inputs the infrared image of the PV module, outputs each PV module pair image, and records the coordinates of the four corners of the PV module pair image.

[0013] As a preferred solution, step S4 specifically includes: The photovoltaic module performs affine transformation on each image in the image set to adjust the viewing angle; Each single-component unit image with a consistent junction box position is cut from the image adjusted for viewing angle.

[0014] Generally, the junction box is located in the middle of a photovoltaic module pair. This scheme performs affine transformation on all images in the photovoltaic module pair image set to adjust them into images with the same viewing angle, cuts the images, and then adjusts them into single-module unit images with the junction box located in the same position. The relative position information of the components in the obtained single-module unit images is retained, so that the model can quickly determine the hot spots of the junction box in the single-module unit images, effectively distinguish between junction box heating and point-like hot spots, and facilitate manual annotation of single-module unit images for training of fine-grained classification models for photovoltaic module defects.

[0015] As a preferred solution, the image is subjected to affine transformation to adjust the viewing angle, specifically including: Adjust the PV panel image to a front view perspective.

[0016] This solution preferably adjusts the photovoltaic component pair image into a front view perspective image through affine transformation.

[0017] As a preferred solution, the cutting to obtain each single component unit image specifically includes: The midpoint connecting line of the two long sides of the photovoltaic module image is cut to obtain two module images, and the module image located below the photovoltaic module image is rotated 180°, and the junction box of each module is located below the image. The two module images finally obtained are single-module unit images.

[0018] The two components in the photovoltaic component pair are symmetrical structures, and the junction box is set between the two components. The photovoltaic component pair image is cut according to the line connecting the midpoints of the two long sides of the image from the front view perspective, and two component images are obtained. The coordinates of the component image after cutting are obtained according to the coordinates of the four corners of the photovoltaic component pair image and the midpoints of the long sides of the four corners of the photovoltaic component pair image. The component image located below the original photovoltaic component pair image is rotated 180°, and the component image is adjusted to an angle consistent with the component image located above the original photovoltaic component pair image, so that the junction box part of each component is located below the image, and the relative position information of the component is retained, which is convenient for the classification model to distinguish the heating of the junction box from the point-shaped hot spots in the single component unit image, and at the same time, the single component unit image is conveniently and quickly accurately labeled. The present invention sets the component defects to include five types, namely normal, point-shaped hot spots, strip-shaped hot spots, plant occlusion, and building occlusion. When labeling, the high temperature area generated by the junction box and the hot spots generated by the fault are distinguished according to the fixed position of the junction box in the single component unit image.

[0019] As a preferred solution, step S5 specifically includes: Scale all single-component unit images to the same size, annotate defect types, and create training sets, validation sets, and test sets; A bilinear convolution model is used to establish a photovoltaic module pair detection model and a photovoltaic module defect fine-grained classification model; The training set is input into the model, and multiple rounds of iterative training are carried out until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the trained PV module defect fine-grained classification model is obtained through verification on the test set.

[0020] This solution annotates single-component unit images, and the annotated defects include five types, namely normal, point-shaped hot spots, strip-shaped hot spots, plant occlusion, and building occlusion. All single-component unit images are scaled to the same size so that the differences in component size caused by different shooting heights will not interfere with model detection, which is beneficial to improving the accuracy of model recognition. The order of annotation and scaling steps for single-component unit images can be adjusted. After all single-component unit images are annotated and scaled, they are divided into training set, validation set, and test set in a ratio of 7:1.5:1.5. The fine-grained classification model for photovoltaic module defects is established using a bilinear convolutional model. The model includes an input layer, a feature extraction layer of two branches, a pooling layer, a classification layer, and an output layer. Both branches use the feature extraction layer of VGG16, that is, the convolution pooling part before the fully connected layer in VGG16. The training of the fine-grained classification model for photovoltaic module defects adopts an end-to-end approach. When the model is initialized, the VGG16 part is loaded with the parameters pre-trained on ImageNet, the training set is input into the model, and the category probability distribution vector of the single component unit image is output. The cross entropy loss is calculated according to the annotation of the single component unit image, and the model parameters are updated by gradient backpropagation based on the loss. Multiple rounds of iterative training are performed until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the final photovoltaic module defect fine-grained classification model is obtained through verification on the test set.

[0021] As a preferred solution, the photovoltaic module defect fine-grained classification model includes: an input layer, a feature extraction layer, a pooling layer, a fully connected layer and an output layer; Input layer, input single component unit image; The feature extraction layer includes a first branch and a second branch. The first branch detects a local area containing rich classification information, and the second branch extracts the features of the local area detected by the first branch. The feature maps output by the two branches are processed by outer product at each position and then enter the pooling layer. The pooling layer performs sum pooling to obtain a bilinear feature vector, and obtains the feature vector by taking the signed square root and L2 normalization, and enters the fully connected layer; The fully connected layer converts the feature vector into a classification feature vector, and obtains the category probability distribution vector through softmax processing; Output layer, outputs the classification results.

[0022] The model of this scheme includes two feature extraction branches: the first branch and the second branch. Both branches use the feature extraction layer of VGG16. The output feature maps of the two branches are outer-producted at each position, and then summed and pooled by the pooling layer and straightened to obtain a bilinear feature vector ξ, and the signed square root and L2 normalization are taken to obtain the feature vector ξ', and the feature vector ξ' is converted into a classification feature vector through a fully connected layer, and the final category probability distribution vector, i.e., the classification result, is obtained through softmax function processing. The present invention designs a fine-grained classification model for photovoltaic module defects based on a bilinear convolutional network, which is conducive to distinguishing defects with similar features and solves the problem that classification in existing methods is easily confused.

[0023] As a preferred solution, it also includes: S7. labeling the infrared image of the photovoltaic module according to the classification result; The classification results and corresponding four-corner coordinates of the single-component unit image are obtained. Different defect types use different colors and are marked on the image to be inspected according to the four-corner coordinates of the single-component unit image.

[0024] This solution sets different colors for different defect types, and marks the image to be inspected based on the classification results and four-corner coordinates of the single-component unit image, so that the defects of the photovoltaic components can be quickly identified.

[0025] A photovoltaic module infrared defect detection system based on fine-grained classification, comprising: An input unit for inputting an infrared image of a photovoltaic module to be inspected; A segmentation unit, which segments the infrared image of the photovoltaic module through the photovoltaic module pair detection model to obtain the photovoltaic module pair image; The conversion unit extracts and obtains a single-component unit image from the photovoltaic component pair image based on the determined junction box position information; The classification unit obtains the classification result through the recognition of the fine-grained classification model of photovoltaic module defects.

[0026] The method of the system of the present invention first divides the components of the image to be inspected and then classifies the infrared defects in a fine-grained manner, avoiding interference from the environment and the shooting height. At the same time, the fine-grained classification model is conducive to distinguishing the types of defects that are easily confused. In the classification process, it is proposed to extract the single component image of the photovoltaic component to determine the position information of the junction box, so that the relative position information between the components can be retained, which is conducive to distinguishing the heating of the junction box from the point-like hot spots, and solves the problem that the existing technology cannot distinguish the heating of the junction box from the point-like hot spots.

[0027] Therefore, the advantages of the present invention are: 1. The method of first segmenting components and then classifying hot spots in a fine-grained manner avoids environmental interference.

[0028] 2. Scale the component images to be inspected to the same size to avoid the impact of inconsistent shooting heights and improve the accuracy of model recognition.

[0029] 3. A fine-grained classification model is used to help distinguish easily confused defect types.

[0030] 4. During the classification process, a single component image extraction method is proposed to determine the position information of the junction box of the photovoltaic module image, so that the relative position information between the components can be retained, which is conducive to distinguishing between junction box heating and point-like hot spots. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of a process of the present invention.

[0032] Figure 2 It is a structural schematic diagram of the fine-grained classification model of photovoltaic module defects in the present invention.

[0033] Figure 3 It is a schematic diagram of annotating the infrared image of a photovoltaic module in the present invention.

[0034] Figure 4 It is a schematic diagram of extracting a single component unit image from an image of a photovoltaic component in the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0036] Embodiment 1: This embodiment provides a photovoltaic module infrared defect detection method based on fine-grained classification, such as Figure 1 As shown, the following steps are included: S1. Photovoltaic module infrared image dataset collection and annotation.

[0037] An infrared camera is used to obtain images of photovoltaic modules, and the outline of the area where each module is located in the image is marked.

[0038] Specifically, the infrared data of photovoltaic components is collected by the infrared camera carried by the drone. Some images are selected from the collected images. Due to the shooting characteristics of the drone, adjacent images usually overlap partially. The overlap between the selected images should be kept as small as possible. The selected infrared images are labeled. It is preferred to use roLabelImg to label the contour of the area where each component pair in the image is located, such as Figure 3 As shown, generally, a photovoltaic module pair is in the shape of a regular rectangle, and a square frame is used to mark the outline of each photovoltaic module pair.

[0039] S2. Build and train the photovoltaic module detection model. The specific process includes the following: S21. Establish a training set, a validation set, and a test set based on the photovoltaic module infrared image dataset. Preferably, the dataset is divided into a training set, a validation set, and a test set in a ratio of 7:1.5:1.5.

[0040] S22. Using the rotation target detection network R 2 CNN builds a photovoltaic module pair detection model; S23. The training set is input into the model, and multiple rounds of iterative training are carried out until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the trained photovoltaic module pair detection model is obtained through verification on the test set.

[0041] The model is trained in an end-to-end manner. When the model is initialized, the pre-trained parameters are loaded first, and then fine-tuned on its own data machine. The model is continuously trained through multiple rounds of iterations until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained and further verified on the test set. After the verification passes, the final photovoltaic module pair detection model is obtained. The photovoltaic module pair detection model inputs the photovoltaic module infrared image, outputs each photovoltaic module pair image, and records the four corner coordinates of the photovoltaic module pair image.

[0042] S3. Use the photovoltaic module pair detection model to segment and obtain a photovoltaic module pair image set.

[0043] The photovoltaic module infrared image data set is input as a whole into the photovoltaic module pair detection model to obtain an image set consisting of photovoltaic module pair images, and the coordinates of the four corners of each photovoltaic module pair image in the corresponding photovoltaic module infrared image are recorded.

[0044] S4. Based on the determined junction box position information, a single component unit image is extracted from the photovoltaic component pair image set.

[0045] Generally, the junction box is located in the middle of a photovoltaic module pair. In this step, all images in the photovoltaic module pair image set are affine transformed into images with the same viewing angle, and the images are cut and adjusted into single-module unit images with the junction box at the same position. The relative position information of the components in the obtained single-module unit image is retained, so that the classification model can quickly determine the hot spots of the junction box in the single-module unit image, effectively distinguish between junction box heating and point-like hot spots, and facilitate manual annotation of single-module unit images for training of fine-grained classification models for photovoltaic module defects.

[0046] Specifically, Figure 4 As shown, this step includes: S41. Performing an affine transformation on each image in the image set by the photovoltaic module to adjust the viewing angle; Preferably, the photovoltaic assembly image is adjusted to a front view perspective image through affine transformation.

[0047] S42. Cutting and obtaining each single component unit image with a consistent junction box position from the image with the adjusted viewing angle.

[0048] The midpoint connecting line of the two long sides of the photovoltaic module image is cut to obtain two module images, and the module image located below the photovoltaic module image is rotated 180°, and the junction box of each module is located below the image. The two module images finally obtained are single-module unit images.

[0049] The two components in the photovoltaic component pair are symmetrical structures, and the junction box is set between the two components. The photovoltaic component pair image is cut according to the line connecting the midpoints of the two long sides of the image from the front view perspective, and two component images are obtained. The coordinates of the component image after cutting are obtained according to the coordinates of the four corners of the photovoltaic component pair image and the midpoints of the long sides of the four corners of the photovoltaic component pair image. The component image located below the original photovoltaic component pair image is rotated 180°, and the component image is adjusted to an angle consistent with the component image located above the original photovoltaic component pair image, so that the junction box part of each component is located below the image, retaining the relative position information of the component, which is convenient for the classification model to distinguish the heating of the junction box from the point-shaped hot spots in the single component unit image, and at the same time, it is convenient to quickly and accurately mark the single component unit image. The present invention sets the component defects to include five types. When marking, the high temperature area generated by the junction box and the hot spots caused by the fault are distinguished according to the fixed position of the junction box in the single component unit image.

[0050] S5. Build and train a fine-grained classification model for photovoltaic module defects. Specifically, the following steps are included: S51. Scale all single component unit images to the same size, mark defect types, and establish training sets, validation sets, and test sets; All single-component unit images are scaled to the same size so that the component size does not interfere with model detection, which is beneficial to improving the accuracy of model recognition. The single-component unit images are also annotated, and the annotated defects include five types, namely normal, point-shaped hot spots, strip-shaped hot spots, plant occlusion, and building occlusion. In this process, the order of annotating and scaling the single-component unit images can be adjusted according to needs. After all single-component unit images are annotated and scaled, they are divided into training set, verification set, and test set in a ratio of 7:1.5:1.5.

[0051] S52. Use bilinear convolution model to establish photovoltaic module pair detection model photovoltaic module defect fine-grained classification model; As a preferred solution of this embodiment, a photovoltaic module defect fine-grained classification model based on a bilinear convolutional network is designed, which is conducive to distinguishing defects with similar features, such as Figure 2As shown, the specific structure of the photovoltaic module defect fine-grained classification model includes an input layer, a feature extraction layer, a pooling layer, a classification layer and an output layer; Input layer, input single component unit image; The feature extraction layer includes a first branch and a second branch. The first branch detects a local area containing rich classification information, and the second branch extracts the features of the local area detected by the first branch. The feature maps output by the two branches are processed by outer product at each position and then enter the pooling layer. The pooling layer performs sum pooling to obtain a bilinear feature vector, and obtains the feature vector by taking the signed square root and L2 normalization, and enters the fully connected layer; The classification layer converts the feature vector into a classification feature vector and obtains the category probability distribution vector through softmax processing; Output layer, outputs the classification results.

[0052] Specifically, the feature extraction layer of the fine-grained classification model for photovoltaic module defects in this embodiment includes a first branch and a second branch, and both branches use the feature extraction layer of VGG16, that is, the convolution pooling part before the fully connected layer in VGG16. For the output feature maps of the two branches, the outer product is performed at each position, and then the bilinear feature vector ξ is obtained by summing, pooling and straightening through the pooling layer, and the signed square root and L2 normalization are taken to obtain the feature vector ξ', and the feature vector ξ' is converted into a classification feature vector through the fully connected layer of the classification layer, and the final category probability distribution vector, that is, the classification result, is obtained through the softmax function processing.

[0053] S53. The training set is input into the model, and multiple rounds of iterative training are carried out until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the trained PV module defect fine-grained classification model is obtained through verification on the test set.

[0054] The training of the fine-grained classification model for photovoltaic module defects adopts an end-to-end training method. When the model is initialized, the VGG16 part is loaded with the parameters pre-trained on ImageNet, the training set is input into the model, and the category probability distribution vector of the single component unit image is output. The cross entropy loss is calculated according to the annotation of the single component unit image, and the model parameters are updated by gradient back propagation based on the loss. Multiple rounds of iterative training are performed until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and further verification is performed on the validation set. The final photovoltaic module defect fine-grained classification model is obtained through verification.

[0055] S6. The image to be inspected is input into the photovoltaic module pair inspection model, and the single module unit image is extracted and then input into the photovoltaic module defect fine-grained classification model to obtain the classification result.

[0056] Specifically, prepare the infrared image of the photovoltaic module to be tested, input the infrared image of the photovoltaic module to be tested into the photovoltaic module pair detection model, obtain the photovoltaic module pair image and save its four-corner coordinates on the original photovoltaic module infrared image. Perform affine transformation on the photovoltaic module pair image to obtain an orthographic view, and perform center cutting and rotation processing to obtain a single component unit image. The corresponding coordinates of each single component unit image in the original photovoltaic module infrared image are obtained by taking the midpoint of the long side according to the four-corner coordinates of the photovoltaic module pair to which it belongs. The single component unit image is input into the photovoltaic module defect fine-grained classification model to obtain its category.

[0057] The present invention adopts a method of first segmenting components and then classifying hot spots in a fine-grained manner, thereby avoiding interference from the environment and shooting height, so that the components are scaled to the same size, which is beneficial to improving the accuracy of model recognition.

[0058] The present invention designs a photovoltaic defect fine-grained classification model based on a bilinear convolutional network, which is conducive to distinguishing defects with similar characteristics.

[0059] In the classification process, the present invention proposes to extract single component images of photovoltaic component pairs to determine the position information of junction boxes, so that the relative position information between components can be retained, which is conducive to distinguishing heating of junction boxes from point-like hot spots.

[0060] Embodiment 2: This embodiment is a photovoltaic module infrared defect detection system based on fine-grained classification, which is used to implement the photovoltaic module infrared defect detection method based on fine-grained classification in the above-mentioned embodiment 1. The system includes an input unit, a segmentation unit, a conversion unit and a classification unit connected in sequence; An input unit for inputting an infrared image of a photovoltaic module to be inspected; A segmentation unit, which segments the infrared image of the photovoltaic module through the photovoltaic module pair detection model to obtain the photovoltaic module pair image; The conversion unit extracts a single-component unit image from the photovoltaic component pair image based on the determined junction box position information; the specific extraction steps include: The photovoltaic module pair image is adjusted into a front view perspective image through affine transformation; The midpoint connecting line of the two long sides of the photovoltaic module image is cut to obtain two module images, and the module image located below the photovoltaic module image is rotated 180°, and the junction box of each module is located below the image. The two module images finally obtained are single-module unit images.

[0061] The classification unit obtains the classification result through the recognition of the fine-grained classification model of photovoltaic module defects.

[0062] Embodiment 3: This embodiment provides another implementation of a photovoltaic module infrared defect detection method based on fine-grained classification, which specifically includes the following steps: S1. Photovoltaic module infrared image dataset collection and annotation; S2. Build and train a photovoltaic module pair detection model; S3. Using the photovoltaic module pair detection model to segment and obtain a photovoltaic module pair image set; S4. Based on the determined junction box position information, a single component unit image is extracted from the photovoltaic component image set; S5. Build and train a fine-grained classification model for photovoltaic module defects; S6. The image to be inspected is input into the photovoltaic module pair inspection model, and the single module unit image is extracted and then input into the photovoltaic module defect fine-grained classification model to obtain the classification result; S7. Label the infrared image of the photovoltaic module according to the classification result.

[0063] The step S7 is added to label the detection results of the fine-grained classification model of photovoltaic module defects, specifically: The classification results and corresponding four-corner coordinates of the single-component unit image are obtained. Different defect types are marked in different colors on the infrared image of the photovoltaic module to be inspected according to the four-corner coordinates of the single-component unit image.

[0064] This solution sets different colors for different defect types, and marks the image to be inspected based on the classification results and four-corner coordinates of the single-component unit image, so that the defects of the photovoltaic components can be quickly identified.

[0065] Steps S1-S6 of this embodiment are consistent with the embodiment, and are specifically as follows: Step S1 specifically includes: An infrared camera is used to obtain images of photovoltaic modules, and the outline of the area where each module is located in the image is marked.

[0066] Photovoltaic component infrared data collection is obtained by using an infrared camera carried by a drone. Some images are selected from the collected images. Due to the shooting characteristics of drones, adjacent images usually overlap partially. The overlap between the selected images should be kept as small as possible. The selected infrared images are labeled. It is preferred to use roLabelImg to label the contour of the area where each component pair in the image is located, such as Figure 3 As shown, generally, a photovoltaic module pair is in the shape of a regular rectangle, and a square frame is used to mark the outline of each photovoltaic module pair.

[0067] Step S2 specifically includes the following process: S21. Establish a training set, a validation set, and a test set based on the photovoltaic module infrared image dataset. Preferably, the dataset is divided into a training set, a validation set, and a test set in a ratio of 7:1.5:1.5.

[0068] S22. Using the rotation target detection network R 2 CNN builds a photovoltaic module pair detection model; S23. The training set is input into the model, and multiple rounds of iterative training are carried out until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the trained photovoltaic module pair detection model is obtained through verification on the test set.

[0069] The model is trained in an end-to-end manner. When the model is initialized, the pre-trained parameters are loaded first, and then fine-tuned on its own data machine. The model is continuously trained through multiple rounds of iterations until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained and further verified on the test set. After the verification passes, the final photovoltaic module pair detection model is obtained. The photovoltaic module pair detection model inputs the photovoltaic module infrared image, outputs each photovoltaic module pair image, and records the four corner coordinates of the photovoltaic module pair image.

[0070] Step S3 specifically includes the following process: The photovoltaic module infrared image data set is input as a whole into the photovoltaic module pair detection model to obtain an image set consisting of photovoltaic module pair images, and the coordinates of the four corners of each photovoltaic module pair image in the corresponding photovoltaic module infrared image are recorded.

[0071] Generally, the junction box is located in the middle of a photovoltaic module pair. In this step, all images in the photovoltaic module pair image set are affine transformed into images with the same viewing angle, and the images are cut and adjusted into single-module unit images with the junction box at the same position. The relative position information of the components in the obtained single-module unit image is retained, so that the hot spot of the junction box in the single-module unit image can be quickly determined, and the heating of the junction box and the point-like hot spots can be effectively distinguished, which facilitates the annotation of the single-module unit image for the training of the fine-grained classification model of photovoltaic module defects.

[0072] Step S4 specifically includes the following process: S41. Performing an affine transformation on each image in the image set by the photovoltaic module to adjust the viewing angle; Preferably, the photovoltaic assembly image is adjusted to a front view perspective image through affine transformation.

[0073] S42. Cutting and obtaining each single component unit image with a consistent junction box position from the image with the adjusted viewing angle.

[0074] The midpoint connecting line of the two long sides of the photovoltaic module image is cut to obtain two module images, and the module image located below the photovoltaic module image is rotated 180°, and the junction box of each module is located below the image. The two module images finally obtained are single-module unit images.

[0075] The two components in the photovoltaic component pair are symmetrical structures, and the junction box is set between the two components. The photovoltaic component pair image is cut according to the line connecting the midpoints of the two long sides of the image from the front view perspective, and two component images are obtained. The coordinates of the component image after cutting are obtained according to the coordinates of the four corners of the photovoltaic component pair image and the midpoints of the long sides of the four corners of the photovoltaic component pair image. The component image located below the original photovoltaic component pair image is rotated 180°, and the component image is adjusted to an angle consistent with the component image located above the original photovoltaic component pair image, so that the junction box part of each component is located below the image, retaining the relative position information of the component, which is convenient for the classification model to distinguish the heating of the junction box from the point-shaped hot spots in the single component unit image, and at the same time, it is convenient to quickly and accurately mark the single component unit image. The present invention sets the component defects to include five types. When marking, the high temperature area generated by the junction box and the hot spots caused by the fault are distinguished according to the fixed position of the junction box in the single component unit image.

[0076] Step S5 specifically includes the following process: S51. Scale all single component unit images to the same size, mark defect types, and establish training sets, validation sets, and test sets; All single-component unit images are scaled to the same size so that the component size does not interfere with model detection, which is beneficial to improving the accuracy of model recognition. The single-component unit images are also annotated, and the annotated defects include five types, namely normal, point-shaped hot spots, strip-shaped hot spots, plant occlusion, and building occlusion. In this process, the order of annotating and scaling the single-component unit images can be adjusted according to needs. After all single-component unit images are annotated and scaled, they are divided into training set, verification set, and test set in a ratio of 7:1.5:1.5.

[0077] S52. Use bilinear convolution model to establish photovoltaic module pair detection model photovoltaic module defect fine-grained classification model; As a preferred solution of this embodiment, a photovoltaic module defect fine-grained classification model based on a bilinear convolutional network is designed, which is conducive to distinguishing defects with similar features, such as Figure 2 As shown, the specific structure of the photovoltaic module defect fine-grained classification model includes an input layer, a feature extraction layer, a pooling layer, a classification layer and an output layer; Input layer, input single component unit image; The feature extraction layer includes a first branch and a second branch. The first branch detects a local area containing rich classification information, and the second branch extracts the features of the local area detected by the first branch. The feature maps output by the two branches are processed by outer product at each position and then enter the pooling layer. The pooling layer performs sum pooling to obtain a bilinear feature vector, and obtains the feature vector by taking the signed square root and L2 normalization, and enters the fully connected layer; The classification layer converts the feature vector into a classification feature vector and obtains the category probability distribution vector through softmax processing; Output layer, outputs the classification results.

[0078] Specifically, the feature extraction layer of the fine-grained classification model for photovoltaic module defects in this embodiment includes a first branch and a second branch, and both branches use the feature extraction layer of VGG16, that is, the convolution pooling part before the fully connected layer in VGG16. For the output feature maps of the two branches, the outer product is performed at each position, and then the bilinear feature vector ξ is obtained by summing, pooling and straightening through the pooling layer, and the signed square root and L2 normalization are taken to obtain the feature vector ξ', and the feature vector ξ' is converted into a classification feature vector through the fully connected layer of the classification layer, and the final category probability distribution vector, that is, the classification result, is obtained through the softmax function processing.

[0079] S53. The training set is input into the model, and multiple rounds of iterative training are carried out until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the trained PV module defect fine-grained classification model is obtained through verification on the test set.

[0080] The training of the fine-grained classification model for photovoltaic module defects adopts an end-to-end training method. When the model is initialized, the VGG16 part is loaded with the parameters pre-trained on ImageNet, the training set is input into the model, and the category probability distribution vector of the single component unit image is output. The cross entropy loss is calculated according to the annotation of the single component unit image, and the model parameters are updated by gradient back propagation based on the loss. Multiple rounds of iterative training are performed until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and further verification is performed on the validation set. The final photovoltaic module defect fine-grained classification model is obtained through verification.

[0081] Step S6 specifically includes: Prepare the infrared image of the photovoltaic module to be tested, input the infrared image of the photovoltaic module to be tested into the photovoltaic module pair detection model, obtain the photovoltaic module pair image and save its four-corner coordinates on the original photovoltaic module infrared image. Perform affine transformation on the photovoltaic module pair image to obtain an orthographic view, and perform center cutting and rotation processing to obtain a single component unit image. The corresponding coordinates of each single component unit image in the original photovoltaic module infrared image are obtained by taking the midpoint of the long side according to the four-corner coordinates of the photovoltaic module pair to which it belongs. Input the single component unit image into the photovoltaic module defect fine-grained classification model to obtain its category.

[0082] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0083] Although the terms such as input unit, segmentation unit, conversion unit, classification unit, etc. are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A photovoltaic module infrared defect detection method based on fine-grained classification, characterized by: The following steps are involved: S1. Photovoltaic module infrared image dataset collection and annotation; S2. Build and train a photovoltaic module pair detection model; S3. Using the photovoltaic module pair detection model to segment and obtain a photovoltaic module pair image set; S4. Based on the determined junction box position information, a single component unit image is extracted from the photovoltaic component image set; S5. Build and train a fine-grained classification model for photovoltaic module defects; S6. The image to be inspected is input into the photovoltaic module pair inspection model, and the single module unit image is extracted and then input into the photovoltaic module defect fine-grained classification model to obtain the classification result.

2. According to claim 1, a photovoltaic module infrared defect detection method based on fine-grained classification is characterized by The step S1 specifically includes: An infrared camera is used to obtain images of photovoltaic modules, and the outline of the area where each module is located in the image is marked.

3. The photovoltaic module infrared defect detection method based on fine-grained classification according to claim 1 is characterized by: The step S2 specifically includes: Establish training set, validation set and test set based on the infrared image dataset of photovoltaic modules; A photovoltaic module pair detection model is established using a rotating target detection network; The training set is input into the model, and multiple rounds of iterative training are carried out until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the trained photovoltaic module pair detection model is obtained through verification on the test set.

4. A photovoltaic module infrared defect detection method based on fine-grained classification according to claim 1, 2 or 3, characterized in that Step S4 specifically includes: The photovoltaic module performs affine transformation on each image in the image set to adjust the viewing angle; Each single-component unit image with a consistent junction box position is cut from the image with the adjusted viewing angle.

5. The photovoltaic module infrared defect detection method based on fine-grained classification according to claim 4 is characterized in that The image is subjected to affine transformation to adjust the viewing angle, specifically comprising: Adjust the PV panel image to a front view perspective.

6. The photovoltaic module infrared defect detection method based on fine-grained classification according to claim 4 is characterized by: The cutting obtains each single component unit image, specifically comprising: The midpoint connecting line of the two long sides of the photovoltaic module image is cut to obtain two module images, and the module image located below the photovoltaic module image is rotated 180°, and the junction box of each module is located below the image. The two module images finally obtained are single-module unit images.

7. The photovoltaic module infrared defect detection method based on fine-grained classification according to claim 4 is characterized by: The step S5 specifically includes: Scale all single-component unit images to the same size, annotate defect types, and create training sets, validation sets, and test sets; A bilinear convolution model is used to establish a photovoltaic module pair detection model and a photovoltaic module defect fine-grained classification model; The training set is input into the model, and multiple rounds of iterative training are carried out until the performance on the validation set converges. The model parameters with the best performance on the validation set are retained, and the trained PV module defect fine-grained classification model is obtained through verification on the test set.

8. The photovoltaic module infrared defect detection method based on fine-grained classification according to claim 7 is characterized by: The photovoltaic module defect fine-grained classification model includes: an input layer, a feature extraction layer, a pooling layer, a classification layer and an output layer; Input layer, input single component unit image; The feature extraction layer includes a first branch and a second branch. The first branch detects a local area containing rich classification information, and the second branch extracts the features of the local area detected by the first branch. The feature maps output by the two branches are processed by outer product at each position and then enter the pooling layer. The pooling layer performs sum pooling to obtain a bilinear feature vector, and obtains the feature vector by taking the signed square root and L2 normalization, and enters the fully connected layer; The classification layer converts the feature vector into a classification feature vector and obtains the category probability distribution vector through softmax processing; Output layer, outputs the classification results.

9. A photovoltaic module infrared defect detection method based on fine-grained classification according to claim 1, 2 or 3, characterized in that Also includes: S7. labeling the infrared image of the photovoltaic module according to the classification result; The classification results and corresponding four-corner coordinates of the single-component unit image are obtained. Different defect types use different colors and are marked on the image to be inspected according to the four-corner coordinates of the single-component unit image.

10. A photovoltaic module infrared defect detection system based on fine-grained classification, implementing the method described in any one of claims 1 to 9, characterized in that: include: An input unit for inputting an infrared image of a photovoltaic module to be inspected; A segmentation unit, which segments the infrared image of the photovoltaic module through the photovoltaic module pair detection model to obtain the photovoltaic module pair image; A conversion unit, based on the determined junction box position information, extracts a single-component unit image from the photovoltaic component pair image; The classification unit obtains the classification results through the recognition of the fine-grained classification model of photovoltaic module defects.

Images