Power station photovoltaic module defect detection method and device based on unsupervised learning

Abstract

The invention provides a power station photovoltaic module defect detection method and device based on unsupervised learning. The method comprises the steps that firstly, a thermal imaging image to be detected and a visible light image in a target area provided with a photovoltaic module are collected; cutting a thermal imaging photovoltaic module image from the thermal imaging image, and cutting a visible light photovoltaic module image from the visible light image; afterwards, a teacher network model and a visible light photovoltaic student network model in the photovoltaic defect detection model are used for processing the visible light photovoltaic module image, and position information of the visible light abnormal photovoltaic module is determined and returned; and the teacher network model and the thermal imaging photovoltaic student network model in the photovoltaic defect detection model are used to process the thermal imaging photovoltaic module image, and the position information of the thermal imaging abnormal photovoltaic module is determined and returned. Wherein the photovoltaic defect detection model is obtained by training in a label-free supervision mode.

Description

Technical Field

[0001] The present disclosure relates to the field of clean energy, and in particular to the field of detection technology, and discloses a method and device for detecting defects of photovoltaic components in a power station based on unsupervised learning. Background Art

[0002] Photovoltaic power generation uses renewable resources such as solar energy to achieve large-scale power generation through clean photoelectric conversion technology, which can significantly reduce dependence on fossil energy and significantly reduce greenhouse gas emissions. In addition, photovoltaic power generation has cost advantages, increases energy diversity, and ensures energy security, which is of great significance to sustainable development and renewable energy strategies.

[0003] Photovoltaic modules in power stations are prone to failure due to long-term exposure to the natural environment. Typical failures such as hot spots will cause the electrical efficiency of the area corresponding to the hot spots to drop significantly, seriously affecting the power generation of the entire photovoltaic module, and may even cause problems with the entire module. Hot spots are temperature-sensitive defects. In addition, photovoltaic modules are also subject to other damages due to long-term exposure to the natural environment, resulting in other temperature-insensitive defects, such as VA shedding, foreign body obstruction, cracks, yellowing, bubbles, etc. These defects also have a great impact on power generation efficiency.

[0004] At present, the detection of the above defects has the following defects: low accuracy, poor robustness, susceptibility to environmental factors, and low detection efficiency. Summary of the invention

[0005] The present disclosure at least provides a method and device for detecting defects of photovoltaic modules in a power station based on unsupervised learning to solve at least one of the above defects.

[0006] According to one aspect of the present disclosure, a method for detecting defects of photovoltaic modules in a power station based on unsupervised learning is provided, comprising:

[0007] Collect thermal imaging images and visible light images to be detected in the target area where the photovoltaic components are set;

[0008] The thermal imaging image and the visible light image are segmented using the trained photovoltaic segmentation model to obtain a first mask image of the region where the photovoltaic components are located in the thermal imaging image and a second mask image of the region where the photovoltaic components are located in the visible light image; the thermal imaging photovoltaic component image is cropped from the thermal imaging image using the first mask image; and the visible light photovoltaic component image is cropped from the visible light image using the second mask image;

[0009] A first test feature matrix of a visible light photovoltaic component image is extracted using a teacher network model in a trained photovoltaic defect detection model, and a second test feature matrix of a visible light photovoltaic component image is extracted using a visible light photovoltaic student network model in the photovoltaic defect detection model, and an error matrix of the first test feature matrix and the second test feature matrix is ​​determined to obtain a visible light test error matrix; a maximum visible light verification error matrix and a minimum visible light verification error matrix for feature extraction of a visible light photovoltaic component verification image set by the teacher network model and the visible light photovoltaic student network model are obtained; a visible light anomaly score matrix of the visible light photovoltaic component image is determined according to the visible light test error matrix, the maximum visible light verification error matrix and the minimum visible light verification error matrix, and a visible light anomaly area is marked in the visible light photovoltaic component image based on an area greater than a first preset threshold in the visible light anomaly score matrix; location information of the visible light abnormal photovoltaic component is determined and returned based on the location information of the visible light abnormal area; wherein the photovoltaic defect detection model is trained in an unlabeled supervised manner;

[0010] The teacher network model is used to extract the third test feature matrix of the thermal imaging photovoltaic component image, and the thermal imaging photovoltaic student network model in the photovoltaic defect detection model is used to extract the fourth test feature matrix of the thermal imaging photovoltaic component image, and the error matrix of the third test feature matrix and the fourth test feature matrix is ​​determined to obtain the thermal imaging test error matrix; the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix of the feature extraction of the teacher network model and the thermal imaging photovoltaic student network model for the thermal imaging photovoltaic component verification image set are obtained; according to the thermal imaging test error matrix, the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix, the thermal imaging anomaly score matrix of the thermal imaging photovoltaic component image is determined, and the thermal imaging abnormal area is marked in the thermal imaging photovoltaic component image based on the area greater than the second preset threshold in the thermal imaging anomaly score matrix; based on the location information of the thermal imaging abnormal area, the location information of the thermal imaging abnormal photovoltaic component is determined and returned.

[0011] In a possible implementation manner, the structures of the teacher network model, the visible light photovoltaic student network model, and the thermal imaging photovoltaic student network model are all as follows:

[0012] A convolutional layer is added between the second and third layers of the patch description network model, and the kernels of all convolutional kernels are odd numbers.

[0013] In a possible implementation, the teacher network model is trained according to the following steps:

[0014] Obtain preset open source data sets and trained preset open source network models;

[0015] For each image in the preset open source data set, the image is input into the preset open source network model to obtain a first feature matrix extracted by the first preset network layer of the preset open source network model and a second feature matrix extracted by the second preset network layer of the preset open source network model; the first feature matrix is ​​subjected to sliding extraction using a sliding window to obtain a third feature matrix; the second feature matrix is ​​subjected to sliding extraction using the sliding window, and the extracted feature matrix is ​​subjected to bilinear interpolation processing to obtain a fourth feature matrix; the step size during adaptive mean pooling is determined using the preset feature matrix dimension, the dimension of the third feature matrix, and the dimension of the fourth feature matrix; the kernel size during adaptive mean pooling is determined using the preset feature matrix dimension, the dimension of the third feature matrix, the dimension of the fourth feature matrix, and the step size during adaptive mean pooling; the first feature matrix and the second feature matrix are subjected to multiple adaptive mean pooling processing using the step size, the kernel size, and the preset complement size to obtain a pooling feature matrix corresponding to the image; the mean of the pooling feature matrix in the channel direction is calculated to obtain a first mean matrix corresponding to the image;

[0016] Calculate the mean of all first mean matrices to obtain a second mean matrix;

[0017] For each image in the preset open source data set, the image is input into the preset open source network model, and a pooling feature matrix corresponding to the image is determined based on the feature matrix extracted by the preset open source network model; the variance of the pooling feature matrix corresponding to the image and the second mean matrix is ​​calculated to obtain a variance matrix;

[0018] Calculate the mean of all variance matrices and perform square root processing on the obtained mean to obtain the standard deviation matrix;

[0019] For each image in the preset open source data set, the image is input into the preset open source network model and the teacher network model respectively, and the pooling feature matrix corresponding to the image is determined based on the feature matrix extracted by the preset open source network model; the mean of the pooling feature matrix corresponding to the image in the channel direction is calculated to obtain a third mean matrix, and the third mean matrix is ​​normalized using the second mean matrix and the standard deviation matrix to obtain a teacher normalized feature matrix corresponding to the image; the teacher normalized feature matrix is ​​used as the label of the teacher feature matrix obtained by extracting features from the image by the teacher network model, and the distance between the teacher normalized feature matrix and the teacher feature matrix is ​​determined to obtain the teacher network training distance;

[0020] The average of all teacher network training distances is taken as the loss value, and the weight parameters of the teacher network model are adjusted using the loss value.

[0021] In a possible implementation, the visible light photovoltaic student network model is trained according to the following steps:

[0022] Input the training images in the visible light photovoltaic module training image set into the teacher network model in a single iterative manner to obtain the first training feature matrix corresponding to each training image; calculate the mean of each first training feature matrix in the channel direction to obtain a first training mean matrix; calculate the mean of all first training mean matrices to obtain a second training mean matrix;

[0023] Input the training images in the visible light photovoltaic module training image set into the teacher network model in a single iterative manner to obtain the second training feature matrix corresponding to each training image; calculate the variance of each second training feature matrix and the second training mean matrix respectively, and calculate the mean of all the obtained variances, perform square root processing on the obtained mean, and obtain the visible light training standard deviation matrix;

[0024] Inputting the training images in the visible light photovoltaic component training image set into the teacher network model in a single iterative manner to obtain a third training feature matrix corresponding to each training image; inputting the training images in the visible light photovoltaic component training image set into the visible light photovoltaic student network model in a single iterative manner to obtain a fourth training feature matrix corresponding to each training image;

[0025] For each training image, the third training feature matrix of the training image is normalized using the second training mean matrix and the visible light training standard deviation matrix to obtain a visible light normalized feature matrix; and the first visible light training loss value corresponding to the training image is determined using the visible light normalized feature matrix and the fourth training feature matrix;

[0026] Performing center cropping on each image in the preset open source data set, and inputting the cropped image into the visible light photovoltaic student network model in a single iterative manner as a penalty image, to obtain a fifth training feature matrix corresponding to each image;

[0027] For each image in the preset open source data set, calculate the square value of the fifth training feature matrix corresponding to the image, and calculate the mean of the obtained square values ​​in the channel direction to obtain the second visible light training loss value corresponding to the image;

[0028] The weight parameters of the visible light photovoltaic student network model are adjusted using the first visible light training loss values ​​and the second visible light training loss values.

[0029] In a possible implementation manner, obtaining a maximum visible light verification error matrix and a minimum visible light verification error matrix for feature extraction of a visible light photovoltaic component verification image set by the teacher network model and the visible light photovoltaic student network model includes:

[0030] Input each verification image in the visible light photovoltaic component verification image set into the teacher network model and the visible light photovoltaic student network model in a single iterative manner, respectively obtain a first verification feature matrix of each verification image in the visible light photovoltaic component verification image set extracted by the teacher network model, respectively obtain a second verification feature matrix of each verification image in the visible light photovoltaic component verification image set extracted by the visible light photovoltaic student network model;

[0031] For each verification image, the first verification feature matrix of the verification image is normalized using the second training mean matrix and the visible light training standard deviation matrix to obtain a visible light verification normalized feature matrix corresponding to the verification image; the difference between the visible light verification normalized feature matrix and the second verification feature matrix is ​​calculated, the square of the difference is calculated, and the mean of the squared matrix in the channel direction is calculated to obtain a visible light verification error matrix;

[0032] The visible light verification error matrices corresponding to each verification image are sorted in descending order, the first n% visible light verification error matrices are intercepted, and the first one of the intercepted visible light verification error matrices is used as the maximum visible light verification error matrix, and the last one of the intercepted visible light verification error matrix is ​​used as the minimum visible light verification error matrix.

[0033] In a possible implementation, the photovoltaic segmentation model uses yolov8-seg as the image encoding part, sets at least one large core attention block between the yolov8-seg backbone network and the neck network, and replaces the segmentation recognition head in yolov8-seg with a visible light segmentation recognition head and a thermal imaging segmentation recognition head; wherein the large core attention block includes multiple convolutions of different sizes and a GELU activation function;

[0034] The connection relationship of the large core attention block from input to output layers is as follows:

[0035] 5×5 convolution; GELU activation function; three parallel convolutional layer branches; 1×1 convolution;

[0036] Among them, the first branch of the three parallel convolutional layer branches includes 1×3 convolutions and 3×1 convolutions connected in series in sequence; the second branch of the three parallel convolutional layer branches includes 1×5 convolutions and 5×1 convolutions connected in series in sequence; the third branch of the three parallel convolutional layer branches includes 1×9 convolutions and 9×1 convolutions connected in series in sequence.

[0037] In a possible implementation, the step of generating a data set for training the photovoltaic segmentation model is further included:

[0038] Collect multiple visible light sample images to obtain a visible light original image set; collect multiple thermal imaging sample images to obtain a thermal imaging original image set;

[0039] For each sample image in the visible light original image set, the photovoltaic module region is annotated to obtain a visible light annotated image set; the visible light original image set and the visible light annotated image set are combined to obtain a visible light segmented image set; the visible light segmented image set is divided according to a first preset ratio to obtain a visible light segmentation training image set and a visible light segmentation verification image set;

[0040] For each sample image in the thermal imaging original image set, the photovoltaic component area is annotated to obtain a thermal imaging annotated image set; the thermal imaging original image set and the thermal imaging annotated image set are merged to obtain a thermal imaging segmentation image set; the thermal imaging segmentation image set is divided according to a second preset ratio to obtain a thermal imaging segmentation training image set and a thermal imaging segmentation verification image set.

[0041] In a possible implementation, the step of training the photovoltaic segmentation model is also included:

[0042] The visible light segmentation training image set is subjected to random data enhancement processing, and the images in the processed image set are used to train the photovoltaic segmentation model, and the weight parameters of the backbone network and the thermal imaging segmentation recognition head are frozen during the training process, and the weight parameters of the structures of the photovoltaic segmentation model other than the backbone network and the thermal imaging segmentation recognition head are adjusted; the weight parameters that make the photovoltaic segmentation model with the highest structure detection accuracy other than the backbone network and the thermal imaging segmentation recognition head are selected using the visible light segmentation verification image set;

[0043] The thermal imaging segmentation training image set is subjected to random data enhancement processing, and the images in the processed image set are used to train the photovoltaic segmentation model. During the training process, the weight parameters of the backbone network and the visible light segmentation recognition head are frozen, and the weight parameters of the structures of the photovoltaic segmentation model other than the backbone network and the visible light segmentation recognition head are adjusted; and the thermal imaging segmentation verification image set is used to select the weight parameters that achieve the highest detection accuracy of the structures of the photovoltaic segmentation model other than the backbone network and the visible light segmentation recognition head.

[0044] In a possible implementation, the step of generating a data set for training the photovoltaic defect detection model is further included:

[0045] According to the annotations in the visible light annotated image set, copy the photovoltaic component area in the corresponding sample image to obtain an initial visible light photovoltaic component image set; select defect-free images from the initial visible light photovoltaic component image set to obtain a visible light photovoltaic component image set; divide the visible light photovoltaic component image set according to a third preset ratio to obtain a visible light photovoltaic component training image set and a visible light photovoltaic component verification image set;

[0046] According to the annotations in the thermal imaging annotated image set, the photovoltaic component area in the corresponding sample image is copied to obtain an initial thermal imaging photovoltaic component image set; defect-free images are screened from the initial thermal imaging photovoltaic component image set to obtain a thermal imaging photovoltaic component image set; the thermal imaging photovoltaic component image set is divided according to a fourth preset ratio to obtain a thermal imaging photovoltaic component training image set and a thermal imaging photovoltaic component verification image set.

[0047] According to another aspect of the present disclosure, a photovoltaic module defect detection device for a power station based on unsupervised learning is provided, comprising:

[0048] An image acquisition and processing module, used for acquiring thermal imaging images and visible light images to be detected in a target area where photovoltaic modules are arranged;

[0049] A photovoltaic component segmentation module is used to segment the thermal imaging image and the visible light image using a trained photovoltaic segmentation model to obtain a first mask image of the region where the photovoltaic components are located in the thermal imaging image and a second mask image of the region where the photovoltaic components are located in the visible light image; using the first mask image to crop a thermal imaging photovoltaic component image from the thermal imaging image; and using the second mask image to crop a visible light photovoltaic component image from the visible light image;

[0050] A photovoltaic defect detection module is used to extract a first test feature matrix of a visible light photovoltaic component image using a teacher network model in a trained photovoltaic defect detection model, extract a second test feature matrix of a visible light photovoltaic component image using a visible light photovoltaic student network model in the photovoltaic defect detection model, determine an error matrix between the first test feature matrix and the second test feature matrix, and obtain a visible light test error matrix; obtain a maximum visible light verification error matrix and a minimum visible light verification error matrix for feature extraction of a visible light photovoltaic component verification image set by the teacher network model and the visible light photovoltaic student network model; determine a visible light anomaly score matrix of the visible light photovoltaic component image according to the visible light test error matrix, the maximum visible light verification error matrix, and the minimum visible light verification error matrix, and mark a visible light anomaly region in the visible light photovoltaic component image based on a region greater than a first preset threshold in the visible light anomaly score matrix; determine and return the location information of the visible light abnormal photovoltaic component based on the location information of the visible light abnormal region; wherein the photovoltaic defect detection model is trained in an unlabeled supervised manner;

[0051] The photovoltaic defect detection module is also used to extract a third test feature matrix of the thermal imaging photovoltaic component image using the teacher network model, extract a fourth test feature matrix of the thermal imaging photovoltaic component image using the thermal imaging photovoltaic student network model in the photovoltaic defect detection model, determine the error matrix of the third test feature matrix and the fourth test feature matrix, and obtain a thermal imaging test error matrix; obtain the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix for feature extraction of the teacher network model and the thermal imaging photovoltaic student network model for the thermal imaging photovoltaic component verification image set; determine the thermal imaging anomaly score matrix of the thermal imaging photovoltaic component image according to the thermal imaging test error matrix, the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix, and mark the thermal imaging abnormal area in the thermal imaging photovoltaic component image based on the area greater than the second preset threshold in the thermal imaging anomaly score matrix; determine and return the location information of the thermal imaging abnormal photovoltaic component based on the location information of the thermal imaging abnormal area.

[0052] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0053] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0054] The disclosed method for detecting photovoltaic module defects in power plants based on unsupervised learning is different from other photovoltaic module defect detection algorithms based on deep learning. It not only uses the information of visible light images, but also uses the temperature information of thermal imaging images. This improves the detection robustness of the disclosed method, is less affected by environmental factors, and can also detect temperature-sensitive photovoltaic defects, such as hot spots, and detect temperature-insensitive photovoltaic defects, such as EVA shedding, foreign body occlusion, cracks, yellowing, bubbles, etc. Such a comprehensive defect detection method can detect photovoltaic defects more comprehensively and improve detection stability.

[0055] The present invention trains a photovoltaic defect detection model based on an unsupervised approach, and does not require manual annotation of defects on photovoltaic panels. When training the model based on an unsupervised approach, only normal visible light photovoltaic component images and normal thermal imaging photovoltaic component images are required, which greatly reduces the time required to produce the data set required for the training model and reduces the initial labor cost.

[0056] The present disclosure is based on a teacher-student architecture and uses an unsupervised strategy to train a teacher model and a student model (visible light photovoltaic student network model, thermal imaging photovoltaic student network model). The teacher model can normally extract feature information of all input images, and the student model can only normally extract feature information in normal photovoltaic component images, but cannot normally extract feature information of images other than normal photovoltaic component images. Therefore, when a defective photovoltaic component image is input into the teacher model and the student model respectively, there will be a large difference in the feature information extracted by the two. By calculating the difference, the purpose of identifying photovoltaic defects can be achieved. This method can not only effectively improve the detection accuracy, but also has a better defect recall rate than the algorithm that specifically detects custom photovoltaic defect categories, because defects are originally an open set, and artificially customized defect categories can easily miss some potential defects.

[0057] The teacher and student models designed in this paper have a simple structure and very few parameters, and can be easily deployed to edge devices. It only takes 3 milliseconds to detect an image on the NVIDIA Jetson Xavier development board, which effectively improves efficiency.

[0058] Compared with the method of detecting photovoltaic defects based on EL images with higher detection accuracy, the present invention does not need to apply voltage to the photovoltaic module, and can perform defect detection when the photovoltaic module is in normal working state, thereby not affecting the service life of the photovoltaic module. The defect detection rate of the present invention can reach 98%, and various types of photovoltaic module defects can be effectively detected.

[0059] In short, by calling the algorithm disclosed in the present invention, it is possible to identify defect anomalies on photovoltaic modules in visible light images and thermal imaging images captured by the camera, and return the location information of defective photovoltaic modules in real time, so that power station staff can maintain defective photovoltaic modules in time, ensure the safe and stable operation of the power station, and extend the service life of photovoltaic modules.

[0060] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0062] Figure 1 is a flow chart of a method for detecting defects in photovoltaic modules of a power station based on unsupervised learning according to the present disclosure;

[0063] Figure 2A It is a schematic diagram of the modified yolov8-seg structure according to the embodiment of the present disclosure;

[0064] Figure 2B is a schematic diagram of the structure of a large core attention block according to an embodiment of the present disclosure;

[0065] Figure 2C is a schematic diagram of the FPVS structure according to an embodiment of the present disclosure;

[0066] Figure 2D is a schematic diagram of the FastSAM structure according to an embodiment of the present disclosure;

[0067] Figure 2E It is a schematic diagram of the structure of yolov8-seg according to an embodiment of the present disclosure;

[0068] Figure 3A is a schematic diagram of a PDN structure according to an embodiment of the present disclosure;

[0069] Figure 3B is a schematic diagram of the New-PDN structure according to an embodiment of the present disclosure;

[0070] Figure 4 is a schematic diagram of the structure of a system for detecting defects in photovoltaic modules of a power station according to an embodiment of the present disclosure;

[0071] Figure 5 is a schematic structural diagram of a photovoltaic module defect detection device for a power station based on unsupervised learning according to the present disclosure;

[0072] Figure 6 is a schematic structural diagram of an electronic device according to the present disclosure. DETAILED DESCRIPTION

[0073] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0074] In view of the defects of low accuracy, poor robustness, susceptibility to environmental factors, low detection efficiency, etc. in the current abnormality detection of photovoltaic components, the present invention proposes a method and device for defect detection of photovoltaic components in a power station based on unsupervised learning. By calling the algorithm of the present invention, it is possible to identify defect anomalies on photovoltaic components in visible light images and thermal imaging images captured by a camera, and return the location information of defective photovoltaic components in real time, effectively solving at least one of the above defects, and enabling power station staff to promptly maintain defective photovoltaic components, thereby ensuring safe and stable operation of the power station and extending the service life of photovoltaic components.

[0075] The technical solution of the present disclosure is described below through specific embodiments.

[0076] like Figure 1 As shown, it is a flow chart of the method for detecting photovoltaic module defects in power plants based on unsupervised learning in this embodiment. The execution subject of this embodiment is a computing device or component with data processing capability. Specifically, the method of this embodiment may include the following steps:

[0077] S110 , collecting thermal imaging images and visible light images to be detected in a target area where photovoltaic modules are installed.

[0078] Specifically, the surrounding environment in the target area can be scanned by a patrol robot. The patrol robot reaches the target point based on the positioning and navigation algorithm, collects video data, and then extracts the thermal imaging image and visible light image in the video frame. The thermal imaging image and visible light image are input into the photovoltaic segmentation model for detection and identification.

[0079] S120, photovoltaic component segmentation step: use the trained photovoltaic segmentation model to segment the thermal imaging image and the visible light image to obtain a first mask image of the area where the photovoltaic components are located in the thermal imaging image and a second mask image of the area where the photovoltaic components are located in the visible light image; use the first mask image to crop the thermal imaging photovoltaic component image from the thermal imaging image; use the second mask image to crop the visible light photovoltaic component image from the visible light image.

[0080] When cropping in this step, the adjacent background outside the area where the photovoltaic modules are located (such as a background of 3 pixels) should be appropriately retained to ensure that the boundary of the area where the photovoltaic modules are located is intact.

[0081] This step uses the photovoltaic segmentation model to identify thermal imaging images and visible light images, and outputs thermal imaging photovoltaic component images and visible light photovoltaic component images.

[0082] like Figure 2A As shown, the photovoltaic segmentation model uses yolov8-seg as the image encoding part, sets at least one large core attention block between the backbone network CNN Backbone of yolov8-seg and the neck network PANet, and replaces the segmentation recognition head in yolov8-seg with a visible light segmentation recognition head and a thermal imaging segmentation recognition head; wherein the large core attention block includes multiple convolutions of different sizes and a GELU activation function. The photovoltaic segmentation model disclosed in the present invention is named FPVS (Fast Photovoltaic segmentation), and the image encoding part is named modified yolov8-seg.

[0083] Specifically, Figure 2B As shown, the connection relationship of the large core attention block from input to output layers is as follows:

[0084] 5×5 convolution; GELU activation function; three parallel convolutional layer branches; 1×1 convolution.

[0085] Among them, the first branch of the three parallel convolutional layer branches includes 1×3 convolutions and 3×1 convolutions connected in series in sequence; the second branch of the three parallel convolutional layer branches includes 1×5 convolutions and 5×1 convolutions connected in series in sequence; the third branch of the three parallel convolutional layer branches includes 1×9 convolutions and 9×1 convolutions connected in series in sequence.

[0086] In summary, the photovoltaic segmentation model in the present disclosure is as follows: Figure 2C As shown, modifiedyolov8-seg is followed by imageembedding. The model input is a thermal imaging image and a visible light image, and the output is a photovoltaic panel position mask, namely the first mask map and the second mask map.

[0087] The above yolov8-seg comes from the FastSAM (Fast Segment Anything Model) model architecture. FastSAM uses the yolov8-seg model based on the YOLACT method for image encoding (Image Encoder), which can process an image in 12ms, with faster speed and fewer parameters. The FastSAM structure is as follows Figure 2DAs shown in Figure 1, it is a model based on the SAM model architecture that can perform interactive segmentation and automatic segmentation. The input is the image to be segmented, such as a visible light image, and the output is a segmentation result mask map, such as a mask map corresponding to the visible light segmentation result. The structure of yolov8-seg is as follows: Figure 2E As shown, there is only one segmentation recognition head in this structure, and the thermal imaging segmentation recognition head and the visible light segmentation recognition head in this application are identical in structure to the segmentation recognition head in this structure.

[0088] Although FastSAM is not as good as the SAM model in the segmentation effect of small objects, its segmentation effect on non-small objects is very close to that of the SAM model. Considering the recognition speed and equipment resource consumption, FastSAM is a very good choice. The present invention improves the FastSAM model for the inspection scenario of photovoltaic components in power stations to obtain the above-mentioned FPVS (Fast Photovoltaic segmentation) model. The FPVS model uses the image encoding part (yolov8-seg) of the FastSAM model as the basic structure, and keeps the original weight parameters unchanged, inheriting its excellent feature extraction ability. Then, a large core attention block is added between the backbone network and the neck network of yolov8-seg to improve the performance of the model in paying attention to the edge contour of the object; finally, the segmentation recognition head of yolov8-seg is optimized to realize the recognition and segmentation of photovoltaic components in thermal imaging images or visible light images. The process of the original FastSAM algorithm requires inputting some location information of the photovoltaic components into the decoded information in order to segment all the photovoltaic components in the input image, which is extremely labor-intensive in industrial applications; and the original FastSAM has poor accuracy in contour segmentation between objects that are close in position. The FPVS model structure designed after the present invention improves FastSAM does not require manual input prompts in the task of photovoltaic component segmentation, and directly outputs the area where the segmented photovoltaic components are located in an end-to-end fast manner, and has good accuracy in edge contour segmentation between photovoltaic components that are close in position.

[0089] The feature information extracted by the backbone network of yolov8-seg in this disclosure is passed through the large core attention module to improve the attention to the edge contour information of the object. In addition, by performing a transposed convolution upsampling operation on the feature information of the original segmentation branch, a feature matrix with richer detail information is obtained, and the original segmentation recognition head is replaced with a visible light segmentation recognition head and a thermal imaging segmentation recognition head to achieve synchronous segmentation of visible light head portraits and thermal imaging images.

[0090] S130, visible light defect anomaly detection step: using the teacher network model in the trained photovoltaic defect detection model to extract the first test feature matrix of the visible light photovoltaic component image, using the visible light photovoltaic student network model in the photovoltaic defect detection model to extract the second test feature matrix of the visible light photovoltaic component image, determining the error matrix of the first test feature matrix and the second test feature matrix, and obtaining the visible light test error matrix; obtaining the maximum visible light verification error matrix and the minimum visible light verification error matrix for feature extraction of the teacher network model and the visible light photovoltaic student network model for the visible light photovoltaic component verification image set; determining the visible light anomaly score matrix of the visible light photovoltaic component image according to the visible light test error matrix, the maximum visible light verification error matrix and the minimum visible light verification error matrix, and marking the visible light anomaly area in the visible light photovoltaic component image based on the area greater than the first preset threshold in the visible light anomaly matrix; determining and returning the location information of the visible light abnormal photovoltaic component based on the location information of the visible light abnormal area; wherein the photovoltaic defect detection model is trained in an unlabeled supervised manner.

[0091] Among them, the structures of the teacher network model, the visible light photovoltaic student network model, and the thermal imaging photovoltaic student network model are as follows:

[0092] A convolutional layer is added between the second and third layers of the Patch Description Network (PDN) model, and the kernels of all convolutional kernels are odd numbers. Figure 3A As shown in the figure, PDN is a network model composed of a small number of convolutions (Conv), mean pooling and ReLU activation functions, and its purpose is to extract the feature information of the input image. When the PDN model inputs an image of size 256×256×3, the model extracts features and outputs a feature information matrix of 64×64×384 dimensions, which contains the feature information extracted by the model from the input image.

[0093] The teacher network model, visible light photovoltaic student network model and thermal imaging photovoltaic student network model disclosed in this disclosure are named New Patch description network (New-PDN), and their structures are as follows: Figure 3BAs shown in the figure. New-PDN uses odd-numbered convolution kernels for convolution operations. Odd-numbered convolution kernels can retain the center position information of the feature map, while even-numbered convolution kernels will ignore a point above / below / left / right when calculating the center point, and cannot fully map the input feature map to the output feature map, resulting in information loss. In addition, compared with the original PDN model, the receptive field of New-PDN is not much different. The improved model has fewer parameters, and all convolution kernels are 3×3, which can better adapt to modern deep learning frameworks and hardware acceleration libraries, and has better hardware support conditions.

[0094] The photovoltaic defect detection model includes a teacher network model, a visible light photovoltaic student network model, and a thermal imaging photovoltaic student network model.

[0095] The above-mentioned determination of the error matrix of the first test characteristic matrix and the second test characteristic matrix to obtain the visible light test error matrix can be specifically implemented by the following steps:

[0096] Calculate the difference between the first test feature matrix and the second test feature matrix; calculate the square of the difference; calculate the mean of the squared result in the channel direction to obtain the above visible light test error matrix. The visible light test error matrix is ​​recorded as error r , which is a matrix of shape [1, 1, 64, 64] in a specific implementation.

[0097] The above-mentioned acquisition of the maximum visible light verification error matrix and the minimum visible light verification error matrix for feature extraction of the teacher network model and the visible light photovoltaic student network model for the visible light photovoltaic component verification image set includes:

[0098] Step 1: Input each verification image in the visible light photovoltaic component verification image set into the teacher network model and the visible light photovoltaic student network model in a single iterative manner, and respectively obtain the first verification feature matrix of each verification image in the visible light photovoltaic component verification image set extracted by the teacher network model, and respectively obtain the second verification feature matrix of each verification image in the visible light photovoltaic component verification image set extracted by the visible light photovoltaic student network model. The visible light photovoltaic component verification image set includes multiple normal visible light photovoltaic component images.

[0099] Step 2: For each verification image, use the second training mean matrix and the visible light training standard deviation matrix to normalize the first verification feature matrix of the verification image to obtain the visible light verification normalized feature matrix corresponding to the verification image; calculate the difference between the visible light verification normalized feature matrix and the second verification feature matrix, calculate the square of the difference, and calculate the mean of the squared matrix in the channel direction to obtain the visible light verification error matrix.

[0100] Here, the normalization formula is as follows:

[0101] M v =(f t -mean t ) / std t

[0102] Where M V represents the normalized feature matrix of visible light verification, f t represents the first validation feature matrix, mean t Represents the second training mean matrix, std t Represents the visible light training standard deviation matrix.

[0103] Step 3: Sort the visible light verification error matrices corresponding to the verification images in descending order, intercept the first n% of the visible light verification error matrices, and take the first one of the intercepted visible light verification error matrices as the maximum visible light verification error matrix, and take the last one of the intercepted visible light verification error matrices as the minimum visible light verification error matrix. Here, n is a positive number, for example, the value can be 10.

[0104] The maximum visible light verification error matrix is ​​recorded as max r And the minimum visible light verification error matrix min r It represents the interval of the first 10% values ​​of the error in the output feature information of the teacher network model and the visible light photovoltaic student network model when extracting features from normal visible light photovoltaic module images.

[0105] The visible light anomaly score matrix score of the visible light photovoltaic module image is determined according to the visible light test error matrix, the maximum visible light verification error matrix and the minimum visible light verification error matrix. r The formula is as follows:

[0106] score r =0.1×(error r -min r ) / (max r -min r )

[0107] In some embodiments, the first preset threshold can be set to 0.2. rIf there is an area greater than 0.2 in the image, it means that the visible light photovoltaic component image is abnormal. According to the area greater than 0.2, the abnormal area (i.e., the visible light abnormal area) is drawn on the visible light photovoltaic component image, and then the position information of the abnormal photovoltaic component is obtained by using the position information of the inspection robot. The position information of the abnormal photovoltaic component and the abnormal visible light photovoltaic component image are returned so that subsequent maintenance personnel can locate the photovoltaic defects and perform relevant maintenance processing.

[0108] S140, thermal imaging defect anomaly detection step: using the teacher network model to extract the third test feature matrix of the thermal imaging photovoltaic component image, using the thermal imaging photovoltaic student network model in the photovoltaic defect detection model to extract the fourth test feature matrix of the thermal imaging photovoltaic component image, determining the error matrix of the third test feature matrix and the fourth test feature matrix, and obtaining the thermal imaging test error matrix; obtaining the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix for feature extraction of the teacher network model and the thermal imaging photovoltaic student network model for the thermal imaging photovoltaic component verification image set; determining the thermal imaging anomaly score matrix of the thermal imaging photovoltaic component image according to the thermal imaging test error matrix, the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix, and marking the thermal imaging anomaly area in the thermal imaging photovoltaic component image based on the area greater than the second preset threshold in the thermal imaging anomaly score matrix; determining and returning the location information of the thermal imaging abnormal photovoltaic component based on the location information of the thermal imaging abnormal area.

[0109] The above-mentioned determination of the error matrix of the third test characteristic matrix and the fourth test characteristic matrix to obtain the thermal imaging test error matrix can be specifically implemented by the following steps:

[0110] Calculate the difference between the third test feature matrix and the fourth test feature matrix; calculate the square of the difference; calculate the mean of the squared result in the channel direction to obtain the above thermal imaging test error matrix. The thermal imaging test error matrix is ​​recorded as error t , which is a matrix of shape [1, 1, 64, 64] in a specific implementation.

[0111] The above-mentioned acquisition of the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix for feature extraction of the teacher network model and the thermal imaging photovoltaic student network model for the thermal imaging photovoltaic component verification image set includes:

[0112] Step 1: Input each verification image in the thermal imaging photovoltaic component verification image set into the teacher network model and the thermal imaging photovoltaic student network model in a single iterative manner, and respectively obtain the third verification feature matrix of each verification image in the thermal imaging photovoltaic component verification image set extracted by the teacher network model, and respectively obtain the fourth verification feature matrix of each verification image in the thermal imaging photovoltaic component verification image set extracted by the thermal imaging photovoltaic student network model. The thermal imaging photovoltaic component verification image set includes multiple normal thermal imaging photovoltaic component images.

[0113] Step 2: For each verification image, the third verification feature matrix of the verification image is normalized using the fourth training mean matrix and the thermal imaging training standard deviation matrix to obtain the thermal imaging verification normalized feature matrix corresponding to the verification image; the difference between the thermal imaging verification normalized feature matrix and the fourth verification feature matrix is ​​calculated, the square of the difference is calculated, and the mean of the squared matrix in the channel direction is calculated to obtain the thermal imaging verification error matrix.

[0114] Here, the normalization formula is as follows:

[0115] M vt =(f tt -mean tt ) / std tt

[0116] Where M vt represents the normalized feature matrix of thermal imaging verification, f tt represents the third verification feature matrix, mean tt Represents the fourth training mean matrix, std tt Represents the thermal imaging training standard deviation matrix.

[0117] Step 3: Sort the thermal imaging verification error matrices corresponding to the verification images in descending order, intercept the first n% of the thermal imaging verification error matrices, and take the first one of the intercepted thermal imaging verification error matrices as the maximum thermal imaging verification error matrix, and take the last one of the intercepted thermal imaging verification error matrices as the minimum thermal imaging verification error matrix. Here, n is a positive number, for example, the value can be 10.

[0118] The maximum thermal imaging verification error matrix is ​​recorded as max t And the minimum thermal imaging verification error matrix min t It represents the interval of the first 10% values ​​of the error in the output feature information of the teacher network model and the thermal imaging photovoltaic student network model when extracting features from normal thermal imaging photovoltaic module images.

[0119] The thermal imaging abnormality score matrix score of the thermal imaging photovoltaic module image is determined according to the thermal imaging test error matrix, the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix. t The formula is as follows:

[0120] score t =0.1×(error t -min t ) / (max t -min t )

[0121] In some embodiments, the second preset threshold can be set to 0.1. t If there is an area greater than 0.1 in the image, it means that the thermal imaging photovoltaic component image is abnormal. According to the area greater than 0.1, the abnormal area (i.e., the thermal imaging abnormal area) is drawn on the thermal imaging photovoltaic component image. Then, the position information of the abnormal photovoltaic component is obtained by using the position information of the inspection robot, and the position information of the abnormal photovoltaic component and the abnormal thermal imaging photovoltaic component image are returned, so that subsequent maintenance personnel can locate the photovoltaic defects and perform relevant maintenance processing.

[0122] The present invention improves the structure of the patch description network (PDN) model, and obtains a photovoltaic defect detection model based on the Student-Teacher (ST) anomaly detection architecture and unsupervised anomaly detection strategy. Compared with the traditional supervised training model to detect custom photovoltaic defects, the present invention adopts an unsupervised training model, which does not require manual labeling of photovoltaic defects to make labels, greatly reducing the labor cost of making data sets; and, thanks to the unsupervised anomaly detection strategy constructed by the present invention, the photovoltaic defect detection model of the present invention can detect all types of photovoltaic defects, specifically, it can distinguish defects in visible light photovoltaic images and thermal imaging photovoltaic images, output defect information, and there is no need to manually define defect categories and collect related defect sample images, avoiding the problem of ignoring potential defect categories when customizing photovoltaic defect categories and the problem of low accuracy in detecting such defects due to a small number of defect samples.

[0123] The following describes the training process of each of the above models and the process of creating the datasets used for training.

[0124] The steps to generate a dataset for training the photovoltaic segmentation model are as follows:

[0125] Step 1: Collect multiple visible light sample images to obtain a visible light original image set; collect multiple thermal imaging sample images to obtain a thermal imaging original image set.

[0126] In good-light weather, the inspection robot equipped with a visible light camera and a thermal imaging camera captures and collects video data of the photovoltaic modules, extracts frames from the collected video data, and obtains a set of visible light original images and a set of thermal imaging original images.

[0127] Step 2: For each sample image in the visible light original image set, the photovoltaic component area is annotated to obtain a visible light annotated image set; the visible light original image set and the visible light annotated image set are merged to obtain a visible light segmentation image set; the visible light segmentation image set is divided according to a first preset ratio to obtain a visible light segmentation training image set and a visible light segmentation verification image set.

[0128] Since the collected sample images include not only photovoltaic modules but also other backgrounds, the open source tool labelme can be used here to annotate the photovoltaic module area.

[0129] Step three: for each sample image in the thermal imaging original image set, annotate the photovoltaic module area to obtain a thermal imaging annotated image set; merge the thermal imaging original image set and the thermal imaging annotated image set to obtain a thermal imaging segmentation image set; divide the thermal imaging segmentation image set according to a second preset ratio to obtain a thermal imaging segmentation training image set and a thermal imaging segmentation verification image set.

[0130] Since the collected sample images include not only photovoltaic modules but also other backgrounds, the open source tool labelme can be used here to annotate the photovoltaic module area.

[0131] Using the visible light segmentation training image set, visible light segmentation verification image set, thermal imaging segmentation training image set, and thermal imaging segmentation verification image set generated above, the steps for training the photovoltaic segmentation model are as follows:

[0132] The visible light segmentation training image set is subjected to random data enhancement processing, and the photovoltaic segmentation model is trained using the images in the processed image set, and the weight parameters of the backbone network and the thermal imaging segmentation recognition head are frozen during the training process, and the weight parameters of the structures of the photovoltaic segmentation model other than the backbone network and the thermal imaging segmentation recognition head are adjusted; the weight parameters that make the photovoltaic segmentation model with the highest structure detection accuracy other than the backbone network and the thermal imaging segmentation recognition head are selected using the visible light segmentation verification image set; the thermal imaging segmentation training image set is subjected to random data enhancement processing, and the photovoltaic segmentation model is trained using the images in the processed image set, and the weight parameters of the backbone network and the visible light segmentation recognition head are frozen during the training process, and the weight parameters of the structures of the photovoltaic segmentation model other than the backbone network and the visible light segmentation recognition head are adjusted; the weight parameters that make the photovoltaic segmentation model with the highest structure detection accuracy other than the backbone network and the visible light segmentation recognition head are selected using the thermal imaging segmentation verification image set.

[0133] The random data enhancement process can be random cropping, random increase of image noise, random Gaussian blur, etc., and the robustness of the model can be increased by this operation. During the training process, the number of iterations can be determined according to the actual situation, for example, 200 iterations.

[0134] There is no structural difference between the above-mentioned thermal imaging segmentation and recognition head and the visible light segmentation and recognition head. The only difference is that in terms of function, the visible light segmentation and recognition head works when the input end inputs the image of the visible light photovoltaic component; the thermal imaging segmentation and recognition head works when the input end inputs the image of the thermal imaging photovoltaic component.

[0135] The photovoltaic segmentation model FPVS model in this disclosure uses Focal Loss as the loss function. Compared with the Binary Cross Entropy loss function, the Focal Loss loss function can better distinguish difficult-to-identify samples and converge faster. Focal Loss calculation formula:

[0136] L fl =-(1-p t ) γ log(p t )

[0137]

[0138] Among them, the values ​​of y are 1 and 0, representing the foreground and background respectively; the value range of p is 0 to 1, which is the probability of predicting that it belongs to the foreground (i.e., the photovoltaic module); p t It reflects the closeness between the predicted result p and the category y, p t The larger the value, the closer p is to category y, that is, the more accurate the classification is; γ>0 is the adjustment factor,

[0139] (1-p t ) γ It can reduce the loss contribution of easy-to-classify samples, thereby increasing the loss proportion of hard-to-classify samples. For example, when p t When it tends to 1, it means that the sample is easy to distinguish. At this time (1-p t ) γ It tends to 0, indicating that the contribution to the loss is small, that is, the loss ratio of easy-to-distinguish samples is reduced. t When it approaches 0, it means that the sample is difficult to distinguish. At this time (1-p t ) γ It tends to 1, indicating that it contributes more to the loss, that is, it increases the loss ratio of difficult-to-distinguish samples. fl Represents the loss value.

[0140] In some implementations, the step of generating a data set for training the photovoltaic defect detection model includes:

[0141] Step 1: According to the annotations in the visible light annotated image set, copy the photovoltaic component area in the corresponding sample image to obtain an initial visible light photovoltaic component image set; select defect-free images from the initial visible light photovoltaic component image set to obtain a visible light photovoltaic component image set; divide the visible light photovoltaic component image set according to a third preset ratio to obtain a visible light photovoltaic component training image set and a visible light photovoltaic component verification image set.

[0142] In some embodiments, the third preset ratio may be 9:1.

[0143] Step 2: According to the annotations in the thermal imaging annotated image set, copy the photovoltaic component area in the corresponding sample image to obtain an initial thermal imaging photovoltaic component image set; screen defect-free images from the initial thermal imaging photovoltaic component image set to obtain a thermal imaging photovoltaic component image set; divide the thermal imaging photovoltaic component image set according to a fourth preset ratio to obtain a thermal imaging photovoltaic component training image set and a thermal imaging photovoltaic component verification image set.

[0144] In some embodiments, the fourth preset ratio may be 9:1.

[0145] The photovoltaic defect detection model is trained using the visible light photovoltaic component training image set, the visible light photovoltaic component verification image set, the thermal imaging photovoltaic component training image set, and the thermal imaging photovoltaic component verification image set. The relevant training steps are described below.

[0146] The steps to train the teacher network model are as follows:

[0147] Step 1: Obtain the preset open source dataset and the trained preset open source network model.

[0148] Before performing this step, you need to use the PyTorch deep learning framework to build a New-PDN model, denoted as T model, that is, to build a teacher network model.

[0149] ImageNet is selected as the preset open source dataset, and WideResNet101 is selected as the preset open source network model. WideResNet101, a preset open source network model trained on ImageNet, is used as the distillation object. Because ImageNet is a very large dataset with a wide range of scenarios, a network model with an excellent structure will have excellent feature extraction capabilities (including data outside the ImageNet dataset) after being trained on the ImageNet dataset. Here, this excellent feature extraction capability is transferred to the constructed simple model, namely the teacher network model, as much as possible through distillation.

[0150] Step 2: For each image in the preset open source data set, input the image into the preset open source network model, obtain the first feature matrix extracted by the first preset network layer of the preset open source network model and the second feature matrix extracted by the second preset network layer of the preset open source network model; use the sliding window to slide the first feature matrix to obtain a third feature matrix; use the sliding window to slide the second feature matrix, and perform bilinear interpolation processing on the extracted feature matrix to obtain a fourth feature matrix; use the preset feature matrix dimension, the dimension of the third feature matrix, and the dimension of the fourth feature matrix to determine the step size when adaptive mean pooling, use the preset feature matrix dimension, the dimension of the third feature matrix, the dimension of the fourth feature matrix and the step size when adaptive mean pooling to determine the kernel size when adaptive mean pooling, use the step size, the kernel size and the preset complement size to perform multiple adaptive mean pooling processing on the third feature matrix and the fourth feature matrix, respectively, to obtain the pooling feature matrix corresponding to the image; calculate the mean of the pooling feature matrix in the channel direction to obtain the first mean matrix corresponding to the image.

[0151] In this step, the images in the preset open source dataset ImageNet are iteratively input into the trained WideResNet101 model and feature information is extracted. Then, a sliding window is used to perform feature sliding extraction on the extracted feature information, and then adaptive mean pooling is performed.

[0152] In some embodiments, the first feature matrix may be feature information extracted by the layer2 module of the WideResNet101 model, and the second feature matrix may be feature information extracted by the layer3 module of the WideResNet101 model.

[0153] In some embodiments, a sliding window of size 3×3 may be used with stride of 1 and padding of 1 to perform sliding feature extraction on the first feature matrix and the second feature matrix respectively.

[0154] In some embodiments, the third feature matrix and the fourth feature matrix may be subjected to two adaptive mean pooling processes, and then a concatenation operation may be performed to obtain a pooled feature matrix F.

[0155] The calculation formula for the parameters required for mean pooling can be adapted:

[0156] s = floor(input s / output s );

[0157] k=input s -(output s -1)×s;

[0158] p=0

[0159] Among them, input s is the width or height of the feature matrix to be processed, that is, the width or height of the third feature matrix or the fourth feature matrix, output s is the width or height of the processed feature matrix, that is, the width or height of the preset feature matrix, floor is the floor function, s is the step size during adaptive mean pooling, k is the kernel size during adaptive mean pooling, and p is the padding size. The purpose of adaptive averaging is to compress the feature information to the desired dimensional shape, such as the shape corresponding to the number of channels of 384. After compression, the feature information matrix F, that is, the pooled feature matrix, is obtained. In some embodiments, its shape is [batch_size, 384, 64, 64]).

[0160] Step 3: Calculate the mean of all first mean matrices to obtain the second mean matrix.

[0161] The second mean matrix can be recorded as mean, which represents the mean of the feature information extracted by the WideResNet101 model on the ImageNet data.

[0162] Step 4: For each image in the preset open source data set, input the image into the preset open source network model, and determine the pooling feature matrix corresponding to the image based on the feature matrix extracted by the preset open source network model; calculate the variance of the pooling feature matrix corresponding to the image and the second mean matrix to obtain the variance matrix.

[0163] The steps for determining the pooling feature matrix of each image here are the same as those in step 2, so they will not be repeated here.

[0164] The above calculation of the variance of the pooling feature matrix corresponding to the image and the second mean matrix is ​​to calculate the distance between the pooling feature matrix corresponding to the image and the second mean matrix.

[0165] Step 5: Calculate the mean of all variance matrices and perform square root processing on the obtained mean to obtain the standard deviation matrix.

[0166] The standard deviation matrix is ​​denoted as std, which represents the standard deviation of the feature information extracted by the WideResNet101 model on the ImageNet data.

[0167] Step six, for each image in the preset open source data set, input the image into the preset open source network model and the teacher network model respectively, and determine the pooling feature matrix corresponding to the image based on the feature matrix extracted by the preset open source network model; calculate the mean of the pooling feature matrix corresponding to the image in the channel direction to obtain a third mean matrix, and use the second mean matrix and the standard deviation matrix to normalize the third mean matrix to obtain the teacher normalized feature matrix corresponding to the image; use the teacher normalized feature matrix as the label of the teacher feature matrix obtained by extracting features from the image by the teacher network model, and determine the distance between the teacher normalized feature matrix and the teacher feature matrix to obtain the teacher network training distance.

[0168] Before performing this step, the model weight parameters of the teacher network model are initialized using the He initialization method.

[0169] In this step, the images in the preset open source dataset ImageNet are iteratively input into the trained WideResNet101 model and the teacher network model respectively.

[0170] The steps of determining the pooling feature matrix corresponding to the image in this step are the same as those in step 2 and will not be repeated here.

[0171] In this step, the third mean matrix is ​​normalized using the second mean matrix and the standard deviation matrix, and the formula for obtaining the teacher normalized feature matrix corresponding to the image is as follows:

[0172] M=(m i -mean) / std

[0173] In the formula, M represents the normalized feature matrix of the teacher, m i represents the third mean matrix.

[0174] Step 7: Take the mean of all teacher network training distances as the loss value, and use the loss value to adjust the weight parameters of the teacher network model.

[0175] The purpose of this step is to make the teacher network model extract as much feature information as possible that is consistent with the pooled feature matrix distribution of the images in the preset open source dataset, that is, to distill the weight parameters of the trained WideResNet101 model into the T model. Steps 6 and 7 implement the distillation of the second mean matrix and standard deviation matrix into the teacher network model to obtain the teacher network model.

[0176] In the above, based on the New-PDN model, i.e. the teacher network model, the teacher network model is obtained by distilling the well-trained model WideResNet101 with excellent performance. The preset open source dataset ImageNet is input into the trained open source WideResNet101 model, the feature information output by the model is compressed, and it is distilled into the teacher network model to obtain the trained teacher network model.

[0177] The training steps for the visible light photovoltaic student network model are as follows:

[0178] Step 1: Input the training images in the visible light photovoltaic component training image set into the teacher network model in a single iterative manner to obtain the first training feature matrix corresponding to each training image; calculate the mean of each first training feature matrix in the channel direction to obtain the first training mean matrix; calculate the mean of all first training mean matrices to obtain the second training mean matrix.

[0179] The second training mean matrix is ​​denoted as mean t , which represents the mean of the feature information extracted by the teacher network model on the normal visible light photovoltaic module training image set.

[0180] Step 2: Input the training images in the visible light photovoltaic component training image set into the teacher network model in a single iterative manner to obtain the second training feature matrix corresponding to each training image; calculate the variance of each second training feature matrix and the second training mean matrix respectively, and calculate the mean of all the variances obtained, perform square root processing on the obtained mean, and obtain the visible light training standard deviation matrix.

[0181] The variance of the second training feature matrix and the second training mean matrix is ​​calculated, that is, the distance between the second training feature matrix and the second training mean matrix is ​​calculated.

[0182] The visible light training standard deviation matrix is ​​denoted as std t , represents the standard deviation of the feature information extracted by the teacher network model on the normal visible light photovoltaic module training image set.

[0183] Step three, input the training images in the visible light photovoltaic component training image set into the teacher network model in a single iterative manner to obtain a third training feature matrix corresponding to each training image; input the training images in the visible light photovoltaic component training image set into the visible light photovoltaic student network model in a single iterative manner to obtain a fourth training feature matrix corresponding to each training image.

[0184] Before performing this step, you need to use the PyTorch deep learning framework to build a New-PDN model, which is the visible light photovoltaic student network model.

[0185] Step 4: For each training image, use the second training mean matrix and the visible light training standard deviation matrix to normalize the third training feature matrix of the training image to obtain a visible light normalized feature matrix; use the visible light normalized feature matrix and the fourth training feature matrix to determine the first visible light training loss value corresponding to the training image.

[0186] The normalization formula is as follows:

[0187] M t =(FF t -mean t ) / std t

[0188] Where M t Represents the normalized feature matrix of visible light, FF t Represents the third training feature matrix.

[0189] The above M t As the output feature information of the visible light photovoltaic student network model, that is, the label of the fourth training feature matrix. Therefore, the above-mentioned use of the visible light normalized feature matrix and the fourth training feature matrix to determine the first visible light training loss value corresponding to the training image can be specifically: calculating the distance from the fourth training feature matrix to the visible light normalized feature matrix to obtain the distance matrix distance1, and then calculating the mean of the distance values ​​of the top 90% in the distance matrix distance1 to obtain the first visible light training loss value, recorded as loss1 hard .

[0190] Step 5: perform center cropping on each image in the preset open source data set, and input the cropped image as a penalty image into the visible light photovoltaic student network model in a single iterative manner to obtain a fifth training feature matrix corresponding to each image.

[0191] Step 6: For each image in the preset open source data set, calculate the square value of the fifth training feature matrix corresponding to the image, and calculate the mean of the obtained square values ​​in the channel direction to obtain the second visible light training loss value corresponding to the image. The mean here is the mean of a single image in the channel direction.

[0192] The second visible light training loss value is recorded as loss1 penalty , which makes the visible light photovoltaic student network model only focus on the feature extraction of normal (defect-free) photovoltaic module images, hindering it from imitating the teacher network model on images other than normal (defect-free) photovoltaic module images.

[0193] Step 7: Use each first visible light training loss value and each second visible light training loss value to adjust the weight parameters of the visible light photovoltaic student network model.

[0194] Specifically, the first visible light training loss value loss1 is used hard And each second visible light training loss value loss1 penalty Adjust the weight parameters of the visible light photovoltaic student network model.

[0195] After iterating the images in the normal visible light photovoltaic component training image set, the weight parameters are saved in the visible light photovoltaic student network model to obtain the visible light photovoltaic student network model.

[0196] The steps for training the thermal imaging photovoltaic student network model are the same as those for training the visible light photovoltaic student network model, except that the dataset used is the thermal imaging photovoltaic module training image set, and the trained thermal imaging photovoltaic student network model can only extract the features of normal (defect-free) thermal imaging photovoltaic module images. The specific training steps are as follows:

[0197] Step 1: Input the training images in the thermal imaging photovoltaic module training image set into the teacher network model in a single iterative manner to obtain the sixth training feature matrix corresponding to each training image; calculate the mean of each sixth training feature matrix in the channel direction to obtain the third training mean matrix; calculate the mean of all third training mean matrices to obtain the fourth training mean matrix.

[0198] The fourth training mean matrix is ​​denoted as mean tt , which represents the mean of the feature information extracted by the teacher network model on the normal thermal imaging photovoltaic module training image set.

[0199] Step 2: Input the training images in the thermal imaging photovoltaic module training image set into the teacher network model in a single iterative manner to obtain the seventh training feature matrix corresponding to each training image; calculate the variance of each seventh training feature matrix and the fourth training mean matrix respectively, and calculate the mean of all the variances obtained, perform square root processing on the obtained mean, and obtain the thermal imaging training standard deviation matrix.

[0200] The variance of the seventh training feature matrix and the fourth training mean matrix is ​​calculated, that is, the distance between the seventh training feature matrix and the fourth training mean matrix is ​​calculated.

[0201] The standard deviation matrix of thermal imaging training is denoted as std tt , represents the standard deviation of the feature information extracted by the teacher network model on the normal thermal imaging photovoltaic module training image set.

[0202] Step three, input the training images in the thermal imaging photovoltaic component training image set into the teacher network model in a single iterative manner to obtain the eighth training feature matrix corresponding to each training image; input the training images in the thermal imaging photovoltaic component training image set into the thermal imaging photovoltaic student network model in a single iterative manner to obtain the ninth training feature matrix corresponding to each training image.

[0203] Before performing this step, you need to use the PyTorch deep learning framework to build a New-PDN model, which is the thermal imaging photovoltaic student network model.

[0204] Step 4: For each training image, the eighth training feature matrix of the training image is normalized using the fourth training mean matrix and the thermal imaging training standard deviation matrix to obtain a thermal imaging normalized feature matrix; and the first thermal imaging training loss value corresponding to the training image is determined using the thermal imaging normalized feature matrix and the ninth training feature matrix.

[0205] The normalization formula is as follows:

[0206] MM t =(FF tt -mean tt ) / std tt

[0207] In the formula, MM t Represents the normalized feature matrix of thermal imaging, FF tt Represents the eighth training feature matrix.

[0208] The above MM t As the output feature information of the thermal imaging photovoltaic student network model, that is, the label of the ninth training feature matrix. Therefore, the above-mentioned use of the thermal imaging normalized feature matrix and the eighth training feature matrix to determine the first thermal imaging training loss value corresponding to the training image can be specifically: calculating the distance from the eighth training feature matrix to the thermal imaging normalized feature matrix to obtain the distance matrix distance2, and then calculating the mean of the distance values ​​of the top 90% in the distance matrix distance2 to obtain the first thermal imaging training loss value, recorded as loss2 hard .

[0209] Step 5: perform center cropping on each image in the preset open source data set, and input the cropped image as a penalty image into the thermal imaging photovoltaic student network model in a single iterative manner to obtain the tenth training feature matrix corresponding to each image.

[0210] Step 6: For each image in the preset open source data set, calculate the square value of the tenth training feature matrix corresponding to the image, and calculate the mean of the obtained square values ​​in the channel direction to obtain the second thermal imaging training loss value corresponding to the image. The mean here is the mean of a single image in the channel direction.

[0211] The second thermal imaging training loss value is recorded as loss2 penalty , which makes the thermal imaging photovoltaic student network model only focus on the feature extraction of normal (defect-free) photovoltaic module images, hindering it from imitating the teacher network model on images other than normal (defect-free) photovoltaic module images.

[0212] Step 7: Use each first thermal imaging training loss value and each second thermal imaging training loss value to adjust the weight parameters of the thermal imaging photovoltaic student network model.

[0213] Specifically, each first thermal imaging training loss value and each second thermal imaging training loss value are used to adjust the weight parameters of the thermal imaging photovoltaic student network model.

[0214] After iterating the images in the normal thermal imaging photovoltaic module training image set, the weight parameters are saved in the thermal imaging photovoltaic student network model to obtain the thermal imaging photovoltaic student network model.

[0215] like Figure 4 As shown, in the technical solution of the present disclosure, the inspection robot is equipped with a visible light camera and a thermal imaging camera. When working, it inspects by row along the arrangement direction of the photovoltaic components, collects videos, and separates thermal imaging images and visible light images from the collected videos; then, the thermal imaging image and the visible light image are processed using the photovoltaic segmentation model, and according to the mask image obtained by the processing, the thermal imaging photovoltaic component image is cropped from the thermal imaging image, and the visible light photovoltaic component image is cropped from the visible light image; then, the visible light photovoltaic component image and the thermal imaging photovoltaic component image are input into the photovoltaic defect detection model, and the input image is processed using the photovoltaic defect detection model to obtain the abnormal information of the photovoltaic component, where the abnormal information of the photovoltaic component at least includes the position information of the abnormal visible light photovoltaic component and / or the position information of the abnormal thermal imaging photovoltaic component. In addition, the present disclosure records and returns the position information of defective photovoltaic components in real time, so that the power station staff can timely maintain the defective photovoltaic components, ensure the safe and stable operation of the power station, and extend the service life of the photovoltaic components.

[0216] The photovoltaic defect detection model disclosed in the present invention includes a teacher network model, a visible light photovoltaic student network model, and a thermal imaging photovoltaic student network model. The structures of these three models are all New-PDN, and the mechanism of the photovoltaic segmentation model is FPVS. Before detecting the image to be detected, it is necessary to collect samples of visible light images and thermal imaging images, and to produce a visible light segmentation image set and a thermal imaging segmentation image set for training FPVS. At the same time, it is necessary to produce a visible light photovoltaic component image set and a thermal imaging photovoltaic component image set for training the photovoltaic defect detection model. In addition, the open source ImageNet needs to be used when training the photovoltaic defect detection model. Figure 4 The process in the dotted line and dotted box is step 0 of the entire algorithm process. After the model is trained, there will be no dotted process when the inspection robot performs inspection tasks.

[0217] Based on the same inventive concept, the present disclosure provides a photovoltaic module defect detection device for a power station based on unsupervised learning. The steps performed by the components of the device are the same or similar to those of the above method, so similar parts are not repeated. Figure 5 As shown, the photovoltaic module defect detection device for a power station based on unsupervised learning in this embodiment includes:

[0218] The image acquisition and processing module 510 is used to acquire thermal imaging images and visible light images to be detected in the target area where the photovoltaic components are set.

[0219] The photovoltaic component segmentation module 520 is used to use the trained photovoltaic segmentation model to segment the thermal imaging image and the visible light image to obtain a first mask image of the area where the photovoltaic components are located in the thermal imaging image and a second mask image of the area where the photovoltaic components are located in the visible light image; use the first mask image to crop the thermal imaging photovoltaic component image from the thermal imaging image; and use the second mask image to crop the visible light photovoltaic component image from the visible light image.

[0220] The photovoltaic defect detection module 530 is used to extract a first test feature matrix of a visible light photovoltaic component image using a teacher network model in a trained photovoltaic defect detection model, extract a second test feature matrix of a visible light photovoltaic component image using a visible light photovoltaic student network model in the photovoltaic defect detection model, determine an error matrix between the first test feature matrix and the second test feature matrix, and obtain a visible light test error matrix; obtain a maximum visible light verification error matrix and a minimum visible light verification error matrix for feature extraction of a visible light photovoltaic component verification image set by the teacher network model and the visible light photovoltaic student network model; determine a visible light anomaly score matrix of the visible light photovoltaic component image according to the visible light test error matrix, the maximum visible light verification error matrix, and the minimum visible light verification error matrix, and mark a visible light anomaly region in the visible light photovoltaic component image based on a region greater than a first preset threshold in the visible light anomaly score matrix; determine and return the location information of the visible light anomaly photovoltaic component based on the location information of the visible light anomaly region; wherein the photovoltaic defect detection model is trained in an unlabeled supervised manner.

[0221] The photovoltaic defect detection module 530 is also used to extract the third test feature matrix of the thermal imaging photovoltaic component image using the teacher network model, extract the fourth test feature matrix of the thermal imaging photovoltaic component image using the thermal imaging photovoltaic student network model in the photovoltaic defect detection model, determine the error matrix of the third test feature matrix and the fourth test feature matrix, and obtain the thermal imaging test error matrix; obtain the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix for feature extraction of the teacher network model and the thermal imaging photovoltaic student network model for the thermal imaging photovoltaic component verification image set; determine the thermal imaging anomaly score matrix of the thermal imaging photovoltaic component image according to the thermal imaging test error matrix, the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix, and mark the thermal imaging abnormal area in the thermal imaging photovoltaic component image based on the area greater than the second preset threshold in the thermal imaging anomaly score matrix; determine and return the location information of the thermal imaging abnormal photovoltaic component based on the location information of the thermal imaging abnormal area.

[0222] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a computer-readable storage medium.

[0223] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0224] like Figure 6 As shown, the device 600 includes a computing unit 610, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 620 or a computer program loaded from a storage unit 680 into a random access memory (RAM) 630. In the RAM 630, various programs and data required for the operation of the device 600 can also be stored. The computing unit 610, the ROM 620, and the RAM 630 are connected to each other via a bus 640. An input / output (I / O) interface 650 is also connected to the bus 640.

[0225] A number of components in the device 600 are connected to the I / O interface 650, including: an input unit 660, such as a keyboard, a mouse, etc.; an output unit 670, such as various types of displays, speakers, etc.; a storage unit 680, such as a disk, an optical disk, etc.; and a communication unit 690, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 690 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0226] The computing unit 610 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 610 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 610 performs the various methods and processes described above. For example, in some embodiments, any of the above methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 680. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 620 and / or a communication unit 690. When the computer program is loaded into RAM 630 and executed by the computing unit 610, one or more steps of any of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 610 may be configured to perform any of the methods described above in any other appropriate manner (e.g., by means of firmware).

[0227] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0228] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0229] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0230] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0231] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0232] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0233] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0234] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for detecting defects in photovoltaic modules of power plants based on unsupervised learning, characterized in that: include: Collect thermal imaging images and visible light images to be detected in the target area where the photovoltaic components are set; The thermal imaging image and the visible light image are segmented using the trained photovoltaic segmentation model to obtain a first mask image of the region where the photovoltaic components are located in the thermal imaging image and a second mask image of the region where the photovoltaic components are located in the visible light image; the thermal imaging photovoltaic component image is cropped from the thermal imaging image using the first mask image; and the visible light photovoltaic component image is cropped from the visible light image using the second mask image; A first test feature matrix of a visible light photovoltaic component image is extracted using a teacher network model in a trained photovoltaic defect detection model, and a second test feature matrix of a visible light photovoltaic component image is extracted using a visible light photovoltaic student network model in the photovoltaic defect detection model, and an error matrix of the first test feature matrix and the second test feature matrix is ​​determined to obtain a visible light test error matrix; a maximum visible light verification error matrix and a minimum visible light verification error matrix for feature extraction of a visible light photovoltaic component verification image set by the teacher network model and the visible light photovoltaic student network model are obtained; a visible light anomaly score matrix of the visible light photovoltaic component image is determined according to the visible light test error matrix, the maximum visible light verification error matrix and the minimum visible light verification error matrix, and a visible light anomaly area is marked in the visible light photovoltaic component image based on an area greater than a first preset threshold in the visible light anomaly score matrix; location information of the visible light abnormal photovoltaic component is determined and returned based on the location information of the visible light abnormal area; wherein the photovoltaic defect detection model is trained in an unlabeled supervised manner; The teacher network model is used to extract the third test feature matrix of the thermal imaging photovoltaic component image, and the thermal imaging photovoltaic student network model in the photovoltaic defect detection model is used to extract the fourth test feature matrix of the thermal imaging photovoltaic component image, and the error matrix of the third test feature matrix and the fourth test feature matrix is ​​determined to obtain a thermal imaging test error matrix; the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix for feature extraction of the teacher network model and the thermal imaging photovoltaic student network model for the thermal imaging photovoltaic component verification image set are obtained; according to the thermal imaging test error matrix, the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix, the thermal imaging anomaly score matrix of the thermal imaging photovoltaic component image is determined, and the thermal imaging anomaly area is marked in the thermal imaging photovoltaic component image based on the area greater than the second preset threshold in the thermal imaging anomaly score matrix; The position information of the photovoltaic assembly with abnormal thermal imaging is determined and returned based on the position information of the abnormal thermal imaging area.

2. The method according to claim 1, characterized in that The structures of the teacher network model, the visible light photovoltaic student network model and the thermal imaging photovoltaic student network model are as follows: A convolutional layer is added between the second and third layers of the patch description network model, and the kernels of all convolutional kernels are odd numbers.

3. The method according to claim 1, characterized in that The teacher network model is trained according to the following steps: Obtain preset open source data sets and trained preset open source network models; For each image in the preset open source data set, input the image into the preset open source network model to obtain a first feature matrix extracted by a first preset network layer of the preset open source network model and a second feature matrix extracted by a second preset network layer of the preset open source network model; Using a sliding window to perform sliding extraction on the first feature matrix to obtain a third feature matrix; The second feature matrix is ​​extracted by sliding the sliding window, and the extracted feature matrix is ​​subjected to bilinear interpolation processing to obtain a fourth feature matrix; Preset the feature matrix dimension, the dimension of the third feature matrix, and the dimension of the fourth feature matrix to determine the step size during adaptive mean pooling, use the preset feature matrix dimension, the dimension of the third feature matrix, the dimension of the fourth feature matrix, and the step size during adaptive mean pooling to determine the kernel size during adaptive mean pooling, use the step size, the kernel size, and the preset complement size to perform multiple adaptive mean pooling processes on the first feature matrix and the second feature matrix to obtain a pooling feature matrix corresponding to the image; Calculate the mean of the pooling feature matrix in the channel direction to obtain the first mean matrix corresponding to the image; Calculate the mean of all first mean matrices to obtain a second mean matrix; For each image in the preset open source data set, the image is input into the preset open source network model, and a pooling feature matrix corresponding to the image is determined based on the feature matrix extracted by the preset open source network model; Calculate the variance of the pooling feature matrix corresponding to the image and the second mean matrix to obtain a variance matrix; Calculate the mean of all variance matrices and perform square root processing on the obtained mean to obtain the standard deviation matrix; For each image in the preset open source data set, the image is input into the preset open source network model and the teacher network model respectively, and a pooling feature matrix corresponding to the image is determined based on the feature matrix extracted by the preset open source network model; Calculate the mean of the pooling feature matrix corresponding to the image in the channel direction to obtain a third mean matrix, and use the second mean matrix and the standard deviation matrix to normalize the third mean matrix to obtain a teacher normalized feature matrix corresponding to the image; The teacher normalized feature matrix is ​​used as a label of a teacher feature matrix obtained by extracting features of the image by the teacher network model, and the distance between the teacher normalized feature matrix and the teacher feature matrix is ​​determined to obtain a teacher network training distance; The mean of all teacher network training distances is taken as the loss value, and the weight parameters of the teacher network model are adjusted using the loss value.

4. The method according to claim 3, characterized in that The visible light photovoltaic student network model is trained according to the following steps: Input the training images in the visible light photovoltaic module training image set into the teacher network model in a single iterative manner to obtain the first training feature matrix corresponding to each training image; calculate the mean of each first training feature matrix in the channel direction to obtain a first training mean matrix; calculate the mean of all first training mean matrices to obtain a second training mean matrix; Inputting the training images in the visible light photovoltaic assembly training image set into the teacher network model in a single-image iterative manner to obtain a second training feature matrix corresponding to each training image; Calculating the variance of each second training feature matrix and the second training mean matrix respectively, calculating the mean of all the obtained variances, performing square root processing on the obtained mean, and obtaining a visible light training standard deviation matrix; Inputting the training images in the visible light photovoltaic component training image set into the teacher network model in a single iterative manner to obtain a third training feature matrix corresponding to each training image; inputting the training images in the visible light photovoltaic component training image set into the visible light photovoltaic student network model in a single iterative manner to obtain a fourth training feature matrix corresponding to each training image; For each training image, normalize the third training feature matrix of the training image using the second training mean matrix and the visible light training standard deviation matrix to obtain a visible light normalized feature matrix; Determine a first visible light training loss value corresponding to the training image by using the visible light normalized feature matrix and the fourth training feature matrix; Performing center cropping on each image in the preset open source data set, and inputting the cropped image into the visible light photovoltaic student network model in a single iterative manner as a penalty image, to obtain a fifth training feature matrix corresponding to each image; For each image in the preset open source data set, calculate the square value of the fifth training feature matrix corresponding to the image, and calculate the mean of the obtained square values ​​in the channel direction to obtain the second visible light training loss value corresponding to the image; The weight parameters of the visible light photovoltaic student network model are adjusted using the first visible light training loss values ​​and the second visible light training loss values.

5. The method according to claim 4, characterized in that The step of obtaining a maximum visible light verification error matrix and a minimum visible light verification error matrix for feature extraction of a visible light photovoltaic component verification image set by the teacher network model and the visible light photovoltaic student network model comprises: Input each verification image in the visible light photovoltaic component verification image set into the teacher network model and the visible light photovoltaic student network model in a single iterative manner, respectively obtain a first verification feature matrix of each verification image in the visible light photovoltaic component verification image set extracted by the teacher network model, respectively obtain a second verification feature matrix of each verification image in the visible light photovoltaic component verification image set extracted by the visible light photovoltaic student network model; For each verification image, the first verification feature matrix of the verification image is normalized using the second training mean matrix and the visible light training standard deviation matrix to obtain a visible light verification normalized feature matrix corresponding to the verification image; the difference between the visible light verification normalized feature matrix and the second verification feature matrix is ​​calculated, the square of the difference is calculated, and the mean of the squared matrix in the channel direction is calculated to obtain a visible light verification error matrix; The visible light verification error matrices corresponding to each verification image are sorted in descending order, the first n% visible light verification error matrices are intercepted, and the first one of the intercepted visible light verification error matrices is used as the maximum visible light verification error matrix, and the last one of the intercepted visible light verification error matrix is ​​used as the minimum visible light verification error matrix.

6. The method according to any one of claims 1 to 5, characterized in that: The photovoltaic segmentation model uses yolov8-seg as the image encoding part, sets at least one large core attention block between the main network and the neck network of yolov8-seg, and replaces the segmentation recognition head in yolov8-seg with a visible light segmentation recognition head and a thermal imaging segmentation recognition head; wherein the large core attention block includes multiple convolutions of different sizes and a GELU activation function; The connection relationship of the large core attention block from input to output layers is as follows: 5×5 convolution; GELU activation function; three parallel convolutional layer branches; 1×1 convolution; Among them, the first branch of the three parallel convolutional layer branches includes 1×3 convolutions and 3×1 convolutions connected in series in sequence; the second branch of the three parallel convolutional layer branches includes 1×5 convolutions and 5×1 convolutions connected in series in sequence; the third branch of the three parallel convolutional layer branches includes 1×9 convolutions and 9×1 convolutions connected in series in sequence.

7. The method according to claim 6, characterized in that The step of generating a data set for training the photovoltaic segmentation model is also included: Collect multiple visible light sample images to obtain a visible light original image set; collect multiple thermal imaging sample images to obtain a thermal imaging original image set; For each sample image in the visible light original image set, the photovoltaic module region is annotated to obtain a visible light annotated image set; the visible light original image set and the visible light annotated image set are combined to obtain a visible light segmented image set; the visible light segmented image set is divided according to a first preset ratio to obtain a visible light segmentation training image set and a visible light segmentation verification image set; For each sample image in the thermal imaging original image set, the photovoltaic component area is annotated to obtain a thermal imaging annotated image set; the thermal imaging original image set and the thermal imaging annotated image set are merged to obtain a thermal imaging segmentation image set; the thermal imaging segmentation image set is divided according to a second preset ratio to obtain a thermal imaging segmentation training image set and a thermal imaging segmentation verification image set.

8. The method according to claim 7, characterized in that Also included is the step of training the photovoltaic segmentation model: The visible light segmentation training image set is subjected to random data enhancement processing, and the images in the processed image set are used to train the photovoltaic segmentation model, and the weight parameters of the backbone network and the thermal imaging segmentation recognition head are frozen during the training process, and the weight parameters of the structures of the photovoltaic segmentation model other than the backbone network and the thermal imaging segmentation recognition head are adjusted; the weight parameters that make the photovoltaic segmentation model with the highest structure detection accuracy other than the backbone network and the thermal imaging segmentation recognition head are selected using the visible light segmentation verification image set; The thermal imaging segmentation training image set is subjected to random data enhancement processing, and the images in the processed image set are used to train the photovoltaic segmentation model. During the training process, the weight parameters of the backbone network and the visible light segmentation recognition head are frozen, and the weight parameters of the structures of the photovoltaic segmentation model other than the backbone network and the visible light segmentation recognition head are adjusted; and the thermal imaging segmentation verification image set is used to select the weight parameters that achieve the highest detection accuracy of the structures of the photovoltaic segmentation model other than the backbone network and the visible light segmentation recognition head.

9. The method according to claim 7, characterized in that: Also included is the step of generating a data set for training the photovoltaic defect detection model: According to the annotations in the visible light annotated image set, copy the photovoltaic component area in the corresponding sample image to obtain an initial visible light photovoltaic component image set; select defect-free images from the initial visible light photovoltaic component image set to obtain a visible light photovoltaic component image set; divide the visible light photovoltaic component image set according to a third preset ratio to obtain a visible light photovoltaic component training image set and a visible light photovoltaic component verification image set; According to the annotations in the thermal imaging annotated image set, the photovoltaic component area in the corresponding sample image is copied to obtain an initial thermal imaging photovoltaic component image set; defect-free images are screened from the initial thermal imaging photovoltaic component image set to obtain a thermal imaging photovoltaic component image set; the thermal imaging photovoltaic component image set is divided according to a fourth preset ratio to obtain a thermal imaging photovoltaic component training image set and a thermal imaging photovoltaic component verification image set.

10. A photovoltaic module defect detection device for a power station based on unsupervised learning, characterized in that: include: An image acquisition and processing module, used for acquiring thermal imaging images and visible light images to be detected in a target area where photovoltaic modules are arranged; A photovoltaic component segmentation module is used to segment the thermal imaging image and the visible light image using a trained photovoltaic segmentation model to obtain a first mask image of the region where the photovoltaic components are located in the thermal imaging image and a second mask image of the region where the photovoltaic components are located in the visible light image; using the first mask image to crop a thermal imaging photovoltaic component image from the thermal imaging image; and using the second mask image to crop a visible light photovoltaic component image from the visible light image; A photovoltaic defect detection module is used to extract a first test feature matrix of a visible light photovoltaic component image using a teacher network model in a trained photovoltaic defect detection model, extract a second test feature matrix of a visible light photovoltaic component image using a visible light photovoltaic student network model in the photovoltaic defect detection model, determine an error matrix between the first test feature matrix and the second test feature matrix, and obtain a visible light test error matrix; obtain a maximum visible light verification error matrix and a minimum visible light verification error matrix for feature extraction of a visible light photovoltaic component verification image set by the teacher network model and the visible light photovoltaic student network model; determine a visible light anomaly score matrix of the visible light photovoltaic component image according to the visible light test error matrix, the maximum visible light verification error matrix, and the minimum visible light verification error matrix, and mark a visible light anomaly region in the visible light photovoltaic component image based on a region greater than a first preset threshold in the visible light anomaly score matrix; determine and return the location information of the visible light abnormal photovoltaic component based on the location information of the visible light abnormal region; wherein the photovoltaic defect detection model is trained in an unlabeled supervised manner; The photovoltaic defect detection module is also used to extract a third test feature matrix of the thermal imaging photovoltaic component image using the teacher network model, extract a fourth test feature matrix of the thermal imaging photovoltaic component image using the thermal imaging photovoltaic student network model in the photovoltaic defect detection model, determine the error matrix of the third test feature matrix and the fourth test feature matrix, and obtain a thermal imaging test error matrix; obtain the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix for feature extraction of the teacher network model and the thermal imaging photovoltaic student network model for the thermal imaging photovoltaic component verification image set; determine the thermal imaging anomaly score matrix of the thermal imaging photovoltaic component image according to the thermal imaging test error matrix, the maximum thermal imaging verification error matrix and the minimum thermal imaging verification error matrix, and mark the thermal imaging anomaly area in the thermal imaging photovoltaic component image based on the area in the thermal imaging anomaly score matrix that is greater than the second preset threshold; The position information of the photovoltaic assembly with abnormal thermal imaging is determined and returned based on the position information of the abnormal thermal imaging area.

Images