Photovoltaic panel defect detection method, data processing device and robot

Abstract

The invention discloses a photovoltaic panel defect detection method, a data processing device and a robot. The method comprises the following steps: acquiring a visible light image and an infrared thermal image of a photovoltaic panel; the visible light image and the infrared thermal image are input into a trained defect recognition model, a plurality of defect types are preset in the defect recognition model, and the defect types comprise at least two types of overall defects, bird droppings, accumulated snow coverage, dust, physical damage and infrared hot spot damage; abnormal features of the visible light image and abnormal features of the infrared thermal image are extracted; and determining the defect type of each abnormal feature based on the texture, shape and color distribution gradient of each abnormal feature. By fusing the information of the visible light image and the information of the infrared thermal image, the defect identification model not only can determine the visible light defect and the infrared thermal defect on the photovoltaic panel and realize the comprehensive detection of the defects of the photovoltaic panel, but also can eliminate the limitation of a single mode, so that the detection precision and reliability of the defect detection of the photovoltaic panel are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method for detecting defects in photovoltaic panels, a data processing device, and a robot. Background Technology

[0002] Centralized photovoltaic power plants are susceptible to damage from natural environments, material aging, and external forces during long-term operation, leading to defects such as microcracks, hot spots, and fragmentation in photovoltaic panels, severely impacting power generation efficiency and system safety. Existing methods for detecting photovoltaic panel defects include automatic detection methods based on visible light cameras, which are easily affected by environmental interference such as changes in illumination, shadows, and reflections, lacking sufficient ability to identify minute defects and exhibiting poor robustness. While infrared imaging detection methods can effectively identify abnormal heating phenomena such as hot spots, their low image resolution makes high-precision classification difficult, and single-spectral information cannot comprehensively cover multiple defect types. Summary of the Invention

[0003] This application provides a photovoltaic panel defect detection method, data processing device, and robot that can comprehensively detect defects in photovoltaic panels.

[0004] In a first aspect, this application provides a method for detecting defects in photovoltaic panels, the method comprising the following steps: Acquire visible light and infrared thermal images of the photovoltaic panel; The visible light image and infrared thermal image are input into the trained defect recognition model. The defect recognition model has multiple preset defect types, including at least two of the following: overall defects, bird droppings, snow cover, dust, physical damage, and infrared hot spot damage. Extracting anomalous features from visible light images and infrared thermal images; The defect type of each abnormal feature is determined based on the gradient of its texture, shape, and color distribution.

[0005] In some feasible implementations, anomalous features are extracted from visible light images and anomalous features from infrared thermal images, including: Convert a visible light image into a visible light pixel image; Convert infrared thermal images into infrared thermal pixel images; Feature maps are obtained by progressively scanning the visible light pixel map and the infrared thermal pixel map; The feature map is divided into multiple regions, and a weight vector for each region is calculated. Compare the weight vectors of each region and select the features of the region with the largest weight vector as the outlier features.

[0006] In some feasible implementations, feature maps are obtained by progressively scanning the visible light pixel map and the infrared thermal pixel map, including: The visible light pixel map and the infrared thermal pixel map are scanned using a sampling window, and the feature values ​​of the scanned area are calculated. Offsets are generated based on feature values ​​to adjust the shape of the sampling window. When the feature value of the scanned area is less than the set value, the generated offset is 0; when the feature value of the scanned area is greater than the set value, an offset greater than 0 is generated for different feature values. When the offset is greater than 0, the sampling window moves along the path of the features in the scanned region to the next region for scanning; The feature map is formed by combining the feature values ​​calculated in each sampling window.

[0007] In some feasible implementations, the defect type of each anomalous feature is determined based on the gradient of its texture, shape, and color distribution, including: Multi-scale feature maps are formed based on the texture, shape, and color distribution gradients of each anomaly feature; Extract the feature vectors of each abnormal feature on the multi-scale feature map; The defect type to which each abnormal feature belongs is determined based on the preset defect type.

[0008] In some feasible implementations, the defect type to which the abnormal feature belongs is determined based on a preset defect type, including: The feature vector is scored based on the preset defect type; The score corresponding to each defect type is converted into a probability value, and the sum of the probability values ​​of all defect types is 1. The defect type with the highest probability value is selected as the defect type corresponding to the abnormal feature.

[0009] In some feasible implementations, the method also includes determining the location of the anomalous features, with steps including: Extract the feature vectors of each abnormal feature on the multi-scale feature map; The feature map coordinates of each abnormal feature relative to the multi-scale feature map are obtained based on the feature vectors; The feature map coordinates of each abnormal feature are converted into the coordinates of each abnormal feature on the photovoltaic panel.

[0010] In some feasible implementations, the feature vector includes the coordinates of the center point of the anomaly, the width of the anomaly, and the height of the anomaly.

[0011] In some feasible implementations, after determining the defect type of the photovoltaic panel, the method also includes: When the defect type of the abnormal feature is bird droppings, snow cover, or dust, output a cleaning command; When the defect type of the abnormal feature is an overall defect, physical damage, or infrared hot spot damage, a maintenance command is output.

[0012] In a second aspect, this application provides a data processing apparatus, including a processor and a memory, wherein the memory is used to store computer-readable instructions, and the processor is used to invoke the instructions stored in the memory to execute the method described in the first aspect.

[0013] Thirdly, this application provides a robot equipped with a camera, an infrared thermal imager, and the data processing device described in the second aspect.

[0014] After acquiring visible light and infrared thermal images of the photovoltaic panel, these images are input into a trained defect recognition model. The model extracts the color, texture, and shape of abnormal features from the visible light image and the color distribution gradient from the infrared thermal image. This information is then fused, specifically the color, texture, and shape of the abnormal features from the visible light image and the color distribution gradient from the infrared thermal image. Based on this fused information and multiple pre-defined defect types within the defect recognition model, the defect type of each abnormal feature is determined, thus identifying the type of defect present on the photovoltaic panel. By fusing information from the visible light and infrared thermal images, the defect recognition model can be cross-validated, enabling comprehensive detection of both visible light and infrared thermal defects on the photovoltaic panel. This also eliminates the limitations of single-modal detection, improving the accuracy and reliability of photovoltaic panel defect detection. Furthermore, the defect detection and recognition model that integrates information from both visible light and infrared thermal images exhibits strong robustness, is less susceptible to environmental changes, and is more adaptable to the complex and variable outdoor environment where photovoltaic panels are located. For example, when visible light is obscured by shadows, infrared thermal images can still reveal hidden internal defects; when changes in ambient temperature cause the absolute temperature of infrared thermal images to be inaccurate, the texture and shape information provided by visible light images can help the model make correct judgments. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0016] Figure 1 A flowchart of the photovoltaic panel defect detection method provided in this application; Figure 2 for Figure 1 The flowchart for implementing step S103 is shown below; Figure 3 A schematic diagram of the structure of the defect identification model provided in this application; Figure 4 for Figure 2 The implementation flowchart of step S203; Figure 5 for Figure 1 The implementation flowchart of step S104; Figure 6 A schematic diagram of the data processing apparatus provided in this application; Figure 7 Recall-precision curve for visible light defects; Figure 8 This is a recall-precision curve for infrared thermal defects.

[0017] Attached image captions: 100 - Input branch, 200 - Feature fusion module, 300 - Detection head, 401 - Processor, 402 - Communication interface, 403 - Memory, 404 - Bus, 405 - Computer-readable instructions. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application will now be described in further detail with reference to the accompanying drawings.

[0019] Please see Figure 1 This application provides a method for detecting defects in photovoltaic panels, the method comprising the following steps: S101 acquires visible light and infrared thermal images of the photovoltaic panel.

[0020] In some feasible implementations, visible light images of the photovoltaic (PV) panel are acquired using a visible light camera, while infrared thermal images are acquired using an infrared thermal imager. The visible light camera and infrared thermal imager can be mounted on the robotic arm of a robot installing the PV panels, or on a drone used to inspect the condition of the PV panels. After the visible light camera and infrared imager capture images of the PV panel at the same time and location, before inputting the visible light and infrared thermal images into the defect identification model, the sizes of the visible light and infrared thermal images are adjusted to ensure precise correspondence between the same physical point in both images; for example, both the visible light and infrared thermal images are adjusted to 640×640 pixels.

[0021] Visible light images provide information about the appearance of the photovoltaic panel, while infrared thermal images provide information about its internal condition. Defects that cannot be detected by a single mode, such as hot spots caused by internal short circuits in a photovoltaic panel that appears normal on the outside, can be easily captured through dual-mode acquisition. This fundamentally broadens the range of detectable defects and provides a complete data foundation for subsequent determination of photovoltaic panel defect types.

[0022] S102, input the visible light image and the infrared thermal image into the trained defect recognition model. The defect recognition model has multiple preset defect types, including at least two of the following: overall defects, bird droppings, snow cover, dust, physical damage, and infrared hot spot damage.

[0023] The defect identification model is a multimodal fusion deep learning network comprising two parallel input branches, a feature fusion module, and a unified detection head. One input branch processes the visible light image, and the other processes the infrared thermal image. The input branches extract shallow and deep features from the visible light and infrared thermal images. Shallow features include the location and shape of edges, corners, textures, etc., while deep features include local patterns, etc. The feature fusion module fuses and organizes all the features extracted by the input branches. For example, after fusing shallow and deep features, the feature fusion layer can determine that a linear crack may exist on the photovoltaic panel. The detection head classifies the fused features and performs bounding box regression, thus determining the defect type and location on the photovoltaic panel.

[0024] In the output layer of the defect identification model, the number of output neurons is set to the total number of preset defect categories, namely, six categories: overall defects, bird droppings, snow cover, dust, physical damage, and infrared hot spot damage. The task of the defect identification model is to determine the defect type of each abnormal feature existing on the photovoltaic panel based on these six categories. It can be understood that overall defects refer to visible light defects other than bird droppings, snow cover, dust, and physical damage. By using the defect identification model for photovoltaic panel defect detection, visible light defects and infrared thermal defects on the photovoltaic panel can be detected comprehensively and accurately, improving the accuracy of defect detection and achieving the standardization of detection standards and the automation of the process.

[0025] S103, extract abnormal features from visible light images and infrared thermal images.

[0026] For extracting anomalous features from visible light images: Convolutional layers learn the RGB value distribution of specific colors, such as the dark brown of bird droppings and the white of snow, to identify the color of anomalous features; sampling windows identify the local arrangement patterns of pixels, such as the linear coarse texture of cracks and the uniform coverage texture of dust, to identify the texture of anomalous features; and combinations of multiple convolutional layers identify the contours and geometric shapes of objects, such as the irregular clumps of bird droppings and the flakes of snow, to identify the shape of anomalous features.

[0027] For extracting anomalous features from infrared thermal images: Model learning maps specific colors in the infrared pseudo-color image, such as bright white and red, to specific high-temperature values ​​to identify the temperature values ​​corresponding to different colors in the infrared thermal image; by calculating the color differences between adjacent pixels, the color distribution gradient of the infrared thermal image is identified, thereby determining areas on the photovoltaic panel where the temperature changes drastically. For example, a hot spot not only has a high temperature itself, but its boundary with the surrounding normal area usually exhibits a large temperature gradient.

[0028] By extracting pixel data from visible light images and infrared thermal images into feature vectors that the defect recognition model can understand and calculate, the subsequent defect recognition model can accurately determine the defect type to which the abnormal features belong.

[0029] S104, determine the defect type of each abnormal feature based on the texture, shape, and color distribution gradient of each abnormal feature.

[0030] The features extracted in step S103 are combined in the fusion module. For example, the texture, shape, and color of abnormal features in the visible light image are combined with the color distribution gradient in the infrared thermal image. The defect recognition model can assign different modal weights to different types of defects. For example, for judging bird droppings, the model mainly relies on the color and shape information provided by the visible light image, with the information provided by the infrared thermal image as an auxiliary; for judging hot spots, the model mainly relies on the color distribution gradient information provided by the infrared thermal image, with the information provided by the visible light image as an auxiliary. The fused features are input into the detection head, which compares this fused feature with the standard patterns of all preset defect types learned by the model during training to determine the defect type to which each abnormal feature on the photovoltaic panel belongs.

[0031] The defect detection method provided in this application, after acquiring visible light and infrared thermal images of a photovoltaic panel, inputs them into a trained defect recognition model. The model extracts the color, texture, and shape of abnormal features from the visible light image and the color distribution gradient from the infrared thermal image, respectively. The information from the visible light and infrared thermal images is then fused, specifically the color, texture, and shape of the abnormal features from the visible light image and the color distribution gradient from the infrared thermal image. Based on the fused information and multiple pre-defined defect types within the defect recognition model, the defect type to which each abnormal feature belongs is determined, thereby identifying the defect type present on the photovoltaic panel. By fusing information from the visible light and infrared thermal images, the defect recognition model can be cross-validated, enabling comprehensive detection of both visible light and infrared thermal defects on the photovoltaic panel, while also eliminating the limitations of single-modality detection, thus improving the accuracy and reliability of photovoltaic panel defect detection. Furthermore, the defect detection and recognition model that fuses information from both visible light and infrared thermal images exhibits strong robustness, is less susceptible to environmental changes, and is more adaptable to the complex and variable outdoor environment in which photovoltaic panels are located. For example, when visible light is obscured by shadows, infrared thermal images can still reveal hidden internal defects; when changes in ambient temperature cause the absolute temperature of infrared thermal images to be inaccurate, the texture and shape information provided by visible light images can help the model make correct judgments.

[0032] For details, please see Figure 2 Step S103, which involves extracting anomalous features from the visible light image and the infrared thermal image, includes the following steps: S201, convert the visible light image into a visible light pixel image.

[0033] A visible light pixel image is a 3-dimensional tensor with a shape of [height, width, 3], where 3 represents the three color channels of RGB. Converting a visible light image into a visible light pixel image allows the defect recognition model to perform mathematical operations.

[0034] S202, convert the infrared thermal image into an infrared thermal pixel image.

[0035] An infrared thermopixel image is a 3-dimensional tensor with a shape of [height, width, 3], where 3 represents the three color channels of RGB. Converting an infrared thermal image into an infrared thermopixel image allows the defect recognition model to perform mathematical operations.

[0036] S203, progressively scan the visible light pixel image and the infrared thermal pixel image to obtain the feature map.

[0037] The defect detection model scans the visible light pixel image and the infrared thermal pixel image using sampling windows, such as convolutional kernels. The sampling window is typically 3×3 pixels or 5×5 pixels in size, and it slides across the entire pixel image from left to right and top to bottom. At each location, the sampling window performs a dot product operation with the pixel region it covers, outputting a numerical value, which is the feature value of the pixels in that region. After the sampling window has scanned the entire image, these values ​​constitute a new image, namely the feature map. Each point on the feature map represents the response intensity of the corresponding region in the original image to a specific feature, such as edges, texture, or color.

[0038] S204 divides the feature map into multiple regions and calculates the weight vector for each region.

[0039] The feature map is spatially divided into multiple regions. For example, a 20×20 feature map can be viewed as 400 independent regions, each of which is a 1×1 feature vector. The weight vector of each region represents the relevance of each region to all other regions. In some feasible implementations, the defect detection model introduces a YOLOv11 Nano Object Detection (YOLOv11n) model to the input branch using a Spatial Pyramid Pooling Fast (SPPF) module and a Cross Stage Partial with Position-Sensitive Attention (C2PSA) module. See also... Figure 3 The input branch 100 of the defect recognition model provided in this application introduces an SPPF module and a C2PSA module on the basis of the input branch of the basic YOLOv11n model. Among them, the newly added SPPF module enhances the defect recognition model's ability to represent features of different sizes by fusing features from different receptive fields through multi-scale pooling; the newly added C2PSA module, through a partial spatial attention mechanism, enables the defect recognition model to focus more on key regions in the image, thereby improving the accuracy of defect detection.

[0040] Specifically, the steps for calculating the weight vector for each region using the C2PSA module include: generating a query vector and a key vector for each region's feature vector through a learnable linear transformation. The query vector represents what information the region needs, and the key vector represents what information the region possesses. A dot product is then performed between the query vector of one region and the key vectors of all regions. The result of this dot product is the attention score; a higher attention score indicates that the region is more distinct from other regions. Finally, all attention scores for a region are transformed into a set of weights that sum to 1 using a softmax function. This set of weights represents the importance distribution of the region, which is the region's weight vector.

[0041] S205: Compare the weight vectors of each region and select the features of the region with the largest weight vector as the abnormal features.

[0042] Among all regions, the regions with the highest weight vectors mean that after the information is aggregated by the attention mechanism module, the feature vectors of these regions are greatly enhanced, and the features of these regions become the most representative features of the anomaly, i.e., the anomalous features.

[0043] By introducing an attention mechanism module into the defect recognition model, the model can actively ignore the background and focus computational resources and attention on anomalous features on the photovoltaic panel that differ from the background. This significantly improves the signal-to-noise ratio of defect detection, allowing even minute and inconspicuous anomalous features to be accurately captured. Simultaneously, the attention mechanism module enables the defect recognition model to discover the correlation between anomalous features and their surrounding anomalous features. For example, if a linear feature is highly correlated with other surrounding linear features, it can be comprehensively determined that it is a complete crack. This makes the defect recognition model less susceptible to the effects of local noise, shading, or uneven illumination, thereby improving the robustness and accuracy of defect detection.

[0044] In some feasible implementations, a variable sampling window module, such as the Asymmetric Kernel Convolution (AKConV) module, is introduced into the input branch of the defect recognition model. The variable sampling window module can define the initial shape of a sampling window of arbitrary size using a novel coordinate generation algorithm during feature extraction. Furthermore, the offset prediction network within the variable sampling window module can generate different offsets based on different feature shapes to adjust the shape of each position within the sampling window for feature extraction.

[0045] Please see Figure 4 In step S203, the defect identification model obtains a feature map by progressively scanning the visible light pixel image and the infrared thermal pixel image through a variable sampling window module, including the following steps: S301, use a sampling window to scan the visible light pixel image and the infrared thermal pixel image, and calculate the feature value of the scanned area.

[0046] First, define the initial shape of the sampling window, such as a 3×3 pixel square grid. The initial sampling window starts from the top left corner of both the visible light pixel image and the infrared thermal pixel image, moving in fixed increments, such as one pixel to the right each time, until it covers the entire image. At each location, the initial sampling window calculates a weighted sum of the nine pixels it covers, resulting in an initial feature value.

[0047] S302, an offset is generated based on the feature value to adjust the shape of the sampling window. When the feature value of the scanned area is less than the set value, the generated offset is 0; when the feature value of the scanned area is greater than the set value, an offset greater than 0 is generated for different feature values.

[0048] The setpoint is learned during training by the defect recognition model and is used to distinguish between background and feature regions. Pixels within the background region have smaller differences, resulting in smaller feature values. Pixels within the feature region have larger differences, resulting in larger feature values. When the initial shape sampling window scans a region where the feature value is lower than the setpoint, it indicates that the scanned region is the background region of the photovoltaic panel. The sampling window does not need to adjust its shape and continues scanning while maintaining its initial shape.

[0049] When the feature value of the scanned region exceeds a set value, it indicates that the sampling window has entered the feature region. The offset prediction network within the variable sampling window is activated, predicting a set of offsets based on the feature values ​​of the currently scanned region. For example, for a 3×3 pixel sampling window, the offset prediction network will predict nine (Δx, Δy) offsets. These nine offsets are then added to the nine sampling points of the original 3×3 pixel sampling window. The sampling window transforms into an irregular, adaptive shape based on the offsets. The defect recognition model then uses this deformed new sampling window to obtain pixel values ​​and performs weighted calculations to obtain a more accurate new feature value.

[0050] S303, when the offset is greater than 0, the sampling window moves along the path of the features in the scanned area to the next area for scanning.

[0051] For the background region, the generated offset is 0, the shape of the sampling window remains unchanged, and the sampling window moves and scans according to the normal step size. For the feature region, the generated offset is not 0, the shape of the sampling window changes, and after the sampling window adjusts its shape according to the offset, its next moving position is along the currently extracted feature direction, such as the direction of crack extension, until the feature of the entire crack is extracted.

[0052] S304, the feature map is formed by combining the feature values ​​calculated in each sampling window.

[0053] Regardless of whether the sampling window is fixed or deformed, the final feature value calculated at each location is recorded. All these feature values ​​together constitute a complete feature map.

[0054] Traditional square sampling windows are inefficient when processing elongated, curved objects, such as cracks and root-like hot spots, due to their tendency to include a large number of irrelevant background pixels. The variable sampling window module introduced in this application can deform itself to perfectly sample along the defect's contour. This significantly improves the detection accuracy and recall for linear and irregular defects, enabling the defect recognition model to more accurately capture the true shape and boundaries of defects. Furthermore, the defect recognition model uses sampling windows of different shapes for scanning different regions. In background areas, it can use an initial shape, while in feature regions, it can generate corresponding offsets based on different feature values ​​to adjust the shape of the sampling window. While ensuring high accuracy in defect detection, it avoids wasting computational power on the background areas of photovoltaic panels, resulting in faster model inference and making it more suitable for deployment on resource-constrained devices.

[0055] For some feasible implementation methods, please refer to Figure 3 The feature fusion module 200 of the defect recognition model adopts a Path Aggregation Network (PANet) structure, including two paths: bottom-up and top-down, each containing six convolutional layers. Through bidirectional path aggregation, it effectively fuses deep semantic information and shallow detailed features. The top-down approach transmits strong semantic information from deep layers, such as "this is a crack," to the shallow layers, allowing them to understand what they are scanning. The bottom-up approach transmits strong localization information from shallow layers, such as the edges of cracks, back up to the deeper layers, enhancing the localization accuracy of deep features. This facilitates subsequent detection heads in determining the type and location of defects.

[0056] The defect recognition model's detection head 300 employs a Non-Maximum Suppression (NMS) design, using a one-to-one detection head for end-to-end inference, meaning each target is matched with only one predicted bounding box. Furthermore, the detection head incorporates a dynamic head mechanism and depthwise separable convolution. The dynamic head mechanism allows the detection head to handle multiple tasks simultaneously, such as classification and localization, and can adaptively adjust the weights of the sampling window based on the current task, allowing the same detection head to play different roles at different times. Depthwise separable convolution refers to splitting standard convolution into depthwise convolution and pointwise convolution, significantly reducing computation and parameter count with almost no loss of accuracy.

[0057] For details, please see Figure 5 Step S104, which determines the defect type of each anomalous feature based on the texture, shape, and color distribution gradient of each anomalous feature, includes the following steps: S401 forms a multi-scale feature map based on the texture, shape, and color distribution gradients of various abnormal features.

[0058] The PANet structure integrates the color, texture, and shape of various anomalous features in visible light images with the color distribution gradients of various anomalous features in infrared thermal images. The PANet structure ultimately outputs a set of highly condensed multi-scale feature maps, which include small-sized, medium-sized, and large-sized anomalous features. Small-sized anomalous features include tiny cracks or small speckles, medium-sized features include patches of bird droppings or small shadows, and large-sized features include large areas of snow or hot spots on an entire component.

[0059] S402 extracts the feature vectors of each abnormal feature on the multi-scale feature map.

[0060] The detection head performs one-to-one predictions directly on multi-scale feature maps. When the detection head scans a region, the dynamic head mechanism kicks in. The detection head includes a classification branch and a regression branch. The classification branch determines the defect type to which the abnormal feature belongs, and the regression branch determines the location of the defect.

[0061] When the detection head uses a classification branch for classification, it adjusts the weights, focusing more on semantic features such as color and texture, and then extracts feature vectors for each anomalous feature on multi-scale feature maps. First, the detection head treats each grid point in each scale feature map as a potential object center. For example, a 40×40 feature map has 1600 potential prediction centers. For each grid point, the feature vector at its location is the feature vector of its potential prediction center. For example, if the feature map has 256 channels, then the feature vector of each grid point on the feature map is a 256-dimensional vector. Based on this 256-dimensional feature vector, the detection head can predict which defect type the center belongs to.

[0062] S403, determine the defect type to which each abnormal feature belongs based on the preset defect type.

[0063] Specifically, the detection head uses lightweight depthwise separable convolution (DSC) to quickly process the feature vectors, outputs the probability of each preset defect type through a softmax function, and takes the one with the highest probability as the final classification. The detection head determines the defect type of a single anomalous feature through the following steps: Feature vectors are scored based on preset defect types; the score for each defect type is converted into a probability value, and the sum of the probability values ​​for all defect types is 1; the defect type with the highest probability value is selected as the defect type corresponding to the abnormal feature. For example, after extracting the feature vector of an abnormal feature, this feature vector is input into a fully connected layer. The number of output neurons in the fully connected layer is equal to the total number of preset defect categories. In this application, the fully connected layer has six output neurons corresponding to six defect types: overall defects, bird droppings, snow cover, dust, physical damage, and infrared hotspot damage. Each output neuron outputs a score, which represents the probability that the abnormal feature corresponding to the current feature vector is the defect type corresponding to that output neuron. The scores output by these output neurons are converted into probability values ​​through Softmax, and the sum of the probability values ​​for all defect types is equal to 1. The defect identification model selects the defect type with the highest probability value as the defect type to which the abnormal feature belongs. For example, the probability values ​​after a feature vector is input into the connection layer and output by each output neuron are [overall defect: 2%, bird droppings: 95%, snow cover: 1%, dust: 1%, physical damage: 0.5%, infrared hot spot damage: 0.5%]. The detection head determines that the abnormal feature corresponding to this feature vector is bird droppings. It should be noted that the defect recognition model of this application can also detect normal areas of the photovoltaic panel without defects, but normal areas do not belong to the defect category, so they are not included in the preset defect types for discussion.

[0064] The defect identification model provided in this application can identify visible light defects such as microcracks, as well as infrared thermal damage such as irregular hot spots, achieving comprehensive coverage of photovoltaic panel defect detection and improving the accuracy of photovoltaic panel defect detection.

[0065] The defect detection method provided in this application also includes determining the location of abnormal features. When the detection head uses a regression branch to determine the location of the defect, the detection head adjusts the weights, paying more attention to spatial features such as edges and shapes. The specific steps include: The process involves extracting feature vectors from each anomalous feature on a multi-scale feature map; obtaining the feature map coordinates of each anomalous feature relative to the multi-scale feature map based on these feature vectors; and converting the feature map coordinates of each anomalous feature into its coordinates on the photovoltaic panel. For example, after extracting the feature vector of an anomalous feature, this feature vector is input into a regression branch. The regression branch predicts the distances from a grid center on the multi-scale feature map to the four edges (top, bottom, left, and right) of the anomalous feature, resulting in four offsets. Based on these four offsets, the bounding box coordinates of the anomalous feature can be directly calculated on the multi-scale feature map, thus obtaining the feature map coordinates of the anomalous feature.

[0066] The detection head determines the scaling factor for each feature map layer from the original visible light image and infrared thermal image based on the step size of the input branch. For example, if the original image is 640×640 pixels and a 40×40 feature map is generated, then the scaling factor = 640 / 40 = 16. This means that one pixel on the feature map corresponds to 16 pixels on the original image. Multiplying the feature map coordinates of the anomalous feature by the corresponding scaling factor yields the pixel coordinates of the anomalous feature. These pixel coordinates directly provide the coordinates of the anomalous feature on the photovoltaic panel, thus determining its location.

[0067] The defect identification model provided in this application can accurately detect and classify defects on photovoltaic panels, and determine the location of each defect, which helps robots, drones, or maintenance personnel to clean and maintain the photovoltaic panels. Simultaneously, the defect identification model outputs precise numerical coordinates that can be stored in a database and compared with historical records. This allows for tracking of a specific defect, such as whether a crack is widening or spreading; it can also assess the effectiveness of repair measures, such as whether the abnormal characteristics at the location have disappeared after repair; and it provides the most basic, precise location data for building a digital twin system for photovoltaic power plants.

[0068] After determining the defect type of the photovoltaic panel, the method further includes: when the defect type of the abnormal feature is bird droppings, snow cover, or dust, a cleaning instruction is output. The cleaning instruction includes manual cleaning, semi-automatic cleaning, or fully automated cleaning. For manual cleaning, after determining the defect type, a work order is generated containing the photovoltaic panel location, the defect type on the photovoltaic panel, the probability of each defect type, and an image of the photovoltaic panel with the defect location marked. This work order is sent to maintenance personnel to facilitate cleaning. For semi-automatic cleaning, the defect identification model sends a cleaning instruction containing the photovoltaic panel coordinates, the type of defect on the photovoltaic panel, and the defect coordinates on the photovoltaic panel to a ground robot or drone equipped with cleaning equipment. The robot or drone automatically navigates to the target photovoltaic panel, and the operator remotely monitors and intervenes if necessary to confirm the start of the cleaning procedure. For fully automated cleaning, the cleaning robot or cleaning drone deployed in the power station automatically navigates to the target photovoltaic panel after receiving the cleaning instruction and uses different cleaning modes for different defects. For example, for bird droppings, a large-scale, low-pressure atomized spray is first used to soak and soften them before cleaning with a brush; for snow cover, the powerful downdraft generated by the drone's propellers is used to blow away the unfrozen dry or wet snow before cleaning with heated cleaning agents; for snow cover, a drone carrying a water tank flies along an array to clean with fan-shaped or high-pressure atomized spray.

[0069] When the defect type of the abnormal feature is a global defect, physical damage, or infrared hot spot damage, a maintenance command is output. When the photovoltaic panel has a global defect, such as when the photovoltaic panel module has a PID effect, a work order can be generated with the location of the photovoltaic panel, the probability of the photovoltaic panel having a global defect, and an image of the photovoltaic panel marked with the location of the global defect. The work order is then sent to maintenance personnel to facilitate maintenance.

[0070] When a photovoltaic panel has physical damage, such as cracks, perforations, or frame deformation in the photovoltaic module glass, backsheet, or frame, the severity of the physical damage should be assessed first. For example, it should be determined whether the crack is on the surface of the photovoltaic glass or has penetrated into the solar cell. Then, a maintenance plan should be specified according to the severity level and sent to the maintenance personnel. For example, for non-emergency damage, such as small chips at the corners of the glass, it can be included in the planned replacement list and dealt with during the next centralized maintenance. For emergency damage, such as large-area shattering or backsheet burn-through, the panel should be taken out of service immediately and replacement should be arranged.

[0071] When a photovoltaic panel has an infrared hot spot, the hot spot is characterized by localized high temperature, which can burn through the backsheet of the module and even cause a fire. Therefore, when infrared hot spot damage is detected, a maintenance command is immediately issued to notify maintenance personnel to carry out maintenance or to control a maintenance robot to perform the operation.

[0072] By issuing cleaning and maintenance instructions, defects on photovoltaic panels can be effectively eliminated, ensuring that every defect on a photovoltaic panel can be handled in the safest, most economical, and most effective way, thereby guaranteeing the long-term health and maximum benefit of the entire photovoltaic power station.

[0073] Please see Figure 6 , Figure 6 This is a schematic diagram of a data processing device provided in this application. The data processing device includes a processor 401 and a memory 403, wherein the memory 403 is used to store computer-readable instructions 405, and the processor 401 is used to call the instructions stored in the memory 403 to execute the aforementioned defect detection method for photovoltaic panels.

[0074] The data processing device can be a server, or it can be a device or component of a photovoltaic power station, or it can be executed by a computer program, etc. For example... Figure 6 As shown, the data processing device includes a processor 401, a communication interface 402, and a memory 403. The processor 401, the communication interface 402, and the memory 403 can be interconnected via a bus 404 or in other ways.

[0075] The processor 401 includes one or more processors, such as one or more central processing units (CPUs). When the processor 401 is a CPU, the CPU can be a single-core CPU or a multi-core CPU. In this embodiment, the processor 401 is used to control the data processing device to implement… Figure 1 The example shown.

[0076] The memory 403 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used to store related instructions and data.

[0077] The communication interface 402 is used to enable communication with other devices. In this embodiment, the memory 403 stores computer-readable instructions 405, and the processor 401 is used to invoke the instructions stored in the memory 403. When the data processing device is a server, the above instructions are used to perform the following steps: (1) Acquire visible light and infrared thermal images of the photovoltaic panel; (2) Input the visible light image and infrared thermal image into the trained defect recognition model. The defect recognition model has multiple defect types preset. The multiple defect types include at least two of the following: overall defects, bird droppings, snow cover, dust, physical damage and infrared hot spot damage. (3) Extracting anomalous features from visible light images and anomalous features from infrared thermal images; (4) Determine the defect type of each abnormal feature based on the gradient of the texture, shape and color distribution of each abnormal feature.

[0078] It should be noted that the data processing device deploys a pre-trained defect identification model. Before deploying the defect identification model to the data processing device, this application first evaluated the defect identification model based on evaluation metrics. Specifically, it used three metrics for evaluation: precision, recall, and mean average precision (mAP). Average precision (AP) and mAP are calculated based on the combined precision and recall. The calculation formulas are as follows:

[0079] In the formula, P represents precision, R represents recall, AP represents average precision, TP represents the defect identification model correctly classifying real positive samples as positive examples, FT represents the defect identification model incorrectly classifying real negative samples as positive examples, and FN represents the defect identification model incorrectly classifying real positive samples as negative examples.

[0080] Please see Figure 7 The calculated recall-precision curves for the defect identification model in detecting visible light defects were obtained. The precision for detecting intact photovoltaic panels was 0.964, and the precision for detecting major defect categories was 0.862. Please refer to [link / reference]. Figure 8 The recall-precision curve of the defect identification model for infrared thermal defect detection was obtained after calculation, with the core parameter, infrared hot spot defect detection precision, being 0.922. Validation confirmed that the overall recognition accuracy of the defect identification model in this application reaches over 90%, indicating that the defect identification model has met the deployment requirements.

[0081] This application also provides a robot equipped with a camera, an infrared thermal imager, and other features. Figure 6The data processing device shown is as follows. The robot can be a ground-based mobile platform with a liftable robotic arm, on which a visible light camera and an infrared thermal imager are mounted for adjusting the shooting angle and height. This ground-based mobile platform with the liftable robotic arm can perform inspections autonomously or along a preset route, detecting the type and location of defects on the photovoltaic panels in real time and uploading the results to a cloud-based control room. Alternatively, the robot can be a multi-rotor drone, with a gimbal integrating a visible light camera and an infrared thermal imager. The drone flies automatically along a planned route, enabling rapid inspection of power plants with large areas covered by photovoltaic panels.

[0082] The robot provided in this application can collect data in real time using cameras and infrared instruments. After analyzing visible light and infrared thermal images through a defect recognition model, it identifies and classifies defects in each abnormal feature on the photovoltaic panel. Through multimodal fusion, it can comprehensively detect defects in the photovoltaic panel, thereby improving the accuracy of defect detection. Based on the defect type of the abnormal feature, it automatically generates cleaning or maintenance instructions and pushes a structured work order containing the defect type, precise coordinates, confidence level, and high-definition image to the operation and maintenance system in real time. For example, a hot spot may be detected by the robot and a maintenance instruction may be generated within minutes of its formation, nipping the problem in the bud and minimizing power generation loss.

[0083] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0084] Furthermore, the use of terms such as "first," "second," etc., in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0085] In this application, unless otherwise expressly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0086] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0087] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting defects in photovoltaic panels, characterized in that, The method includes the following steps: Acquire visible light and infrared thermal images of the photovoltaic panel; The visible light image and the infrared thermal image are input into the trained defect recognition model. The defect recognition model has multiple preset defect types, including at least two of the following: overall defects, bird droppings, snow cover, dust, physical damage, and infrared hot spot damage. Extract the abnormal features of the visible light image and the abnormal features of the infrared thermal image; The defect type of each abnormal feature is determined based on the texture, shape, and color distribution gradient of each abnormal feature.

2. The defect detection method as described in claim 1, characterized in that, The extraction of anomalous features from the visible light image and the infrared thermal image includes: Convert the visible light image into a visible light pixel image; The infrared thermal image is converted into an infrared thermal pixel image; A feature map is obtained by progressively scanning the visible light pixel map and the infrared thermal pixel map; The feature map is divided into multiple regions, and a weight vector for each region is calculated. Compare the weight vectors of each region and select the features of the region with the largest weight vector as the outlier features.

3. The defect detection method as described in claim 2, characterized in that, The stepwise scanning of the visible light pixel map and the infrared thermal pixel map to obtain the feature map includes: The visible light pixel map and the infrared thermal pixel map are scanned using a sampling window, and the feature values ​​of the scanned area are calculated. Offsets are generated based on feature values ​​to adjust the shape of the sampling window. When the feature value of the scanned area is less than the set value, the generated offset is 0; when the feature value of the scanned area is greater than the set value, an offset greater than 0 is generated for different feature values. When the offset is greater than 0, the sampling window moves along the path of the features in the scanned region to the next region for scanning; The feature map is formed by combining the feature values ​​calculated in each sampling window.

4. The defect detection method as described in claim 1, characterized in that, The step of determining the defect type of each of the aforementioned abnormal features based on the texture, shape, and color distribution gradients includes: A multi-scale feature map is formed based on the texture, shape, and color distribution gradients of each of the aforementioned abnormal features; Extract the feature vectors of each abnormal feature on the multi-scale feature map; The defect type to which each abnormal feature belongs is determined based on the preset defect type.

5. The defect detection method as described in claim 4, characterized in that, Determining the defect type of the abnormal feature based on a preset defect type includes: The feature vectors of abnormal features are scored based on the preset defect types. The score corresponding to each defect type is converted into a probability value, and the sum of the probability values ​​of all defect types is 1. The defect type with the highest probability value is selected as the defect type corresponding to the abnormal feature.

6. The defect detection method as described in claim 4, characterized in that, The method further includes determining the location of the abnormal feature, the steps of which include: Extract the feature vectors of each abnormal feature on the multi-scale feature map; Based on the feature vectors, the feature map coordinates of each abnormal feature relative to the multi-scale feature map are obtained; The feature map coordinates of each abnormal feature are converted into the coordinates of each abnormal feature on the photovoltaic panel.

7. The defect detection method as described in claim 1, characterized in that, After determining the defect type based on the abnormal characteristics, the method further includes: When the defect type of the abnormal feature is bird droppings, snow cover, or dust, a cleaning command is output.

8. The defect detection method as described in claim 7, characterized in that, After determining the defect type of the photovoltaic panel, the method further includes: When the defect type of the abnormal feature is an overall defect, physical damage, or infrared hot spot damage, a maintenance command is output.

9. A data processing apparatus, characterized in that, The method includes a processor and a memory, wherein the memory is used to store computer-readable instructions, and the processor is used to invoke the instructions stored in the memory to perform the method of any one of claims 1-8.

10. A robot, characterized in that, The robot is equipped with a camera, an infrared thermal imager, and a data processing device as described in claim 9.

Images