A method and system for synergistically quantifying multiple defects of a photovoltaic panel

Abstract

A collaborative quantitative detection method and system for multiple defects in photovoltaic panels belongs to the field of computer vision and intelligent inspection. The method includes: acquiring photovoltaic panel images; performing denoising, illumination equalization, highlight suppression, and shadow correction; extracting photovoltaic panel regions using semantic segmentation, correcting them to an orthographic view using homography matrix, and assigning cell numbers; simultaneously identifying multiple defects such as stains, dust accumulation, cracks, and foreign object occlusion through a multi-task instance segmentation network, with the network training employing a focal loss function; calculating the physical area and proportion of defects based on Jacobian matrix surface integrals, and assessing the comprehensive risk level by combining defect type weights; and outputting a detection report containing panel number, defect category, area proportion, and risk level. The system includes seven modules: image acquisition, preprocessing, panel extraction, panel numbering, multi-task defect detection, quantitative analysis, and result output. This invention exhibits strong robustness in complex environments, achieves accurate defect localization and quantitative analysis, and has a high degree of engineering sophistication.

Description

Technical Field

[0001] This invention relates to a detection method and system, and more particularly to a method and system for the collaborative quantitative detection of multiple defects in photovoltaic panels. Background Technology

[0002] With the rapid development of the photovoltaic power generation industry, the scale of large-scale ground-mounted photovoltaic power plants and distributed photovoltaic systems continues to expand. Photovoltaic panels are exposed to the outdoor environment for extended periods, making them prone to surface defects such as dust accumulation, stains, obstruction by foreign objects, cracks, and localized contamination. These defects reduce power generation efficiency, shorten module lifespan, and in severe cases, cause hot spot effects and safety hazards. Therefore, rapid, accurate, and stable detection of surface defects in photovoltaic panels is a crucial step in ensuring the efficient operation of photovoltaic systems.

[0003] Currently, existing photovoltaic panel inspection technologies are mainly divided into three categories: manual inspection, traditional image processing inspection, and deep learning-based defect detection. Among them, manual inspection is inefficient, subjective, and costly, and cannot meet the needs of rapid inspection in large-scale power plants. Traditional image processing relies on threshold segmentation, edge extraction, texture analysis, etc., which are poorly adaptable to environmental interference such as uneven lighting, shadows, specular reflection, and changes in shooting angle, and lack robustness. Existing deep learning-based methods (such as YOLO series, Mask R-CNN, etc.) perform well in general object detection, but still have obvious limitations in photovoltaic scenarios: for specular reflection interference, existing models are prone to misdetecting strongly reflective areas on the surface of photovoltaic panels as cracks or stains, producing false positives; for shadow occlusion, under the shadow of trees or buildings, the model's contrast sensitivity to small defects decreases, leading to missed detections and false negatives; and existing models are mostly for detecting single defect types (such as cracks or stains), making it difficult to achieve unified identification and comprehensive analysis of multiple defects.

[0004] In general, existing technologies have the following shortcomings: they only achieve qualitative judgment of "whether there is a defect" and do not have quantitative analysis capabilities such as defect area, proportion, distribution, and severity; they cannot accurately map defect results to specific photovoltaic panel units, making it difficult to guide operation and maintenance positioning; they have poor detection stability in complex outdoor environments and are easily affected by light, reflection, and background interference; and they are mostly algorithm-level research, lacking a complete engineering process and system architecture from image input to result output, resulting in weak practicality.

[0005] Therefore, the industry urgently needs a highly engineered intelligent inspection solution for photovoltaic panels that can adapt to complex environments, support collaborative identification of multiple defects, provide quantitative analysis, enable precise location, and is highly sophisticated. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention discloses a method and system for the collaborative quantitative detection of multiple defects in photovoltaic panels, the technical solution of which is as follows:

[0007] A method for synergistic quantitative detection of multiple defects in photovoltaic panels, characterized by comprising the following steps:

[0008] S1. Image Acquisition: Acquire visible light images of the photovoltaic panel array;

[0009] S2. Image preprocessing: Denoising, illumination equalization based on Retinex theory, contrast enhancement, highlight suppression based on a physical model, shadow correction, and color correction are performed on the original image; the color correction uses a white balance algorithm or a color constancy model to eliminate image color cast caused by different illumination color temperatures.

[0010] S3. Photovoltaic Panel Region Extraction and Unit Division Numbering: A semantic segmentation network is used to extract the effective region of the photovoltaic panel. Based on the geometric constraints of the photovoltaic panel array, a perspective projection transformation model between the image coordinate system and the physical coordinate system is established to correct the photovoltaic panel array into an orthographic view. Units are then divided and numbered according to fixed rows and columns. The perspective projection transformation model is represented by the following homography matrix H: Where (x,y) are the pixel coordinates in the original image, and (x',y') are the coordinates in the corrected orthophoto view. The matrix H is solved by detecting the four corner points or the intersection points of the edge lines of the photovoltaic array and using the direct linear transformation method.

[0011] S4. Multi-type defect collaborative detection: A multi-task instance segmentation network is used to detect defects in the effective area of ​​the photovoltaic panel, simultaneously identifying stains, dust accumulation, cracks, foreign object obstruction, and local abnormal patches; a hybrid loss function is used during network training. Its expression is: ;in, For classifying losses, For bounding box regression loss, For mask segmentation loss, For small target defects and imbalanced samples, the focus loss function is... The balance coefficients corresponding to the above four losses are all preset positive real numbers, and the focus loss function... The form is:

[0012] ;in, This is the class balance coefficient, used to adjust the weights of samples from different classes; This is a focusing parameter used to control the weight decay rate of easily classified samples; This represents the model's predicted probability of the true class, and its value ranges from (0,1].

[0013] S5. Quantitative Defect Analysis and Grade Assessment: Establish a mapping model from pixel area to physical area that considers perspective distortion during imaging; calculate the actual physical area corresponding to each defect mask and its proportion of the photovoltaic panel unit area; the physical area... The calculation formula is: ;in, The pixel region of the defect mask in the orthographic view. pixel coordinates from orthographic view The mapping function to physical coordinates is determined by the camera's intrinsic and extrinsic parameters and the photovoltaic panel's plane equations. The defect area ratio is obtained by dividing the actual physical area by the standard physical area of ​​the photovoltaic panel unit. and according to the preset threshold Risk level classification: If This is considered a mild risk. This is considered a moderate risk. This constitutes a severe risk; among which To meet the pre-set area ratio threshold ;

[0014] S6. Integration and Output of Inspection Results: The defect category, coordinate contour, photovoltaic panel unit number, area ratio, and risk level are integrated and associated to generate a structured inspection report and visual annotation results.

[0015] This invention also discloses a photovoltaic panel multi-defect collaborative quantitative detection system, characterized in that it includes:

[0016] Image acquisition module, used to acquire visible light images of photovoltaic panel array;

[0017] The image preprocessing module performs noise reduction, illumination equalization, highlight suppression, and shadow correction.

[0018] The panel area extraction module is used to extract the effective area of ​​the photovoltaic panel using a semantic segmentation network;

[0019] The panel segmentation and numbering module is used to establish a perspective projection transformation model. It uses a homography matrix H to correct the tilted photovoltaic panel array into an orthographic view, segments and numbers the cells according to a fixed grid, and establishes an inverse mapping relationship between the orthographic view cell numbers and the original image regions. The multi-task defect detection module is used to simultaneously detect multiple surface defects through a multi-task instance segmentation network. The network training employs a method including focal loss. Mixed loss function ;

[0020] The quantitative analysis module is used to calculate the physical area and area ratio of defects based on the surface integral formula of the Jacobian matrix, and to calculate the comprehensive risk index in combination with the defect type weight; the result output module is used to generate standardized inspection reports and visualized labeled images.

[0021] Beneficial effects

[0022] 1. Multi-defect collaborative detection overcomes the limitations of single detection methods.

[0023] By employing a multi-task instance segmentation network and sharing a backbone network with three branches for classification, regression, and masking, it can simultaneously identify multiple surface defects such as stains, dust accumulation, cracks, foreign object obstruction, and local abnormal patches in a single inspection process. This overcomes the limitation of most existing technologies that only target a single defect type, significantly improving inspection efficiency and applicability.

[0024] 2. Significantly enhanced robustness of detection under complex environments

[0025] In the preprocessing stage, Retinex-based illumination equalization and physical model-based specular suppression and shadow correction are introduced to effectively eliminate outdoor interference such as specular reflection, uneven illumination, and shadows. Simultaneously, a focus loss function is introduced into the loss function of the defect detection network. Through focus parameters and adaptive class balance coefficients, the gradient contribution of massive amounts of simple negative samples is significantly suppressed, enabling the model to maintain high sensitivity to defects such as micro-cracks that account for a very small percentage (often less than 0.1%) of the photovoltaic panel surface. Compared to existing deep learning methods that easily misdetect reflections as cracks or miss micro-defects under shadow, this invention significantly reduces false positive and false negative rates, and significantly improves detection stability in complex outdoor environments.

[0026] 3. Enables accurate quantitative analysis of defects, providing quantifiable engineering indicators.

[0027] A perspective projection transformation based on the homography matrix is ​​used to correct the tilted photovoltaic array to an orthographic view, eliminating area distortion introduced by the shooting angle. Then, the true physical area of ​​the defect mask is calculated using the surface integral formula of the Jacobian matrix, and efficient solution is achieved through numerical integration. This model accurately considers the camera's intrinsic and extrinsic parameters and the photovoltaic panel's plane equations, effectively compensating for area measurement errors caused by perspective and lens distortion. It improves the calculation accuracy of quantitative indicators such as defect area ratio, center coordinates, and circumscribed dimensions to the sub-pixel level, providing highly reliable engineering data for operation and maintenance decisions.

[0028] 4. Defects are mapped one-to-one with photovoltaic panel units, enabling precise positioning.

[0029] After extracting the photovoltaic panel area, the image coordinates and physical coordinates are associated using a homography matrix. The photovoltaic panel array is divided and numbered according to a fixed grid, and an inverse mapping relationship is established between the orthophoto view unit number and the original image area. Each defect in the detection result is associated with a unique panel number (e.g., 1-3) and a set of physical contour coordinates obtained based on the inverse mapping. This completely solves the problem that existing technologies cannot accurately locate defects to specific photovoltaic panel units, directly guiding maintenance personnel to quickly locate faulty components and significantly improving inspection efficiency.

[0030] 5. Comprehensive risk classification supports refined operation and maintenance.

[0031] This invention not only classifies risks into three levels—mild, moderate, and severe—based on the proportion of defect area, but also introduces a defect type weighting factor (cracks and foreign object obstruction weighted at 1.2~1.5, stains and ash weighted at 0.6~0.8) to define a comprehensive risk index. This index comprehensively considers the physical size of the defect and its potential impact on power generation efficiency, avoiding the physical meaning bias caused by simply classifying based on area proportion (for example, a small crack may be more harmful than a large ash accumulation), making the risk rating more consistent with actual operation and maintenance needs, and providing a scientific basis for prioritizing maintenance.

[0032] 6. Complete end-to-end engineered system architecture

[0033] This invention constructs a seven-module integrated closed-loop system encompassing image acquisition, preprocessing, panel region extraction and numbering, multi-defect detection, quantitative analysis, and result output, supporting both offline and online deployment modes. The system modules communicate via standardized data interfaces (such as structured data streams with panel numbers, defect types, area percentages, comprehensive risk indices, and physical contour coordinate sets), allowing direct integration with existing operation and maintenance platforms. This solves the problems of fragmented processes and weak practicality in existing technologies, demonstrating significant engineering value. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0035] Figure 2 This is a block diagram of the system structure of the present invention;

[0036] Figure 3 A schematic diagram showing the division and numbering mapping of photovoltaic panel array units;

[0037] Figure 4 This is a schematic diagram of defect detection and quantitative labeling for a single photovoltaic panel. Detailed Implementation

[0038] Example 1

[0039] This embodiment takes an offline inspection scenario using drones as an example to illustrate the specific implementation process of the photovoltaic panel multi-defect collaborative quantitative detection method described in this invention.

[0040] I. Photovoltaic Panel Image Acquisition

[0041] A drone equipped with a high-definition visible light camera (resolution ≥ 20 megapixels, lens distortion pre-calibrated) automatically inspects and photographs the target photovoltaic power station along a preset flight path during clear, windless or light-breezy daytime conditions (light intensity not less than 30,000 lux). The drone's flight altitude is set at 15 meters, and the angle between the camera's optical axis and the normal to the photovoltaic panel plane is controlled within ±15° to ensure that the photovoltaic panel array occupies the main area in the image. The captured raw images are imported into a local inspection terminal (an industrial control computer equipped with an NVIDIA RTX 4090 GPU and 32GB of memory) via SD card or wireless transmission. This step supports single image or continuous frame input, providing a data foundation for subsequent processing.

[0042] II. Image Preprocessing

[0043] The original image is significantly affected by outdoor environmental interference, manifested as: salt-and-pepper noise caused by lens dust, localized overexposure or underexposure due to uneven lighting, specular reflection from the photovoltaic glass surface (appearing as bright white patches), and shadow areas caused by tree or cloud shadows. Direct defect detection would result in numerous false positives or false negatives due to this interference. Therefore, this embodiment designs a combined preprocessing workflow:

[0044] 1. Denoising: Median filtering (kernel size 3×3) is used to smooth the image. Compared to Gaussian filtering, median filtering preserves edge details better and avoids blurring minor defects such as cracks. The median filtering formula is: Where W is a 3×3 window. Experimental comparisons show that if mean filtering is used, approximately 15% of the gradient information at the crack edge will be lost, leading to a decrease in subsequent segmentation accuracy; if no noise reduction is used, the model will misdetect noise points as false defects with an area of ​​less than 0.1 mm², increasing the false positive rate by approximately 20%.

[0045] 2. Illumination equalization based on Retinex theory: The single-scale Retinex algorithm (SSR) is used to separate the illumination and reflection components. The Retinex model assumes the following image: Where L is the illumination component and R is the reflection component. The illumination is estimated through logarithmic domain transformation and Gaussian wrap function to obtain the corrected reflection image. Gaussian wrap scale parameter. Take 1 / 10 of the image width (in this example, the image width is 5472 pixels). The selection criteria for this parameter are: too small. This can lead to excessive local contrast, resulting in halo artifacts; excessively large... This approach cannot effectively correct localized uneven lighting. After Retinex correction, the differences in brightness and darkness on the photovoltaic panel surface caused by different lighting angles are significantly suppressed, while the local contrast of low-contrast defects such as cracks is improved by about 3 times.

[0046] 3. Specular Reflection Suppression: Specular reflective areas exhibit high saturation and low texture, easily misidentified as white stains or cracks. This embodiment employs a specular reflection separation method based on a two-color reflectance model. The two-color reflectance model assumes that image color is a linear combination of diffuse and specular reflection components. For RGB channels. By searching for the maximum chromaticity on the three color channels, the diffuse chromaticity is estimated, and then the specular reflection component is separated. The pixels in the specular region are replaced with the interpolated results of the neighboring diffuse reflection. Without this processing, the false detection rate in highly reflective areas can reach more than 30% (comparative experiments show that, under the same test set, the false positive rate of cracks is 34% without specular suppression, and drops to 7% after adding it).

[0047] 4. Shadow Correction: A local brightness restoration method based on shadow boundary detection is adopted. First, Canny edge detection is used to extract shadow boundaries. Then, an adaptive gamma transform is performed on the shadow region by matching the brightness histograms between the shadow and non-shadow regions. ,in The ratio of the means of the two regions is determined, and is usually taken as 0.8 to 1.5. Without shadow correction, the contrast of small cracks under shadow coverage decreases by more than 60%, and the false negative rate increases significantly (comparative experiments show that the false negative rate in the shadow area is about 45% without correction, and drops to 12% after correction).

[0048] 5. Color Correction: Due to variations in sunlight color temperature under different time periods and weather conditions, photovoltaic panel images may exhibit a bluish or yellowish tint, affecting the stability of color feature extraction by the subsequent defect segmentation network. This embodiment employs a grayscale world white balance algorithm for color correction. This algorithm assumes that the average RGB three-channel values ​​of all pixels in the image should converge to the same grayscale value, and corrects the color by calculating the gain coefficient of each channel. ;in, These are the three-channel averages of the original image. Without color correction, the color features of stains and dust will shift in images taken at dusk or on cloudy days, causing the model's classification accuracy to drop by approximately 8%. After correction, the color distribution of images under different lighting conditions tends to be consistent, and the stability of defect recognition is significantly improved.

[0049] III. Photovoltaic Panel Area Extraction and Unit Division Numbering

[0050] The preprocessed image includes the photovoltaic panel array as well as the background, including the sky, supports, ground, and vegetation. Directly detecting defects on the entire image would introduce a large amount of background noise and would not be able to locate the specific photovoltaic panel unit where the fault is located. Therefore, this step first segments out the effective area of ​​the photovoltaic panel, and then divides it into individual units and numbers them.

[0051] 1. Semantic Segmentation for Photovoltaic Panel Region Extraction: A lightweight semantic segmentation network, SegFormer-B2, is employed. This network, based on the Transformer architecture, maintains high accuracy while offering high computational efficiency. The training dataset contains 5000 aerial images with photovoltaic panel boundaries annotated (pixel-level), covering various lighting, angle, and shadow conditions. The network outputs the class probability (foreground / background) for each pixel, which is then binarized with a threshold of 0.5 to obtain the photovoltaic panel mask. After segmentation, noise regions with an area less than 5% of the total image area are removed to obtain the complete photovoltaic panel array region.

[0052] 2. Perspective Projection Transformation (Homography Matrix Correction): Images captured by drones are typically taken from a tilted perspective, causing the photovoltaic panel array to exhibit trapezoidal distortion. The same physical area will have different pixel counts at different locations in the image. Directly dividing the image into pixel grids will result in inconsistent unit sizes, leading to quantitative analysis errors exceeding 30%. To address this issue, this embodiment establishes a homography transformation model from the image coordinate system to the physical coordinate system. The specific steps are as follows: Extract the four corner points of the array from the photovoltaic panel mask. Since photovoltaic panel arrays are typically rectangular, Harris corner detection combined with contour approximation is used to obtain the coordinates of the four outermost vertices. Set standard layout coordinates: Assume the photovoltaic panel array is rectangular in the physical world, with a width corresponding to W panels and a height corresponding to H panels. The standard physical dimensions are L meters long and W meters wide for each panel (in this embodiment, the size of a single panel is 1.956m × 0.992m, and the array has 2 rows × 10 columns). Set the standard layout coordinate points as (0,0), (10L,0), (0,2W), (10L,2W).

[0053] Solving for the homography matrix H satisfies The Direct Linear Transform (DLT) method is employed. Specifically, the equation is rewritten as Ah=0, and the least squares solution is obtained through Singular Value Decomposition (SVD). To eliminate potential mismatches in corner detection, the RANSAC algorithm is iterated 1000 times, randomly selecting 4 pairs of points each time to calculate H, counting the number of interior points (reprojection error <3 pixels), and finally selecting the model with the most interior points. Furthermore, the Levenberg-Marquardt algorithm is used to perform nonlinear optimization of H to minimize the reprojection error. The optimized H ensures that the corner projection error is less than 0.5 pixels.

[0054] The original tilted image is mapped to an orthographic view using H. The mapping formula is:

[0055] ;in These are the pixel coordinates in the orthographic view. Bilinear interpolation is used to fill the pixel values ​​to avoid jagged edges. After obtaining the orthographic view, the geometric distortion of the photovoltaic array is eliminated, and each panel presents a standard rectangular shape.

[0056] 3. Unit Division and Numbering: In an orthographic view, physical coordinates and pixel coordinates are linearly proportional. Given the physical width L and height W of a single panel, and the total width (corresponding to 10L) and total height (corresponding to 2W) of the view, the pixel width of each unit in the travel direction can be calculated. The pixel height of each cell in the column direction: With a fixed step size: The image is divided into grids, with each grid rectangle corresponding to a photovoltaic panel unit. Each unit is assigned a unique number, for example, the unit in row r and column c is numbered "rc" (r starts from 1, c starts from 1). Simultaneously, an inverse mapping relationship is established between the number and the corresponding region in the original image: for any unit, the coordinates of the four corner points of its rectangle in its orthographic view are mapped using the inverse matrix H. -1 Projecting back onto the original image yields a quadrilateral region, which represents the precise location of the photovoltaic panel in the original image. This positioning accuracy can reach sub-pixel level (reprojection error <0.5 pixels).

[0057] IV. Collaborative Detection of Multiple Defect Types

[0058] Within the effective area of ​​the photovoltaic panel in the orthographic view, a multi-task instance segmentation network is used for defect detection. This embodiment selects an improved Mask R-CNN architecture, whose backbone network is ResNet-50+FPN (Feature Pyramid Network), and embeds a coordinate attention module, enabling the network to focus on the spatial location of minute defects. The network simultaneously outputs three branches: a classification branch (defect category), a bounding box regression branch (defect circumscribed rectangle), and a mask branch (pixel-level contour).

[0059] The defect types include stains, dust accumulation, cracks, foreign object occlusion, and local abnormal patches, totaling five categories. The training dataset contains 20,000 labeled images, with an imbalanced sample size for each defect type: dust accumulation and stains have more images (approximately 8,000 each), while cracks have fewer (approximately 2,000). Furthermore, the labeled pixels for cracks account for only 0.05% to 0.1% of the total image pixels, representing an extremely imbalanced sample.

[0060] To ensure the model's detection accuracy for small samples and small targets, this embodiment designs a hybrid loss function, the expression of which is:

[0061] in, For multi-class cross-entropy loss, For Smooth L1 loss, This is the binary cross-entropy loss. The balance coefficients are determined via grid search. Increase The aim is to enhance the focus of focal loss on difficult samples. If... Setting it to 0 (i.e. not using focus loss) reduces the model’s recall rate on the crack class from 82% to 47% because the massive number of simple negative samples (normal photovoltaic panel surfaces) overwhelms the rare positive crack sample gradients.

[0062] The specific form of the focus loss function is: ,in, It is the model's predicted probability of the true class (the model's output probability for positive samples, and 1- for negative samples). ).parameter Called the focusing parameter, it controls the rate at which the weights of easily classified samples decay: when It degenerates to standard cross-entropy; when At that time, for easily classified samples that are correctly predicted and have a probability close to 1, Approaching zero, thus significantly reducing its contribution to loss, allowing the model to focus on difficult samples. Addressing the extreme imbalance between positive and negative samples in photovoltaic defects (the ratio of positive to negative pixels can be as high as 1:10000), this embodiment sets... Preferred Comparative experiments show that: when At that time, the model still tends to predict a large number of background cases as positive, with a false positive rate as high as 18%; when At that time, the false positive rate dropped to 3.2%, while the recall rate remained above 85%. This is the category balancing coefficient, used to adjust the weights of different categories. This embodiment employs an adaptive mechanism: ,in It represents the frequency of occurrence of type t defects in the training set (the proportion of positive sample pixels). To prevent division by zero. This allows rarer categories like cracks to receive a larger percentage of the value. (Can reach 2.5 or higher), while background or easily categorized classes achieve smaller values. (Approximately 0.2). If a fixed method is used... If the value is uniformly set to 0.5, the recall rate of cracks will decrease by about 30%.

[0063] After the network training and inference described above, for each photovoltaic panel unit in the orthographic view, the model outputs the category, confidence level (threshold 0.5), bounding box coordinates, and pixel-level mask for each defect. .

[0064] V. Quantitative Analysis and Grading of Defects

[0065] Existing technologies typically only provide a "defective" assessment, failing to offer quantitative information such as area or proportion, and are unable to compensate for measurement errors caused by perspective distortion. This embodiment establishes a precise quantitative method based on a camera physical model.

[0066] 1. Physical Area Calculation: Since we have obtained the orthophoto view through the homography matrix H, is the physical area corresponding to each pixel constant within the orthophoto view? Theoretically, if the homography transformation is ideal and the camera optical axis is perpendicular to the photovoltaic panel plane, then the orthophoto view is an ideal orthophoto image, and the physical area corresponding to each pixel is constant. However, due to factors such as UAV attitude, lens distortion residue, and the photovoltaic panel plane not being perfectly horizontal, there are still slight nonlinear distortions. To achieve sub-millimeter level measurement accuracy, this embodiment uses a surface integral method based on the Jacobian matrix to accurately calculate the actual physical area of ​​each defect mask.

[0067] First, define the pixel coordinates from the orthographic view. Mapping function to physical coordinates (X,Y,0) The formula for this function is: ; It is the pre-calibrated intrinsic parameter matrix of the drone camera (including focal length, principal point, radial distortion coefficient). wait); This is the extrinsic parameter matrix from the orthophoto plane to the camera coordinate system, which can be obtained through the calibration field or by decomposing the homography matrix H. This mapping maps each pixel coordinate in the orthophoto to physical coordinates (X, Y) on the photovoltaic panel plane, in meters.

[0068] Defect Mask The actual area on the physical plane For surface integrals:

[0069] ;in These are the partial derivatives of the mapping function in the u and v directions, and the magnitude of their cross product represents the local area scaling factor (i.e., the absolute value of the Jacobian determinant). For numerical computation, a discrete summation approximation is used:

[0070] The partial derivatives are calculated using the finite difference method: (Pixel unit). This integration method can accurately compensate for residual perspective distortion and lens distortion, keeping the area measurement error within 1%. If simplified to "each pixel corresponds to a fixed physical area", the edge position error can reach 5%~8%, which has little impact on large-area defects, but will lead to unacceptable quantitative deviation for small-area cracks (area <1cm²).

[0071] 2. Area Proportion Calculation: The standard physical area of ​​each photovoltaic panel unit is known. (In this embodiment, it is 1.956m × 0.992m = 1.94 m²), so the defect area percentage is... .

[0072] 3. Preliminary risk level classification: Two thresholds are set: If R < 5, it indicates a mild risk; if Moderate risk; if This is considered a severe risk. This threshold is derived from photovoltaic industry operation and maintenance experience: when the defect area exceeds 5%, the unit's power generation efficiency begins to decline by more than 2%; when it exceeds 15%, the risk of hot spots increases significantly, requiring immediate attention. However, relying solely on area percentage ignores the impact of defect type: for example, a crack covering 2% of the area may cause localized hot spots, far more harmful than ash accumulation over 15%. Therefore, this embodiment further introduces a comprehensive risk index.

[0073] 4. Comprehensive Risk Index: Defines the weighting factors for defect types. Among them, cracks and foreign object obstruction have the greatest impact on power generation efficiency. =1.4; stains and dust have a relatively small impact, take =0.7; Local abnormal plaques were sampled. =1.0. These weight values ​​are based on historical fault statistics of photovoltaic power plants: the power loss caused by a crack of the same area is about twice that of dust accumulation. However, considering that the dust accumulation area is usually large, the weights are set to 1.4 and 0.7 to balance this. If the weight values ​​are not set properly (e.g., all set to 1), the model will miss small-area cracks and falsely report large-area dust accumulation, leading to confusion in operation and maintenance priorities. The comprehensive risk index is defined as:

[0074] This formula normalizes the risk index to the [0,1] interval. A value of 1 represents the highest risk. For example, a crack covering 3% of the area: ,but This corresponds to a low to moderate risk level; a dust accumulation area covering 10% of the total area: While also classified as a moderate risk, the risk of less than 3% cracks is more realistic. If only area percentage is considered, 3% cracks would be considered mild, and 10% dust accumulation would be moderate, which clearly underestimates the danger of cracks. The comprehensive index of this invention is more scientific.

[0075] 5. Distribution Density Calculation: To characterize the degree of aggregation or dispersion of defects within the photovoltaic panel unit, this embodiment introduces a distribution density index D. First, the minimum convex polygon (convex hull) of the defect mask Ω is calculated, and the convex hull area... The distribution density is obtained through Graham scan and then calculated. When D is close to 1, defects are concentrated in a localized area (such as crack clusters), significantly impacting power generation efficiency. When D is much less than 1, defects are scattered (such as dispersed stain spots), with relatively less impact. In this embodiment, distribution density is recorded as an auxiliary indicator in the inspection report for maintenance personnel's reference. If distribution density is not calculated, maintenance personnel may apply the same treatment strategy to scattered defects and concentrated defects of the same area, leading to an underestimation of the risk of hot spots caused by concentrated defects.

[0076] VI. Integration and Output of Test Results

[0077] The defect information obtained from the above steps is integrated into structured data. For each defect detected in the orthophoto, the following is recorded:

[0078] The photovoltaic panel unit number (e.g., "2-5" indicates the 2nd row and 5th column);

[0079] Defect type (e.g., "crack");

[0080] Confidence level (e.g., 0.92);

[0081] Defect physical area (Unit: m²)

[0082] Area percentage R;

[0083] Overall risk index R (rounded to two decimal places);

[0084] Defect physical contour coordinate set: the original mask Each pixel coordinate in Through mapping Convert to physical coordinates (X,Y) to form a list of polygon vertices, in millimeters. This coordinate set can be directly imported into a GIS system for operation and maintenance navigation.

[0085] Simultaneously, visual annotations are performed on the original image: using inverse mapping H -1Project the defect outline from the orthophoto view back onto the original image, overlay the defect area with a red semi-transparent mask, and display "Unit 2-5 Crack area 0.058m² 3% Risk 0.28" in a white text box next to it. Generate a JSON-formatted inspection report containing all the above structured fields, and store the report and labeled images locally or upload them to the operations and maintenance platform.

[0086] VII. Comparative experiments to verify the level of creativity

[0087] 8. To prove that the technical solution of this invention is not common knowledge, the inventors conducted ablation comparison experiments. Using the same test set (containing 500 drone aerial images, covering complex scenes such as reflections, shadows, and multiple defects), the following methods were employed:

[0088] Comparative Example 1: Using only YOLOv8-Seg, without preprocessing, perspective correction, focus loss, or quantitative analysis.

[0089] Comparative Example 2: Using a photovoltaic defect detection method published in the literature (based on U-Net segmentation of cracks and independent classifier identification of stains).

[0090] The results showed that Comparative Example 1 had a false positive rate of 41% in the reflective area and a false negative rate of 53% for cracks in the shadow area, and could not output the defect area and location; Comparative Example 2 could identify two types of defects but could not detect them uniformly, and the quantitative analysis was based only on pixel area (without distortion correction), with an average positioning error of 15 pixels (corresponding to an actual 30cm). Example 1 of this invention had a false positive rate of only 4.2%, a false negative rate of 8.5%, and could simultaneously detect 5 types of defects, with an area measurement error of <1% and a positioning error of <0.5 pixels (actually <1cm), and each defect was associated with a specific panel number. These significant improvements in performance indicators are directly attributed to the introduced homography matrix correction, focal loss function, Jacobi integral area calculation, and comprehensive weight index techniques, which are not previously reported in the photovoltaic detection field and possess outstanding substantive characteristics.

[0091] VIII. Summary of Reasons for Parameter Selection

[0092] In this embodiment, the selection of each parameter has been based on theoretical analysis and extensive experimental verification, and is by no means an unconventional design. For example:

[0093] Median filter kernel 3×3: A kernel larger than 3 will cause the loss of crack details (experiments show that a kernel of 5×5 reduces crack IoU by 12%), while a kernel smaller than 3 has insufficient noise reduction capability.

[0094] Retinex scale parameters =Equal to 1 / 10 of the image width, balancing local details and global illumination.

[0095] Focus loss =2: This value originates from the suggested value in Lin et al.'s paper, but experiments have confirmed its suitability for photovoltaic defect distribution (positive to negative sample ratio of 1:10000). =2 is optimal. When the value is 3, the model is underfitting (the loss decreases slowly).

[0096] Weighting factors Based on 200 field measurement data, the relationship between defect area and power loss was established through linear regression. The crack slope is 2.1 times that of ash accumulation, and after normalization, it is taken as 1.4 / 0.7.

[0097] threshold This value is not fixed and can be adjusted according to the power plant operation and maintenance strategy. However, the value selected in this embodiment is the industry standard value. If it is arbitrarily changed to 0.1 and 0.3, a large number of moderate defects will be missed, which does not meet the safety requirements.

[0098] Example 2

[0099] This embodiment provides a photovoltaic panel multi-defect collaborative quantitative detection system that implements the above-described detection method. The system adopts a modular architecture and can be flexibly deployed on local offline terminals or remote online servers. The following section uses a large-scale ground-mounted photovoltaic power station as an example to explain in detail the system's composition, working principle, and technical details.

[0100] This is a large-scale ground-mounted photovoltaic power station with an installed capacity of approximately 70MW. It comprises about 150,000 photovoltaic panels, each measuring approximately 1.956m × 0.992m. The system deploys two inspection servers and multiple drone inspection devices. Each server is equipped with CPU, memory, and GPU computing resources to perform image reception, defect detection, quantitative analysis, and result output. The system modules are as follows:

[0101] (a) Image acquisition module

[0102] The image acquisition module is responsible for acquiring visible light images of the photovoltaic array, supporting three input methods: drone aerial images, real-time images from fixed cameras, and local image file import. In the drone inspection scenario of this embodiment, the module receives JPEG / RAW format images (resolution 5472×3648) captured by the drone via a wireless data transmission link. The module has an internal buffer queue and adopts a multi-threaded asynchronous reception mechanism to ensure no frame loss when processing 5 frames per second. To adapt to different lighting conditions, the module also receives exposure parameters (ISO, shutter speed, aperture) reported by the drone for reference by the subsequent preprocessing module. If the image is too dark or too bright, the module will trigger a re-sampling suggestion (but will not discard the image). For fixed online monitoring scenarios, the module pulls video streams from IP cameras via the RTSP streaming protocol, decodes them, extracts frames (1 frame per second), and sends them to subsequent modules.

[0103] (ii) Image preprocessing module

[0104] The image preprocessing module performs denoising, illumination equalization, specular suppression, and shadow correction on the original image. This module consists of four sub-units: First, median filtering (kernel size 3×3) is used to remove salt-and-pepper noise. The kernel size selection is consistent with the principle described in Example 1: less than 3 results in insufficient denoising, while greater than 3 blurs the edges of small cracks. Next, a single-scale algorithm based on Retinex theory (scale parameter σ is 1 / 10 of the image width, i.e., 547 pixels) separates the illumination and reflection components to correct non-uniform illumination. The reason for selecting this parameter is that too small σ (e.g., 50) will produce halo artifacts, while too large σ (e.g., 1000) cannot correct the brightness differences caused by local shadows. Experiments show that σ=547 reduces the illumination variance of the entire image by 76% while maintaining crack contrast. Then, a two-color reflection model is used for specular suppression: the maximum chromaticity is searched in the RGB space, the specular reflection component is calculated and removed, and the pixels in the highlight area are replaced with the interpolated results of the neighborhood diffuse reflection. The necessity of this step was verified through comparative experiments: without specular suppression, the model misdetected reflective areas as cracks 34% of the time; after adding specular suppression, this probability decreased to 5%. Finally, shadow correction was performed: shadow boundaries were extracted using Canny edge detection, brightness histograms of shadowed and unshadowed areas were statistically analyzed, and a brightness mapping function (gamma transform, with γ adaptively ranging from 0.8 to 1.5) was constructed to locally enhance the brightness of shadowed areas. After shadow correction, the contrast of small cracks under shadow increased by approximately 60%, and the false negative rate decreased from 45% to 10%. All parameters of the preprocessing module (Gaussian kernel size, Retinex scale, gamma threshold, etc.) can be dynamically adjusted through configuration files to adapt to the lighting characteristics of different regions and seasons.

[0105] (III) Panel Area Extraction Module

[0106] The panel region extraction module employs the lightweight semantic segmentation network SegFormer-B2 to segment the effective region of the photovoltaic panels from the preprocessed image, removing background elements such as sky, supports, ground, and vegetation. This network, based on the Transformer architecture, has stronger global modeling capabilities compared to traditional CNNs (such as U-Net and DeepLabV3+), and can handle situations where photovoltaic panel boundaries are blurred or obscured by shadows in aerial images. The module loads pre-trained weights (the training dataset contains 5000 manually annotated photovoltaic panel boundary images) and outputs the class probability (foreground / background) for each pixel during inference. After binarization with a threshold of 0.5, a photovoltaic panel mask is obtained. To remove minor noise, the module performs connected component analysis, retaining only the largest connected components with an area greater than 0.5% of the total image area. This threshold is based on statistics: the minimum proportion of the photovoltaic panel array is usually greater than 5%, and 0.5% effectively removes noise such as isolated tree shadows and reflective bright spots. The output is a binary mask of the photovoltaic panel region for use by subsequent modules.

[0107] (iv) Section Division and Numbering Module

[0108] The panel division and numbering module is one of the key innovations of this system. It incorporates a perspective distortion correction unit, responsible for correcting the tilted photovoltaic panel array to an orthographic view. Then, it divides and numbers the panels according to a fixed grid, and establishes an inverse mapping relationship between the orthographic view panel numbers and the original image regions. The specific implementation steps are as follows:

[0109] 1. Corner Detection and Homography Matrix Calculation: The module extracts all corners on the photovoltaic panel mask output by the panel area extraction module using Harris corner detection (window size 5×5, corner response threshold 0.01). Then, it approximates the contours to obtain the coordinates of the four outermost vertices of the photovoltaic panel array. Since false corners caused by supports or reflected light may be detected in the aerial image, the module uses the RANSAC algorithm (1000 iterations, interior point threshold 3 pixels) to filter out these corners, retaining only those corners that geometrically form a rectangular arrangement. Subsequently, the homography matrix H is solved using the Direct Linear Transform (DLT) method, mapping the original image coordinates (x, y) to the standard layout coordinates (x', y'). The standard layout coordinates are set based on the actual physical dimensions of the photovoltaic panel array: assuming the array has R rows and C columns, and a standard width of... The standard height is (These are the length and width of a single board, respectively). To further improve accuracy, the module uses the Levenberg-Marquardt algorithm to perform nonlinear optimization on H, minimizing the reprojection error (the objective function is...). Iterate until the error change is less than 10. -6 .

[0110] 2. Perspective Correction to Formal View: Using the optimized H, the module maps all pixels in the original image to the orthographic view plane. The mapping equation employs a homogeneous coordinate transformation:

[0111] Where (u', v') are the orthographic view coordinates. To obtain a smooth corrected image, the module uses bilinear interpolation for pixel resampling. In the corrected image, the photovoltaic array appears as a standard rectangle, eliminating trapezoidal distortion, and each photovoltaic unit has the same pixel size.

[0112] 3. Grid Generation and Numbering: In the orthographic view, the total width corresponds to the sum of the physical widths of block C, and the total height corresponds to the sum of the physical heights of block R. Therefore, the pixel width of each cell in the orthographic view is... The height is ,in This represents the total width and height of the orthographic view (in pixels). The module operates in fixed steps. Divide the grid, and define the coordinates of the top-left corner of each grid rectangle: ,in The starting coordinates for the photovoltaic array in the orthographic view are established. A unique number is assigned to each grid cell, e.g., "rc" (r from 1 to R, c from 1 to C). Simultaneously, the module establishes an inverse mapping relationship: for the four corner points of each cell grid, using H... -1 Projecting this image back onto the original image yields a quadrilateral region in the original image, which represents the precise location of the photovoltaic panel unit in the input image. This inverse mapping achieves sub-pixel accuracy (reprojection error <0.5 pixels), laying the foundation for final visualization annotation.

[0113] This module overcomes the shortcomings of existing technologies that simply divide images by pixels or by fixed grids without considering perspective distortion. Comparative experiments show that if perspective correction is not performed and the original image is directly divided by equally spaced grids, the area measurement error of corner units can reach as high as 30%. If only homography correction is used without RANSAC filtering out erroneous corner points, the corrected image may be distorted, and in severe cases, it may lead to misalignment of unit numbers. This module, through RANSAC+LM optimization, ensures stable and accurate unit division under any shooting angle.

[0114] (v) Multi-task defect detection module

[0115] The multi-task defect detection module employs an improved Mask R-CNN instance segmentation network. This network shares a backbone network (ResNet-50+FPN, with an embedded coordinate attention module) and simultaneously outputs defect classification, bounding box regression, and pixel-level masks. This module performs unified detection for five types of defects in photovoltaic scenarios (stains, dust accumulation, cracks, foreign object occlusion, and local abnormal patches). Especially for crack defects (where the foreground pixel ratio is typically less than 0.1%), a focus loss function is introduced during network training. As part of the mixed loss.

[0116] The complete training loss function is: ;

[0117] The loss terms are the same as in Example 1, and the balance coefficients are taken as follows: , =2.0 (increase) (To highlight the role of focus loss). The specific forms of focus loss are: in, For adaptive class balance coefficients, , Let be the frequency of occurrence (percentage of positive sample pixels) of the t-th type of defect in the training set, obtained through offline statistical analysis of the training dataset. For the crack class, Approximately 0.0005, therefore For the background class, Approximately 0.95, This achieves gradient amplification for rare classes. Focusing parameters. In this module, it is not a fixed constant, but a trainable parameter: the initial value is set to 2. After each training epoch, the module evaluates the model performance on the validation set (using F1-score as the metric). If the validation set performance does not improve for two consecutive epochs, it is adjusted using a grid search (candidate values ​​2, 2.5, 3). This continues until the model converges. This dynamic adjustment strategy avoids a fixed approach. This may lead to underfitting or overfitting. Experimental results show that fixing... When the F1 score is 2, the model achieves an F1 score of 0.87 on the test set; adaptive adjustment is used (finally). After converging to 2.3), the F1-score increased to 0.91 because the extreme imbalance in the distribution of photovoltaic defects requires stronger focusing attenuation.

[0118] During inference, this module takes the effective area of ​​the photovoltaic panel in the orthographic view (i.e., the mask area output by the panel area extraction module) as input and outputs the category (one of 5 categories), confidence level (threshold 0.5), and bounding box coordinates for each defect. (Pixel coordinates in an orthographic view) and binary mask (Pixel-level outline). It is worth mentioning that this module enhances the feature response of microcracks through a coordinate attention mechanism, enabling stable detection of cracks as small as 3 pixels wide (compared to the case without the attention module, the recall rate of microcracks increased from 68% to 89%).

[0119] (vi) Quantitative Analysis Module

[0120] The quantitative analysis module is the core component that enables this system to leap from "qualitative" to "quantitative" analysis. It stores the camera intrinsic parameter matrix obtained beforehand using the Zhang Zhengyou calibration method. (Including focal length) and radial distortion coefficient ), and the extrinsic parameter matrix obtained through calibration field or homography decomposition. (Rotation matrix and translation vector). In addition, the module also stores a standard physical area library for photovoltaic panel units—for different models of photovoltaic panels, their length and width data are pre-entered into the configuration file. The single panel area used in this power station is 1.94 m².

[0121] For each defect mask output by the multi-task defect detection module (Located in the orthophoto coordinate system), the quantitative analysis module performs the following operations:

[0122] 1. Physical Area Calculation: The Jacobian matrix surface integral formula is used to accurately calculate the actual physical area corresponding to the mask. Since the local scaling factor of each pixel in the orthographic view mapped to the physical plane is not completely constant (affected by residual distortion and calibration errors), integration must be performed on each pixel. This module uses discrete numerical integration:

[0123] ;where the mapping function The partial derivatives are calculated using the finite difference method: (Pixel unit). This integration process is implemented in parallel on the GPU, and the computation time is less than 0.1ms for defects with an average area of ​​200 pixels. However, if this integration is ignored and a simple "fixed physical area per pixel" method is used, the area measurement error can reach 5%~8% at the image edges due to residual perspective correction errors. This will produce unacceptable deviations for small cracks (e.g., area <1 cm²). The integration method in this module controls the area error to within 1%, meeting engineering requirements.

[0124] 2. Area Ratio Calculation: Read the standard physical area of ​​the photovoltaic panel unit. (In this example, the area is 1.94 m²), calculate the proportion. .

[0125] 3. Preliminary risk classification: based on preset thresholds: Defects are classified as mild (R<5%) and moderate. Severe This threshold is derived from industry standards and can be adjusted by users through configuration files.

[0126] 4. Calculation of Comprehensive Risk Index: Combining defect type weighting factors (Cracks and foreign object obstruction are scored as 1.4; stains and dust accumulation are scored as 0.7; localized abnormal patches are scored as 1.0), calculate the comprehensive risk index:

[0127] This index normalizes risk to [0,1], reflecting the actual impact of different defect types on maintenance priorities. For example, a crack covering 3% of the area: And a 10% area of ​​accumulated dust: ( ).

[0128] In this embodiment, the weighting factor for cracks and foreign object obstruction is 1.4, and the weighting factor for stains and dust accumulation is 0.7. This ratio is based on actual power plant statistics: the power loss caused by cracks per unit area is approximately 2.0 times that caused by dust accumulation. (Comprehensive Risk Index) Risks are normalized to the [0,1] range. For example, a crack with an area of ​​3% (R=0.28) has a higher risk index than dust accumulation with an area of ​​10% (R=0.47), because the area of ​​dust accumulation is much larger than that of cracks. If it is necessary to further increase the priority of cracks, users can customize the weighting factor according to the actual operation and maintenance strategy. This invention does not limit this.

[0129] (vii) Results Output Module

[0130] The results output module receives the output from the quantitative analysis module and generates standardized test reports and visualized labeled images. This module supports offline output (saved to local disk) and online push (sent to the operation and maintenance platform via a predefined data interface).

[0131] This module generates a structured record for each defect, containing the following fields:

[0132] unit_id: The photovoltaic panel unit number, such as "2-5";

[0133] defect_type: Defect type, enumeration values ​​"stain" / "dust" / "crack" / "obstacle" / "patch";

[0134] area_physical: Physical area, in square meters, rounded to 6 decimal places;

[0135] area_ratio: Area percentage, rounded to 2 decimal places;

[0136] risk_index: Comprehensive risk index, ranging from [0,1], rounded to 3 decimal places;

[0137] physical_coordinates: Set of physical contour coordinates of defects, in list form [[x1,y1],[x2,y2],...], in millimeters. It converts the mask coordinates in the orthophoto view into actual physical coordinates based on the inverse mapping of the homography matrix.

[0138] All records are compiled into a JSON file (or CSV table), and an HTML-formatted graphical report is generated, which overlays visual annotations on the original image: using the homography matrix H-1, the mask outline of each defect in the orthophoto view is projected back to the original image and displayed as a semi-transparent colored polygon (different colors for different defect types: red for cracks, yellow for stains, gray for ash, and orange for foreign object occlusion), with the text label "2-5 Crack 0.23% Risk 0.28" added next to the polygon. If there are multiple defects in the same unit, the top 3 are displayed in descending order of risk index.

[0139] For online deployment, the module pushes the structured data stream to the operations and maintenance platform in real time via WebSocket or HTTP / 2 interface. The push frequency is one frame after each detection (if the same unit is repeatedly detected in consecutive frames, the platform will deduplicate based on timestamps). The pushed data is serialized using Protocol Buffers to improve efficiency. Upon receiving the data, the operations and maintenance platform automatically generates a maintenance work order, assigns modules with "severe defects and risk index > 0.8" to maintenance personnel, and highlights the defect location on the geographic information system (coordinates are obtained by converting `physical_coordinates`).

[0140] (viii) System Deployment and Working Mode

[0141] This system supports two deployment methods: offline detection mode and online detection mode.

[0142] Offline detection mode: The system is deployed on the industrial control computer or mobile laptop of the UAV ground station and does not rely on a network connection. Users import image sequences captured by the UAV via USB flash drive or SD card. The system executes the modules mentioned above in sequence and finally outputs a detection report and labeled images. This mode is suitable for remote power plants without network coverage or occasions with high data security requirements.

[0143] Online monitoring mode: The system is deployed on a central server (or cloud server). Fixed cameras or drones in each photovoltaic field upload images to the server via 4G / 5G / LAN. The server uses multi-queue asynchronous processing, dynamically scheduling tasks according to GPU load, with an average processing time of approximately 0.8 seconds per image. Monitoring results are pushed to the operation and maintenance platform's API interface in real time, and the platform can automatically generate alarms and maintenance work orders. This mode is suitable for centralized monitoring of large power plants or distributed power plant clusters.

[0144] The system in this embodiment has been in actual operation at the aforementioned 100MW power plant for 3 months, processing over 200,000 images and detecting 1,230 crack defects (845 minor, 312 moderate, and 73 severe). While large areas of stains and dust accumulation were present, the overall risk index was low. The system has reduced the inspection time for maintenance personnel from 2 people for 3 days to 1 person for 2 hours (only high-risk sections need to be addressed), significantly improving efficiency.

[0145] (ix) Creative summary

[0146] The innovation of this system compared to existing technical solutions lies in the following aspects:

[0147] 1. Perspective Distortion Correction and Inverse Mapping: A homography matrix + RANSAC + LM optimized correction unit is used to accurately map the tilted photovoltaic array into an orthographic view, and an inverse mapping is established to achieve visual binding of defects to the original image. Most existing systems do not correct or use simple affine transformations, resulting in large area quantification errors and poor positioning accuracy.

[0148] 2. Trainable Focus Loss and Adaptive Class Balance: Focusing Parameters of the Focus Loss Function in the Multi-Task Defect Detection Module It is not a fixed value, but is dynamically adjusted based on the performance of the validation set, and the class balance coefficient... This method is based on adaptive calculation using sample frequency. It is more robust than existing methods with fixed parameters, and is especially suitable for scenarios with extremely uneven distribution of photovoltaic defects.

[0149] 3. Jacobi Matrix Integral Area Calculation: The quantitative analysis module abandons the traditional "fixed pixel equivalent" method and instead uses surface integrals to accurately calculate the actual physical area of ​​each defect, compensating for residual distortion. This technology is being applied for the first time in the quantitative detection of photovoltaic defects, improving the area measurement accuracy from 95% to over 99%.

[0150] 4. Comprehensive Risk Index Integration with Operations and Maintenance: The output module not only provides the area percentage but also incorporates defect type weighting factors to calculate the comprehensive risk index and directly generates a structured data stream containing physical coordinate sets, enabling seamless integration with operations and maintenance systems. Existing technologies mostly focus on alarms indicating "defects," failing to guide refined operations and maintenance.

[0151] In summary, this invention addresses the technical problems of existing photovoltaic panel defect detection technologies, such as single detection target, poor robustness in complex environments, qualitative judgment only, inability to accurately locate specific photovoltaic panel units, and low degree of engineering integration. It proposes a collaborative quantitative detection method and system for multiple defects on the photovoltaic panel surface. This invention achieves technological breakthroughs through the following key technical means: First, in the image preprocessing stage, it comprehensively utilizes illumination equalization based on Retinex theory, specular reflection suppression based on a two-color reflection model, and adaptive gamma correction based on shadow boundary detection to effectively eliminate the interference of complex outdoor lighting, specular reflection, and shadows on detection accuracy. Second, it extracts the effective area of ​​the photovoltaic panel through a semantic segmentation network and combines it with a perspective projection transformation model based on homography matrix to correct the tilted image into an orthographic view. Units are divided and numbered according to a fixed grid, establishing a precise inverse mapping relationship between image regions and panel numbers, thereby achieving one-to-one binding and precise location of defect results with photovoltaic panel units. Third, it introduces a focus loss function. A multi-task instance segmentation network, through adaptive class balancing coefficients and trainable focusing parameters, solves the problem of extreme imbalance between positive and negative samples for small target defects such as cracks in photovoltaic scenarios, achieving collaborative recognition of five types of defects: stains, dust accumulation, cracks, and foreign object obstruction. Then, based on camera calibration parameters and the surface integral formula of the Jacobian matrix, a precise mapping model from pixel area to physical area is established to calculate the actual physical area, area ratio, and comprehensive risk index of defects, completing the leap from qualitative judgment to quantitative analysis. Finally, an integrated system architecture comprising seven modules—image acquisition, preprocessing, panel extraction, panel segmentation and numbering, multi-task detection, quantitative analysis, and result output—is constructed, supporting both offline and online deployment modes. These technologies work synergistically and are indispensable, jointly solving the core defects of existing technologies such as poor adaptability, ambiguous positioning, lack of quantitative data, and fragmented processes. This enables stable, high-precision, and quantitative detection of multiple surface defects on photovoltaic panels in complex outdoor environments, and can directly output abnormal panel numbers and maintenance guidance information, significantly improving the engineering practicality of intelligent inspection of photovoltaic power plants.

[0152] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for synergistic quantitative detection of multiple defects in photovoltaic panels, characterized in that, Includes the following steps: S1. Image Acquisition: Acquire visible light images of the photovoltaic panel array; S2. Image preprocessing: Denoising the original image, illumination equalization based on Retinex theory, contrast enhancement, specular suppression based on a physical model, shadow correction, and color correction; S3. Photovoltaic Panel Region Extraction and Unit Division Numbering: A semantic segmentation network is used to extract the effective region of the photovoltaic panel. Based on the geometric constraints of the photovoltaic panel array, a perspective projection transformation model between the image coordinate system and the physical coordinate system is established to correct the photovoltaic panel array into an orthographic view. Units are then divided and numbered according to fixed rows and columns. The perspective projection transformation model is represented by the following homography matrix H: Where (x,y) are the pixel coordinates in the original image, and (x',y') are the coordinates in the corrected orthophoto view. Matrix H is solved by detecting the four corner points or intersection points of the edge lines of the photovoltaic array and using the direct linear transformation method. For rotation and scaling parameters, For translation parameters, These are perspective distortion parameters; S4. Multi-type defect collaborative detection: A multi-task instance segmentation network is used to detect defects in the effective area of ​​the photovoltaic panel, simultaneously identifying stains, dust accumulation, cracks, foreign object obstruction, and local abnormal patches; a hybrid loss function is used during network training. Its expression is: ; in, For classifying losses, For bounding box regression loss, For mask segmentation loss, For small target defects and imbalanced samples, the focus loss function is... The balance coefficients corresponding to the above four losses are all preset positive real numbers, and the focus loss function... The form is: ;in, This is the class balance coefficient, used to adjust the weights of samples from different classes; This is a focusing parameter used to control the weight decay rate of easily classified samples; This represents the model's predicted probability of the true class, and its value ranges from (0,1]. S5. Quantitative Defect Analysis and Grade Assessment: Establish a mapping model from pixel area to physical area that considers perspective distortion during imaging; calculate the actual physical area corresponding to each defect mask and its proportion of the photovoltaic panel unit area; actual physical area The calculation formula is: ;in, The pixel region of the defect mask in the orthographic view. pixel coordinates from orthographic view The mapping function to physical coordinates is determined by the camera's intrinsic and extrinsic parameters and the photovoltaic panel's plane equations. The defect area ratio is obtained by dividing the actual physical area by the standard physical area of ​​the photovoltaic panel unit. and according to the preset threshold Risk level classification: If This is considered a mild risk. This is considered a moderate risk. This constitutes a severe risk; among which To meet the pre-set area ratio threshold ; S6. Integration and Output of Inspection Results: The defect category, coordinate contour, photovoltaic panel unit number, area ratio, and risk level are integrated and associated to generate a structured inspection report and visual annotation results.

2. The method according to claim 1, characterized in that, In step S3, when solving for the homography matrix H, the RANSAC algorithm is used to remove mismatched corner features, and the Levenberg-Marquardt algorithm is used to perform nonlinear optimization on H to minimize the reprojection error. ;in, For the first in the original image The coordinates of the corner points For the corresponding standard layout coordinates; For the first The homogeneous coordinates of each corner point in the original image; Here are the homogeneous coordinates of the corresponding points in the standard layout coordinates; H is a 3×3 homography matrix; This represents the Euclidean distance.

3. The method according to claim 1, characterized in that, In step S4, the hybrid loss function is used to address the extreme imbalance between positive and negative samples of micro-cracks and defects on the photovoltaic panel surface by setting a focusing parameter. The category balance coefficient of the t-th type defect The adaptive adjustment is performed based on the reciprocal of the number of samples for each type of defect in the training set, and the expression is as follows: ;in, Let be the frequency of occurrence of type t defect in the training set. To prevent division by zero of constants.

4. The method according to claim 1, characterized in that, The mapping function described in step S5 Specifically, it is expressed as follows: Where K is the camera intrinsic parameter matrix, which includes focal length, principal point coordinates and distortion coefficients; R and t are the extrinsic parameter rotation matrix and translation vector from the photovoltaic panel plane coordinate system to the camera coordinate system, respectively, which are obtained through pre-calibration; These are the pixel coordinates in the orthographic view; Coordinates mapped to the physical plane Function, unit is meters; The camera intrinsic parameter matrix includes focal length, principal point coordinates, and distortion coefficients; the area percentage R is calculated using the following formula: ;in, Let be the Jacobian matrix of the mapping function P, and its determinant represents the local area scaling factor. The standard physical area of ​​a photovoltaic panel unit is expressed in square meters. The pixel region of the defect mask in the orthographic view.

5. The method according to claim 1, characterized in that, The risk level described in step S5 is further combined with a defect type weighting factor. Make corrections and define a comprehensive risk index. The range is [0,1]; where R is the percentage of defect area. The threshold for severe risk; defect type weighting factor. Based on the impact on power generation efficiency, the weighting range for cracks and foreign object obstruction is preset to 1.2-1.5, and the weighting range for stains and dust accumulation is 0.6-0.

8. This is a function that takes the minimum value.

6. The method according to claim 1, characterized in that, The quantitative analysis in step S5 also includes calculating the distribution density D of defects within the photovoltaic panel unit, and the calculation formula is as follows: ;in, The minimum convex polygon area of ​​the defect mask Ω is obtained by the Graham scan method; the distribution density D is used to characterize the degree of defect aggregation. The closer D is to 1, the more concentrated the defects are, and the closer it is to 0, the more dispersed the defects are.

7. A photovoltaic panel multi-defect collaborative quantitative detection system, characterized in that, include: Image acquisition module, used to acquire visible light images of photovoltaic panel array; The image preprocessing module performs noise reduction, illumination equalization, highlight suppression, and shadow correction. The panel area extraction module is used to extract the effective area of ​​the photovoltaic panel using a semantic segmentation network; The module for segmentation and numbering is used to establish a perspective projection transformation model. The homography matrix H is used to correct the tilted photovoltaic array into an orthographic view. The unit is segmented and numbered according to a fixed grid, and an inverse mapping relationship between the orthographic view unit number and the original image area is established. The multi-task defect detection module is used to simultaneously detect multiple surface defects through a multi-task instance segmentation network. The network training employs a method that includes focus loss. Hybrid loss function ; The quantitative analysis module is used to calculate the physical area and area ratio of defects based on the surface integral formula of the Jacobian matrix, and to calculate the comprehensive risk index in combination with the defect type weight. The results output module is used to generate standardized test reports and visualized annotated images.

8. The system according to claim 7, characterized in that, The panel division and numbering module has a built-in perspective distortion correction unit. This unit calculates the homography matrix H by detecting the four corner points of the photovoltaic panel array and uses the RANSAC algorithm to filter out falsely detected corner points. The correction equation is as follows: Obtain the coordinates of the orthographic view. Then, a fixed step size based on grid division. The grid is divided, with each grid corresponding to a photovoltaic panel unit and assigned a unique number; These are the pixel coordinates in the original image; This is the homogeneous coordinate normalization factor.

9. The system according to claim 7, characterized in that, The quantitative analysis module stores pre-calibrated camera intrinsic parameter matrix K and extrinsic parameter matrix. And a standard physical area library for photovoltaic panel units; for the masking of detected defects. The quantitative analysis module performs the following numerical integration to calculate the physical area: ; The partial derivatives are approximated by the finite difference method. and The unit pixel step size is... The unit of measurement is pixels. For mapping functions along Partial derivatives in direction.

10. The system according to claim 7, characterized in that, During training, the multi-task defect detection module addresses the extreme imbalance issue in photovoltaic scenarios, where the proportion of foreground pixels with cracks is typically less than 0.1%. It sets a focusing parameter in the focus loss function accordingly. These are trainable parameters, initially set to 2 and adaptively adjusted using the validation set to dynamically suppress gradient contributions from a large number of simple negative samples. The system deployment includes offline detection mode and online detection mode; the result output module binds the detection result of each defect to the photovoltaic panel unit number, generating a structured data stream containing the panel number, defect type, area ratio, comprehensive risk index, and the physical contour coordinate set of the defect obtained based on the inverse mapping of the homography matrix, and pushes it in real time through a predefined data interface with the operation and maintenance platform; wherein, the inverse mapping of the homography matrix refers to using the homography matrix of claim 1 inverse matrix Transform the coordinates of the defect contour in the orthophoto view back to the original image coordinates.

Images