Lightweight end-to-end defect detection method based on rail-mounted photovoltaic cleaning robot

Abstract

The present invention provides a lightweight end-to-end defect detection method based on a rail-mounted photovoltaic cleaning robot. The photovoltaic cleaning robot collects the original image of the photovoltaic panel surface and the position corresponding to the original image; after a first round of preprocessing of the original image, the frame line image and grid line image of the photovoltaic panel within the ROI region are extracted to form a spatial composition contour map corresponding to the original image; template matching is performed to determine whether there is a risk; and based on the network connection status, defect detection is performed using a visual language model to obtain a first defect detection result, or an edge computing device is used to execute a backup defect detection mechanism to obtain a second defect detection result. The present invention can effectively handle complex safety detection tasks in on-site cleaning tasks of photovoltaic power stations, significantly improving the accuracy and reliability of detection.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent operation and maintenance of photovoltaic power stations, and specifically to a three-stage collaborative defect detection method for edge computing scenarios such as photovoltaic cleaning robots. The method integrates dynamic template matching based on spatial composition contours, color space analysis, and visual language model (VLM) technology. It aims to realize a lightweight, highly robust, and highly secure photovoltaic module defect recognition system that is independent of environmental perception. The method is particularly suitable for photovoltaic panel cleaning operation systems in dynamic and complex scenarios. Background Art

[0002] With the widespread adoption of photovoltaic power generation, it is becoming a widely used clean energy source worldwide. However, PV modules are exposed to complex environments (such as high temperature, humidity, and dust accumulation) for a long time, which can lead to surface defects such as hidden cracks and stains. These defects significantly impact the module's power generation efficiency, necessitating efficient inspection methods.

[0003] Traditional PV module defect detection methods often combine data from various sensors, as well as infrared, radar, and other modules, to meet the defect detection needs of PV station components based on computer vision systems, deep learning, and visual language models (VLM). As shown in the table below, relevant companies and universities have conducted in-depth research and exploration to improve PV module defect identification by combining different detection technologies.

[0004]

[0005] Early solutions relied on infrared thermal imaging (IR) and electroluminescence (EL) imaging technologies to detect defects through hot spot location and current characteristic analysis. However, these methods required specialized imaging equipment, resulting in high equipment costs, a limited range of defect types, and a high risk of missed and false detections. A second-generation solution based on grayscale histogram segmentation and template matching achieved a 200ms response time in structured scenes, but suffered from a high misjudgment rate for defects with similar colors (such as bird droppings and delamination with a ΔE < 5).

[0006] Existing pure deep learning and visual language models use strategies such as confidence fusion for defect identification, but have the disadvantage of hindering implementation. In particular, the YOLO series of models (such as the improved YOLOv5) achieves 93.5% mAP in laboratory environments, but their computing power requirements far exceed the carrying capacity of the photovoltaic cleaning robot's main control chip. The actual mAP and frame rate will drop sharply, resulting in a loss of real-time detection and bringing great difficulties to actual industrial application deployment.

[0007] Existing mainstream detection methods still mainly rely on a single model or a simple combination of models in industrial applications. They have not broken through the limitations of the multimodal collaborative framework, are highly dependent on manual labor, are difficult to deploy, and are difficult to adapt to complex dynamic scenarios such as photovoltaic cleaning robots.

[0008] Existing technologies fail to effectively integrate rapid edge-based screening with refined cloud-based analysis capabilities. Especially when the network is unstable, traditional solutions cannot guarantee the continuity of basic detection, posing operational and maintenance safety risks to cleaning robots while they are in motion.

[0009] In the current industry context of large-scale deployment and intelligent transformation of photovoltaic power stations' operation and maintenance, photovoltaic cleaning robots, as core equipment for improving power generation efficiency, urgently need a detection technology that is adaptable to the complex environment of photovoltaic power stations, with safety and real-time performance when cleaning robots are moving, and can adapt to the limitations of robot deployment. Summary of the Invention

[0010] This invention aims to address four core issues in the photovoltaic defect detection field: 1) Single detection type: Methods based on infrared and EL imaging are costly, require multiple sensors, are susceptible to environmental interference, and are not easily deployed in large-scale photovoltaic power plants. 2) Edge-based fast response requirements hinder edge deployment, with insufficient computing resources, particularly those that restrict the deployment of deep learning architectures. 3) Robustness in dynamic scenarios: Existing template matching technologies primarily focus on identifying and localizing single target objects (such as fingerprints and numbers). Due to the kinematic characteristics of photovoltaic cleaning robots, limited field of view, and the demand for efficient data processing in industrial applications, it is difficult to accurately locate a single target panel within their field of view. The high noise levels in photovoltaic panel scenes and the limited number of images available to accurately and completely capture the panels under inspection pose challenges to the robustness of traditional single-target matching. 4) Network instability: The method integrates rapid edge screening with refined cloud-based analysis to leverage their respective strengths and ensure reliable operation in unstable network conditions.

[0011] To solve the above technical problems, the present invention provides a lightweight end-to-end defect detection method for photovoltaic power plants. This method is based on the collaborative work of edge computing devices (deployed on photovoltaic cleaning robots) and cloud servers. Its core lies in a three-stage collaborative detection framework, including: integrity screening based on template matching on the edge, stain detection based on color space on the edge (as part of the backup mechanism), and refined defect analysis based on the visual language model (VLM) on the cloud.

[0012] In the present invention, the photovoltaic panel array refers to an array composed of several photovoltaic panels of the same size arranged in a line. When the photovoltaic cleaning robot is working, it runs on the surface of the photovoltaic panel array along a straight line parallel to the direction of the photovoltaic panel array.

[0013] The present invention provides a lightweight end-to-end defect detection method based on a rail-mounted photovoltaic cleaning robot, comprising the following steps:

[0014] Step 1. The photovoltaic cleaning robot runs in a linear cycle on a long strip of photovoltaic panels arranged in a straight line according to the following preset rules: stop, run a distance L1, stop, and run a distance L2;

[0015] The photovoltaic cleaning robot includes an edge computing device and an image acquisition device. The viewing angle of the image acquisition device is adjusted so that the width of the photovoltaic panels in its field of view is not less than the width of the photovoltaic cleaning robot, and the length of the photovoltaic panels in its field of view is not less than the total length L of n photovoltaic panels arranged in a line, and L is not less than the sum of L1 and L2;

[0016] During the stopping, an original image of the photovoltaic panel surface is acquired, and the position of the photovoltaic cleaning robot relative to the photovoltaic panel array is used as the position corresponding to the original image;

[0017] The original image includes a trapezoidal image of photovoltaic panels arranged in a line with a length of L;

[0018] Step 2. After the original image is preprocessed for the first time, a ROI region is obtained by using the four vertices of the trapezoidal image region; the frame line image and grid line image of the photovoltaic panel within the ROI region are extracted to form a spatial composition contour map corresponding to the original image;

[0019] Step 3. On the edge computing device, the spatial composition outline is matched with a template at a corresponding position and a template at a distance L1 from the corresponding position in a preset template library to determine whether there is a risk.

[0020] Step 4. If there is a risk, the photovoltaic cleaning robot stops moving; otherwise, based on the network connection status, the visual language model is used to perform defect detection to obtain a first defect detection result, or the edge computing device is used to execute the backup defect detection mechanism to obtain a second defect detection result.

[0021] Preferably, the determining whether there is a risk comprises the following steps:

[0022] Use the normalized correlation coefficient method to calculate the matching score, and make judgments based on the matching score;

[0023] Match the spatial composition contour map with a template at a corresponding position and a template at a distance L1 from the corresponding position in a preset template library. If the matching score between at least one template and the spatial composition contour map reaches a preset first threshold, it indicates that the photovoltaic panel in the original image is not missing or deformed; otherwise, it indicates that there is a risk.

[0024] The risk mentioned above refers to the risk that the photovoltaic cleaning robot may not be able to clean properly or fall off the photovoltaic panel due to missing or deformed photovoltaic panels;

[0025] The template includes: a spatial composition outline template generated based on a trapezoidal image of a plurality of photovoltaic panels without defects and deformation arranged in a straight line with a length of L.

[0026] Preferably, the calculation process of the matching score includes the following steps:

[0027] The normalized correlation coefficient R(x, y) between the spatial composition outline image I and each template T is calculated according to the following formula:

[0028]

[0029]

[0030]

[0031] Where (x, y) is the reference coordinate of the starting point of the current match in the spatial composition contour image I, (i, j) is the offset relative to the reference coordinate, and its value range is i∈(0, w−1), j∈(0, h−1), where w and h are the width and height of the template T, respectively. I(x, y) represents the pixel value of a point in the sub-block area of ​​the spatial composition contour image I with (x, y) as the upper left corner and the same size as the template image T; μ I is the grayscale mean of the current sub-block area, μ T is the grayscale mean of the template image T. I(x+i, y+j) represents an image in I that is consistent with T o The pixel value of a point in a sub-block of the same size, with its upper left corner at (x, y) and its lower right corner at (x+w-1, y+h-1), will be the same as T o Similarity is calculated pixel by pixel. T(i, j) corresponds one-to-one to I(x+i, y+j), representing the pixel value at offset (i, j) in the template image. R(x, y) ranges from -1 to 1, with values ​​closer to 1 indicating a higher degree of match.

[0032] Preferably, step 3 further comprises the following steps:

[0033] When the matching score is greater than the first preset threshold, the matching area with the highest matching score is extracted, and the Mahalanobis distance D between the area and the corresponding template is calculated; if D is less than the preset second threshold, the pixel value, mean and variance of the corresponding template are fused and updated using the weighted sliding average method; if D is not less than the preset second threshold, the old template is retained, and this image is marked as having no missing parts but containing instantaneous interference.

[0034] Preferably, the image acquisition device is equipped with a polarizer; the first round of preprocessing includes illumination compensation and noise filtering. Image acquisition, the first round of preprocessing, and the backup defect detection mechanism during the previous cleaning robot stop are executed in parallel with image acquisition during the next stop. A double-buffered queue is used to manage the image data to be processed, thereby reducing waiting time between tasks.

[0035] Preferably, in step 4, the method of performing defect detection using a visual language model according to the network connection status to obtain a first defect detection result, or performing a backup defect detection mechanism using an edge computing device to obtain a second defect detection result, specifically includes:

[0036] When the edge computing device is connected to the cloud server network normally, the image determined to be risk-free is input into the cloud visual language model, and defect detection is performed on the image in combination with the photovoltaic defect semantic information to obtain the first defect detection result;

[0037] When the network connection is abnormal, images determined to be risk-free are cached on the edge computing device, and a backup detection mechanism is executed. The backup detection mechanism includes stain defect detection based on color space to obtain a second defect detection result; and after the network connection is restored to normal, the cached image is uploaded to the visual language model on the cloud to obtain a first defect detection result, and the first defect detection result and the second defect detection result are fused to obtain a final defect detection result.

[0038] Preferably, before inputting the risk-free images into the cloud-based visual language model, the process further includes performing a second round of preprocessing on the risk-free images, wherein the second round of preprocessing includes an inverse perspective deformation transformation. The inverse perspective deformation transformation transforms the image from a trapezoidal shape back to a normal shape, making it easier for the VLM to identify defects.

[0039] Preferably, the fusion of the first and second defect detection results includes the following steps: the final defect detection result includes two defect categories: stains and glass cracks; the stain category further includes bird droppings and dirt; determining whether the first defect detection result indicates glass cracks; if so, the final defect detection result indicates glass cracks; otherwise, determining whether the VLM detected stains; if not, trusting the second defect detection result; and determining whether the confidence level of the VLM in detecting stains is less than a third threshold; if so, trusting the second defect detection result; otherwise, trusting the first defect detection result. This design takes into account that the edge backup mechanism (second defect detection result) is primarily based on color space and is more sensitive to "stain" defects (such as bird droppings and dirt). However, for "glass crack" defects, the edge color space backup mechanism has weaker detection capabilities for "glass cracks" and primarily relies on the first defect detection result.

[0040] Preferably, in step 1, the value of n is 3, 4, or 5. With the above values, the trapezoidal image is of moderate size, can reflect common defects such as bird droppings, dirt, and glass cracks, and satisfies the composition of the spatial composition outline. When n is too small, such as 1, it is impossible to form a spatial composition outline. When the value of n is too large, the proportion of each photovoltaic panel in the formed spatial composition outline is too small, making it difficult to detect defects thereon.

[0041] In step 2, the frame line image and the grid line image of the photovoltaic panel in the ROI region are extracted, including: the image of the ROI region is processed by an edge detection algorithm and a Hough line segment detection algorithm to obtain the frame line image and the grid line image of the photovoltaic panel in the ROI region. By using the above two edge detection algorithms and the Hough line segment detection algorithm, the extracted frame line image and the grid line image are made clearer, which is conducive to template comparison and risk identification in the later stage.

[0042] Preferably, the visual language model is a Qwen-VL model, and instruction fine-tuning is performed using a dataset containing photovoltaic module images and their corresponding defect descriptions.

[0043] Said defects include broken glass, bird droppings, and dirt;

[0044] The prompt word framework of the visual language model includes the following dimensions:

[0045] Spatial dimension: description of the relative position of the defect on the component;

[0046] Morphological dimension: crack direction or stain diffusion morphological characteristics;

[0047] Time series dimension: Compare and analyze with historical test results.

[0048] Beneficial effects of the present invention:

[0049] 1. Significantly improve the reliability and safety of structural risk detection: Based on the "spatial composition contour" matching strategy and dynamic template update mechanism, it can more accurately identify high-risk risks such as missing photovoltaic panels and severe deformation, effectively ensuring the safe operation of equipment such as cleaning robots.

[0050] 2. Enhanced robustness to gradual environmental changes and transient interference: Combining the GPS scenario-based dynamic template library and the Mahalanobis distance-based update mechanism, the system can adapt to gradual changes in photovoltaic panels due to aging and dust accumulation, and effectively resist transient interference such as sudden changes in illumination and short-term occlusion, ensuring long-term stability and accuracy of detection.

[0051] 3. Achieves lightweight deployment and efficient collaboration: Computation-intensive tasks (such as VLM analysis) are placed in the cloud, and fast integrity screening, dynamic template management, and backup detection are performed on the edge, making the overall solution lightweight. It is suitable for embedded platforms with limited computing power and reduces hardware deployment costs.

[0052] 4. Multimodal advantages complement each other to enhance comprehensive detection capabilities: The integration of template matching (sensitive to structural defects and deformation), color space analysis (sensitive to specific stains) and VLM (powerful image and text understanding and fine-grained recognition capabilities) overcomes the limitations of a single model and improves the comprehensive detection capabilities of various types of defects.

[0053] 5. Excellent engineering characteristics and application potential: The entire method has a clear process and a modular design that is easy to deploy and integrate. It can adapt to the complex and dynamic scenarios in actual industrial environments and promote the intelligent and automated level of photovoltaic power station operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic flow diagram of the present invention;

[0055] Figure 2 It is a schematic diagram of the VLM model flow of the present invention;

[0056] Figure 3 This is a walking diagram of the photovoltaic cleaning robot of the present invention;

[0057] Figure 4 This is a diagram showing the effect of detecting different types of stains and defects using the backup detection mechanism of the present invention;

[0058] Figure 5 This is an example of a field photovoltaic panel after pre-processing and ROI area template image. DETAILED DESCRIPTION

[0059] The present invention provides a lightweight end-to-end defect detection method based on a rail-mounted photovoltaic cleaning robot, such as Figure 1 The process diagram shown specifically includes the following steps:

[0060] S1, image acquisition, the image acquisition device (such as a camera) deployed on the photovoltaic cleaning robot captures the image data of the photovoltaic component and obtains the position of the photovoltaic cleaning robot relative to the photovoltaic panel array as the position corresponding to the original image.

[0061] S2, image preprocessing, in this stage, multiple processes are performed on the original image to provide high-quality small images for subsequent template matching and defect analysis.

[0062] Lighting compensation and noise filtering (first round of preprocessing):

[0063] 1) Illumination compensation: For images where defects need to be detected, the contrast-limited adaptive histogram equalization (CLAHE) algorithm is used to adjust the local contrast of the image and mitigate the effects of uneven illumination.

[0064] 2) Noise filtering: The non-local means denoising algorithm is used to effectively filter out image noise while preserving edge and texture details.

[0065] 3) Delineate the region of interest (ROI): Based on the actual situation of the photovoltaic power station, the ROI is obtained by using the four vertices of the trapezoidal image area of ​​the photovoltaic panels arranged in a straight line with a length of L in the original image. This ROI delineation method takes into account the actual shooting needs of the photovoltaic cleaning robot. The frame line image and grid line image of the photovoltaic panels in the ROI area are extracted to form a spatial composition contour map corresponding to the original image, such as Figure 5 ROI delineation in .

[0066] Inverse perspective distortion correction (second round of preprocessing, after integrity screening): Before risk-free images are fed into the cloud-based visual language model, camera internal and external references are used to correct the perspective image to a top-down view without perspective. The innovation lies in explicitly specifying that this step is performed after the initial component integrity screening, avoiding preprocessing failures due to missing or deformed components that prevent feature point extraction.

[0067] S3, Edge-based template matching completeness fast screening and tracking based on spatial position composition

[0068] This patent abandons traditional single-target template matching and instead compares the "spatial composition" formed by the outlines of multiple photovoltaic panels within the current location's ROI to see if it matches the composition in the template library at that location. This is one of the key innovations of this patent. When a target photovoltaic panel is missing or severely deformed, its "absence" itself becomes part of the spatial composition, enabling more accurate judgment of "missing" or "serious structural risk." Due to the limited field of view during robot movement, the opportunity to accurately capture a single panel is limited, making multi-panel composition matching even more essential.

[0069] "Component missing and deformation" means that when the cleaning robot is walking on the photovoltaic panels, it may encounter structural factors such as partial missing photovoltaic panels, deformation of photovoltaic panels, and tilt of photovoltaic panels that affect the robot's walking.

[0070] For photovoltaic panels confirmed to have no missing deformation, the robot collects and generates negative sample images of "no missing deformation" and their GPS latitude and longitude information during its first walk, forming an initial template library. After the first round of image preprocessing, the spatial composition outline template is extracted. During template matching, the system selects the template image corresponding to the current shooting position and the template at a distance L1 from the corresponding position from the template library to form the candidate template library for this matching based on the robot's current GPS location.

[0071] On the edge computing device, a template matching algorithm pre-set for the spatial composition outline of the photovoltaic module is used to analyze the spatial composition outline of the image after the first round of preprocessing. The matching score between the current image and the candidate template library is calculated, and a first preset threshold is set.

[0072] Once the matching score calculated using the normalized correlation coefficient method exceeds the first threshold, the Mahalanobis distance D between the current matching area and the template image is further calculated. Only when the Mahalanobis distance is less than a preset second threshold is the match considered valid and free of significant transient interference, at which point the template is updated. Template pixels are updated using a weighted sliding average method. Simultaneously, the template's pixel-level mean and variance are dynamically updated, enabling the template to track gradual changes in the target while simultaneously resisting transient interference and preventing template contamination. If D exceeds the preset second threshold, the old template is retained, and the image is marked as free of missing data but containing transient interference.

[0073] S4, Edge - Emergency Risk Assessment and Initial Triage

[0074] Make a judgment based on the matching score in step S3:

[0075] If the matching score between at least one template and the spatial composition outline for two consecutive frames fails to reach a preset first threshold, the current inspection area is deemed to be at risk. This risk refers to the risk of the PV cleaning robot failing to clean properly or falling off the panels due to missing or deformed panels. The edge device immediately triggers the robot's stop mechanism and reports the risk status, terminating the current inspection process to ensure robot safety.

[0076] If the matching score between at least one template and the spatial composition outline in two consecutive frames reaches a preset first threshold, the photovoltaic panel in the original image is intact. The current inspection area is determined to have no significant risk of missing or deformed images. Depending on network connectivity, defect detection is performed using a visual language model to obtain a first defect detection result, or a backup defect detection mechanism is executed using an edge computing device to obtain a second defect detection result. Simultaneously, the cleaning robot can continue moving along the predetermined path to capture the next image of the photovoltaic panel. At this point, if a matching score exceeds the first preset threshold, the matching region with the highest score is extracted, and the Mahalanobis distance D between this region and the corresponding template is calculated. If D is less than a preset second threshold, the pixel values, mean, and variance of the corresponding template are updated using a weighted sliding average fusion method. If D is greater than the preset second threshold, the old template is retained, and the image is marked as intact but containing transient interference.

[0077] S5, Cloud-VLM refined defect detection (main path), e.g. Figure 2 Schematic diagram of the VLM model process;

[0078] Determine the network connection status between the edge device and the cloud server:

[0079] If the network connection is normal, the preprocessed image is uploaded to the cloud server. The cloud-based Visual Language Model (VLM) receives the image and, combined with pre-defined photovoltaic defect semantics (input via optimized prompt words, whose framework encompasses spatial, morphological, and temporal dimensions), performs detailed defect detection and classification (identifying glass cracks and stains). The VLM directly outputs detailed defect recognition results as the primary defect detection result.

[0080] If the network connection is interrupted or unstable, the image cannot be immediately uploaded to the cloud. In this case, the image, deemed safe, is cached in the local storage of the edge device and re-uploaded to the cloud VLM for analysis after the network is restored. Simultaneously, to ensure that essential defect detection is not interrupted, a backup detection mechanism is activated on the edge (step S6).

[0081] S6, Edge - Alternative Detection Mechanism and Result Fusion (Alternative Path)

[0082] In the event of a network outage or as supplemental verification, the following backup detection process is performed on the edge device:

[0083] 6a. Stain Defect Detection: The image is converted to the HSV (Hue, Saturation, Value) color space. Using a multi-channel adaptive segmentation algorithm, the system distinguishes the normal PV panel surface from areas with stains, bird droppings, and other attachments based on color and saturation characteristics. This generates a second defect detection result. Until network connectivity is restored, the PV panel inspection results are output according to this algorithm.

[0084] 6b. Result Temporary Storage and Decision Fusion: The detection results from step 6a are temporarily stored. When the network connection is restored, the locally cached image is successfully uploaded to the cloud-based VLM for analysis (performed in step S5), resulting in a first defect detection result. After obtaining the VLM's detection results, the first and second defect detection results are fused based on a third confidence level to obtain the final defect detection result. This fusion strategy ensures that preliminary detection can be performed even in poor network conditions, and that confirmation and correction can be performed using the VLM's high-precision analysis capabilities after the network is restored.

[0085] Preferably, in S2, template matching (S3) does not rely on the component's four corner feature points for inverse perspective correction. This is because if the component is missing or deformed, the feature points may not be extracted, resulting in preprocessing failure. Therefore, illumination compensation and noise filtering are used as the first round of preprocessing. After the missing and deformed components are detected as safe, perspective transformation is used as the second round of image preprocessing and input into the next analysis (S5, S6).

[0086] Furthermore, in S3, since it is difficult for the camera to accurately and individually capture the target photovoltaic panel, the original image captured will include the target photovoltaic panel and 2-4 adjacent photovoltaic panels. When the target photovoltaic panel is missing or deformed, ordinary template matching will also determine that there is no missing deformation due to matching the integrity of the adjacent photovoltaic panels. However, under ideal conditions, assuming that the photovoltaic panel contour structure is complete, the 3-5 photovoltaic panels in the images captured by the camera at different locations only have differences in spatial composition. Therefore, considering the actual situation of photovoltaic power stations, we shift the core of template matching detection to whether the spatial composition of the photovoltaic panel contour within the current position ROI area matches the template library at this location.

[0087] Based on this, the core of the template matching algorithm preset for the spatial composition contour of photovoltaic panels is to completely match the contour structure of the target photovoltaic panel and the photovoltaic panel composed of adjacent positions in the matching contour template library, without the need for additional sensor data or target detection methods. Its core structure includes:

[0088] 1. Data preparation and preprocessing module: responsible for collecting and screening templates / test samples in different regions, and performing the first round of preprocessing and template size normalization.

[0089] 2. Template library generation module: The robot is equipped with a camera to collect negative sample images of the first row of photovoltaic panels in the photovoltaic station and obtain the latitude and longitude of the photovoltaic panels. After the first round of preprocessing, a "no missing deformation" negative sample image is generated. Light compensation, noise filtering, edge detection, Hough line segment detection and ROI delineation are performed to generate a set of spatial composition contour templates (such as Figure 5 - spatial composition outline template), constituting the template library of the first row of photovoltaic panels {T pre,neg_1 , T pre,neg_2 ,...,T pre,neg_N}.

[0090] 3. Test image processing module: performs preprocessing on the input image to be tested consistent with the training sample.

[0091] 4. Template matching calculation module:

[0092] The robot's GPS positioning module provides location information, conveying the latitude and longitude of the current image capture location. During the first round of walking capture, the robot selects the location and the three template images following it for matching. Starting with the second round, each template matching attempt selects the template corresponding to the current capture location and a template at a distance L1 from the corresponding location from the template library to form the template library for this matching. This is the core embodiment of spatial position-based composition.

[0093] In the template matching stage, the normalized cross-correlation algorithm (NCC) is used to calculate the matching score between the photovoltaic panel spatial composition outline of the test image and the photovoltaic panel spatial composition outline of each template in the template library.

[0094] 5. Decision-making and dynamic template fusion module:

[0095] In order to solve the problem of matching failure caused by various changes in traditional static template matching during long-term operation, the present invention introduces a mechanism that combines template matching based on spatial composition contours with dynamic template library updating.

[0096] The matching score calculated based on each template image is compared with the preset first threshold. When there is a matching score greater than the first threshold, the matching area with the highest matching score is extracted, and the Mahalanobis distance D between the current area and the template image is calculated and compared with the preset second threshold.

[0097] Template update condition: If D < the preset second threshold, the match is considered valid and there is no large instantaneous interference. At this time, the mean, variance and template are dynamically updated. (Pcurrent (where D is the current matching area), the image is marked as complete. If D > a preset second threshold, the old template is retained and the image is marked as complete but with transient interference. If the condition of "the matching score between at least one template and the spatial composition outline in two consecutive frames reaches the preset first threshold" is not met, a risk is determined and an alarm is triggered.

[0098] Preferably, the VLM model selects a large visual language model from the Qwen-VL series (such as Qwen-VL-Max). To improve its performance in photovoltaic defect detection tasks, instruction fine-tuning is performed based on photovoltaic defect text-image pairs to optimize cross-modal alignment. The processing flow includes:

[0099] 1. Instruction fine-tuning: Fine-tune the pre-trained VLM using a dataset containing PV module images and their corresponding defect descriptions (text) to enhance the model's understanding of PV-specific defect scenarios and cross-modal alignment capabilities.

[0100] 2. Standardized API interface design: Design a standardized API interface for data transmission (image upload, command issuance) and result analysis (defect type, location, confidence level, etc.) between edge devices and cloud-based VLM to ensure system integration.

[0101] 3. Multi-image comparison mechanism (optional): During VLM analysis, historical images or standard reference images related to the component under inspection can be loaded from the cloud. By comparing similarity with the input image, it can assist in determining whether the defect is new or evolving, or help identify similar defect patterns, thereby improving diagnostic accuracy. This mechanism is configurable.

[0102] 4. Dynamic Hint Injection: Leveraging an attention-guided hint word optimization mechanism, structured photovoltaic defect semantic templates (e.g., a pre-defined 3D text description framework encompassing spatial location, morphological characteristics, and possible causes) are dynamically injected into the VLM's input prompts. This helps direct the VLM's attention to specific defect features, compensates for blind spots in edge detection, and improves the recognition accuracy of complex or subtle defects (such as specific types of glass cracks).

[0103] Preferably, the photovoltaic defect semantic template library can contain multi-dimensional information, for example: spatial dimension prompts: such as "detect whether there is a linear crack in the upper left edge area of ​​the photovoltaic module"; morphological dimension prompts: such as "find whether there is a fragmented area with a white spider-web-like radial shape and blurred boundaries"; context / time dimension prompts (if there is historical data): such as "compare historical images to confirm whether the previously marked stain area has expanded."

[0104] Preferably, to ensure the real-time performance of the system, an asynchronous pipeline architecture can be adopted during deployment: tasks such as image preprocessing (step S2) and edge detection (steps S3 and S6a) are performed in parallel with image acquisition (step S1). A double-buffered queue or similar mechanism is used to manage the image data to be processed, reducing the waiting time between tasks and achieving near-zero-wait data flow. If the edge device hardware supports it (such as including an NPU), some edge model inference tasks (such as the computationally intensive parts of template matching or color segmentation) can be assigned to dedicated hardware cores for execution, further accelerating processing. VLM inference in the cloud can also utilize GPUs or dedicated AI accelerators.

[0105] Preferably, the result fusion strategy (step S6b) also incorporates dynamic fusion of confidence levels. A requirement for the VLM to output confidence levels is added to the VLM prompt. The question template is modified so that the VLM outputs the confidence level of its image analysis. When detecting "stain" defects, if the VLM output confidence level is less than a third threshold (typically set to 0.7 during training), the results of the backup detection mechanism are used. Otherwise, the VLM model detection results are output, thus avoiding the subjectivity of semantic analysis. When detecting "cracked glass" defects, the backup mechanism based on color space has weak detection capabilities, so the VLM model results are fully used.

[0106] In order to better describe the working principle and details of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings to more clearly illustrate its objectives, technical solutions and positive impacts.

[0107] The image acquisition device deployed on the photovoltaic cleaning robot (for example, a camera model HX2-TC2) has an additional polarizing film in front of its lens to capture image data of the photovoltaic panels. When the photovoltaic cleaning robot is working, it moves along a straight line parallel to the direction of the photovoltaic panel array and on the surface of the photovoltaic panel array (for example, grasping the guide rails on the left and right sides of the photovoltaic panel array). Figure 3 The robot's normal walking speed is set to 18m / min. The shooting logic is as follows:

[0108] Fixed-point shooting sequence: The robot then stops every 30 cm (i.e., every second, based on a speed of 18 m / min ≈ 30 cm / s) to capture a fixed-point image. At each fixed-point position, the camera captures one image.

[0109] Movement and Repeated Shooting: After completing two fixed-point captures (two images), the robot continues to move forward for 1.4 meters without capturing any images. It then reenters the fixed-point capture sequence described above (i.e., stopping and capturing one image every 30 cm).

[0110] Image usage: Each image taken at a fixed point will serve as the original image data for subsequent steps (especially S3) to detect defects such as deformation and missing on the next photovoltaic panel or the current area.

[0111] This specific walking and shooting logic ensures that images can be acquired at a certain interval and redundancy (taking one image at each point) during the robot's movement, while also taking into account operational efficiency (traveling without shooting for a specific distance).

[0112] The following first round of preprocessing operations are performed on the acquired images, including illumination compensation and noise filtering, to improve image quality and eliminate environmental interference:

[0113] Light compensation processing (such as Figure 5 - Lighting Compensation): Uses the contrast-limited adaptive histogram equalization (CLAHE) algorithm to adjust the local contrast of the image, improve detail expression, and enhance the recognizability of defects.

[0114] Noise filtering (such as Figure 5 - NLM Denoising): Use the non-local means (NLM) denoising algorithm to remove random noise and environmental interference in image acquisition while ensuring the integrity of edge and texture details.

[0115] Traditional single-target template matching is prone to failure when the target photovoltaic panel is missing or severely deformed. The core of the "spatial composition contour" matching strategy adopted by the present invention is to use the contour arrangement characteristics (i.e., "spatial composition") formed by the target photovoltaic panel and its 2-4 adjacent photovoltaic panels within the ROI as the matching object. Even if the target photovoltaic panel is missing, its "missing" state itself will change the characteristics of this spatial composition, so that this "missing" can be identified, rather than causing matching failure or incorrect matching to the intact photovoltaic panel next to it. Taking into account that the cleaning robot has a limited field of view during movement and that the two sides of the photovoltaic panel are usually overhead, it is difficult to stably achieve accurate capture of only a single target panel at the physical level, and there is often only one chance to accurately capture the target panel. Therefore, matching based on the "spatial composition" that contains the contour information of the surrounding environment (adjacent photovoltaic panels) is a necessary means to improve detection robustness and accuracy.

[0116] After the first round of preprocessing, the spatial composition outline image is used to quickly screen the integrity of edge components. A preset component spatial composition outline template matching algorithm is deployed on the edge computing device. The matching degree between the spatial composition outline image after the first round of preprocessing and the template is calculated to obtain a matching score. A predefined first threshold is used to determine the risk of missing components. An abnormality is detected when the matching degree is less than this threshold.

[0117] In order to solve the defects of traditional static template matching during long-term operation, such as the offset of the target photovoltaic panel position in the test image due to lighting changes, manual target adjustment, camera parameter changes, and errors in fixed-point shooting time, a combination of template matching based on spatial composition contours and dynamic template library updates was invented. The actual camera is installed on the cleaning robot, and the viewing angle inclination and installation angle are fixed, without the interference of common rotation invariance problems. In addition, through edge learning, the template can track the gradual changes of the target while resisting instantaneous interference (such as short-term occlusion and noise).

[0118] The specific testing process is as follows:

[0119] When the robot performs its first walking operation in a specific area of ​​the photovoltaic array (such as the first row of photovoltaic panels), it will use the onboard camera to collect "no missing deformation" baseline images of various positions along the way. At the same time, the precise latitude and longitude positions (Latitude, Longitude) corresponding to these images are obtained through the GPS positioning module. The collected original image first undergoes the first round of preprocessing in S2, including illumination compensation and noise filtering, and then the ROI image is obtained after ROI delineation. Subsequently, the ROI image is subjected to edge detection (Canny operator or other edge detection algorithm) and Hough line detection to extract the contour information of the photovoltaic panel (and its adjacent panels), and it can be converted into a binary spatial composition contour template (such as Figure 5 These template images containing spatial composition outline information are stored together with their corresponding GPS location information, as well as the initial pixel-level mean and variance statistics for subsequent Mahalanobis distance calculation, to form the initial template library of the fixed point location {T pre,neg_1， T pre,neg_2 ,...,T pre,neg_N}.

[0120] During subsequent routine inspections, when the cleaning robot moves to a new location and captures images, the system first obtains the robot's current GPS location information (latitude, longitude). Based on this location information, the system then selects a template image at the current shooting location (latitude, longitude) from a pre-stored template library, along with a template at a distance L1 from the current location in the template library, to form the candidate template library for this matching operation. This method of selecting candidate templates based on GPS location embodies the core concept of "spatial position composition," making matching more targeted and adaptable to subtle differences in lighting or photovoltaic panel arrangement that may exist in different geographic locations.

[0121] For the current frame image to be detected, the first round of preprocessing in S2, which is consistent with the template library sample generation, is first performed to extract its spatial composition contour information within the ROI. Then, the normalized cross-correlation algorithm is used to calculate the matching score R(x, y) between the spatial composition contour of the ROI of the image to be detected and the spatial composition contour of each template in the candidate template set. The normalized correlation coefficient formula is as follows:

[0122]

[0123]

[0124]

[0125] (x, y): The reference coordinates of the current match in the spatial composition contour image (e.g., the upper left corner of the photovoltaic panel area), indicating the starting point of the current match. (i, j): The offset relative to the reference coordinates (traversing the entire template size), with a value range of i∈(0, w−1), j∈(0, h−1), where w×h is T o I (x, y) means the top left corner of I and T o μ is the pixel value of a point in a sub-block area of ​​the same size. I is the grayscale mean of the sub-block in the current spatial composition contour image, μ T T o The grayscale mean of . I(x+i, y+j) represents the grayscale mean of one of I and T o The pixel value of a point in a sub-block of the same size, with its upper left corner at (x, y) and its lower right corner at (x+w-1, y+h-1), will be the same as T o Calculate similarity pixel by pixel. T(i, j) corresponds one-to-one to I(x+i, y+j), representing the pixel value at offset (i, j) in the template image.

[0126] The value range of R(x, y) is [-1,1]. The closer the value is to 1, the higher the matching degree.

[0127] For each template in the candidate template group, the calculated NCC matching score is compared with a preset first threshold (for example, usually set to 0.7). If among all the candidate templates, there is at least one template with the highest NCC matching score greater than the first threshold, then the preliminary match is considered successful, and the spatial composition contour information I in the current frame ROI is extracted. current and its corresponding original template T oldIf the highest NCC matching scores of all candidate templates are less than the first threshold, it means that the matching degree between the current image to be detected and all templates representing "no missing deformation" samples is low. Therefore, it is determined that there may be missing photovoltaic modules or serious structural risks in the current area, and the subsequent S4 alarm process is triggered.

[0128] For the area I where the initial matching is successful current and template T old The weighted sliding average method is used to retain slowly changing factors such as light, dust, and photovoltaic panel aging. In addition, the pixel-level mean and variance of the maintenance template are added to take into account the transient interference of birds and local reflections in the field. Combined with the Mahalanobis distance constraint, the specific formula is as follows:

[0129]

[0130] In the matching stage, the Mahalanobis distance between the spatial composition contour image and the template image is calculated. Only when D(x,y) < the second threshold (based on the training data, usually set to 3.0) is the match considered valid and the template is allowed to be updated.

[0131]

[0132] The weighted sliding average formula, whose parameter α∈[0,1] is the forgetting factor, generally selected as 0.9 (slow update, adapt to the scene of gradual change of light and mechanical wear, and avoid sudden interference), T old is the pixel value of the old template at position (x, y) (T new Similarly), I current is the pixel value of the matching area in the current spatial composition contour image, and the position is (x, y). T_new is neither T_old nor I current Instead, the new template obtained by fusing the two through Mahalanobis distance will replace T_old and store it back in the template library at the corresponding GPS position.

[0133]

[0134]

[0135] The mean and variance are the core parameters of the Mahalanobis distance. They dynamically evaluate the difference between the current data and the template and decide whether to update the location (latitude, longitude) template. The mean update is consistent with the weighted average and reflects the long-term trend. The variance update consists of two parts: 1. The local variance σ of the current window 2 current (Measures the noise intensity of the current area). 2. The square difference between the new and old means (μ current − μ old ) 2(A measure of the magnitude of gradual change.) In this way, the template's criteria for determining "normal change" are dynamically adjusted, making it more adaptable to gradual changes in the environment and aging of components.

[0136] When D(x,y) ≧ the second threshold, it indicates that although the matching score is high, the current matching area I current With template T old There are significant differences in statistical characteristics, which may be caused by large transient interference or atypical, non-structural changes. In order to prevent these abnormalities from contaminating the purity of the template, the system chooses not to update the template and retains the old template T old . And mark the current image as "no missing but contains instantaneous interference". This set includes a mechanism for preliminary matching, secondary verification of Mahalanobis distance and dynamic update. Its core advantages are: 1) Introducing Mahalanobis distance as a key constraint for template update and fusion, rather than just using Mahalanobis distance to directly judge the matching target. 2) Through the synchronous dynamic update of the template pixels themselves and their statistical characteristics (mean, variance), the template can actively track and adapt to the gradual changes of the target (such as the slow accumulation of dust and color fading on the surface of photovoltaic panels due to long-term use), and can effectively resist erroneous updates caused by instantaneous interference factors such as short-term occlusion and sudden changes in light, thereby ensuring the long-term effectiveness and robustness of the template library.

[0137] Subsequently, the edge side performs emergency risk judgment and preliminary diversion. If the condition of "the matching score of at least one template in two consecutive frames and the spatial composition contour map reaches the preset first threshold" is not met, it is determined that the current photovoltaic component is at risk. The edge device immediately triggers the emergency parking mechanism of the cleaning robot to prevent structural risks from causing damage to the robot or accidents. The risk status is reported to the monitoring platform to terminate the current detection process; if the matching score of at least one template in two consecutive frames and the spatial composition contour map reaches the preset first threshold, it is determined that there is no obvious missing risk in the current detection area, and the pre-processed image is marked for further analysis. Subsequently, the system will execute the subsequent S5 cloud-based refined defect detection or S6 edge-side backup detection mechanism based on the current network connection status of the edge device and the cloud server. The cleaning robot can continue to move along the predetermined path to collect images of the next photovoltaic component.

[0138] Next, confirm that the image without missing conditions is subjected to the second step of perspective transformation image preprocessing (such as Figure 5 - IPM transformation): To address the perspective distortion caused by wide-angle lenses, a fixed homography matrix H can be calculated using the pre-measured camera internal parameters and the camera external parameters relative to the photovoltaic panel plane. ipmThrough this transformation matrix, the ROI image (part of it) after the first round of preprocessing (and passing the S3 integrity screening) is corrected from the original perspective view to a top view (bird's eye view) without perspective effect. The key to this step is that it is performed after the preliminary component integrity screening. This can avoid the problem of inability to accurately extract feature points (such as corner points) used for real-time calculation of the transformation matrix due to the actual absence or severe deformation of the target photovoltaic panel components, thereby avoiding the problem of preprocessing failure. Directly use the known camera internal and external parameters and the fixed transformation matrix H ipm Transformation is performed without real-time corner detection, which improves the robustness and processing efficiency of image correction. Let the transformation matrix be H ipm , the inverse perspective formula is as follows:

[0139]

[0140] Among them (W std , H std ) are the standardized output image width and height, and a “top-down” view of the image is obtained on the photovoltaic panel plane. The pixel coordinates of the ROI in the original image are selected, and then the homography matrix is ​​used for transformation without corner detection.

[0141] When the network connection between the edge device and the cloud server is determined to be normal, VLM refined defect processing is performed. The edge device uploads the second round of pre-processed images (i.e., after inverse perspective correction) to the cloud server. A large-scale visual language model (VLM) is deployed on the cloud, such as a model from the Qwen-VL series (such as Qwen-VL-Max). To improve its performance in photovoltaic defect detection tasks, this VLM can be pre-tuned using a dataset containing PV module images and their corresponding defect descriptions (text). This enhances the model's understanding of specific PV defect scenarios and its cross-modal alignment capabilities.

[0142] After receiving the image to be inspected, the VLM combines preset, optimized photovoltaic defect semantic information (input through a dynamic prompt injection mechanism) to perform detailed defect detection and classification on the image, mainly identifying two types of defects, such as glass cracks and stains (bird droppings, dirt). These prompt words can include multiple dimensions, such as:

[0143] 1) System prompt (setting roles and tasks): "You are a professional photovoltaic panel inspection assistant. You need to carefully analyze the photovoltaic panels in the image for two types of defects: stains (bird droppings, dirt) and glass cracks. You must strictly follow the following format: 1. First, describe your observations in detail. 2. Then, provide your final judgment. The format must be [X,X], where the first X represents the crack condition (A = cracked, B = not cracked), and the second X represents the stain condition (A = stains, B = not stains). If you are unsure about a judgment, use B. For example, [B,B] represents no cracks and no stains."

[0144] 2) Initial question prompt (for specific defects): "Please carefully analyze this photo of a photovoltaic panel and answer according to the following steps: 1. Describe your observations in detail. 2. Ignore the background environment and provide a final judgment on whether the photovoltaic panel is collapsed, cracked, or stained. The format must be [X,X]..."

[0145] 3) Spatial dimension prompts: such as "Check whether there is a linear crack in the upper left corner of the photovoltaic module."

[0146] 4) Morphological dimension prompts: For example, "Look for fragmented areas with white spider-web-like radiating shapes and blurred boundaries."

[0147] 5) Contextual / temporal prompts (if historical data is available): For example, "Compare historical images to confirm whether the previously marked stain area has expanded." This dynamic prompt injection helps guide the VLM's attention to specific defect features, compensates for potential blind spots in edge detection, and improves the recognition accuracy of complex or subtle defects.

[0148] During analysis, the VLM can selectively load historical cloud-based images or standard reference images related to the component being inspected. By comparing similarities with the current input image, it can assist in determining whether a defect is new or evolving, or help identify similar defect patterns, further improving diagnostic accuracy. Finally, a standardized API is designed for data transmission (image upload, command issuance) and result analysis (defect type, location, confidence level, etc.) between edge devices and the cloud-based VLM, ensuring system integration.

[0149] When the network connection between the edge device and the cloud server is interrupted or unstable, the image information is stored in the local cache of the edge device, and the backup defect detection mechanism is executed to obtain a second defect detection result as a supplementary verification method for the cloud detection result to ensure that basic defect detection is not interrupted.

[0150] To address network instability or to supplement cloud-based detection results, perform the following backup detection on the edge device:

[0151] Stain defect detection: The input is the photovoltaic module image that has been processed by inverse perspective correction, illumination compensation and denoising. The RGB color image is converted to HSV color space, and the pixel matrices of the H, S, and V channels are extracted respectively. The local threshold is calculated by the mean weighted method. The window size for calculating the local threshold is defined to adapt to the uneven illumination of the photovoltaic panel surface. A multi-channel adaptive segmentation algorithm is used to distinguish the normal photovoltaic panel surface from stains, bird droppings and other attachments based on hue, saturation and value features. The three binary images are fused according to logical rules, and the area covered by the three images is extracted according to the rules to improve the detection accuracy. Alternatively, a logical OR combination can be used according to the actual situation to ensure a more complete detection (such as Figure 4 ).

[0152] The defect detection results obtained by the edge backup detection mechanism are temporarily stored in the local cache. When the network connection is restored, the original locally cached images will be successfully uploaded to the cloud VLM for more detailed analysis to obtain the first defect detection result. After obtaining the detection results from the cloud VLM, the system will fuse the first defect detection result with the second defect detection result to obtain the final defect detection result.

[0153] The final defect detection results include two types of defects: stains and glass cracks. During fusion, the first defect detection result is determined to be glass cracks. If so, the final defect detection result is glass cracks. Otherwise, for stains, the edge-side backup mechanism (second defect detection result) is primarily based on color space and is more sensitive to stains (such as bird droppings and dirt). If the VLM detects stains with a confidence level ≥ the third threshold, the VLM is used. If the VLM does not detect stains or detects stains with a confidence level less than the third threshold, but the edge-side HSV color space analysis detects a stain signal, the edge-side result is used. For glass cracks, the edge-side color space backup mechanism has weaker detection capabilities, and this type of defect relies entirely on the VLM.

[0154] To ensure the real-time performance of the system, an asynchronous pipeline architecture can be adopted during deployment. The image acquisition, image preprocessing, edge detection and other tasks when the cleaning robot stopped the previous time are executed in parallel with the image acquisition when it stops the next time. A double-buffered queue or a similar mechanism can be used to manage the image data to be processed, thereby reducing the waiting time between tasks and achieving almost zero-wait data flow. If the edge device hardware supports it (for example, it includes an NPU), some edge model inference tasks (such as the computationally intensive part of template matching or color segmentation) can be assigned to dedicated hardware cores for execution to further accelerate processing. Similarly, VLM inference in the cloud can also use GPUs or dedicated AI accelerators to improve efficiency.

[0155] The intelligent collaborative image defect detection mechanism described above significantly improves the stability and accuracy of defect detection in the complex and dynamic environments of photovoltaic power plants. This mechanism not only ensures accuracy for rapid industrial deployment but also effectively overcomes the shortcomings of existing single-detection technologies, which are susceptible to uncontrollable factors such as ambient lighting variations and insufficient dataset diversity. By leveraging the complementary advantages of multimodal and multi-stage approaches, it provides a highly robust and reliable intelligent photovoltaic module defect detection solution.

[0156] This invention effectively solves the problem of rapid and accurate identification of missing and deformed photovoltaic panels through an innovative dynamic template matching algorithm based on spatial position composition. Through a three-stage collaborative framework between the edge and the cloud, it cleverly combines the rapidity of template matching, the targeted nature of color space analysis, and the sophisticated analytical capabilities of visual language models. The dynamic template update mechanism enhances the system's robustness to gradual environmental changes and transient interference. The timing optimization of inverse perspective correction and specific optimization strategies for VLM (such as dynamic prompt injection) are both important innovative components of this patented solution, together forming an efficient, reliable, lightweight photovoltaic module defect detection solution that adapts to dynamic and complex scenarios. This technical solution has good engineering feasibility and scalability, adapts to actual industrial application needs, and promotes the improvement of the intelligent and automated level of photovoltaic power station operation and maintenance.

[0157] Table 1 shows the comparison of the accuracy of detecting all types of defects using the VLM model (without using reference images), VLM (using reference images), HSV multi-channel adaptive algorithm, and Deficiency-TM (spatial composition contour template matching) in the present invention.

[0158] surface

[0159]

[0160] As can be seen from the table above, the system, which combines the advantages of all models, can achieve good detection accuracy for all categories in both disconnected and connected conditions. This demonstrates that the system can effectively handle the complex safety inspection tasks involved in on-site cleaning of photovoltaic power plants, significantly improving detection accuracy and reliability.

Claims

1. A lightweight end-to-end defect detection method based on a rail-mounted photovoltaic cleaning robot, characterized in that: The following steps are involved: Step 1. The photovoltaic cleaning robot runs in a linear cycle on a long strip of photovoltaic panels arranged in a straight line according to the following preset rules: stop, run a distance L1, stop, and run a distance L2; The photovoltaic cleaning robot includes an edge computing device and an image acquisition device. The viewing angle of the image acquisition device is adjusted so that the width of the photovoltaic panels in its field of view is not less than the width of the photovoltaic cleaning robot, and the length of the photovoltaic panels in its field of view is not less than the total length L of n photovoltaic panels arranged in a line, and L is not less than the sum of L1 and L2; During the stopping, an original image of the photovoltaic panel surface is acquired, and the position of the photovoltaic cleaning robot relative to the photovoltaic panel array is used as the position corresponding to the original image; The original image includes a trapezoidal image of photovoltaic panels arranged in a line with a length of L; Step 2. After the original image is preprocessed for the first time, a ROI region is obtained by using the four vertices of the trapezoidal image region; the frame line image and grid line image of the photovoltaic panel within the ROI region are extracted to form a spatial composition contour map corresponding to the original image; The contour arrangement features formed by the target photovoltaic panel and its adjacent photovoltaic panels in the ROI area are used as matching objects; the missing state of the target photovoltaic panel changes the characteristics of the spatial composition, making the missing state recognizable; Step 3. On the edge computing device, the spatial composition outline is matched with a template at a corresponding position and a template at a distance L1 from the corresponding position in a preset template library to determine whether there is a risk. Step 4. If there is a risk, the photovoltaic cleaning robot stops moving; otherwise, based on the network connection status, the visual language model is used to perform defect detection to obtain a first defect detection result, or the edge computing device is used to execute the backup defect detection mechanism to obtain a second defect detection result.

2. The method according to claim 1, characterized in that The determination of whether there is a risk includes the following steps: Use the normalized correlation coefficient method to calculate the matching score, and make judgments based on the matching score; Match the spatial composition contour map with a template at a corresponding position and a template at a distance L1 from the corresponding position in a preset template library. If the matching score between at least one template and the spatial composition contour map reaches a preset first threshold, it indicates that the photovoltaic panel in the original image is not missing or deformed; otherwise, it indicates that there is a risk. The risk mentioned above refers to the risk that the photovoltaic cleaning robot may not be able to clean properly or fall off the photovoltaic panel due to missing or deformed photovoltaic panels; The template includes: a spatial composition outline template generated based on a trapezoidal image of a plurality of photovoltaic panels without defects and deformation arranged in a straight line with a length of L.

3. The method according to claim 2, characterized in that The calculation process of the matching score includes the following steps: The normalized correlation coefficient R(x, y) between the spatial composition outline image I and each template T is calculated according to the following formula: ； ； ； Where (x, y) is the reference coordinate of the starting point of the current match in the spatial composition contour image I, (i, j) is the offset relative to the reference coordinate, and its value range is i∈(0, w−1), j∈(0, h−1), where w and h are the width and height of the template T, respectively. I(x, y) represents the pixel value of a point in the sub-block area of ​​the spatial composition contour image I with (x, y) as the upper left corner and the same size as the template image T; μ I is the grayscale mean of the current sub-block area, μ T is the grayscale mean of the template image T; I(x+i, y+j) represents an image in I that is consistent with T o The pixel value of a point in a sub-block of the same size, with its upper left corner at (x, y) and its lower right corner at (x+w-1, y+h-1), will be the same as T o Similarity is calculated pixel by pixel; T(i, j) corresponds one-to-one to I(x+i, y+j), representing the pixel value at offset (i, j) in the template image; the value range of R(x, y) is [-1, 1], and the closer the value is to 1, the higher the match.

4. The method according to claim 1, characterized in that Step 3 also includes the following steps: When the matching score is greater than the first preset threshold, the matching area with the highest matching score is extracted, and the Mahalanobis distance D between the area and the corresponding template is calculated; if D is less than the preset second threshold, the pixel value, mean and variance of the corresponding template are fused and updated using the weighted sliding average method; if D is not less than the preset second threshold, the old template is retained, and the image is marked as having no missing parts but containing instantaneous interference.

5. The method according to claim 1, characterized in that The image acquisition device is provided with a polarizing plate; the first round of preprocessing includes illumination compensation and noise filtering; The image acquisition, first round of preprocessing and backup defect detection mechanism during the previous cleaning robot stop are executed in parallel with the image acquisition during the next stop. A double-buffered queue is used to manage the image data to be processed, thereby reducing the waiting time between tasks.

6. The method according to claim 1, characterized in that In step 4, the method of performing defect detection using a visual language model based on the network connection status to obtain a first defect detection result, or performing a backup defect detection mechanism using an edge computing device to obtain a second defect detection result, specifically includes: When the edge computing device is connected to the cloud server network normally, the image determined to be risk-free is input into the cloud visual language model, and defect detection is performed on the image in combination with the photovoltaic defect semantic information to obtain the first defect detection result; When the network connection is abnormal, images determined to be risk-free are cached on the edge computing device, and a backup detection mechanism is executed. The backup detection mechanism includes stain defect detection based on color space to obtain a second defect detection result; and after the network connection is restored to normal, the cached image is uploaded to the visual language model on the cloud to obtain a first defect detection result, and the first defect detection result and the second defect detection result are fused to obtain a final defect detection result.

7. The method according to claim 6, characterized in that Before inputting the risk-free image into the cloud-based visual language model, the following steps are also included: The images determined to be risk-free are subjected to a second round of preprocessing, which includes an inverse perspective transformation.

8. The method according to claim 6, characterized in that The fusing of the first defect detection result and the second defect detection result comprises the following steps: The final defect detection results include two types of defects: stains and glass cracks; Determine whether the first defect detection result is glass breakage; if so, the final defect detection result is glass breakage; otherwise: Determine whether VLM detects stains. If not, the second defect detection result is adopted. It is determined whether the confidence level of the stain detected by the VLM is less than a third threshold. If so, the second defect detection result is adopted; if not, the first defect detection result is adopted.

9. The method according to claim 1, characterized in that In step 1, the value of n is 3, 4, or 5; In step 2, the frame line image and grid line image of the photovoltaic panel in the ROI area are extracted, including: processing the image of the ROI area through an edge detection algorithm and a Hough line segment detection algorithm to obtain the frame line image and grid line image of the photovoltaic panel in the ROI area.

10. The method according to claim 1, characterized in that The visual language model is the Qwen-VL model, and is fine-tuned using a dataset containing photovoltaic module images and their corresponding defect descriptions. Said defects include broken glass, bird droppings, and dirt; The prompt word framework of the visual language model includes the following dimensions: Spatial dimension: description of the relative position of the defect on the component; Morphological dimension: crack direction or stain diffusion morphological characteristics; Time series dimension: Compare and analyze with historical test results.

Images