Detection method for detecting string gap in real time of photovoltaic cleaning vehicle

Abstract

The invention discloses a detection method for detecting string gaps in real time of a photovoltaic cleaning vehicle, and relates to the technical field of photovoltaic detection. According to the method, an original image is obtained through multiple cameras and preprocessed, and linear segment data are output through LSD linear detection; the real-time position and the driving direction of the cleaning vehicle are obtained by means of an RTK system, and corresponding template images are loaded for matching; fusing and filtering the detection result, and retaining a front related straight line segment; and intercepting the sub-images based on the linear segment features, and calculating a gray difference to judge the gap. The method improves the detection accuracy and efficiency, adapts to different working conditions, and has universality and flexibility; the detection result can provide an instruction for cleaning the rolling brush, the cleaning effect is improved, the operation and maintenance cost is reduced, and the photovoltaic power generation efficiency is guaranteed.

Description

Technical Field

[0001] The present application belongs to the field of photovoltaic detection technology, and in particular relates to a method for detecting gaps in photovoltaic strings in real time using a photovoltaic cleaning vehicle. Background Art

[0002] As a highly promising renewable energy source, solar energy is gaining increasing attention and favor worldwide due to its significant advantages, including unlimited reserves, widespread distribution, clean utilization, and economical practicality. With the deepening implementation of the global sustainable development strategy, vigorously developing the photovoltaic industry and actively exploiting solar energy resources have become important components of national energy strategies and have received unprecedented attention worldwide.

[0003] However, photovoltaic power generation technology still faces numerous challenges in practical application. Compared to traditional power generation methods, the efficiency of photovoltaic power generation is significantly affected by external factors such as the environment, climate, and topography. Among these, the accumulation of pollutants such as dust and debris on the surface of photovoltaic modules is a major factor contributing to reduced power generation efficiency. Specifically, the theoretical power generation efficiency of photovoltaic modules can reach 25%, but in actual operation it is typically only 23%. When the modules are contaminated, their power generation efficiency drops sharply to 17% to 18%. Therefore, maintaining the cleanliness of photovoltaic modules is crucial to maintaining the high efficiency of photovoltaic power plants.

[0004] As photovoltaic power plants continue to expand, the number of modules often reaches millions, covering millions of acres, and installed capacity continues to climb annually. Against this backdrop, cleaning photovoltaic modules has become an extremely laborious and labor-intensive task in the operation and management of photovoltaic power plants. Traditional cleaning methods rely primarily on manual inspection and operation, which is not only inefficient but also prone to missed inspections, making it difficult to ensure comprehensive and thorough cleaning. Summary of the Invention

[0005] The purpose of this application is to provide a real-time detection method for photovoltaic panel gap detection by a cleaning vehicle. The method uses a multi-step fine detection process to acquire and pre-process original images with multiple cameras, perform LSD line detection and fusion filtering, and calculate grayscale differences based on straight line segment features to determine the gaps between the strings. The method combines the RTK system to acquire real-time position and driving direction to accurately load template image matching, thereby improving the accuracy, adaptability and pertinence of photovoltaic panel gap detection and reducing misjudgments and missed judgments.

[0006] To achieve the above objectives, the present invention provides a method for detecting gaps in photovoltaic strings in real time using a photovoltaic cleaning vehicle. The photovoltaic cleaning vehicle is equipped with multiple cameras and an RTK system. The original images are acquired by the multiple cameras and pre-processed to obtain processed images.

[0007] Perform LSD line detection on the processed image and output straight line segment data;

[0008] The RTK system is used to obtain the RTK data of the photovoltaic cleaning vehicle and convert the RTK data into the UTM coordinate system to obtain the real-time position status of the photovoltaic cleaning vehicle and determine the driving direction of the photovoltaic cleaning vehicle;

[0009] Load the corresponding template image according to the driving direction of the photovoltaic cleaning vehicle, normalize the template image and the processed image, and calculate the feature descriptor of the template image; traverse the processed image through a sliding window, calculate the similarity, and determine the best matching position;

[0010] The straight line segment data is fused and filtered to retain the straight line segments related to the gaps in the photovoltaic panels in front of the photovoltaic cleaning vehicle;

[0011] Based on the length and inclination angle of the straight line segment, a sub-image is intercepted from the processed image, and the inclination angle of the straight line segment in the sub-image is calibrated to zero degrees; the sub-image is divided into two parts, upper and lower parts, according to the straight line segments in the sub-image, and the grayscale difference between the upper and lower parts is calculated. If the grayscale difference is greater than a first preset threshold, it is considered that there is a group gap in the straight line segment; if the grayscale difference is less than or equal to the first preset threshold, it is considered that there is no group gap in the straight line segment.

[0012] The above method according to the embodiment of the present application may also have the following additional technical features:

[0013] Furthermore, the original images are acquired through the multi-channel cameras and pre-processed to obtain the processed images, including: connecting the multi-channel cameras to the domain controller through the GMSL coaxial cable, and saving the configuration parameters of the multi-channel cameras as a YAML configuration file; starting the ROS node and opening multiple threads, each of the multiple threads corresponding to one of the multi-channel cameras; using the frame capture interface function provided by the SDK of the multi-channel camera, periodically capturing the frame image in the BGRA format, converting the frame image into a three-channel image in the RGB format, and performing dedistortion and grayscale processing on the three-channel image to obtain the processed image, and publishing the processed image in real time according to the configuration parameters in the YAML configuration file.

[0014] Furthermore, the frame image is converted into a three-channel image in RGB format, and the three-channel image is subjected to dedistortion processing and grayscale processing to obtain a processed image, including: converting the frame image in BGRA format into a three-channel image in RGB format through a CUDA function of OpenCV's color space conversion, and dedistorting and grayscale processing the three-channel image in RGB format according to distortion correction parameters of multiple cameras to obtain a processed image.

[0015] Furthermore, LSD line detection is performed on the processed image to output straight line segment data, including: Gaussian downsampling of the processed image, calculating the gradient value and direction of each point in the processed image, and sorting all points according to the size of the gradient value; setting the status of points whose gradient values ​​are less than a second preset threshold to USED, setting the status of the remaining points to UNUSED, and extracting points whose status is set to UNUSED; setting a maximum tolerance value for direction change, performing region growing processing on the points whose status is set to UNUSED, and obtaining the density of homogeneous points. If the density of homogeneous points is less than the truncation threshold, the region is truncated; if the density of homogeneous points is greater than the truncation threshold, calculating the error control function NFA, if the NFA is less than the error control threshold, outputting straight line segment data, and detecting whether there are unprocessed points whose status is set to UNUSED. If so, continue processing; if not, outputting the final straight line segment data RECT; if the NFA is greater than or equal to the error control threshold, no processing is required.

[0016] Furthermore, the RTK data of the photovoltaic cleaning vehicle is obtained through the RTK system, and the RTK data is converted into a UTM coordinate system to obtain the real-time position status of the photovoltaic cleaning vehicle, and the driving direction of the photovoltaic cleaning vehicle is determined, including: obtaining the RTK data of the photovoltaic cleaning vehicle through the RTK system, and converting the RTK data into a UTM coordinate system to obtain the real-time position status of the photovoltaic cleaning vehicle; obtaining the relative position change of the UTM coordinate of the photovoltaic cleaning vehicle within a time period of 2 seconds, and calculating the unit vector cosine angle of the two relative points; if the unit vector cosine angle is greater than zero, the driving direction of the photovoltaic cleaning vehicle is from west to east; if the unit vector cosine angle is less than zero, the driving direction of the photovoltaic cleaning vehicle is from east to west.

[0017] Furthermore, a corresponding template image is loaded according to the driving direction of the photovoltaic cleaning vehicle, the template image and the processed image are normalized, and a feature descriptor of the template image is calculated; the processed image is traversed through a sliding window, the similarity is calculated, and the best matching position is determined, including: loading a corresponding template image according to the driving direction of the photovoltaic cleaning vehicle, normalizing the template image and the processed image, and calculating the feature descriptor of the template image; defining a sliding window of the same size as the template image on the processed image, and traversing the sliding window on the processed image; for each window position, extracting the image area within the window, calculating the similarity between the image area and the template image through a similarity measurement method, determining the image area with the highest similarity as the best matching position, and judging whether the highest similarity is higher than a third preset threshold value; if higher, the match is successful, and the best matching position and the highest similarity value are output; if lower than or equal to, no processing is required.

[0018] The method for detecting gaps in photovoltaic strings in real time using a photovoltaic cleaning vehicle provided in the embodiments of the present application has the following beneficial technical effects compared to the prior art:

[0019] The embodiment of the present application obtains the original image through multiple cameras and performs preprocessing to remove noise and distortion, improve image quality, and provide clear and accurate materials for subsequent detection; then performs LSD line detection and outputs line segment data, which can accurately identify the line features in the image and help find the line segments related to the photovoltaic panel gaps; the detection results are then fused and filtered to retain the relevant line segments in the driving direction, further improving the targeted detection; finally, based on the line segment features, a sub-image is intercepted and the grayscale difference is calculated. The presence of a string gap is determined when the grayscale difference is greater than a preset threshold. This multi-step fine detection process greatly improves the accuracy of photovoltaic panel gap detection, can effectively identify actual gaps, and reduce false positives and missed detections; the RTK system is used to obtain the real-time position status of the photovoltaic cleaning vehicle and determine the driving direction, and the corresponding template image is loaded according to the driving direction for matching; since the presentation status of the photovoltaic panel may be different under different driving directions, this processing method combining real-time position and direction makes template matching more accurate, can more accurately detect photovoltaic panel gaps related to the driving direction, and improves the adaptability and accuracy of detection under different working conditions.

[0020] In the image acquisition and preprocessing stage, the embodiment of the present application connects multiple cameras to the domain controller through the GMSL coaxial cable, and starts the ROS node to open multiple threads, each thread corresponding to one camera, thereby realizing parallel processing of multiple cameras; at the same time, the frame capture interface function provided by the SDK of the multiple cameras is used to regularly capture frame images and perform real-time processing and publishing, thereby ensuring the real-time nature and high efficiency of processing of image data, being able to quickly acquire and process a large amount of image data, and improving the overall detection efficiency; in LSD line detection, Gaussian downsampling is performed on the processed image to reduce the amount of data and improve the processing speed; the points are sorted by calculating the gradient value and direction, the state is set and the unprocessed points are taken out, and region growing processing is performed, and whether to output the straight line segment data is determined based on the density of the isotropic points and the error control function NFA. This optimized algorithm process can quickly and effectively extract straight line segment features, reduce unnecessary calculations, and improve detection efficiency.

[0021] The embodiment of the present application can adapt to the characteristic changes of photovoltaic panels under different driving directions by loading the corresponding template image according to the driving direction of the photovoltaic cleaning vehicle; this flexible template matching method enables the system to be applied to photovoltaic power stations with different layouts and installation directions, thereby improving the versatility and adaptability of the system; the configuration parameters of multiple cameras are saved as a YAML configuration file and published in real time according to the parameters in the configuration file, so that the system can flexibly adjust the camera parameters such as resolution, frame rate, etc. according to actual needs to adapt to different detection scenarios and requirements, thereby enhancing the flexibility and scalability of the system.

[0022] After accurately detecting the gaps in the photovoltaic panels, the embodiment of the present application can provide an instruction basis for the lifting and lowering action of the cleaning roller brush, so that the cleaning roller brush can accurately align the gaps for cleaning, thereby improving the cleaning effect, ensuring the cleanliness of the photovoltaic panels, and thus improving the efficiency of photovoltaic power generation; through real-time video stream detection, photovoltaic panel gap problems can be discovered in a timely manner, avoiding the accumulation of problems leading to more serious failures, and reducing the cost of subsequent maintenance and replacement of photovoltaic panels; at the same time, the efficient detection process also reduces the workload and time cost of manual detection, improves the operation and maintenance efficiency of photovoltaic power stations, and reduces the overall operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a method for detecting gaps between photovoltaic strings in real time using a photovoltaic cleaning vehicle according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some, rather than all, structures related to the present application are shown in the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] As used herein, the terms "comprise," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0026] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0027] like Figure 1 As shown, the embodiment of the present application provides a method for detecting gaps in photovoltaic strings in real time using a photovoltaic cleaning vehicle, comprising the following steps:

[0028] Step 101: A photovoltaic cleaning vehicle is equipped with multiple cameras and an RTK system. The original image is acquired through the multiple cameras and pre-processed to obtain a processed image.

[0029] Step 101 describes how a PV cleaning vehicle uses multiple cameras to capture raw images, which are then processed through a series of preprocessing operations to produce the final processed image. This process forms the foundation of the entire PV cleaning vehicle-based real-time video stream detection method for string gaps, providing high-quality image data for analysis in subsequent steps.

[0030] In this embodiment, multiple cameras are connected to a domain controller via GMSL (Gigabit Multimedia Serial Link) coaxial cable. A domain controller, in "domain" mode, is a server responsible for authenticating every computer and user connected to the network. It acts like a security guard for an organization and is referred to as a "Domain Controller (DC)." GMSL coaxial cable offers high speed and bandwidth, meeting the requirements for real-time transmission of large amounts of image data from multiple cameras, ensuring stable transmission of image data.

[0031] Save the configuration parameters of multiple cameras as YAML (YAML Ain't Markup Language) configuration files. YAML is an intuitive, human-readable data serialization format that facilitates the management and modification of camera parameters. These configuration parameters may include camera resolution, frame rate, exposure time, etc. The configuration files allow for flexible adjustment of camera operating conditions to suit different inspection environments and requirements.

[0032] Start a ROS (Robot Operating System) node. ROS is an open-source framework that supports robotics research and development. It provides a wealth of tools and libraries to facilitate robotics software development. After starting a ROS node, you can use ROS's communication mechanisms to enable data exchange and collaborative work between different modules.

[0033] Start multiple threads, one for each camera. Multithreading improves image acquisition efficiency, ensuring that each camera can independently and efficiently capture and process images. For example, if there are four cameras, start four threads, one for each camera, to avoid delays and blockages that may occur with single-threaded processing.

[0034] The multi-channel camera SDK (Software Development Kit) provides a frame capture interface function to periodically capture BGRA format frames. SDKs are software development kits provided by camera manufacturers that include various camera function interfaces to facilitate development. Periodic image capture ensures continuous and stable video stream data, providing sufficient samples for subsequent processing.

[0035] Convert the captured BGRA frame image to a three-channel RGB image. The BGRA format is an image format that contains blue, green, red, and alpha channels, while the RGB format is a commonly used three-channel color image format. The conversion is performed using the CUDA function of OpenCV's color space conversion. OpenCV (Open Source Computer Vision Library) is a cross-platform computer vision library that provides a rich set of image processing and computer vision algorithms. CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model launched by NVIDIA. Leveraging the parallel computing capabilities of GPUs can greatly increase the speed of image processing.

[0036] Dedistort three-channel RGB images using the distortion correction parameters of multiple cameras. Camera lens distortion during the imaging process can cause distortion in the shape of objects in the image. Using the distortion correction parameters, images can be corrected to restore the true shape of objects, improving image accuracy and usability.

[0037] Grayscale the dedistorted image to obtain the processed image. Grayscale conversion is the process of converting a color image into a grayscale image. Grayscale images contain only brightness information, not color information. Grayscale processing can simplify image data and reduce computational complexity. For some inspection tasks that focus solely on object outlines and structures, grayscale images are sufficient.

[0038] The processed image is published in real time according to the configuration parameters in the YAML configuration file. This allows the processed image data to be delivered in a timely manner, ensuring the real-time and consistency of the entire detection process. For example, the image can be published to a specific ROS topic, and other modules that have subscribed to the topic can obtain the processed image data in real time. Publishing the video stream as a topic means that the domain controller acts as a publisher and publishes the video stream data obtained from multiple sensors in a specific format and method to a specific "topic". This topic is like a public "channel", and other nodes that need to obtain this video stream data can act as subscribers and receive the video stream data by subscribing to this topic.

[0039] In summary, image acquisition and preprocessing in step 101 are critical steps in the entire string gap detection method. High-quality image data is the foundation for subsequent steps such as LSD line detection and template matching. Unclear image acquisition and inadequate preprocessing can lead to inaccurate detection results, impacting the accuracy and reliability of string gap detection. Standardized camera connection, configuration, image capture, format conversion, dedistortion, and grayscale processing ensure that image data suitable for subsequent analysis is obtained, improving the performance and effectiveness of the entire detection method.

[0040] Step 102: Perform LSD line detection on the processed image and output line segment data.

[0041] In this embodiment of the present application, step 102 performs line segment detection using the Line Segment Detector (LSD) on the processed image and outputs line segment data. This step aims to accurately extract line segment information from the pre-processed image, providing basic data for subsequent photovoltaic panel gap detection. LSD is a line detection algorithm that can quickly and accurately detect straight lines in an image. The algorithm determines the direction and position of a line by analyzing the gradient information in the image.

[0042] The embodiments of the present application perform Gaussian downsampling on the processed image primarily to reduce the image data size and computational complexity, while also smoothing the image to a certain extent and reducing the impact of noise on line detection. Gaussian downsampling blurs the image by applying a Gaussian filter and then downsampling the image at a specific sampling rate. The Gaussian filter can effectively smooth the image and remove high-frequency noise, while the sampling rate selection requires finding a balance between preserving important image features and reducing the data size.

[0043] Calculating the gradient value and direction at each point in an image is intended to determine the direction of the most significant grayscale change, as straight lines typically correspond to areas of significant grayscale change. Common gradient calculation methods, such as the Sobel operator, calculate the rate of grayscale change in the horizontal and vertical directions of an image to determine the gradient value and direction at each point. The gradient value reflects the intensity of the grayscale change, while the direction indicates the direction of the most rapid grayscale change.

[0044] All points are sorted by gradient value, and points with gradient values ​​less than a second preset threshold are set to USED, while the remaining points are set to UNUSED. This is done to filter out points that may contribute to line detection, reducing the computational effort in subsequent processing. Points with larger gradient values ​​typically correspond to edges or lines in the image and are therefore more likely to belong to straight line segments. By setting a second preset threshold, points with smaller gradient values ​​that do not contribute much to line detection can be excluded, thereby improving detection efficiency.

[0045] Region growing is performed on points set to UNUSED to cluster points that may belong to the same line segment, forming a support region for the line segment. Region growing begins with a seed point and, based on certain growth criteria (such as the maximum tolerance for directional changes), adds adjacent points that meet the criteria to the region. By setting the maximum tolerance for directional changes, the directional consistency of region growing can be controlled, resulting in a more accurate support region for the line segment.

[0046] Calculating the homogeneous point density and determining whether it is less than the cutoff threshold is to determine whether the area is dense enough to support the existence of a straight line segment. The homogeneous point density reflects the density of points in the area. If the homogeneous point density is less than the cutoff threshold, it means that the points in the area are too sparse to form a reliable straight line segment, so the area is cut off.

[0047] If the density of homogeneous points exceeds the cutoff threshold, the error control function (NFA) is calculated to assess the credibility of the line segment and determine whether it is a true line segment. NFA (Number of False Alarms) is a statistical method used to evaluate line segment detection results. It takes into account factors such as the length and gradient of the line segment and calculates the probability of a similar line segment appearing in a random image. If the NFA is less than the error control threshold, the line segment has a high credibility and can be considered a true line segment.

[0048] Outputting line segment data and checking for any unprocessed points set to UNUSED ensures that all possible line segments have been processed. If the NFA is less than the error control threshold, the line segment data is output. Next, checking for any unprocessed points set to UNUSED is performed. If so, processing continues until all points have been processed. Finally, outputting the RECT data for all detected line segments. If the NFA is greater than or equal to the error control threshold, no further processing is required.

[0049] In summary, step 102 achieves the goal of accurately extracting straight line segment information from the image by performing a series of operations on the processed image, including Gaussian downsampling, calculating gradient values ​​and directions, point sorting and state setting, region growing, determining homogeneous point density, calculating the error control function (NFA), and outputting and subsequently processing the straight line segment data. This straight line segment data will provide an important basis for subsequent photovoltaic panel gap detection, helping to improve detection accuracy and reliability.

[0050] Step 103: Obtain RTK data of the photovoltaic cleaning vehicle through the RTK system, convert the RTK data into a UTM coordinate system, obtain the real-time position status of the photovoltaic cleaning vehicle, and determine the driving direction of the photovoltaic cleaning vehicle.

[0051] In the embodiment of the present application, step 103 is to obtain the RTK data of the photovoltaic cleaning vehicle through the RTK system, and convert these data into the UTM coordinate system, thereby obtaining the real-time position status of the photovoltaic cleaning vehicle and determining its driving direction.

[0052] The RTK (Real-Time Kinematic) system is a high-precision positioning technology that provides precise real-time location information for photovoltaic cleaning vehicles. RTK data provides the basis for subsequent position determination and driving direction assessment. The RTK system receives satellite signals and utilizes carrier phase differential technology to achieve centimeter- or even millimeter-level positioning accuracy. The RTK receiver on the photovoltaic cleaning vehicle receives and processes these signals in real time, generating RTK data containing information such as position, speed, and time.

[0053] Converting RTK data to the UTM (Universal Transverse Mercator) coordinate system converts geographic coordinates (such as longitude and latitude) into planar coordinates for ease of subsequent calculations and analysis. The UTM coordinate system is a rectangular coordinate system based on the Transverse Mercator projection. It divides the Earth's surface into multiple 6-degree wide projection zones, each with a separate central meridian as the coordinate origin. Using specific conversion formulas, geographic coordinates in RTK data can be converted to planar coordinates in the UTM coordinate system.

[0054] The real-time location of the PV cleaning vehicle is determined to provide a spatial reference for determining its current location and detecting gaps. After conversion to the UTM coordinate system, the location information in the RTK data can be used to directly represent the vehicle's real-time location. This location information typically includes the horizontal coordinate (distance easting), the vertical coordinate (distance northing), and altitude.

[0055] The purpose of determining the direction of travel of the PV cleaning vehicle is to load the corresponding template image based on the direction of travel for subsequent template matching and gap detection. By obtaining the relative position change of the PV cleaning vehicle's UTM coordinates within a 2-second interval, the vehicle's movement direction and distance during this period can be calculated. Specifically, the cosine angle between the unit vectors of two relative points can be calculated, which reflects the changing trend of the cleaning vehicle's travel direction.

[0056] If the cosine angle of the unit vector is greater than zero, it means that the cleaning vehicle mainly moves eastward during this period, so its driving direction is judged to be from west to east.

[0057] If the cosine angle of the unit vector is less than zero, it means that the cleaning vehicle mainly moves to the west during this period, so its driving direction is judged to be from east to west.

[0058] If the unit vector cosine angle is zero, the PV cleaning vehicle may be stationary or traveling at a constant speed in a straight line. Since the embodiments of this application focus on determining the direction of travel and do not mention speed information, it is impossible to determine whether the vehicle is stationary or traveling at a constant speed in a straight line based solely on this angle value. In actual applications, it is necessary to combine other sensor data (such as a speed sensor) to comprehensively determine the driving state of the PV cleaning vehicle.

[0059] In summary, step 103 uses the RTK system to acquire RTK data from the PV cleaning vehicle and converts this data into a UTM coordinate system. This data then determines the vehicle's real-time position and direction of travel. This step provides important spatial reference and directional information for subsequent template matching and gap detection, and is crucial for ensuring the accuracy and reliability of the entire detection method.

[0060] Step 104 , load the corresponding template image according to the driving direction of the photovoltaic cleaning vehicle, normalize the template image and the processed image, and calculate the feature descriptor of the template image; traverse the processed image through a sliding window, calculate the similarity, and determine the best matching position.

[0061] In this embodiment, step 104 involves loading a template image based on the direction of travel of the PV cleaning vehicle, normalizing the template image and the processed image, calculating feature descriptors for the template image, and traversing the processed image using a sliding window to calculate similarity to determine the optimal matching location. This step utilizes template matching technology to accurately locate the area in the real-time video stream that most closely resembles the template image, providing a positioning foundation for subsequent PV panel gap detection.

[0062] Loading a template image based on the vehicle's travel direction ensures that the template image matches the layout and characteristics of the PV panels in that direction, thereby improving template matching accuracy. Because PV panels can exhibit different layouts and characteristics (such as tilt angle and arrangement) depending on the vehicle's travel direction, it's important to select the most appropriate template image for matching based on the vehicle's travel direction.

[0063] Normalizing the template image and processed image eliminates differences in scale, lighting, and other aspects of the images, making template matching more accurate and stable. Normalization typically involves operations such as image scaling and grayscale adjustment, converting the images to a uniform scale and grayscale range. This ensures that the template image and processed image are matched at the same scale, reducing matching errors caused by scale differences.

[0064] Calculating feature descriptors for the template image is to extract key feature information from the template image for subsequent similarity calculations. Feature descriptors are typically extracted from images using specific algorithms (such as SIFT and SURF) and describe local features within the image. By calculating the feature descriptors of the template image, they can be compared with the features in the processed image to find the most similar regions.

[0065] The goal of using a sliding window to traverse the processed image is to find the area within the image that is most similar to the template image, region by region. Sliding windows are a common image processing technique that traverses the entire image by moving a fixed-size window across the image. At each window position, the image region within the window is extracted and its similarity is calculated with the template image.

[0066] The goal of calculating the similarity between an image region and a template image and determining the optimal matching position is to find the image region that is most similar to the template image, i.e., the possible location of the photovoltaic panel. This similarity calculation is typically achieved using a specific metric (such as Euclidean distance or cosine similarity). For each image region extracted at each window position, its similarity to the template image is calculated, and the image region with the highest similarity is determined as the optimal matching position.

[0067] The system determines whether the highest similarity exceeds a third preset threshold. If so, the match is successful, and the best match location and highest similarity value are output. This ensures the reliability and accuracy of the matching results. The third preset threshold is a value set based on experimentation and experience, and is used to determine whether the similarity is high enough to consider a match successful. If the highest similarity exceeds this threshold, an area sufficiently similar to the template image is considered found, meaning the match is successful. The best match location and highest similarity value are output for subsequent processing. If it is lower than or equal to the threshold, no processing is required.

[0068] In summary, step 104 loads a corresponding template image based on the direction of travel of the PV cleaning vehicle, normalizes the template image and the processed image, calculates feature descriptors for the template image, and calculates similarity using a sliding window across the processed image to determine the optimal matching location. This step fully leverages the advantages of template matching technology, accurately identifying the area in the real-time video stream that most closely resembles the template image, providing an important positioning foundation for subsequent PV panel gap detection. Furthermore, by setting a third preset threshold to determine whether a match is successful, the reliability and accuracy of the matching results are further improved.

[0069] Step 105 , performing fusion filtering on the straight line segment data, and retaining the straight line segments related to the gaps of the photovoltaic panels in the driving direction of the photovoltaic cleaning vehicle.

[0070] In this embodiment of the present application, step 105 involves fusing and filtering the line segment data, retaining the line segments associated with the photovoltaic panel gaps in the direction of travel of the photovoltaic cleaning vehicle. This step aims to filter out the line segments truly associated with the photovoltaic panel gaps from the numerous line segments obtained by LSD line detection, providing an accurate data foundation for subsequent gap effectiveness assessment. In this embodiment of the present application, fusing and filtering involves calculating the cosine angle between the unit vector angle of the line segment and the vehicle's travel direction, retaining line segments with a result greater than 0.98.

[0071] LSD line detection may detect a large number of straight line segments in an image, many of which may not be related to gaps in photovoltaic panels. Fusion filtering can remove these irrelevant straight line segments, improving detection accuracy. Retaining straight line segments related to gaps in photovoltaic panels can reduce the computational complexity of subsequent processing and improve algorithm efficiency.

[0072] Based on the direction of travel of the PV cleaning vehicle, we can determine which straight lines are likely to be ahead. For example, if the PV cleaning vehicle is traveling from west to east, straight lines on the east side of the image that align with the orientation of the PV panels are more likely to be associated with gaps in the panels. In addition to direction of travel, other features such as the length, tilt angle, and position of the straight lines can also be used for filtering. For example, thresholds can be set to retain only straight lines with lengths within a certain range and tilt angles that match the PV panel installation angle.

[0073] For the retained straight line segments, further determination is needed to determine whether they are related to gaps in the photovoltaic panels. This can be achieved by analyzing the image features surrounding the straight line segment, for example, detecting whether there are significant grayscale differences or edge features on either side of the straight line segment. Because photovoltaic cleaning vehicles may encounter different situations during actual driving (such as changes in lighting and photovoltaic panel arrangement), the fusion filtering strategy may need to be dynamically adjusted. For example, thresholds or filtering conditions can be adjusted based on real-time image features or detection result feedback.

[0074] During the fusion and filtering process, the LSD line detection results need to be correlated and matched with information such as the PV cleaning vehicle's direction and location. This can be achieved through methods such as coordinate transformation and image registration. To improve the efficiency of fusion and filtering, the algorithm can be optimized and accelerated. For example, parallel computing and GPU acceleration can be used to speed up processing.

[0075] In summary, step 105 achieves the goal of selecting the true line segments related to PV panel gaps from among numerous line segments by fusing and filtering the line segment data and retaining the line segments associated with the PV panel gaps in the direction of travel of the PV cleaning vehicle. This step not only improves detection accuracy but also reduces the computational complexity of subsequent processing, providing an accurate data foundation for subsequent gap effectiveness assessment. In practical applications, the fusion and filtering strategy needs to be dynamically adjusted to suit different environments and conditions.

[0076] Step 106: Based on the length and inclination angle of the straight line segment, a sub-image is captured on the processed image, and the inclination angle of the straight line segment in the sub-image is calibrated to zero degrees; the sub-image is divided into two parts, an upper part and an lower part, according to the straight line segment in the sub-image, and the grayscale difference between the upper and lower parts is calculated. If the grayscale difference is greater than a first preset threshold, it is considered that there is a group gap in the straight line segment; if the grayscale difference is less than or equal to the first preset threshold, it is considered that there is no group gap in the straight line segment.

[0077] In the embodiment of the present application, step 106 is directly related to the accurate identification of the string gap.

[0078] Sub-images are extracted from the processed image based on the length and tilt angle of the line segments. This allows the detection to focus on areas that may contain string gaps, reducing the computational complexity of subsequent processing and improving detection accuracy. The line segment data obtained through LSD line detection includes information such as the line segment's length and tilt angle. This information allows the accurate location and extraction of sub-images containing line segments from the processed image. When extracting sub-images, it is important to ensure that the sub-images are large enough to include the line segment and its surrounding context, but not too large to avoid introducing unnecessary noise.

[0079] The tilt angle of the line segments in the sub-graph is calibrated to zero degrees to simplify subsequent processing steps and make the division between the upper and lower parts more accurate and convenient. By rotating the sub-graph, the line segments are aligned with the horizontal direction, that is, the tilt angle is zero degrees. This allows the subsequent division between the upper and lower parts to proceed directly along the horizontal direction without considering the tilt angle of the line segments.

[0080] The purpose of dividing a sub-image into two parts based on the straight line segments within the sub-image is to calculate the grayscale difference between the two parts and thus determine whether there is a string gap in the line segment. After the straight line segment tilt angle is calibrated to zero degrees, the sub-image can be divided into two parts horizontally. When dividing the sub-image, ensure that the areas of the two parts are equal or similar to ensure more accurate and reliable grayscale difference calculation.

[0081] The grayscale difference between the upper and lower parts is calculated to determine whether a string gap exists within the line segment. The grayscale difference reflects the difference in brightness between the upper and lower image parts. If a gap exists within a photovoltaic string, the image brightness (i.e., grayscale value) above and below the gap may differ significantly. By calculating the grayscale difference and comparing it with a first preset threshold, it can be determined whether a string gap exists within the line segment.

[0082] If the grayscale difference is greater than a first preset threshold, the line segment is considered to have a string gap. This ensures accurate detection results and provides guidance for subsequent cleaning roller brush raising and lowering actions. The first preset threshold, determined based on experimental data or experience, determines whether the grayscale difference is large enough to indicate the presence of a string gap. If the grayscale difference is greater than this threshold, it is assumed that a string gap does exist in the line segment, requiring the detection result to be output and the appropriate processing action to be triggered. If the grayscale difference is less than or equal to the first preset threshold, the line segment is considered to have no string gap.

[0083] In summary, step 106 achieves accurate identification of PV string gaps through a series of operations, including extracting a sub-image based on line segment features, calibrating the line segment's tilt angle to zero degrees, dividing the sub-image into upper and lower parts based on the line segments, calculating the grayscale difference between the upper and lower parts, and determining string gaps. These operations not only improve detection accuracy but also reduce the computational complexity of subsequent processing, providing effective technical support for string gap detection using real-time video streams from PV cleaning vehicles.

[0084] It should be noted that, in the present application, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0085] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for detecting gaps in photovoltaic strings in real time using a photovoltaic cleaning vehicle, characterized in that: The photovoltaic cleaning vehicle is equipped with a multi-channel camera and an RTK system, and the original image is obtained by the multi-channel camera and pre-processed to obtain a processed image; Performing LSD line detection on the processed image and outputting line segment data; The RTK system is used to obtain the RTK data of the photovoltaic cleaning vehicle, and the RTK data is converted into a UTM coordinate system to obtain the real-time position status of the photovoltaic cleaning vehicle and determine the driving direction of the photovoltaic cleaning vehicle; Loading a corresponding template image according to the driving direction of the photovoltaic cleaning vehicle, normalizing the template image and the processed image, and calculating a feature descriptor of the template image; traversing the processed image through a sliding window, calculating similarity, and determining the best matching position; Performing fusion filtering on the straight line segment data to retain the straight line segments related to the photovoltaic panel gaps in front of the photovoltaic cleaning vehicle in the driving direction; Based on the length and inclination angle of the straight line segment, a sub-image is intercepted on the processed image, and the inclination angle of the straight line segment in the sub-image is calibrated to zero degrees; the sub-image is divided into two parts, upper and lower parts, according to the straight line segments in the sub-image, and the grayscale difference between the upper and lower parts is calculated. If the grayscale difference is greater than a first preset threshold, it is considered that there is a group gap in the straight line segment; if the grayscale difference is less than or equal to the first preset threshold, it is considered that there is no group gap in the straight line segment.

2. The method for detecting gaps in photovoltaic strings in real time by a photovoltaic cleaning vehicle according to claim 1, wherein: The method of acquiring original images through the multi-channel cameras and performing pre-processing to obtain processed images includes: The multi-channel camera is connected to the domain controller via a GMSL coaxial cable, and the configuration parameters of the multi-channel camera are saved as a YAML configuration file; the ROS node is started and multiple threads are opened, each of the multiple threads corresponds to one camera in the multi-channel camera; the frame image in BGRA format is captured regularly through the frame capture interface function provided by the SDK of the multi-channel camera, and the frame image is processed to obtain a processed image, and the processed image is published in real time according to the configuration parameters in the YAML configuration file.

3. The method for detecting the gap between photovoltaic strings in real time by a photovoltaic cleaning vehicle according to claim 2, characterized in that: The step of periodically capturing a frame image in a BGRA format and processing the frame image to obtain a processed image includes: The frame image in BGRA format is converted into a three-channel image in RGB format through the CUDA function of OpenCV's color space conversion. The three-channel image in RGB format is dedistorted and grayscaled according to the distortion correction parameters of the multi-channel camera to obtain a processed image.

4. The method for detecting gaps in photovoltaic strings in real time by a photovoltaic cleaning vehicle according to claim 1, wherein: The performing LSD line detection on the processed image and outputting line segment data includes: Performing Gaussian downsampling on the processed image, calculating the gradient value and direction of each point in the processed image, and sorting all points according to the magnitude of the gradient value; setting the status of points whose gradient values ​​are less than a second preset threshold to USED, setting the status of the remaining points to UNUSED, and extracting the points whose status is set to UNUSED; Set the maximum tolerance value of the direction change, perform region growing processing on the points whose status is set to UNUSED, and obtain the density of homogeneous points. If the density of homogeneous points is less than the truncation threshold, truncate the region; if the density of homogeneous points is greater than the truncation threshold, calculate the error control function NFA. If NFA is less than the error control threshold, output the straight line segment data, and detect whether there are unprocessed points whose status is set to UNUSED. If so, continue processing; if not, output the final straight line segment data RECT; if NFA is greater than or equal to the error control threshold, no processing is required.

5. The method for detecting gaps in photovoltaic strings in real time using a photovoltaic cleaning vehicle according to claim 1, wherein: The RTK system is used to obtain RTK data of the photovoltaic cleaning vehicle, and the RTK data is converted into a UTM coordinate system to obtain the real-time position status of the photovoltaic cleaning vehicle, and to determine the driving direction of the photovoltaic cleaning vehicle, including: The RTK system is used to obtain the RTK data of the photovoltaic cleaning vehicle, and the RTK data is converted into a UTM coordinate system to obtain the real-time position status of the photovoltaic cleaning vehicle; the relative position change of the UTM coordinate of the photovoltaic cleaning vehicle within a time period of 2 seconds is obtained, and the unit vector cosine angle of the two relative points is calculated; if the unit vector cosine angle is greater than zero, the driving direction of the photovoltaic cleaning vehicle is from west to east; if the unit vector cosine angle is less than zero, the driving direction of the photovoltaic cleaning vehicle is from east to west.

6. The method for detecting gaps in photovoltaic strings in real time using a photovoltaic cleaning vehicle according to claim 1, wherein: The step of loading a corresponding template image according to the driving direction of the photovoltaic cleaning vehicle, normalizing the template image and the processed image, and calculating a feature descriptor of the template image; Traversing the processed image through a sliding window, calculating similarity, and determining the best matching position, including: A corresponding template image is loaded according to the driving direction of the photovoltaic cleaning vehicle, the template image and the processed image are normalized, and the feature descriptor of the template image is calculated; a sliding window with the same size as the template image is defined on the processed image, and the sliding window is traversed on the processed image; for each window position, the image area within the window is extracted, and the similarity between the image area and the template image is calculated by a similarity measurement method, the image area with the highest similarity is determined as the best matching position, and it is judged whether the highest similarity is higher than a third preset threshold. If it is higher, the match is successful, and the best matching position and the highest similarity value are output; if it is lower than or equal to, no processing is required.

Images