Joint application strategy of laser radar and image recognition in clearance radar

Abstract

The invention discloses a combined application strategy of a laser radar and image recognition in a clearance radar, and relates to the technical field of wind power, and the strategy comprises the steps: monitoring the data size and discrete degree of ground data of the laser radar to judge the current weather condition, and starting image recognition to carry out auxiliary judgment when the weather condition is bad; when the weather condition is good, video images and laser radar data of the fan blades are collected, and an image recognition reference object database is established in combination with the corresponding laser radar data; when the weather condition is bad, collecting a current leaf video image and laser radar data, and performing comparative analysis on leaf features extracted from the video image and reference objects in the image recognition reference object database; and determining the clearance value and accuracy evaluation of the fan blade according to the comparison and analysis result and the laser radar data, the method can utilize the image information to carry out validity verification and supplement on the laser radar data, and greatly improves the reliability and accuracy of clearance monitoring in severe weather.

Description

Technical Field

[0001] This invention relates to the field of wind power technology, and in particular to a strategy for the combined application of lidar and image recognition in airspace radar. Background Technology

[0002] Wind turbines are a crucial component of the new energy sector. With advancements in wind turbine technology, blade length and tower height are continuously increasing in pursuit of higher wind energy utilization. However, the increased blade length makes them more susceptible to significant bending under complex wind conditions or extreme weather, leading to a reduction in the clearance between the blade tip and the tower (i.e., the "clearance value"). Once the clearance value falls below a safe threshold, a "tower-sweeping" accident (blade impacting the tower) is highly likely. Such accidents can range from minor blade damage to the complete failure of the wind turbine generator set, causing substantial economic losses and serious safety hazards. Therefore, real-time and accurate monitoring of the wind turbine blade clearance value is of paramount importance for ensuring safe turbine operation, extending equipment lifespan, and improving power generation efficiency.

[0003] Currently, most mainstream air clearance radars on the market employ the principle of multi-channel single-point laser ranging, calculating the clearance value by measuring the distance from the laser to the blade surface. However, this solution, which relies solely on laser radar, suffers from the following serious drawbacks and insurmountable technical bottlenecks in practical applications, especially under complex and severe weather conditions: Laser signal attenuation and failure: When the radar window is condensed, icy, dusty, or dirty, or blocked by foreign objects, the laser beam cannot be emitted or received normally, resulting in missing or severely distorted measurement data. In practical applications, especially in high humidity, low temperature, or dusty environments, this situation occurs frequently, rendering the lidar completely inoperable.

[0004] Severe interference under adverse weather conditions: In weather conditions such as heavy fog and dense fog, laser beams will be strongly scattered and reflected (i.e., "backlight") by fog particles during propagation.

[0005] "False target" interference: The laser may generate a backlight signal in the dense fog area before it reaches the blade, which is enough to trigger the receiver, causing the radar to misjudge the fog as the blade and measure a completely wrong distance value.

[0006] Ground signal loss: In dense fog, the laser beam that should reach the ground may be strongly reflected by the fog near the blade area, causing the radar to be unable to detect the ground reference signal. All measurement data fall within the possible range of blade distance, making the system unable to distinguish between blade signal and fog interference, and the clearance value calculation completely fails.

[0007] Signal submersion and confusion: Even if the laser can penetrate some of the fog to reach the blades, the continuous reflection signal generated by the fog can easily be misidentified as a blade signal or mixed in with the real blade signal, making it impossible for the system to accurately determine the start and end boundaries of the blade signal, resulting in a huge deviation in the calculation of the clearance value.

[0008] These problems indicate that the monitoring performance of existing technologies drops sharply or even fails completely when faced with severe weather conditions such as heavy fog, rain, snow, and sandstorms, creating "blind spots" in wind turbine safety monitoring and failing to meet the needs of all-weather, high-reliability safety monitoring. Summary of the Invention

[0009] This invention provides a strategy for the combined application of lidar and image recognition in airspace radar to solve the technical problems of low accuracy, poor reliability, or even complete failure of lidar measurement when used alone under adverse weather conditions, especially dense fog, patchy fog, and window pollution.

[0010] In view of the above problems, the present invention provides a joint application strategy of lidar and image recognition in airspace radar, including: The amount and dispersion of ground data from the lidar are monitored to determine the current weather conditions. When the weather conditions become severe to a preset level, image recognition is activated to assist in the judgment. When the weather conditions are good, video images of the wind turbine blades and synchronous lidar data are collected. The video images are processed to extract the blade motion features, and the corresponding lidar data is combined to establish an image recognition reference object database. When the weather conditions are severe, current leaf video images and lidar data are collected, and the leaf features extracted from the current video images are compared and analyzed with reference objects in the image recognition reference object database to evaluate the accuracy of lidar data and assist in calculating the leaf clearance value. Based on the results of the comparative analysis and the lidar data, the clearance value and accuracy assessment of the wind turbine blades are determined.

[0011] Preferably, after determining the reliability of the current lidar data, the method further includes: if the degree of conformity is lower than the first degree of conformity threshold, then the current lidar data is likely to be incorrect.

[0012] Preferably, after constructing the image recognition reference object database, the method further includes: periodically or automatically updating the image recognition reference object database according to weather changes.

[0013] The technical solution provided in this application has at least the following technical effects or advantages: By intelligently activating image recognition under adverse weather conditions and establishing and utilizing an image recognition reference object database, the system verifies and corrects lidar data or provides alternative measurements when lidar fails. This significantly improves the accuracy, reliability, and all-weather capability of wind turbine blade clearance monitoring under adverse weather conditions such as dense fog, rain, snow, and window pollution. It effectively overcomes the fatal defects of existing technologies that rely solely on lidar, such as "monitoring blind spots" and "prone to misjudgment." In particular, it addresses the problem of lasers generating backlight within the actual blade area, which can lead to the inability to detect the actual blade or determine whether the distance value corresponds to the actual blade. This provides technical support for ensuring the safe operation of wind turbine generators. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the combined application strategy of lidar and image recognition in the airspace radar of this invention. Detailed Implementation

[0015] This invention relates to a strategy for the combined application of lidar and image recognition in airspace radar to address the technical problems of low measurement accuracy, poor reliability, or even complete failure of lidar alone under adverse weather conditions (especially dense fog, patchy fog, window pollution, etc.).

[0016] The above technical solutions will be described in detail below with reference to the accompanying drawings and specific embodiments to better understand them. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments used only to explain the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Furthermore, it should be noted that, for ease of description, only the parts related to the present invention are shown in the drawings, not all of them.

[0017] like Figure 1 The flowchart of the joint application strategy of lidar and image recognition in airspace radar is shown, including the following steps: The amount and dispersion of ground data from the lidar are monitored to determine the current weather conditions; when the weather conditions become severe to a preset level, image recognition is activated to assist in the judgment.

[0018] Specifically, the air clearance radar continuously acquires environmental echo signals through its lidar module. When the wind turbine blades are not in the path of the laser beam, a portion of the laser beam (e.g., a specially designed ground detection beam, or the main detection beam that illuminates the ground during blade gaps) will illuminate the ground and generate echoes. The main control unit analyzes the quality and quantity of these ground echo signals in real time.

[0019] Specifically, the preset conditions for judging severe weather conditions can be based on one or more of the following aspects: (1) The amount of ground data is significantly reduced: for example, due to dense fog absorbing or scattering lasers, or the radar window being contaminated (such as condensation, icing, or dust), the laser cannot effectively reach the ground or receive ground echoes. (2) The distance values ​​of ground data fluctuate drastically and irregularly: for example, dense fog randomly generates strong echoes nearby, which are mistaken for ground signals, resulting in extremely unstable measured ground distance values. When the monitored conditions meet these preset conditions, it indicates that the measurement reliability of the lidar has significantly decreased. At this time, the image recognition module is automatically activated, and the operating mode is switched to the joint application mode of lidar and image recognition.

[0020] In some embodiments, the monitoring of the ground data volume and dispersion of the lidar to determine the current weather conditions includes: Monitor the amount of ground data received by each channel of the lidar per unit time and compare it with a first preset threshold. Monitor the dispersion of ground data distance values ​​measured by each channel of the lidar and compare them with a second preset threshold; When the amount of ground data is lower than the first preset threshold, or the dispersion of the ground data distance value is higher than the second preset threshold, the current weather condition is determined to be severe and image recognition is initiated.

[0021] Specifically, for ground data monitoring, the main control unit counts, for example, the number of valid ground echo points received per second by a specific channel of the lidar (e.g., the ground should be illuminated when the blades are not obstructing it). The first preset threshold is set as a percentage (e.g., lower than 30% of the normal value) or an absolute lower limit (e.g., less than 500 valid surface points per second, this value depends on the lidar's measurement frequency, such as 20kHz) based on the number of ground points that should be received under normal weather conditions (e.g., obtained through historical data or calibration).

[0022] Specifically, for monitoring the dispersion of ground data, the main control unit analyzes multiple ground distance values ​​measured by the same laser channel within a certain time window (e.g., 1 second or several consecutive measurement cycles). These values ​​are very close (considering ground flatness and minor radar fluctuations). The standard deviation, range, or interquartile range of these distance values ​​are calculated and compared with a second preset threshold (a maximum permissible dispersion, e.g., a standard deviation greater than 0.5 meters, this value needs to be determined based on the radar's inherent accuracy and installation stability experiments).

[0023] Furthermore, when the ground data volume of any laser channel is lower than the first preset threshold, or the dispersion of its ground data distance values ​​is higher than the second preset threshold, it is determined that the current weather conditions significantly interfere with the lidar measurement, thus meeting the conditions for activating image recognition. This judgment logic ensures sensitivity to the impact of various severe weather conditions.

[0024] When the weather conditions are good, video images of the wind turbine blades and synchronous lidar data are collected. The video images are processed to extract the blade motion features, and the corresponding lidar data is combined to establish an image recognition reference object database.

[0025] Specifically, once the weather conditions are confirmed to be favorable (i.e., sufficient and stable ground-based lidar data), the main control unit instructs the camera module (built into the air clearance radar) to continuously capture video images of the wind turbine blades rotating through the lidar's predetermined detection area. Simultaneously, the lidar module operates normally, recording data such as distance and signal strength measured by each beam channel as the blades pass. Both video and lidar data are precisely timestamped to ensure synchronization.

[0026] Specifically, the process of processing the video images to extract blade motion features includes image preprocessing (such as noise reduction and contrast enhancement), blade segmentation (separating the blade region from the background), and feature extraction (such as blade edge contours, blade tip positions, blade surface textures, and the shape and speed of the blade as it passes through specific areas). These features, along with synchronously recorded LiDAR data (such as whether the corresponding areas of beams 1, 2, and 3 in the video are obscured by blades and their corresponding precise distance values), environmental parameters (such as wind speed and direction, which can be obtained from the wind turbine's SCADA system), and wind turbine operating parameters (such as blade rotation speed), are organized and stored in the image recognition reference object database. This database uses wind speed, blade rotation speed, etc., as indexes to store the corresponding blade image feature sequences and LiDAR measurement templates.

[0027] In some embodiments, the step of acquiring video images of wind turbine blades and synchronous lidar data when weather conditions are favorable, processing the video images to extract blade motion features, and establishing an image recognition reference object database in conjunction with the corresponding lidar data includes: The acquired video images are processed to grayscale. The adjacent frame difference method is used to analyze the grayscale processed video image to identify the blade motion contour; By combining synchronized lidar data, effective blade motion video clips are selected; The blade motion trajectory, edge features, and morphological changes are extracted from the effective blade motion video clips and correlated with the corresponding lidar data and parameters such as wind speed and blade rotation speed to construct the image recognition reference object database.

[0028] Specifically, converting color video frames into single-channel grayscale images aims to reduce computational load, eliminate the interference of color information on motion analysis under certain lighting changes, and improve the accuracy of subsequent frame difference analysis.

[0029] Specifically, by calculating the pixel differences between two or more consecutive grayscale images, moving areas in the image can be effectively highlighted. Since the wind turbine blades are the primary moving objects, their motion contours will appear as high-brightness areas in the difference image, making them easy to segment and identify.

[0030] Specifically, selecting valid video clips of blade movement is crucial for ensuring the quality of the reference data. For example, if LiDAR data confirms that the current video frame actually captures a blade (e.g., the measurement values ​​of LiDAR beams 1, 2, or 3 indicate that the blade is present in the detection area), and the blade image is clear and free of severe occlusion (which can be determined using an image quality assessment algorithm), then the clip is considered valid. Invalid clips (such as blades not entering the detection area, blurry images, or overexposure or underexposure due to drastic changes in lighting) will be discarded.

[0031] Furthermore, more refined feature extraction is performed on the blade image sequences from the selected valid video clips. For example, edge detection algorithms (such as Canny and Sobel) are used to extract blade edges, and Hough transform or contour tracking algorithms are used to obtain the movement trajectory of the blade tip. The morphology (such as width and angle of attack) of the blade in specific areas of the video (especially the area covered by the laser beam) changes over time. These visual features are stored together with the precise distance values ​​measured by the synchronous lidar, the width of the blade at that distance (which can be indirectly estimated), the length of time the blade is in contact with the laser beam, and recorded parameters such as wind speed and blade rotation speed. The entries in the database can be understood as: "At a certain wind speed and a certain blade rotation speed, when the blade passes through the beam X area, its visual feature sequence is A, and the corresponding lidar clearance value change pattern is B."

[0032] In some embodiments, after constructing the image recognition reference object database, the method further includes: The image recognition reference object database is updated periodically or automatically based on weather changes.

[0033] Specifically, in order to adapt to gradual changes in the surface condition of wind turbine blades (such as dirt, slight icing), differences in seasonal lighting conditions, or slight drifts in radar parameters, the image recognition reference object database needs to be updated regularly. For example, it can be set to automatically re-execute the reference object learning and building process every certain period of time (such as weekly or monthly), or after the weather conditions are detected to change from severe to good and remain stable for a period of time, supplementing or replacing the old reference data with new data.

[0034] When the weather conditions are severe, current leaf video images and lidar data are collected, and the leaf features extracted from the current video images are compared and analyzed with reference objects in the image recognition reference object database to evaluate the accuracy of the lidar data or to assist in calculating the leaf clearance value.

[0035] Specifically, once the severe weather joint application mode is entered, the main control unit continuously instructs the camera module and lidar module to synchronously collect data. Each frame or video sequence currently acquired undergoes grayscale processing and preliminary blade area identification (the results may be poor due to severe weather). Then, based on the current wind turbine operating status (such as the estimated blade speed, which can be obtained from historical data or SCADA), the most matching reference object (i.e., the blade motion feature sequence and lidar data template under similar wind speeds and speeds) is retrieved from the image recognition reference object database.

[0036] Specifically, the comparative analysis involves matching the visual features of the blades (such as motion trajectory and edge morphology) extracted from the current video (which may be incomplete or interfered with) with the visual features of the reference object, and calculating a degree of consistency or confidence score. Simultaneously, the data measured by the current LiDAR (if a signal is still present) is compared with the LiDAR data template of the reference object.

[0037] In some embodiments, when the weather conditions are severe, acquiring current leaf video images and lidar data, and comparing and analyzing the leaf features extracted from the current video images with reference objects in the image recognition reference object database to evaluate the accuracy of the lidar data or assist in calculating the leaf clearance value, includes the following when the lidar can still acquire periodic leaf data: Estimate the time window for the blade to reach the detection area, and simultaneously acquire current blade video images and lidar data within that time window; Retrieve the reference trajectory corresponding to the currently estimated blade rotation speed from the image recognition reference object database; The leaf features extracted from the current video image are compared with the reference trajectory to calculate the degree of consistency. The reliability of the current lidar data is determined based on whether the degree of conformity is higher than the first degree of conformity threshold.

[0038] Specifically, even in adverse weather conditions, if the lidar can intermittently capture signals related to the rotation cycle of the wind turbine blades (“periodic blade data”), this periodicity can be used to estimate the approximate time window for the next blade to pass through the detection area. This allows for concentrated processing resources, focusing on enhancing the simultaneous acquisition and analysis of video and lidar data only within that window.

[0039] Specifically, the reference trajectory is a sequence of leaf visual features contained in the selected reference object (such as the expected position sequence of the leaf tip in the image, and the expected morphological change sequence of the leaf edge).

[0040] Specifically, the degree of conformity can be calculated using various methods. For example, it can be done by calculating the overlap between the blade edge identified in the current video frame and the expected edge in the reference trajectory (e.g., IoU - Intersection over Union), or by comparing the similarity between the current blade tip trajectory and the reference trajectory (e.g., using the Dynamic Time Warping (DTW) algorithm). The degree of conformity is a quantified score, for example, between 0 and 1.

[0041] Specifically, the first conformity threshold is an empirical value (e.g., 0.6 or 0.7) used to define the level of consistency between the image analysis results and the reference model. If the calculated conformity is higher than this threshold, it is considered that the current video image clearly captures the blade, and its movement is basically consistent with the normal mode. In this case, the corresponding blade signal measured by the lidar (if it exists and the value is within a reasonable range) is considered reliable.

[0042] In some embodiments, after determining the reliability of the current lidar data, the process further includes: If the degree of conformity is lower than the first degree of conformity threshold, then the current lidar data is likely to be incorrect.

[0043] Specifically, if the image analysis results have a very low degree of agreement with the reference model, it indicates that the current video image may not have clearly captured the blades, or that the blade movement pattern differs significantly from normal conditions (possibly due to extreme weather or severe image interference). In this case, even if the lidar provides some readings that appear to show blades, the authenticity of these readings is questionable and should be marked as "highly likely to be wrong" or "low confidence," reducing their weight or excluding them from subsequent clearance value calculations.

[0044] In some embodiments, when the weather conditions are severe, acquiring current blade video images and lidar data, and comparing and analyzing the blade features extracted from the current video images with reference objects in the image recognition reference object database to evaluate the accuracy of the lidar data or assist in calculating the blade clearance value, includes the following when the lidar does not have obvious periodic blade data but confirms that the blade is still rotating: Acquire current video images of the blades within the estimated time window during which the blades may pass; By analyzing the chromaticity and brightness changes of the current video image, the edge region of the blade and its motion trajectory are fitted. The fitted blade edge motion trajectory is compared with the reference trajectory in the image recognition reference object database, and the degree of conformity is calculated. When the degree of compliance is higher than the second degree of compliance threshold, the blade clearance value is estimated mainly based on the image analysis results.

[0045] Specifically, when severe weather (such as extremely dense fog) renders the lidar completely unable to detect blade signals, or when the signals are completely drowned out by noise ("no obvious periodic blade data"), but the blades are still known to be moving through other means (such as the wind turbine's SCADA system confirming that the wind turbine is still running and the blades are rotating), image analysis will become more crucial.

[0046] Specifically, since the image quality may also be poor, more robust image processing techniques will be attempted. For example, analyzing whether there are large areas of abrupt changes in color or brightness in the image that match the approximate shape and expected direction of movement of the leaf, in order to try to fit the possible edges and trajectory of the leaf. This may involve background subtraction, motion estimation algorithms, etc.

[0047] Specifically, the blade motion trajectory, which is fitted to the best of its ability under harsh conditions, is compared with the trajectory in the reference database, and the degree of conformity is calculated.

[0048] Specifically, the second conformity threshold is typically lower than the first conformity threshold (e.g., 0.4 or 0.5) because, in this extreme case, the expectation for image analysis is to provide some indication, albeit minimally. If the conformity still reaches this lower threshold, it indicates that the image analysis has at least captured some weak but consistent evidence related to blade motion. At this point, since lidar data is completely unavailable, the blade clearance value will be estimated primarily (or even entirely) based on the results of this image analysis (e.g., through the blade position in the image and the pre-defined camera-to-tower geometry).

[0049] Based on the results of the comparative analysis and the lidar data, the clearance value and accuracy assessment of the wind turbine blades are determined.

[0050] Specifically, the analysis results from the above steps are considered as follows: If the LiDAR data is deemed reliable (e.g., the compliance level is higher than the first compliance threshold), the net clearance value measured by the LiDAR is primarily used, with image analysis results used for verification. If the LiDAR data is deemed highly likely to be erroneous, or if there is no valid LiDAR data but the compliance level of the image analysis is higher than the second compliance threshold, the net clearance value estimated based on image analysis is used. If neither is reliable, a "cannot be measured" or a conservative estimate based on the worst-case scenario may be output, with a very low accuracy assessment.

[0051] Specifically, the accuracy assessment can be qualitative (e.g., high, medium, low confidence) or quantitative (e.g., a confidence score). This assessment result will be output along with the net void value for maintenance personnel to reference.

[0052] In some embodiments, determining the air clearance value of the wind turbine blades and its accuracy assessment based on the results of the comparative analysis and the lidar data includes: If the lidar data is deemed reliable based on the results of the comparative analysis, then the blade clearance value is determined by combining the lidar data and the results of the comparative analysis. If the comparison analysis results indicate that the lidar data is likely to be incorrect, or if there is no obvious periodic blade data in the lidar, then the blade clearance value should be estimated mainly based on the comparison analysis results. The determined or estimated blade clearance value is compared and verified with historical clearance values ​​and clearance value change trends. Output the final blade clearance value, along with an accuracy assessment level based on the degree of compliance and historical data comparison verification results.

[0053] Specifically, for the first scenario (where the lidar data is reliable), the final clearance value can be directly taken from the lidar measurement results, or subject to minor, image-analysis-based corrections. The accuracy assessment level is "high".

[0054] Specifically, in the second scenario (where lidar data is unreliable and image analysis is relied upon), the clearance value is calculated based on the blade position in the image, camera parameters (such as focal length, installation position, and angle), and the geometric model of the wind turbine tower. The accuracy assessment level may be "medium" or "low," depending on the degree of agreement between the image analysis and the target data.

[0055] Specifically, regardless of how the current net clearance value is obtained, it will be compared with the stored historical net clearance value sequence to check whether it is within the expected range of change (e.g., the net clearance value will not undergo drastic changes far beyond physical probability in a short period of time). If the current value deviates significantly from the historical trend, its accuracy assessment will be further reduced or an alarm will be triggered.

[0056] Specifically, the accuracy assessment level of the output can be divided into, for example, "high confidence" (the LiDAR data is valid and verified by images), "medium confidence" (mainly relies on image analysis, and the degree of conformity is acceptable and verified by historical trends), "low confidence" (the degree of conformity of image analysis is low, or there is a certain deviation from historical trends, but it is still the best estimate at present) or "invalid data".

[0057] Okay, no problem. Let's construct a specific scenario example and demonstrate the calculation process in detail to more clearly understand how the strategy of this invention works.

[0058] In one specific scenario, a coastal wind farm is located in an area where advection fog is common in the early mornings of autumn and winter. A normally operating wind turbine has its blade clearance radar equipped with the software of the strategy described in this invention. The radius of the tower at the laser detection height is 5.0 meters.

[0059] Under normal operating conditions: In clear weather, the clearance between the blade tip and the tower (safe clearance value) remains stable at approximately 8.0 meters. Therefore, the blade distance value measured by the lidar should be approximately 13.0 meters (5.0-meter tower radius + 8.0-meter clearance). The ground distance is approximately 85 meters.

[0060] Fan status: The fan is running stably with a blade speed of 18 revolutions per minute.

[0061] System preset parameters: LiDAR measurement frequency: 20,000Hz First preset threshold for ground data volume: 1,000 points / second Second preset threshold (standard deviation) for ground data dispersion: 0.5 meters The first threshold for the degree of consistency in the credibility judgment is 0.6. The second threshold for compliance in the "best effort" judgment is 0.4. Image recognition reference object database: It was learned and established under previous clear weather conditions.

[0062] 2. Implementation process Phase 1: Weather Deterioration and Triggering Image Recognition Time: 6:30 a.m., sea fog begins to spread.

[0063] Phenomenon: The main control unit of the airspace radar continuously monitors the ground data from the lidar.

[0064] Ground data volume monitoring: 6:29 AM (before fog): The main control unit counted approximately 5,500 effective area data points per second. 5500 > 1000, so the weather condition is judged as "good".

[0065] 6:31 AM (Fog begins): Due to the absorption and scattering of laser light by the fog, the main control unit counted a sharp drop in the number of effective ground data points per second to 720.

[0066] Calculation and judgment: 720 < 1000 (first preset threshold), the first trigger condition is met.

[0067] Ground data dispersion monitoring: 6:29 AM (before fog): 100 ground distance samples measured within 1 second are {85.1, 85.0, 85.1, 85.2, 85.1, ...}, and their standard deviation is calculated to be 0.08 meters. 0.08 < 0.5, indicating stable data.

[0068] 6:31 AM (Fog begins): Due to some lasers generating echoes in the nearby fog, the 100 ground distance sample values ​​within 1 second become {85.1, 45.3, 62.8, 84.9, 55.0, ...}, and their standard deviation is calculated to be 15.2 meters.

[0069] Calculation and judgment: 15.2 > 0.5 (second preset threshold), the second triggering condition is met.

[0070] System Action: Since both conditions have been met, the system automatically determines the current weather condition as "severe" and officially launches the image recognition joint application strategy.

[0071] Phase Two: Joint Detection - Scenario A (LiDAR still has some signal) Time: 6:33 AM, a leaf is passing through the inspection area.

[0072] Data collection: LiDAR data: Within 0.05 seconds of the blade passing by, the lidar received a set of chaotic data, which included some points between 12.8 meters and 13.3 meters (possibly real blades), but more noise points distributed between 20 meters and 50 meters (generated by fog).

[0073] Video image data: The camera simultaneously captured a sequence of video frames as the blades passed by.

[0074] Calculation and analysis process: Image feature extraction: The main control unit performs grayscale processing and adjacent frame difference method on the video frames to clearly extract the edge contour of the currently moving blade and obtain the actual motion trajectory point set P_current={(120,300),(125,308),(130,316),...} of the blade tip in the image coordinate system.

[0075] Reference trajectory retrieval: Based on the current blade speed of 18 revolutions per minute, the system retrieves the standard reference trajectory point set P_reference={(121,301),(126,309),(131,317),...} from the image recognition reference object database for the corresponding working condition.

[0076] Similarity calculation: The system compares the similarity between P_current and P_reference point by point. For simplicity, we use the average normalized inverse distance as the calculation method for similarity. For the first pair of points (120, 300) and (121, 301), the pixel distance d1 = sqrt((121-120)^2 + (301-300)^2) = sqrt(2) ≈ 1.414. The score s1 = 1 / (1+1.414) ≈ 0.414.

[0077] For the second pair of points (125, 308) and (126, 309), the pixel distance d2 = sqrt((126-125)^2 + (309-308)^2) = sqrt(2) ≈ 1.414. The fraction s2 ≈ 0.414.

[0078] ...(The matching degree of subsequent points is very high, the distance is close to 0, and the score is close to 1.) Assume the average score of all points along the entire trajectory is 0.88.

[0079] Judgment: The calculated compliance score is 0.88. 0.88 > 0.6 (first compliance threshold). System judgment: The image clearly captures the blade, and its movement trajectory is normal. Therefore, the blade signal measured by the lidar within this time period is reliable.

[0080] Data fusion and output: The system filters LiDAR data within a precise time window (0.05 seconds) determined by image analysis.

[0081] Filter out obvious noise (such as data points at a distance > 20 meters).

[0082] The average distance of the remaining valid points between 12.8 meters and 13.3 meters was obtained by averaging the distances, resulting in an average distance of 13.1 meters.

[0083] Compare this value with the historical trend (~13.0 meters) to verify its rationality.

[0084] Final output: Blade clearance: 13.1 meters (total distance) - 5.0 meters (tower radius) = 8.1 meters Accuracy assessment: High reliability Phase 3: Joint Detection - Scenario B (LiDAR signals are completely submerged) Time: 6:40 AM, the fog has become extremely thick.

[0085] Data collection: LiDAR data: Within the estimated time window of the blade's passage, the LiDAR received no echoes within the 10-15 meter range; all signals were displayed beyond 25 meters, presenting a "noise wall." The system determined that "there is no obvious periodic blade data."

[0086] Video image data: The images captured by the camera have extremely low contrast, and the boundary between the leaves and the background is very blurry.

[0087] Calculation and analysis process: Image Feature Extraction: Due to the poor performance of the frame difference method, the system instead analyzes the brightness and chromaticity distribution of the image. It detects a large, moving gray area (leaf) on the expected motion path and attempts to estimate a rough leaf edge trajectory P_fitted={(115,295),(122,305),(129,315),...} through region growing and morphological fitting.

[0088] Reference trajectory retrieval: Similarly, retrieve the standard reference trajectory P_reference={(121,301),(126,309),(131,317),...} at 18 revolutions per minute.

[0089] Compliance calculation: For the first pair of points (115, 295) and (121, 301), the pixel distance d1 = sqrt((121-115)^2 + (301-295)^2) = sqrt(36 + 36) ≈ 8.485. The score s1 = 1 / (1 + 8.485) ≈ 0.105.

[0090] Due to the low fitting accuracy and the large average distance between points, we assume that the final calculated average score of the entire trajectory is 0.45.

[0091] Judgment: The calculated degree of agreement is 0.45.

[0092] 0.45 < 0.6 (first compliance threshold), therefore any weak lidar signal is unreliable.

[0093] 0.45 > 0.4 (second consistency threshold), the system believes that although the confidence level of image analysis is not high, it is still the only available and somewhat reliable source of information.

[0094] Estimating net clearance based on images: The system at this point depends entirely on the fitted blade trajectory P_fitted.

[0095] Assuming the system is pre-calibrated: when the blade tip is at position 310 pixels in the Y coordinate of the image, without bending, its real-world straight-line distance from the radar is 13.0 meters. Image analysis shows that due to blade bending, when the blade tip is at position 310 pixels in the Y coordinate, its X coordinate is offset by 15 pixels compared to its normal position.

[0096] Calculated using the camera's geometric model (perspective transformation matrix), this 15-pixel offset corresponds to a blade displacement of -0.4 meters toward the radar (i.e., the blade is closer to the radar).

[0097] Estimated distance: 13.0 meters (baseline distance) - 0.4 meters (bending displacement) = 12.6 meters.

[0098] The estimated value is considered to be within a reasonable range when compared with the historical trend (13.0 meters).

[0099] Final output: Blade clearance: 12.6 meters (estimated distance) - 5.0 meters (tower radius) = 7.6 meters Accuracy assessment: Medium confidence level (because it is based on image analysis with low consistency). It should be understood that the embodiments disclosed in this invention and the above description enable those skilled in the art to implement this invention. However, this invention is not limited to the embodiments mentioned above. It should be understood that those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention, and should all be included within the protection scope of this invention.

Claims

1. A strategy for the combined application of lidar and image recognition in airspace clearance radar, characterized in that, The strategy includes: The amount and dispersion of ground data from the lidar are monitored to determine the current weather conditions. When the weather conditions become severe to a preset level, image recognition is activated to assist in the judgment. When the weather conditions are good, video images of the wind turbine blades and synchronous lidar data are collected. The video images are processed to extract the blade motion features, and the corresponding lidar data is combined to establish an image recognition reference object database. When the weather conditions are severe, current leaf video images and lidar data are collected, and the leaf features extracted from the current video images are compared and analyzed with reference objects in the image recognition reference object database to evaluate the accuracy of lidar data and assist in calculating the leaf clearance value. Based on the results of the comparative analysis and the lidar data, the clearance value and accuracy assessment of the wind turbine blades are determined.

2. The joint application strategy of lidar and image recognition in airspace radar as described in claim 1, characterized in that, The monitoring of the ground data volume and dispersion of the lidar is used to determine the current weather conditions, including: Monitor the amount of ground data received by each channel of the lidar per unit time and compare it with a first preset threshold. Monitor the dispersion of ground data distance values ​​measured by each channel of the lidar and compare them with a second preset threshold; When the amount of ground data is lower than the first preset threshold, or the dispersion of the ground data distance value is higher than the second preset threshold, the current weather condition is determined to be severe and image recognition is initiated.

3. The joint application strategy of lidar and image recognition in airspace radar as described in claim 1, characterized in that, When weather conditions are favorable, video images of the wind turbine blades and synchronous lidar data are acquired. The video images are processed to extract blade motion features, and a database of image recognition reference objects is established by combining the corresponding lidar data. This includes: The acquired video images are processed to grayscale. The adjacent frame difference method is used to analyze the grayscale processed video image to identify the blade motion contour; By combining synchronized lidar data, effective blade motion video clips are selected; The blade motion trajectory, edge features, and morphological changes are extracted from the effective blade motion video clips and correlated with the corresponding lidar data and wind speed and blade rotation speed parameters to construct the image recognition reference object database.

4. The joint application strategy of lidar and image recognition in airspace radar as described in claim 1, characterized in that, When the weather conditions are severe, the process involves acquiring current leaf video images and lidar data, and comparing the leaf features extracted from the current video images with reference objects in the image recognition reference object database to assess the accuracy of the lidar data or assist in calculating the leaf clearance value. When the lidar can still acquire periodic leaf data, the process includes: Estimate the time window for the blade to reach the detection area, and simultaneously acquire current blade video images and lidar data within that time window; Retrieve the reference trajectory corresponding to the currently estimated blade rotation speed from the image recognition reference object database; The leaf features extracted from the current video image are compared with the reference trajectory to calculate the degree of consistency. The reliability of the current lidar data is determined based on whether the degree of conformity is higher than the first degree of conformity threshold.

5. The joint application strategy of lidar and image recognition in the airspace radar as described in claim 1, characterized in that, When the weather conditions are severe, the process involves acquiring current blade video images and lidar data, and comparing the blade features extracted from the current video images with reference objects in the image recognition reference object database to assess the accuracy of the lidar data or assist in calculating the blade clearance value. This includes situations where the lidar does not show obvious periodic blade data but confirms that the blade is still rotating. Acquire current video images of the blades within the estimated time window during which the blades may pass; By analyzing the chromaticity and brightness changes of the current video image, the edge region of the blade and its motion trajectory are fitted. The fitted blade edge motion trajectory is compared with the reference trajectory in the image recognition reference object database, and the degree of conformity is calculated. When the degree of compliance is higher than the second degree of compliance threshold, the blade clearance value is estimated mainly based on the image analysis results.

6. The joint application strategy of lidar and image recognition in airspace radar as described in claim 1, characterized in that, The determination of the wind turbine blade clearance value and accuracy assessment based on the results of the comparative analysis and the lidar data includes: If the lidar data is deemed reliable based on the results of the comparative analysis, then the blade clearance value is determined by combining the lidar data and the results of the comparative analysis. If the comparison analysis results indicate that the lidar data is likely to be incorrect, or if there is no obvious periodic blade data in the lidar, then the blade clearance value should be estimated based on the comparison analysis results. The determined or estimated blade clearance value is compared and verified with historical clearance values ​​and clearance value change trends. Output the final blade clearance value, along with an accuracy assessment level based on the degree of compliance and historical data comparison verification results.

7. The joint application strategy of lidar and image recognition in the airspace radar as described in claim 4, characterized in that, After determining the credibility of the current lidar data, the method further includes: if the degree of conformity is lower than the first degree of conformity threshold, then the current lidar data is likely to be incorrect.

8. The joint application strategy of lidar and image recognition in airspace radar as described in claim 3, characterized in that, After constructing the image recognition reference object database, the method further includes: periodically or automatically updating the image recognition reference object database according to weather changes.

Images