Wind turbine blade three-dimensional trajectory dynamic monitoring method fusing image and laser point cloud

Abstract

This invention belongs to the field of blade trajectory monitoring technology, specifically involving a method for dynamic monitoring of the three-dimensional trajectory of wind turbine blades by fusing imagery and laser point cloud data. The steps include: fixing a camera and scanner with a rigid connection device for use as a dynamic monitoring system for the three-dimensional trajectory of the wind turbine blade; calibrating the camera; calibrating the relative position and attitude between the camera and scanner; acquiring image data and laser point cloud data during normal operation of the wind turbine; performing motion compensation correction on the laser point cloud data; calculating the three-dimensional coordinates corresponding to the monitoring point; obtaining a time-series image-side coordinate sequence of the monitoring point; and calculating the three-dimensional coordinate sequence corresponding to the time-series image-side coordinate sequence of the monitoring point, i.e., the three-dimensional motion trajectory of the monitoring point on the wind turbine blade during operation. This invention uses imagery and laser point cloud technology to monitor the three-dimensional trajectory of wind turbine blades, requiring no auxiliary markers, eliminating complex calibration, offering flexible deployment and high accuracy, and efficiently supporting blade safety assessment and maintenance decisions.

Description

Technical Field

[0001] This invention belongs to the field of blade trajectory monitoring technology, specifically involving a method for dynamic monitoring of the three-dimensional trajectory of wind turbine blades by fusing images and laser point clouds, applicable to the dynamic monitoring of the three-dimensional trajectory of in-service wind turbine blades. Background Technology

[0002] Three-dimensional operational trajectory data of in-service wind turbine blades is a crucial foundation for analyzing the dynamic stress on the blades, assessing their structural health, and providing early warnings of potential fatigue damage. It directly impacts the long-term safe and stable operation of wind turbine blades and serves as core data support for structural safety assessments, lifespan predictions, and maintenance decisions within wind power systems. Currently, the industry mainstream methods for measuring the three-dimensional trajectory of in-service wind turbine blades include stereo vision and laser scanning technologies. While these methods can achieve basic trajectory measurements, a highly efficient monitoring solution adapted to the actual operating scenarios of wind turbine blades has not yet been developed.

[0003] Traditional measurement methods have significant technical bottlenecks and application defects in practical applications, making it difficult to meet the high-precision, long-term, and interference-free monitoring requirements of wind turbine blades. Stereo vision systems not only require complex field calibration processes, which are difficult to calibrate and require cumbersome maintenance, but their measurement accuracy is also significantly affected by ambient light, weather conditions such as sunshine, rain, and snow, resulting in insufficient stability in complex outdoor wind fields. Laser scanning methods require additional auxiliary devices such as reflective targets or markers to be installed on the blade surface, which not only significantly increases the manpower and material costs of on-site installation and daily maintenance, but also damages the original structural design of the blade, interferes with the aerodynamic performance of the blade, and even affects its structural integrity, which is contrary to the actual operating requirements of wind turbine blades. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a method for dynamic monitoring of the three-dimensional trajectory of wind turbine blades by fusing images and laser point clouds. The method of monitoring the three-dimensional trajectory of wind turbine blades by fusing images and laser point clouds has no auxiliary markers, does not require complex calibration, is flexible in deployment and has high accuracy, and can efficiently support blade safety assessment and maintenance decisions.

[0005] To achieve the above objectives, this invention provides a method for dynamic monitoring of the three-dimensional trajectory of wind turbine blades by fusing imagery and laser point clouds, comprising the following steps:

[0006] S1. The camera and scanner are fixed by a rigid connection device and used as a dynamic monitoring system for the three-dimensional trajectory of wind turbine blades. The scanner is a three-dimensional laser scanner.

[0007] S2. Use a planar calibration plate to calibrate the camera and obtain the camera's precise interior orientation elements;

[0008] S3. Calibrate the relative position and orientation between the camera and the scanner;

[0009] S4. Using the calibrated dynamic monitoring system, monitor the three-dimensional trajectory of the wind turbine blades during normal operation of the wind turbine, and acquire image data and laser point cloud data.

[0010] S5. Perform motion compensation correction on a frame-by-frame basis for the monitored laser point cloud data;

[0011] S6. Determine the neighboring three-dimensional points of each monitoring point in the laser point cloud data, and calculate the three-dimensional coordinates of the monitoring point. The monitoring point is a feature point to be tracked selected on the wind turbine blade.

[0012] S7. In the initial frame of the image data, for the selected monitoring point, the monitoring point is continuously tracked in each subsequent frame of the image to obtain the time series monitoring point image coordinate sequence.

[0013] S8. Calculate the three-dimensional coordinate sequence corresponding to the image coordinate sequence of the time series monitoring point. The three-dimensional coordinate sequence completely describes the continuous position change of the monitoring point in space, that is, the three-dimensional motion trajectory of the monitoring point on the wind turbine blade during operation.

[0014] As a preferred embodiment of the present invention, in S1, an aluminum alloy profile splicing rigid connection device is used to fix the industrial camera and scanner. 6061 aluminum alloy square tube is used as the frame. According to the external dimensions and installation interface of the camera and scanner, they are spliced ​​into a suitable integrated bracket. The integrated bracket is reserved with threaded holes that are precisely matched with the camera gimbal interface and the bottom mounting hole of the scanner. It is fastened by stainless steel bolts.

[0015] The integrated bracket features a thick aluminum alloy rib plate in the middle for triangular reinforcement, and a cutout channel is reserved at the corresponding position of the camera lens and the scanner's scanning port to provide a field of view for data acquisition. The bottom of the integrated bracket is equipped with an adjustable positioning bracket for seamless docking with an external monitoring bracket / pan-tilt unit, preventing relative displacement or shaking of the camera and scanner during data acquisition.

[0016] As a preferred embodiment of the present invention, in S3, at least four cylinders are used as calibration references to calibrate the relative position and orientation between the camera and the scanner. First, the cylinders are arranged and data is collected synchronously. Then, the two-dimensional center coordinates of the circular end face of the cylinder are extracted based on the image, and the three-dimensional center coordinates of the circular end face of the cylinder are extracted based on the laser point cloud. Finally, the relative position and orientation of the camera relative to the scanner are calculated.

[0017] As a preferred embodiment of the present invention, the specific steps for calibrating the relative position and orientation between the camera and the scanner in step S3 are as follows:

[0018] S3.1 Arrange cylinders and collect data synchronously: Orient the circular end face of the cylinders toward the camera and scanner, and distribute each cylinder evenly in the camera's imaging field; synchronously trigger the camera and scanner to acquire images containing the cylinders and laser point cloud data respectively;

[0019] S3.2 Extracting the 2D center coordinates of the circular end face of the cylinder based on the image: Edge detection is performed on the acquired image to extract the contour edges of each cylindrical end face; the extracted contour edges are fitted using an ellipse fitting method, and the center coordinates of the fitted ellipse are taken as the 2D center coordinates of the end face in the image coordinate system, denoted as . , where i is the cylinder's index. , , Let X and Y represent the X and Y coordinates of the center of the circular end face of the i-th cylinder in the image coordinate system, respectively, where n is the number of cylinders.

[0020] S3.3 Extracting the 3D center coordinates of the cylindrical end face based on laser point cloud: The cylindrical end face is extracted from the laser point cloud. The extracted end face is then contoured to obtain its contour points. These contour points are then fitted with a circle to obtain the center coordinates of the cylindrical end face. These center coordinates are the 3D coordinates of the cylindrical end face's center in the scanner coordinate system. , , , These represent the X, Y, and Z coordinates of the center of the circular end face of the i-th cylinder in the scanner coordinate system.

[0021] S3.4 Calculate the relative position and orientation of the camera with respect to the scanner: based on the corresponding center coordinates in the image and the laser point cloud. and By combining the collinearity condition equation in photogrammetry and using the spatial resection algorithm, the relative position between the camera and the scanner can be calculated. and attitude rotation matrix , , , These represent the X, Y, and Z coordinates of the camera's projection center in the scanner's coordinate system, respectively. , , These represent the camera's yaw angle, pitch angle, and roll angle relative to the scanner, respectively.

[0022] As a preferred embodiment of the present invention, in S4, during the monitoring process, the camera and scanner are triggered simultaneously when each frame of data is acquired, so as to obtain image frames and laser point cloud data that are strictly aligned in time.

[0023] As a preferred embodiment of the present invention, the motion compensation correction process in S5 is as follows:

[0024] S5.1, with the center of the main axis of wind turbine blade rotation as the origin of the coordinate system. Taking the plane of rotation where the wind turbine blades are located as A plane, and the direction perpendicular to that plane is The axial direction, the direction of any wind turbine blade is Axial direction, based on Axial direction and Axis direction determined Establish a rectangular coordinate system for the wind turbine body along the axis direction. All scan point coordinates are transformed to this coordinate system, where a scan point is each independent three-dimensional laser sampling point contained in a single frame of laser point cloud data;

[0025] S5.2, in In the plane, with Establish a polar coordinate system with the origin, and , Coordinate transformation from rectangular coordinate system to polar coordinate system Down, The coordinates remain unchanged, where, The polar radius represents To the scan point The straight-line distance between projection points in the plane; The polar angle indicates the direction from which the polar angle originates. Around the positive direction of the axis Rotate counterclockwise to the angle between the line connecting the origin and the line connecting the projection point of the scan point;

[0026] S5.3 Calculate the time interval between adjacent scan points based on the scanner's scanning frequency, field of view, and angular resolution. The angular velocity of the wind turbine blades is , combined and Calculate the angle correction in polar coordinates. ,right Perform reverse compensation to obtain the corrected polar coordinates. ;

[0027] S5.4 Restore the corrected polar coordinates to Cartesian coordinates, and the original The coordinates are reconstructed into a three-dimensional rectangular coordinate system. This new three-dimensional rectangular coordinate system is then transformed back into the scanner coordinate system to obtain the motion-compensated three-dimensional coordinates. , , , These represent the X, Y, and Z coordinates of the j-th scan point in the k-th frame of laser point cloud data, respectively, in the scanner coordinate system.

[0028] As a preferred embodiment of the present invention, in step S6, the process of searching for neighboring three-dimensional points for each monitoring point in a single frame of laser point cloud data is as follows:

[0029] S6.1. Based on the camera imaging model, convert the motion-compensated three-dimensional coordinates... Projected onto the image plane of the corresponding frame, we obtain The corresponding two-dimensional projected coordinates are denoted as , , These represent the X-axis and Y-axis coordinates of the j-th scan point in the k-th frame of laser point cloud data, respectively, in the image coordinate system.

[0030] S6.2 For a certain monitoring point on the k-th frame image, find P points in the laser scanning point projection coordinate set corresponding to the frame that are closest to the monitoring point in Euclidean distance and are not collinear as projection points, P≥10, where the k-th frame image and the k-th frame laser point cloud data are spatiotemporally aligned data pairs that are synchronously triggered and acquired.

[0031] S6.3. Based on the obtained P projection points, obtain the corresponding 3D laser point coordinates in the scanner coordinate system, denoted as . , , , These represent the three-dimensional laser point coordinate components corresponding to the p-th projection point in the k-th frame image in the scanner coordinate system.

[0032] As a preferred embodiment of the present invention, in step S6, calculating the three-dimensional coordinates corresponding to the monitoring point specifically involves:

[0033] S6.4. Using P projection points, fit the curve using the quadratic surface equation:

[0034] (1);

[0035] In the formula, A, B, C, D, E, F, G, H, I, and J are the coefficients to be solved. The coefficients are obtained by solving the least squares method, and the geometric model of the blade surface near the monitoring point of the kth frame is obtained.

[0036] S6.5 The image point of the monitoring point in the image and the center of the camera projection form a spatial ray. The intersection of this spatial ray and the geometric model of the blade surface at the monitoring point is the three-dimensional coordinate of the monitoring point.

[0037] As a preferred embodiment of the present invention, in S6.5, the equation for spatial rays is:

[0038] (2);

[0039] (3);

[0040] In the formula, , These are the X-axis and Y-axis coordinates of the monitoring point in the k-th frame on the camera imaging plane, respectively. , , These are the X-axis, Y-axis, and Z-axis coordinates of the monitoring point in the k-th frame in the Cartesian coordinate system of the wind turbine body. , These are the focal lengths along the camera's X and Y axes, respectively. , These are the X-axis and Y-axis coordinates of the projection point of the camera's optical axis onto the camera's imaging plane, respectively. It consists of nine orthogonal rotation matrix elements composed of the direction cosines of the camera attitude angle;

[0041] Establish the fitting equation:

[0042] (4);

[0043] Solving the system of equations (2), (3), and (4) yields the monitoring points in the k-th frame. Corresponding three-dimensional coordinates .

[0044] The algorithm involved in this invention can be executed by an electronic device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor executes the software to implement the above-mentioned algorithm calculation.

[0045] The beneficial effects of this invention are:

[0046] This invention achieves a deep integration of optical imaging and laser point cloud technology. By simultaneously acquiring high-frame-rate optical images and high-precision laser point cloud data, the spatiotemporal information of the two technologies complements each other, significantly improving the accuracy and comprehensiveness of three-dimensional trajectory monitoring of wind turbine blades. Furthermore, the solution eliminates the need for any auxiliary markings on the blade surface, avoiding complex on-site calibration procedures. This saves on the manpower and material costs of marking installation and maintenance, and prevents auxiliary devices from interfering with the aerodynamic performance and structural integrity of the blades.

[0047] This invention features flexible deployment and strong environmental adaptability, and is not limited by complex outdoor environmental factors such as light and weather. It can achieve long-term stable monitoring of in-service wind turbine blades, and can efficiently and accurately reconstruct the three-dimensional motion trajectory of the blades under real wind loads. It provides reliable and accurate data support for the structural safety assessment, life prediction and maintenance decision-making of wind turbine blades, and further helps to improve the intelligence and overall safety of wind power system operation, and is adapted to the actual field monitoring needs of the wind power industry. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the principle of this invention. Detailed Implementation

[0049] The embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0050] Example 1: As Figure 1 As shown, the method for dynamic monitoring of the three-dimensional trajectory of wind turbine blades by fusing imagery and laser point clouds includes the following steps:

[0051] S1. The camera and scanner are fixed by a rigid connection device and used as a dynamic monitoring system for the three-dimensional trajectory of wind turbine blades. The scanner is a three-dimensional laser scanner.

[0052] S2. Use a planar calibration plate to calibrate the camera and obtain the camera's precise interior orientation elements;

[0053] S3. Calibrate the relative position and orientation between the camera and the scanner;

[0054] S4. Using the calibrated dynamic monitoring system, monitor the three-dimensional trajectory of the wind turbine blades during normal operation of the wind turbine, and acquire image data and laser point cloud data.

[0055] S5. Perform motion compensation correction on a frame-by-frame basis for the monitored laser point cloud data;

[0056] S6. Determine the neighboring three-dimensional points of each monitoring point in the laser point cloud data, and calculate the three-dimensional coordinates of the monitoring point. The monitoring point is a feature point to be tracked selected on the wind turbine blade.

[0057] S7. In the initial frame of the image data, for the selected monitoring point, the monitoring point is continuously tracked in each subsequent frame of the image to obtain the time series monitoring point image coordinate sequence.

[0058] S8. Calculate the three-dimensional coordinate sequence corresponding to the image coordinate sequence of the time series monitoring point. The three-dimensional coordinate sequence completely describes the continuous position change of the monitoring point in space, that is, the three-dimensional motion trajectory of the monitoring point on the wind turbine blade during operation.

[0059] In S1, an aluminum alloy profile splicing rigid connection device is used to fix the industrial camera and scanner. 6061 aluminum alloy square tube is used as the frame. According to the external size and installation interface of the camera and scanner, they are spliced ​​into a matching integrated bracket. The integrated bracket is reserved with threaded holes that are precisely matched with the camera gimbal interface and the bottom mounting hole of the scanner. The installation is fastened with stainless steel bolts.

[0060] The integrated bracket features a thick aluminum alloy rib plate in the middle for triangular reinforcement, and a cutout channel is reserved at the corresponding position of the camera lens and the scanner's scanning port to provide a field of view for data acquisition. The bottom of the integrated bracket is equipped with an adjustable positioning bracket for seamless docking with an external monitoring bracket / pan-tilt unit, preventing relative displacement or shaking of the camera and scanner during data acquisition.

[0061] The rigid connection device ensures that the relative spatial position and orientation between the camera and the scanner remain unchanged throughout the monitoring process.

[0062] In S2, a high-precision planar calibration plate is used to calibrate the camera, obtaining the camera's precise interior orientation elements, i.e. , , , Camera calibration can be performed using well-known techniques.

[0063] In S3, at least four cylinders are used as calibration references to calibrate the relative position and orientation between the camera and the scanner. The specific steps are as follows:

[0064] S3.1 Arrange cylinders and collect data synchronously: Orient the circular end face of the cylinders toward the camera and scanner, and distribute each cylinder evenly in the camera's imaging field; synchronously trigger the camera and scanner to acquire images containing the cylinders and laser point cloud data respectively;

[0065] S3.2 Extracting the 2D center coordinates of the circular end face of the cylinder based on the image: Edge detection is performed on the acquired image to extract the contour edges of each cylindrical end face; the extracted contour edges are fitted using an ellipse fitting method, and the center coordinates of the fitted ellipse are taken as the 2D center coordinates of the end face in the image coordinate system, denoted as . , where i is the cylinder's index. , , Let X and Y represent the X and Y coordinates of the center of the circular end face of the i-th cylinder in the image coordinate system, respectively, where n is the number of cylinders.

[0066] S3.3 Extracting the 3D center coordinates of the cylindrical end face based on laser point cloud: The cylindrical end face is extracted from the laser point cloud. The extracted end face is then contoured to obtain its contour points. These contour points are then fitted with a circle to obtain the center coordinates of the cylindrical end face. These center coordinates are the 3D coordinates of the cylindrical end face's center in the scanner coordinate system. , , , These represent the X, Y, and Z coordinates of the center of the circular end face of the i-th cylinder in the scanner coordinate system.

[0067] S3.4 Calculate the relative position and orientation of the camera with respect to the scanner: based on the corresponding center coordinates in the image and the laser point cloud. and By combining the collinearity condition equation in photogrammetry and using the spatial resection algorithm, the relative position between the camera and the scanner can be calculated. and attitude rotation matrix , , , These represent the X, Y, and Z coordinates of the camera's projection center in the scanner's coordinate system, respectively. , , These represent the camera's yaw angle, pitch angle, and roll angle relative to the scanner, respectively.

[0068] In S4, during the monitoring process, in order to ensure that the two types of data are strictly aligned in time, the camera and scanner are triggered simultaneously when each frame of data is acquired, so as to obtain image frames and laser point cloud data that are strictly aligned in time.

[0069] Due to the high-speed rotation of wind turbine blades, the point cloud data acquired by the laser scanner within a single frame scan time is affected by the blade motion, resulting in motion blur errors. Therefore, it is necessary to perform motion compensation correction on the coordinates of each frame of 3D laser points by combining parameters such as the blade rotation angular velocity, the distance from the scanning point to the center of the blade rotation axis, the scanner's scanning frequency, the field of view, and its angular resolution.

[0070] In S5, the motion compensation correction process is as follows:

[0071] S5.1 Establish the wind turbine body coordinate system and transform the coordinates of the laser scanning points to this coordinate system: with the center of the wind turbine blade rotation axis as the coordinate origin. Taking the plane of rotation where the wind turbine blades are located as A plane, and the direction perpendicular to that plane is The axial direction, the direction of any wind turbine blade is Axial direction, based on Axial direction and Axis direction determined Establish a rectangular coordinate system for the wind turbine body along the axis direction. All scan point coordinates are transformed to this coordinate system, where a scan point is each independent three-dimensional laser sampling point contained in a single frame of laser point cloud data;

[0072] S5.2, Wind turbine body coordinates , Coordinate transformation to polar coordinates: In the plane, with Establish a polar coordinate system with the origin, and , Coordinate transformation from rectangular coordinate system to polar coordinate system Down, The coordinates remain unchanged, where, The polar radius represents To the scan point The straight-line distance between projection points in the plane; The polar angle indicates the direction from which the polar angle originates. Around the positive direction of the axis Rotate counterclockwise to the angle between the line connecting the origin and the line connecting the projection point of the scan point;

[0073] S5.3 Calculate motion correction in polar coordinates: Based on the scanner's scanning frequency, field of view, and angular resolution, calculate the time interval between adjacent scan points. The angular velocity of the wind turbine blades is , combined and Calculate the angle correction in polar coordinates. ,right Perform reverse compensation to obtain the corrected polar coordinates. ;

[0074] S5.4 Coordinate Restoration: Restore the corrected polar coordinates to... Cartesian coordinates, and the original The coordinates are reconstructed into a three-dimensional rectangular coordinate system. This new three-dimensional rectangular coordinate system is then transformed back into the scanner coordinate system to obtain the motion-compensated three-dimensional coordinates. , , , These represent the X, Y, and Z coordinates of the j-th scan point in the k-th frame of laser point cloud data, respectively, in the scanner coordinate system.

[0075] In S6, within a single frame of laser point cloud data, the process of searching for the nearest 3D points of each monitoring point is as follows:

[0076] S6.1. Based on the camera imaging model, convert the motion-compensated three-dimensional coordinates... Projected onto the image plane of the corresponding frame, we obtain The corresponding two-dimensional projected coordinates are denoted as , , These represent the X-axis and Y-axis coordinates of the j-th scan point in the k-th frame of laser point cloud data, respectively, in the image coordinate system.

[0077] S6.2 For a certain monitoring point on the k-th frame image, find P points in the laser scanning point projection coordinate set corresponding to the frame that are closest to the monitoring point in Euclidean distance and are not collinear as projection points, P≥10. The k-th frame image and the k-th frame laser point cloud data are spatiotemporally aligned data pairs that are synchronously triggered for acquisition. The total number of frames (total number of acquisition times) is set to K.

[0078] S6.3. Based on the obtained P projection points, obtain the corresponding 3D laser point coordinates in the scanner coordinate system, denoted as . , , , They represent the p-th projection point in the k-th frame image, respectively. The coordinate components of the three-dimensional laser point in the scanner coordinate system.

[0079] The specific steps for calculating the three-dimensional coordinates of the monitoring point are as follows:

[0080] S6.4. Using P projection points, fit the curve using the quadratic surface equation:

[0081] (1);

[0082] In the formula, A, B, C, D, E, F, G, H, I, and J are the coefficients to be solved. The coefficients are obtained by solving the least squares method, and the geometric model of the blade surface near the monitoring point of the kth frame is obtained.

[0083] S6.5 The image point of the monitoring point in the image and the center of the camera projection form a spatial ray. The intersection of this spatial ray with the geometric model of the blade surface at the monitoring point (near the monitoring point) is the three-dimensional coordinate of the monitoring point.

[0084] The equation for spatial rays is:

[0085] (2);

[0086] (3);

[0087] In the formula, , These are the X-axis and Y-axis coordinates of the monitoring point on the camera imaging plane in the k-th frame (the k-th synchronous acquisition time); , , These are the X-axis, Y-axis, and Z-axis coordinates of the monitoring point in the k-th frame in the Cartesian coordinate system of the wind turbine body. , These are the focal lengths along the camera's X and Y axes, respectively. , These are the X-axis and Y-axis coordinates of the projection point of the camera's optical axis onto the camera's imaging plane, respectively. These are nine orthogonal rotation matrix elements composed of the direction cosines of the camera attitude angles:

[0088] ;

[0089] Establish the fitting equation:

[0090] (4);

[0091] The coefficients in equation (4) are the same as those in equation (1). The simultaneous equations (2), (3), and (4) are solved using numerical methods (e.g., Newton's iteration method, Gauss-Newton method, quasi-Newton method, etc.) to obtain the monitoring points of the k-th frame. Corresponding three-dimensional coordinates .

[0092] In S7, a feature point (i.e., a monitoring point) on a wind turbine blade is selected in the initial frame of the image sequence. A Kalman filter and image correlation algorithm are used to continuously track this monitoring point in each subsequent frame, thereby obtaining the time-series image coordinate sequence of the monitoring point. , , ...... .

[0093] Image correlation algorithms can employ various mature image matching and tracking techniques, including: gray-level-based correlation algorithms, such as Normalized Cross-Correlation (NCC), Sum of Absolute Differences (SAD), and Sum of Squared Differences (SSD), which achieve cross-frame localization of monitoring points by selecting feature windows in the reference frame and calculating similarity measures within the search area of ​​subsequent frames; feature-based matching algorithms, such as SIFT, SURF, and ORB, which first extract significant features such as corners and edges in the image and generate feature descriptors, then achieve stable tracking of feature points through descriptor matching; and optical flow-based algorithms, such as the Lucas-Kanade optical flow method, which assumes constant gray levels between adjacent frames, solves the optical flow equation to obtain the motion vector of pixels, thereby achieving dense or sparse tracking of monitoring points. These algorithms can be flexibly selected according to scene lighting, leaf texture, and real-time requirements, providing reliable two-dimensional coordinate observations for Kalman filtering to improve the accuracy and robustness of three-dimensional leaf trajectory monitoring.

[0094] In S8, based on S6, the three-dimensional coordinate sequence corresponding to the image-side coordinate sequence of the time series monitoring points is calculated. .

[0095] Example 2: The difference between this example and Example 1 is that, in finding the P projection points that are closest to the monitoring point in Euclidean distance and are not collinear, a hierarchical neighborhood search combined with collinearity verification is used. The specific steps are as follows:

[0096] Centered on the coordinates of the monitoring point on the image plane, three circular neighborhoods are set from the inside out. The radius of the inner neighborhood is 5 to 8 times the resolution of the image pixels, the radius of the middle neighborhood is twice that of the inner neighborhood, and the radius of the outer neighborhood is twice that of the middle neighborhood. Projection points are selected first from the inner neighborhood. If the number of inner projection points is insufficient, they are added layer by layer to the outside until no less than 15 candidate projection points are selected.

[0097] Collinearity check is performed on the candidate projection points. Three candidate projection points are selected in turn to construct a straight line equation. The vertical distance from the remaining candidate projection points to the straight line is calculated. If the distance is less than three times the image pixel resolution, it is determined to be a collinear point and is removed. This operation is repeated until no three points among the candidate projection points are collinear.

[0098] From the candidate projection points after collinearity verification, select the P points with the smallest Euclidean distance as the final projection points, where P is between 12 and 16.

[0099] Hierarchical neighborhood search avoids near-point selection bias caused by uneven local density in the laser point cloud, ensuring the spatial correlation between projected points and monitoring points and improving the accuracy of subsequent geometric model fitting. Adding three-point collinearity verification and increasing the pixel distance threshold for collinearity removal creates a quantifiable and executable verification standard, preventing collinear points from interfering with quadratic surface fitting. Further optimization of the P-value to 12-16 balances computational complexity and data validity while meeting fitting accuracy requirements.

[0100] Example 3: Based on Example 1, when continuously tracking the monitoring points, a robust illumination compensation and inter-frame verification mechanism is introduced: First, local brightness adaptive equalization is performed on each frame of image to eliminate feature tracking offset caused by sudden changes in illumination, shadows and reflections in the wind field environment; then, the image coordinates of the monitoring points are checked for inter-frame displacement consistency by taking three adjacent frames as a group. If the deviation between the current frame coordinates and the linear extrapolation position of the previous and next frames exceeds the preset pixel threshold, it is determined to be a tracking anomaly, and the current frame coordinates are corrected by the weighted average of the previous and next frames to complete stable tracking.

[0101] By using local brightness adaptive equalization, the interference of complex outdoor lighting, shadows, and blade surface reflections on feature point tracking is effectively reduced, improving tracking stability and robustness under different weather and lighting conditions. At the same time, by using the displacement consistency verification and anomaly correction mechanism of adjacent three frames, tracking drift, mismatch and other problems are automatically identified and corrected, avoiding erroneous coordinates from entering the three-dimensional trajectory calculation, which can improve the continuity reliability and calculation accuracy of the three-dimensional motion trajectory of wind turbine blades.

Claims

1. A wind turbine blade three-dimensional trajectory dynamic monitoring method of fusing images and laser point clouds, characterized in that Includes the following steps: S1. The camera and scanner are fixed by a rigid connection device and used as a dynamic monitoring system for the three-dimensional trajectory of wind turbine blades. The scanner is a three-dimensional laser scanner. S2. Use a planar calibration plate to calibrate the camera and obtain the camera's precise interior orientation elements; S3. Calibrate the relative position and orientation between the camera and the scanner. The specific steps are as follows: S3.1 Arrange cylinders and collect data synchronously: Orient the circular end face of the cylinders toward the camera and scanner, and distribute each cylinder evenly in the camera's imaging field; synchronously trigger the camera and scanner to acquire images containing the cylinders and laser point cloud data respectively; S3.2 Extracting the 2D center coordinates of the circular end face of the cylinder based on the image: Edge detection is performed on the acquired image to extract the contour edges of each cylindrical end face; the extracted contour edges are fitted using an ellipse fitting method, and the center coordinates of the fitted ellipse are taken as the 2D center coordinates of the end face in the image coordinate system, denoted as . , where i is the cylinder's index. , , Let X and Y represent the X and Y coordinates of the center of the circular end face of the i-th cylinder in the image coordinate system, respectively, where n is the number of cylinders. S3.3 Extracting the 3D center coordinates of the cylindrical end face based on laser point cloud: The cylindrical end face is extracted from the laser point cloud. The extracted end face is then contoured to obtain its contour points. These contour points are then fitted with a circle to obtain the center coordinates of the cylindrical end face. These center coordinates are the 3D coordinates of the cylindrical end face's center in the scanner coordinate system. , , , These represent the X, Y, and Z coordinates of the center of the circular end face of the i-th cylinder in the scanner coordinate system. S3.4 Calculate the relative position and orientation of the camera with respect to the scanner: based on the corresponding center coordinates in the image and the laser point cloud. and By combining the collinearity condition equation in photogrammetry and using the spatial resection algorithm, the relative position between the camera and the scanner can be calculated. and attitude rotation matrix , , , These represent the X, Y, and Z coordinates of the camera's projection center in the scanner's coordinate system, respectively. , , These represent the camera's yaw, pitch, and roll angles relative to the scanner, respectively. S4. Using the calibrated dynamic monitoring system, monitor the three-dimensional trajectory of the wind turbine blades during normal operation of the wind turbine, and acquire image data and laser point cloud data. S5. Perform motion compensation correction on a frame-by-frame basis for the monitored laser point cloud data; S6. Determine the neighboring three-dimensional points of each monitoring point in the laser point cloud data, and calculate the three-dimensional coordinates of the monitoring point. The monitoring point is a feature point to be tracked selected on the wind turbine blade. S7. In the initial frame of the image data, for the selected monitoring point, the monitoring point is continuously tracked in each subsequent frame of the image to obtain the time series monitoring point image coordinate sequence. S8. Calculate the three-dimensional coordinate sequence corresponding to the image coordinate sequence of the time series monitoring point. The three-dimensional coordinate sequence completely describes the continuous position change of the monitoring point in space, that is, the three-dimensional motion trajectory of the monitoring point on the wind turbine blade during operation.

2. The method of claim 1, wherein the method further comprises: In S1, an aluminum alloy profile splicing rigid connection device is used to fix the industrial camera and scanner. 6061 aluminum alloy square tube is used as the frame. According to the external size and installation interface of the camera and scanner, they are spliced ​​into a matching integrated bracket. The integrated bracket is reserved with threaded holes that are precisely matched with the camera gimbal interface and the bottom mounting hole of the scanner. It is fastened by stainless steel bolts. The integrated bracket features a thick aluminum alloy rib plate in the middle for triangular reinforcement, and a cutout channel is reserved at the corresponding position of the camera lens and the scanner's scanning port to provide a field of view for data acquisition. The bottom of the integrated bracket is equipped with an adjustable positioning bracket for seamless docking with an external monitoring bracket / pan-tilt unit, preventing relative displacement or shaking of the camera and scanner during data acquisition.

3. The method of claim 1, wherein the method further comprises: In S3, at least four cylinders are used as calibration references to calibrate the relative position and orientation between the camera and the scanner. First, the cylinders are arranged and data is collected synchronously. Then, the two-dimensional center coordinates of the circular end face of the cylinder are extracted based on the image, and the three-dimensional center coordinates of the circular end face of the cylinder are extracted based on the laser point cloud. Finally, the relative position and orientation of the camera relative to the scanner are calculated.

4. The method of claim 1, wherein the method further comprises: In S4, during the monitoring process, the camera and scanner are triggered simultaneously when each frame of data is acquired, so as to obtain image frames and laser point cloud data that are strictly aligned in time.

5. The method of claim 1, wherein the method further comprises: In S5, the motion compensation correction process is as follows: S5.1, with the center of the main axis of rotation of the wind turbine blade as the origin of the coordinate system. Taking the plane of rotation where the wind turbine blades are located as A plane, and the direction perpendicular to that plane is The axial direction, the direction of any wind turbine blade is Axial direction, based on Axial direction and Axis direction determined Establish a rectangular coordinate system for the wind turbine body along the axis direction. All scan point coordinates are transformed to this coordinate system, where a scan point is each independent three-dimensional laser sampling point contained in a single frame of laser point cloud data; S5.2, in In the plane, with Establish a polar coordinate system with the origin, and , Coordinate transformation from rectangular coordinate system to polar coordinate system Down, The coordinates remain unchanged, where, The polar radius represents To the scan point The straight-line distance between projection points in the plane; The polar angle indicates the direction from which the polar angle originates. Around the positive direction of the axis Rotate counterclockwise to the angle between the line connecting the origin and the line drawn from the projection point of the scan point; S5.3、According to the scanning frequency, field of view angle and angular resolution of the scanner, the time interval between adjacent scanning points is calculated , the angular velocity of the wind turbine blade in operation is , combined with and , the angle correction amount in polar coordinates is calculated , the reverse compensation is performed on to obtain the corrected polar coordinates ; S5.4, restore the corrected polar coordinates to rectangular coordinates, and combine them with the original coordinates to form three-dimensional rectangular coordinates. Convert the new three-dimensional rectangular coordinates back to the scanner coordinate system to obtain the motion-compensated three-dimensional coordinates , , , Xk j, Yk j, Zk j represent the X-axis, Y-axis, and Z-axis coordinate values of the jth scanning point in the kth frame of laser point cloud data in the scanner coordinate system, respectively.

6. The method of claim 5, wherein the method further comprises: In S6, the process of searching for neighboring 3D points for each monitoring point in a single frame of laser point cloud data is as follows: S6.1、According to the camera imaging model, the motion compensated three-dimensional coordinates projected to the image plane of the corresponding frame to obtain the corresponding two-dimensional projection coordinates, denoted as , , respectively represent the X-axis and Y-axis coordinate values of the jth scanning point in the kth frame of laser point cloud data in the image coordinate system. S6.2 For a certain monitoring point on the k-th frame image, find P points in the laser scanning point projection coordinate set corresponding to the frame that are closest to the monitoring point in Euclidean distance and are not collinear as projection points, P≥10, where the k-th frame image and the k-th frame laser point cloud data are spatiotemporally aligned data pairs that are synchronously triggered and acquired. S6.3, based on the obtained P projection points, obtain the corresponding three-dimensional laser point coordinates in the scanner coordinate system, denoted as , 、 、 respectively represent the three-dimensional laser point coordinate components corresponding to the pth projection point in the kth frame of image in the scanner coordinate system.

7. The method of claim 6, wherein the method further comprises: In S6, the calculation of the three-dimensional coordinates corresponding to the monitoring point is specifically as follows: S6.

4. Using P projection points, fit the curve using the quadratic surface equation: （1）； In the formula, A, B, C, D, E, F, G, H, I, and J are the coefficients to be solved. The coefficients are obtained by solving the least squares method, and the geometric model of the blade surface near the monitoring point of the kth frame is obtained. S6.5 The image point of the monitoring point in the image and the center of the camera projection form a spatial ray. The intersection of this spatial ray and the geometric model of the blade surface at the monitoring point is the three-dimensional coordinate of the monitoring point.

8. The method for dynamic monitoring of the three-dimensional trajectory of wind turbine blades by fusing images and laser point clouds according to claim 7, characterized in that, In S6.5, the equation for a spatial ray is: （2）； （3）； In the formula, , These are the X-axis and Y-axis coordinates of the monitoring point in the k-th frame on the camera imaging plane, respectively. , , These are the X-axis, Y-axis, and Z-axis coordinates of the monitoring point in the k-th frame in the Cartesian coordinate system of the wind turbine body. , These are the focal lengths along the camera's X and Y axes, respectively. , These are the X-axis and Y-axis coordinates of the projection point of the camera's optical axis onto the camera's imaging plane, respectively. It consists of nine orthogonal rotation matrix elements composed of the direction cosines of the camera attitude angle; Establish the fitting equation: （4）； Solving the system of equations (2), (3), and (4) yields the monitoring points in the k-th frame. Corresponding three-dimensional coordinates .

Images