Method for monitoring three-dimensional displacement of offshore wind turbine blade based on visual depth estimation

Abstract

This invention discloses a method for monitoring the three-dimensional displacement of offshore wind turbine blades based on visual depth estimation, belonging to the field of wind turbine condition monitoring and machine vision image measurement technology. The method includes: acquiring video of wind turbine blade operation using a monocular camera to obtain the two-dimensional pixel coordinates of key points on the blade; inputting the video frames into a pre-trained monocular depth estimation model to extract the depth values ​​of the key points, and converting the two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system by combining the camera intrinsic parameter matrix; selecting a reference target for camera pose estimation and motion compensation to obtain the spatial coordinates in the reference coordinate system; and calculating the three-dimensional displacement of the key points on the blade based on the compensated coordinates. This invention achieves three-dimensional displacement monitoring of the blade without the need for binocular cameras and contact sensors, and eliminates false displacements caused by camera shaking during offshore operations through motion compensation, providing accurate data support for offshore wind turbine blade condition monitoring and anomaly early warning.

Description

Technical Field

[0001] This invention relates to the field of wind turbine condition monitoring and machine vision image measurement technology, and in particular to a method for monitoring the three-dimensional displacement of offshore wind turbine blades based on visual depth estimation. Background Technology

[0002] Offshore wind turbine blades are subjected to strong winds, waves, and alternating loads over long periods, making them prone to dynamic responses such as flapping, oscillating, and torsional displacements during operation. Blade displacement, especially the three-dimensional displacement at the blade tip and midsection, reflects the blade's load-bearing state, structural stiffness changes, and potential abnormal vibrations. Therefore, non-contact, continuous three-dimensional displacement monitoring of offshore wind turbine blades has become a key focus in this field. Currently, wind turbine blade displacement or vibration monitoring mainly employs contact sensors and laser ranging methods. Contact sensors typically need to be installed on or inside the blade surface, making deployment and maintenance difficult, and they are easily affected by the high humidity and strong winds of the offshore environment. Laser ranging methods can usually only measure distance changes in specific directions or local locations, making it difficult to obtain the three-dimensional displacement of key points across the entire blade.

[0003] In contrast, visual methods offer advantages such as simple equipment, low cost, and flexible installation, making them suitable for long-distance monitoring of offshore wind turbines. However, existing conventional visual methods often rely on target detection, template matching, or optical flow tracking, typically only obtaining two-dimensional displacement of the blades within the image plane, which is insufficient to fully reflect the true three-dimensional motion of the blades. Some studies employ binocular vision methods to acquire spatial position information, but binocular systems require two cameras for simultaneous acquisition and precise calibration, increasing system cost and installation complexity. Furthermore, under low-light conditions and long-distance observation at sea, matching accuracy is easily affected by environmental factors. Therefore, how to acquire blade depth displacement while maintaining the advantages of simple structure and flexible deployment of monocular cameras, and how to eliminate the spurious displacement effects caused by the displacement and attitude changes of the offshore camera itself, are urgent technical problems that need to be solved in the current three-dimensional displacement monitoring of offshore wind turbine blades. Summary of the Invention

[0004] The purpose of this invention is to provide a method for monitoring the three-dimensional displacement of offshore wind turbine blades based on visual depth estimation. The method aims to use a monocular camera combined with a depth estimation model to realize the three-dimensional coordinate inversion of the blades, and to correct pseudo-displacement through camera pose calculation, thereby achieving low-cost and highly reliable non-contact three-dimensional displacement monitoring.

[0005] To achieve the above objectives, this invention provides a method for monitoring the three-dimensional displacement of offshore wind turbine blades based on visual depth estimation, comprising the following steps: S1. Use a monocular camera to acquire video of the operation of offshore wind turbine blades, preprocess the video frames to obtain a corrected image frame sequence; identify the preset blade key points in the corrected image frame sequence and obtain the two-dimensional pixel coordinates of the blade key points. S2. Input the corrected image frame sequence into the pre-trained monocular depth estimation model to obtain the depth map corresponding to the blade region of each image frame; extract the depth value corresponding to each blade key point in the depth map based on the two-dimensional pixel coordinates of the blade key points; combine the pre-calibrated monocular camera intrinsic parameter matrix to convert each blade key point from two-dimensional pixel coordinates to three-dimensional coordinates in the camera coordinate system. S3. Select a reference target in the corrected image frame sequence, extract stable feature points, estimate the pose change of the monocular camera between consecutive frames, the pose change includes rotation matrix and translation vector, and use rotation matrix and translation vector to perform motion compensation on the three-dimensional coordinates in the camera coordinate system to obtain the spatial coordinates in the reference coordinate system. S4. Based on the spatial coordinates after motion compensation, calculate the three-dimensional displacement information of key points of each blade.

[0006] Preferably, in S2, the pre-trained monocular depth estimation model is constructed in the following manner: Collect images or video frames of offshore wind turbine blades under different operating conditions, blade angles, shooting distances, lighting conditions, weather conditions, and sea conditions to establish a wind turbine blade image sample set; the image sample set shall include at least the overall blade outline, blade tip, blade root, leading edge, trailing edge, blade centerline, and pre-defined blade key point annotation information. For a subset of samples in the sample set, corresponding true depth labels, relative depth labels, or geometric scale constraint labels are generated by combining wind turbine blade design dimensions, blade 3D models, UAV close-up measurement data, simulation rendering images, or on-site calibration data. Relative depth labels are annotation information that does not contain actual spatial scale but only represents the relative size relationship between depth values. The actual spatial scale refers to the real distance scale expressed in physical length units. Relative depth labels only reflect the relative size of depth values ​​between different pixels or regions, not their actual physical distance. The image frames in the sample set are arranged in chronological order, with temporal relationships between adjacent frames. Obtain the intrinsic parameter matrix, distortion parameters, and geometric scale information of the wind turbine blades from the monocular camera, and establish the correspondence between image coordinates, depth estimates, and actual spatial scales. The geometric scale information of the wind turbine blades includes one or more of the following: blade length, blade root diameter, blade chord length distribution, blade preset key point spacing, hub center position, tower feature dimensions, or nacelle structural feature dimensions.

[0007] Preferably, the monocular depth estimation model takes a monocular image frame as input and outputs a depth map of the blade region; the monocular depth estimation model adopts an encoder-decoder structure, including an image feature extraction module, a blade structure prior fusion module, a scale perception module and a depth decoding module; The image feature extraction module is used to extract the relationship features between the leaf contour, edges, texture, and background. The blade structure prior fusion module is used to introduce blade continuity, slender structure and key point topological relationships; The scale sensing module is used to introduce blade geometric scale information; The depth decoding module is used to output a depth map of the blade area; For the Frame input image The depth output of the monocular depth estimation model is expressed as follows: ; in, This represents a monocular depth estimation model. For model parameters, For the first Frame input image, This is the corresponding depth estimation result.

[0008] Preferably, during the training of the monocular depth estimation model, depth supervision constraints are introduced based on the real depth label, relative depth constraints are introduced based on the relative depth label, depth smoothing constraints are introduced, leaf geometric scale constraints are introduced based on the geometric scale constraint label, and video temporal consistency constraints are introduced based on the temporal relationship between adjacent frames. The total loss function of the model is expressed as: ; in, For the total loss function, For deep supervision loss or relative depth loss, For deep smoothing loss, For blade geometry constraint loss, For video temporal consistency loss, , , and These are the corresponding weighting coefficients.

[0009] Preferably, S1 specifically includes: S11. Perform distortion correction, image enhancement, image stabilization preprocessing, and time synchronization processing on the video frames to obtain the corrected image frame sequence; S12. In the corrected image frame sequence, the preset blade key points are identified by target detection, semantic segmentation, key point detection, template matching or optical flow tracing methods, and the two-dimensional pixel coordinates of each blade key point in the image coordinate system are obtained. No. The first frame The pixel coordinates of each blade key point are represented as follows: ; in, Indicates the first The first frame The horizontal pixel coordinates of each leaf key point in the image coordinate system Indicates the first The first frame The vertical pixel coordinates of each leaf key point in the image coordinate system; The two-dimensional displacement of the same leaf key point in adjacent frames is represented as follows: ; in, For the first The key points of the blade are in the first The frame is a two-dimensional displacement of the image plane relative to the previous frame.

[0010] Preferably, the camera intrinsic parameter matrix in S2 is represented as follows: ; in, For the camera intrinsic parameter matrix, and These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image, respectively. and The coordinates of the principal point in the image; The key points of each blade are converted from two-dimensional pixel coordinates to three-dimensional coordinates in the camera coordinate system, based on the following relationship: ; in, Indicates the first Frame number The three-dimensional position of each blade key point in the camera coordinate system , and The first Frame number The coordinate components of each blade key point in the camera coordinate system along the X, Y, and Z axes.

[0011] Preferably, S3 specifically includes: S31. Select at least one of the following as reference targets: wind turbine tower, nacelle outer contour, hub area, offshore platform fixed structure or artificial marker point, and extract stable feature points in the reference area. S32. Estimate the pose change of a monocular camera in consecutive frames using at least one of the following methods: PnP pose calculation based on reference target feature points, visual odometry, multi-frame feature matching, or inertial measurement unit data fusion. The rotation matrix and translation vector of the frame camera relative to the reference coordinate system are respectively and ; S33. Transform the key points of each blade from the camera coordinate system to the reference coordinate system. The transformation relationship is expressed as follows: ; in, For the first Frame number The spatial coordinates of each blade key point in the reference coordinate system after motion compensation.

[0012] Preferably, the three-dimensional displacement information in S4 is calculated in the following way: The three-dimensional displacement of the blade keypoints between adjacent frames or the reference frame is represented as follows: ; in, For the first The spatial coordinates of each blade key point in the reference frame (i.e., the initial state); The three-dimensional displacement amplitude is expressed as: ; in, For the first Frame number Three-dimensional displacement amplitude of key points on the blade , and The first Frame number The displacement components of each blade key point in three spatial directions.

[0013] Preferably, the preset key points of the leaf include at least one point in the leaf tip, the middle of the leaf, and the root region of the leaf.

[0014] Therefore, compared with the prior art, the present invention adopts the above-mentioned three-dimensional displacement monitoring method for offshore wind turbine blades based on visual depth estimation, which has the following beneficial effects: (1) Reduce equipment costs and deployment difficulty. This invention can realize three-dimensional displacement monitoring of blades using only a monocular camera, without the need for simultaneous acquisition and complex calibration of binocular cameras, and without the need to install contact sensors on the blade surface. This effectively reduces the hardware cost, installation complexity and post-maintenance difficulty of the monitoring system, making it more suitable for long-distance and long-term monitoring applications of offshore wind turbines.

[0015] (2) Achieve accurate monitoring of three-dimensional displacement of the blade. This invention uses a monocular depth estimation model to obtain the depth values ​​of key points of the blade, and combines the camera intrinsic parameter matrix to convert the two-dimensional pixel coordinates into three-dimensional coordinates in the camera coordinate system. This overcomes the limitation of ordinary monocular vision, which can only obtain two-dimensional displacement of the image plane, and enables the monocular system to obtain the displacement of the blade in the depth direction, thereby fully reflecting the true three-dimensional motion state of the blade.

[0016] (3) Eliminating pseudo-displacement interference caused by camera sway at sea. This invention addresses the problem of camera movement caused by wind, waves, platform vibration, or ship sway in offshore wind turbine monitoring. By selecting a fixed reference target for camera pose calculation and motion compensation, pseudo-displacement caused by camera movement is separated from blade observation displacement. This effectively improves the stability and reliability of blade three-dimensional displacement monitoring in complex offshore environments, providing accurate data support for blade structural health monitoring and anomaly early warning.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the three-dimensional displacement monitoring method for offshore wind turbine blades based on visual depth estimation according to the present invention. Figure 2 This is a schematic diagram of the three-dimensional geometric model of the wind turbine blade and the spatial coordinate annotation of key points of the blade according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the deployment and target marking of the monocular camera on the offshore wind turbine according to an embodiment of the present invention; Figure 4 This is a schematic diagram of key frame image extraction from wind turbine blade rotation video in an embodiment of the present invention; Figure 5 This is a time history diagram of the two-dimensional displacement results of key points of the blade in an embodiment of the present invention; Figure 6 This is a time history diagram of the three-dimensional displacement results of key points of the blade in an embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example like Figure 1 As shown in the figure, this embodiment provides a method for monitoring the three-dimensional displacement of offshore wind turbine blades based on visual depth estimation, including the following steps: S1. Use a monocular camera to acquire video of the operation of offshore wind turbine blades, preprocess the video frames to obtain a corrected image frame sequence; identify preset blade key points in the corrected image frame sequence and obtain the two-dimensional pixel coordinates of the blade key points; the preset blade key points include points at least one location in the blade tip, blade middle and blade root regions.

[0022] Specifically, it includes: S11. Perform distortion correction, image enhancement, image stabilization preprocessing, and time synchronization processing on the video frames to obtain the corrected image frame sequence; S12. In the corrected image frame sequence, the preset blade key points are identified by target detection, semantic segmentation, key point detection, template matching or optical flow tracing methods, and the two-dimensional pixel coordinates of each blade key point in the image coordinate system are obtained. In this embodiment, as Figure 4 As shown, a video of the blade rotation and vibration of a wind turbine in operation was obtained through simulation, and the two-dimensional motion information of key points on the blade was extracted based on a visual displacement algorithm. The simulated wind turbine is a 15MW wind turbine with a support structure height of 150m, a rotor diameter of 240m, a rotor elevation angle of 6°, and a video duration of 2s.

[0023] No. The first frame The pixel coordinates of each blade key point are represented as follows: ; in, Indicates the first The first frame The horizontal pixel coordinates of each leaf key point in the image coordinate system Indicates the first The first frame The vertical pixel coordinates of each leaf key point in the image coordinate system; The two-dimensional displacement of the same leaf key point in adjacent frames is represented as follows: ; in, For the first The key points of the blade are in the first The frame is a two-dimensional displacement of the image plane relative to the previous frame.

[0024] like Figure 5 The image shows the two-dimensional displacement tracking results of preset leaf key points in the image coordinate system. Specifically, the leaf region and preset leaf key points (such as leaf tip, leaf root, etc.) are first identified in the corrected image frame, and optical flow tracing or template matching algorithms are used to perform temporal tracking of the pixel position changes of the same key point in consecutive frames, thereby drawing the leaf key points in the horizontal direction of the image ( (axis) and the vertical direction of the image ( The two-dimensional pixel displacement time history curve (axis). Figure 5 It can be seen that the two-dimensional displacement maintains a certain fluctuation during the normal monitoring period, but a feature mutation occurs in a specific frame (around 1.7 seconds). This mutation is mainly caused by the change in image features when the blade rotates to a specific angle, indicating that pure image plane tracking is easily affected by changes in blade attitude.

[0025] Assuming the starting position of the video is (0,0), the two-dimensional displacement of the leaf tip key point in the first second on the image plane is: ; The two-dimensional displacement time history can be used to obtain the two-dimensional position of the blade key points in the coordinate system of each frame image, providing input for subsequent three-dimensional coordinate inversion by combining monocular depth estimation results.

[0026] S2. Input the corrected image frame sequence into the pre-trained monocular depth estimation model to obtain the depth map corresponding to the blade region of each image frame; extract the depth value corresponding to each blade key point in the depth map based on the two-dimensional pixel coordinates of the blade key points; combine the pre-calibrated monocular camera intrinsic parameter matrix to convert each blade key point from two-dimensional pixel coordinates to three-dimensional coordinates in the camera coordinate system. The pre-trained monocular depth estimation model is constructed in the following way: a. Collect images or video frames of offshore wind turbine blades under different operating conditions, blade angles, shooting distances, lighting conditions, weather conditions, and sea conditions, and establish a wind turbine blade image sample set; the image sample set shall include at least the overall outline of the blade, blade tip, blade root, leading edge, trailing edge, blade centerline, and the annotation information of preset blade key points; In this embodiment, image samples can be constructed through multiple channels, including on-site monocular camera acquisition, UAV inspection images, and rendered images of 3D models of wind turbine blades. The offshore wind turbine blade image sample set includes, but is not limited to, one or more images during normal operation, shutdown, startup, shutdown, yaw adjustment, and pitch control, as well as images of the blades in vertically upward, vertically downward, horizontal, and at different tilt angles.

[0027] b. For a subset of samples in the sample set, combine the wind turbine blade design dimensions, blade 3D model, UAV close-up measurement data, simulation rendering images or on-site calibration data to generate corresponding real depth labels, relative depth labels or geometric scale constraint labels; the image frames in the sample set are arranged in chronological order, and there is a temporal relationship between adjacent frames. c. Obtain the intrinsic parameter matrix, distortion parameters, and geometric scale information of the wind turbine blades from the monocular camera, and establish the correspondence between image coordinates, depth estimates, and actual spatial scales; the geometric scale information of the wind turbine blades includes one or more of the following: blade length, blade root diameter, blade chord length distribution, blade preset key point spacing, hub center position, tower feature dimensions, or nacelle structural feature dimensions.

[0028] In addition, it is necessary to obtain the geometric location information of key points on the actual wind turbine blades. The geometric location information of key points on the blades can be determined based on blade design parameters, the 3D model of the blades, or the spanwise scale of the blades. For example... Figure 2 As shown, several key points can be set along the blade spanwise direction, including the leaf root point, the 25% leaf length position point, the 50% leaf length position point, the 75% leaf length position point, and the leaf tip point. By annotating these key points, the correspondence between the fixed leaf root reference, the blade spanwise scale, and the spatial positions of each key point can be established, providing geometric constraints for subsequent 3D blade position inversion. Simultaneously, as... Figure 3 As shown, a fixed reference point is set at the center of the wind turbine hub, and preset key points of the blade (such as blade tip and blade root) are marked on the blade to provide a reference benchmark for subsequent blade 3D coordinate inversion and camera pose calculation.

[0029] The monocular depth estimation model takes monocular image frames as input and outputs a depth map of the blade region. The monocular depth estimation model adopts an encoder-decoder structure, including an image feature extraction module, a blade structure prior fusion module, a scale perception module, and a depth decoding module. The image feature extraction module is used to extract the relationship features between the leaf contour, edges, texture, and background. The blade structure prior fusion module is used to introduce blade continuity, slender structure and key point topological relationships; The scale sensing module is used to introduce blade geometric scale information; The depth decoding module is used to output a depth map of the blade area; Monocular depth estimation models can employ convolutional neural networks, visual Transformer networks, or encoder-decoder structures that combine convolutional networks with attention mechanisms.

[0030] For the Frame input image The depth output of the monocular depth estimation model is expressed as follows: ; in, This represents a monocular depth estimation model. For model parameters, For the first Frame input image, This is the corresponding depth estimation result.

[0031] During the training of the monocular depth estimation model, depth supervision constraints are introduced based on the real depth labels, relative depth constraints are introduced based on the relative depth labels, depth smoothing constraints are introduced, leaf geometric scale constraints are introduced based on the geometric scale constraint labels, and video temporal consistency constraints are introduced based on the temporal relationship between adjacent frames. The total loss function of the model is expressed as: ; in, For the total loss function, For deep supervision loss or relative depth loss, For deep smoothing loss, For blade geometry constraint loss, For video temporal consistency loss, , , and These are the corresponding weighting coefficients.

[0032] By applying the above constraints, the blade depth results output by the model are made to conform to the actual geometric scale and continuous structural characteristics of the blade, thereby reducing the impact of sea surface reflection, fog, changes in sea level, background occlusion and illumination fluctuations on the depth estimation results.

[0033] Before formal data acquisition or processing, the monocular camera needs to be calibrated to obtain the camera intrinsic parameter matrix and distortion parameters. Taking a certain model of NV200 camera as an example, the calibration board used is the GP240 calibration board, with a black and white checkerboard grid of 12×9, and each grid side length is 15mm. The camera calibration program in the MATLAB toolbox is used to calibrate the camera on multiple black and white checkerboard images. After importing the acquired checkerboard images into the MATLAB camera calibration program, the actual size of each checkerboard grid is input. In this embodiment, the size of each checkerboard grid is set to 15mm. After importing the images, the program automatically extracts the corner points of the checkerboard grid and performs camera calibration based on the corner point coordinates. After completing the calibration by clicking Calibrate, the camera's intrinsic parameters and distortion parameters can be viewed. To improve calibration accuracy, images with large pixel errors can be deleted based on the reprojection error results, and the calibration can be performed again to obtain more accurate camera intrinsic parameters and distortion parameters.

[0034] like Figure 3 The diagram shown illustrates the deployment and monitoring target of a monocular camera on an offshore wind turbine, as provided in an embodiment of the present invention. The monocular camera is installed on an offshore monitoring platform (such as an offshore maintenance platform, the external structure of the wind turbine tower, an offshore substation platform, or a maintenance vessel), and its field of view covers the blade rotation area, enabling continuous acquisition of video footage of blade operation.

[0035] The camera intrinsic parameter matrix is ​​represented as follows: ; in, For the camera intrinsic parameter matrix, and These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image, respectively. and The coordinates of the principal point in the image; The camera calibration results obtained in this embodiment are as follows: ; In the above calibration results, the focal length and The unit is pixels, and the principal point coordinates are... and The unit is also pixels.

[0036] By using camera intrinsic parameters and distortion parameters, distortion correction can be performed on the acquired images. Furthermore, by combining blade geometric scale information such as blade length, blade root diameter, blade keypoint spacing, and hub center position, a correspondence between image coordinates, depth estimates, and actual spatial scale can be established, providing a scale basis for subsequent blade 3D coordinate inversion.

[0037] The key points of each blade are converted from two-dimensional pixel coordinates to three-dimensional coordinates in the camera coordinate system, based on the following relationship: ; in, Indicates the first Frame number The three-dimensional position of each blade key point in the camera coordinate system , and The first Frame number The coordinate components of each blade key point in the camera coordinate system along the X, Y, and Z axes. The depth value output by the monocular depth estimation model is directly assigned.

[0038] In this embodiment, video frames are input into the trained monocular depth estimation model to obtain a depth map of the leaf region, and the first video frame is read. Depth value corresponding to each key point of the blade In this embodiment, the camera intrinsic parameter matrix of the NV200 described above is used. Simultaneously, the depth value at the key position at the tip of the blade is taken. , For time. Combined with the camera intrinsic parameter matrix, the keypoints are transformed from two-dimensional image coordinates to three-dimensional coordinates in the camera coordinate system.

[0039] Taking the first second as an example, the three-dimensional coordinate inversion relationship is expressed as follows: ; in, , and They are respectively the 1st second and the 1st second. The three coordinate components of a key point (i.e., the leaf tip position) in the camera coordinate system; This indicates the three-dimensional position of the key point in the camera coordinate system.

[0040] This allows us to obtain the initial three-dimensional displacement results of the blade keypoints in the X, Y, and Z directions. This result includes both displacement components within the image plane and displacement components in the depth direction, serving as the foundational input for subsequent camera pose calculation, motion compensation, and three-dimensional displacement recognition.

[0041] S3. Select a reference target in the corrected image frame sequence, extract stable feature points, estimate the pose change of the monocular camera between consecutive frames. The pose change includes rotation matrix and translation vector. Use rotation matrix and translation vector to perform motion compensation on the three-dimensional coordinates in the camera coordinate system to obtain the spatial coordinates in the reference coordinate system. The reference coordinate system is a fixed spatial coordinate system established based on a selected reference target (such as the hub center as the origin). The reference target is a physical structure that remains stationary within the camera's field of view (such as a wind turbine tower, the outer contour of the nacelle, or the hub area). Therefore, the reference coordinate system does not change with camera movement. Camera pose calculation is to solve for the rotation matrix and translation vector of the camera coordinate system relative to this fixed reference coordinate system.

[0042] Stable feature points refer to image feature points that can be stably monitored and matched in consecutive video frames, including corner points, edge points, or scale-invariant feature transformation points. These feature points still have a high repetition rate under conditions of illumination change, viewpoint change, or partial occlusion.

[0043] Specifically, it includes: S31. Select at least one of the following as reference targets: wind turbine tower, nacelle outer contour, hub area, offshore platform fixed structure or artificial marker point. Use fixed positions such as hub center, tower bottom center or nacelle top center on the reference target as fixed reference points and extract stable feature points in the reference area. S32. Estimate the pose change of a monocular camera in consecutive frames using at least one of the following methods: PnP pose calculation based on reference target feature points, visual odometry, multi-frame feature matching, or inertial measurement unit data fusion. The rotation matrix and translation vector of the frame camera relative to the reference coordinate system are respectively and ; S33. Transform the key points of each blade from the camera coordinate system to the reference coordinate system. The transformation relationship is expressed as follows: ; in, For the first Frame number The spatial coordinates of each blade key point in the reference coordinate system after motion compensation.

[0044] As can be seen from the camera calibration, the rotation matrix and translation vector of the camera relative to the reference coordinate system are respectively... and Since the simulated video does not have lens distortion, , The relationship between the key points of the blade in the first second and their transformation from the camera coordinate system to the reference coordinate system can be expressed as follows: ; in, For the 1st second The spatial coordinates of each blade key point (i.e., the blade tip position) after camera motion compensation; it should be noted that in actual video shooting, the camera may have problems such as attitude changes, position changes and lens distortion, so the rotation matrix and translation vector need to be determined based on the actual camera calibration and pose estimation results.

[0045] Through the above coordinate transformation, the pseudo-displacement caused by the camera's own translation, pitch, roll and yaw motion can be separated from the blade observation results, so that the compensated three-dimensional coordinates mainly reflect the blade's own vibration and deformation.

[0046] S4. Based on the spatial coordinates after motion compensation, calculate the three-dimensional displacement information of key points of each blade.

[0047] Calculated in the following way: Using the first frame of the video sequence as the reference frame, record the initial spatial coordinates of the key points of each blade. The three-dimensional displacement of the blade keypoints between adjacent frames or the reference frame is expressed as: ; in, For the first The spatial coordinates of each blade key point in the reference frame (i.e., the initial state); Using the first frame of the video sequence as the reference frame, record the initial spatial coordinates of the key points of each blade. Assume the spatial position of the leaf tip in the reference frame. If the value is 0, then the three-dimensional displacement of the blade keypoint in the first second between adjacent frames or the reference frame is expressed as: ; Based on the compensated three-dimensional coordinates of the key points on the blade, calculate the three-dimensional displacement information of the blade tip, the middle of the blade, or other monitoring points, including lateral displacement, vertical displacement, and depth displacement.

[0048] The three-dimensional displacement amplitude is expressed as: ; in, For the first Frame number Three-dimensional displacement amplitude of key points on the blade , and The first Frame number The displacement components of each blade key point in three spatial directions.

[0049] like Figure 6As shown, this is the three-dimensional displacement time history result of the key points of the blade in the spatial reference coordinate system after processing in steps S2 and S3. Specifically, it combines the depth values ​​extracted by the monocular depth estimation model with the camera intrinsic parameter matrix. Figure 5 The two-dimensional pixel coordinates in the image are converted to three-dimensional coordinates in the camera coordinate system; subsequently, a fixed reference point (such as...) is introduced. Figure 3 Using a fixed reference point at the center of the wheel hub (as shown), camera pose calculation and motion compensation are performed between consecutive frames to eliminate errors caused by camera platform shaking. Figure 6 The X, Y, and Z axis displacement curves clearly show that the key points of the blade exhibit regular periodic vibrations in the spatial coordinate system (such as the reciprocating motion along the Z-axis corresponding to the blade's axial direction), and the three-dimensional displacement results effectively correct for... Figure 5 The method of the present invention can accurately extract the three-dimensional spatial displacement information of the blade under the unsteady conditions of the marine camera by tracking the sudden fluctuations in a specific frame.

[0050] The three-dimensional displacement amplitude of the key point at the blade tip position in the first second is: ; Therefore, this embodiment can obtain the three-dimensional displacement components and amplitude of the blade's key points at the first second. For the complete video, the three-dimensional displacement time history of the blade tip or other key points can be calculated frame by frame using the same method, thereby realizing the three-dimensional displacement monitoring of offshore wind turbine blades.

[0051] Therefore, this invention adopts the above-mentioned three-dimensional displacement monitoring method for offshore wind turbine blades based on visual depth estimation. It uses a monocular camera combined with a depth estimation model, eliminating the need for simultaneous calibration of binocular systems and installation of contact sensors, significantly reducing equipment costs and deployment difficulty. This enables non-contact monitoring of the blade's three-dimensional displacement, overcoming the limitation of ordinary monocular vision, which can only acquire two-dimensional displacement. At the same time, it introduces a camera pose calculation and motion compensation mechanism to effectively separate the pseudo-displacement of the camera's own motion caused by sea waves, platform vibrations, etc., improving the stability and reliability of displacement monitoring in harsh marine environments. This provides accurate data support for blade vibration status identification, structural health monitoring, and anomaly early warning, and has good engineering application value.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for monitoring the three-dimensional displacement of offshore wind turbine blades based on visual depth estimation, characterized in that, Includes the following steps: S1. Use a monocular camera to collect video of the operation of offshore wind turbine blades, preprocess the video frames, and obtain a corrected image frame sequence. In the corrected image frame sequence, pre-defined blade key points are identified, and the two-dimensional pixel coordinates of the blade key points are obtained; S2. Input the corrected image frame sequence into the pre-trained monocular depth estimation model to obtain the depth map corresponding to the leaf region of each image frame. Based on the two-dimensional pixel coordinates of the blade key points, the depth values ​​corresponding to each blade key point are extracted from the depth map. By combining the pre-calibrated monocular camera intrinsic parameter matrix, the key points of each blade are transformed from two-dimensional pixel coordinates to three-dimensional coordinates in the camera coordinate system; S3. Select a reference target in the corrected image frame sequence, extract stable feature points, estimate the pose change of the monocular camera between consecutive frames, the pose change includes rotation matrix and translation vector, use rotation matrix and translation vector to perform motion compensation on the three-dimensional coordinates in the camera coordinate system, and obtain the spatial coordinates in the reference coordinate system. S4. Based on the spatial coordinates after motion compensation, calculate the three-dimensional displacement information of each blade key point; In S2, the pre-trained monocular depth estimation model is constructed in the following way: Collect images or video frames of offshore wind turbine blades under different operating conditions, blade angles, shooting distances, lighting conditions, weather conditions, and sea conditions to establish a wind turbine blade image sample set; the image sample set shall include at least the overall blade outline, blade tip, blade root, leading edge, trailing edge, blade centerline, and pre-defined blade key point annotation information. For a subset of samples in the sample set, combine wind turbine blade design dimensions, blade 3D models, UAV close-up measurement data, simulation rendering images, or on-site calibration data to generate corresponding real depth labels, relative depth labels, or geometric scale constraint labels. The image frames in the sample set are arranged in chronological order, and there is a temporal relationship between adjacent frames; Obtain the intrinsic parameter matrix, distortion parameters, and geometric scale information of the wind turbine blades from the monocular camera, and establish the correspondence between image coordinates, depth estimates, and actual spatial scales; Relative depth labels are annotation information that does not contain actual spatial scale information and is only used to represent the relative size relationship between depth values; The geometric dimensional information of wind turbine blades includes one or more of the following: blade length, blade root diameter, blade chord length distribution, blade preset key point spacing, hub center position, tower feature dimensions, or nacelle structural feature dimensions.

2. The method according to claim 1, characterized in that, The monocular depth estimation model takes monocular image frames as input and outputs a depth map of the blade region. The monocular depth estimation model adopts an encoder-decoder structure, including an image feature extraction module, a blade structure prior fusion module, a scale perception module, and a depth decoding module. The image feature extraction module is used to extract the features of leaf contour, edge, texture and background; The blade structure prior fusion module is used to introduce blade continuity, slender structure and key point topological relationships; The scale sensing module is used to introduce blade geometric scale information; The depth decoding module is used to output a depth map of the blade area; For the Frame input image The depth output of the monocular depth estimation model is expressed as follows: ； in, This represents a monocular depth estimation model. For model parameters, For the first Frame input image, This is the corresponding depth estimation result.

3. The method according to claim 1, characterized in that, During the training of the monocular depth estimation model, depth supervision constraints are introduced based on the real depth label, relative depth constraints are introduced based on the relative depth label, depth smoothing constraints are introduced, leaf geometric scale constraints are introduced based on the geometric scale constraint label, and video temporal consistency constraints are introduced based on the temporal relationship between adjacent frames. The total loss function of the model is expressed as: ； in, For the total loss function, For deep supervision loss or relative depth loss, For deep smoothing loss, For blade geometry constraint loss, For video temporal consistency loss, , , and These are the corresponding weighting coefficients.

4. The method according to claim 1, characterized in that, S1 specifically includes: S11. Perform distortion correction, image enhancement, image stabilization preprocessing, and time synchronization processing on the video frames to obtain the corrected image frame sequence; S12. In the corrected image frame sequence, the preset blade key points are identified by target detection, semantic segmentation, key point detection, template matching or optical flow tracing methods, and the two-dimensional pixel coordinates of each blade key point in the image coordinate system are obtained. No. The first frame The pixel coordinates of each blade key point are represented as follows: ； in, Indicates the first The first frame The horizontal pixel coordinates of each leaf key point in the image coordinate system Indicates the first The first frame The vertical pixel coordinates of each leaf key point in the image coordinate system; The two-dimensional displacement of the same leaf key point in adjacent frames is represented as follows: ； in, For the first The key points of the blade are in the first The frame is a two-dimensional displacement of the image plane relative to the previous frame.

5. The method according to claim 4, characterized in that, The camera intrinsic parameter matrix described in S2 is represented as follows: ； in, For the camera intrinsic parameter matrix, and These are the equivalent focal lengths of the camera in the horizontal and vertical directions of the image, respectively. and The coordinates of the principal point in the image; The key points of each blade are converted from two-dimensional pixel coordinates to three-dimensional coordinates in the camera coordinate system, based on the following relationship: ； in, Indicates the first Frame number The three-dimensional position of each blade key point in the camera coordinate system , and The first Frame number The coordinate components of each blade key point in the camera coordinate system along the X, Y, and Z axes.

6. The method according to claim 1, characterized in that, S3 specifically includes: S31. Select at least one of the following as reference targets: wind turbine tower, nacelle outer contour, hub area, offshore platform fixed structure or artificial marker point, and extract stable feature points in the reference area. The pose change of a monocular camera in consecutive frames is estimated using at least one of the following methods: PnP pose calculation based on reference target feature points, visual odometry, multi-frame feature matching, or inertial measurement unit data fusion. The rotation matrix and translation vector of the frame camera relative to the reference coordinate system are respectively and ; S33. Transform the key points of each blade from the camera coordinate system to the reference coordinate system. The transformation relationship is expressed as follows: ； in, For the first Frame number The spatial coordinates of each blade key point in the reference coordinate system after motion compensation.

7. The method according to claim 1, characterized in that, The three-dimensional displacement information described in S4 is calculated in the following way: The three-dimensional displacement of the blade keypoints between adjacent frames or the reference frame is represented as follows: ； in, For the first Spatial coordinates of key points of each blade in the reference frame; The three-dimensional displacement amplitude is expressed as: ； in, For the first Frame number Three-dimensional displacement amplitude of key points on the blade , and The first Frame number The displacement components of each blade key point in three spatial directions.

8. The method according to claim 1, characterized in that, The preset key points of the leaf include at least one point in the leaf tip, the middle of the leaf, and the root region of the leaf.

Images