Offshore wind power blade-oriented stain following and unmanned aerial vehicle disturbance detection method

Abstract

The invention provides an offshore wind power blade-oriented stain following and unmanned aerial vehicle disturbance detection method, which comprises the following steps of: acquiring a fan blade image by using an unmanned aerial vehicle airborne camera, and preprocessing the fan blade image to obtain a fan blade stain track; performing time sequence processing on the stain track of the fan blade, extracting time and space characteristics of the stain track, decoupling an unmanned aerial vehicle offshore wind power blade cleaning interference source, and obtaining a disturbance detection result; and according to a disturbance detection result, an unmanned aerial vehicle flight control strategy is adjusted, and the stability of the unmanned aerial vehicle during fan blade stain cleaning is kept. According to the method, a plurality of disturbance identification modes are adopted for track time sequence processing from visual identification, identification is carried out without depending on a sensor of the unmanned aerial vehicle, the disturbance type is directly identified through the position change of the stain, the change in a time sequence signal has high identifiability and decoupling performance on the track, disturbance identification is efficiently carried out, and the disturbance identification efficiency is improved. And meanwhile, the unmanned aerial vehicle follows the blade positions.

Description

Technical Field

[0001] The present invention relates to the technical field of wind power generation, and in particular to a stain following and drone disturbance detection method for offshore wind power blades. Background Art

[0002] The blades of offshore wind turbines are the core components of offshore wind farms, and they are the key components for converting wind energy into mechanical energy. However, the complexity of the offshore environment, especially factors such as sea breeze, salt spray, humidity, and bird activities, makes it easy for dirt to accumulate on the surface of the blades, such as bird droppings, salt deposition, and the attachment of marine organisms. These dirt not only affect the aerodynamic performance of the blades, resulting in a decrease in wind power generation efficiency, but may also accelerate the corrosion of the blade material and shorten the life of the wind turbine.

[0003] Therefore, regular cleaning of blades has become an important task in wind farm maintenance. Drone technology provides an automated and efficient solution for cleaning wind turbine blades. By using drones equipped with high-pressure water guns or other cleaning tools, cleaning can be performed without stopping the blades, thus reducing downtime.

[0004] In practical applications, drone cleaning also faces multiple technical challenges:

[0005] 1) The wind speed at sea is usually high. The sea wind not only affects the flight stability of the drone, but also may interfere with the precise positioning of the cleaning nozzle. Under high wind speed conditions, it is difficult for the drone to maintain a fixed position, which may lead to a decrease in cleaning accuracy.

[0006] 2) The fan blades continue to rotate while in operation, which increases the difficulty of cleaning. The drone must be able to sense the blade position and movement trajectory in real time to ensure accurate spraying of cleaning fluid during the blade rotation process. This requires the drone to have high-precision tracking and positioning technology.

[0007] 3) UAVs usually use high-pressure water guns or other high-pressure nozzles. However, the recoil generated by the nozzle may affect the attitude control of the UAV, causing it to deviate from the target during the cleaning process. Therefore, the UAV needs to have good anti-interference ability and attitude stability to ensure the continuous stability of the cleaning process.

[0008] Faced with the above three different interferences, the disturbances of the stain position identified by the drone have different obvious characteristics. First of all, for sea breeze disturbances, the wind direction of the sea breeze usually remains relatively unchanged in a short period of time, resulting in the identified stain position trajectory showing a curve along the coastal wind direction in space when the drone is disturbed by the sea breeze; for the rotation of the wind turbine blades, the identified stain position trajectory is an arc trajectory around the rotor of the unit relative to the world coordinate; the recoil generated by the airborne cleaning device during the cleaning operation will cause the stain coordinate trajectory to move in the opposite direction of the water spray. In the actual working environment, the above three interferences will be simultaneously affected, resulting in a large irregular randomness in the measurement estimate, which is roughly the result of the superposition of the three interferences. When performing flight control, if the three interference causes can be decoupled, and then the corresponding flight control anti-interference control is performed according to different sources of disturbance, it will greatly reduce the computing resources generated during flight stability control, and can be adjusted more stably, and the drone position can be maintained more quickly and stably. Therefore, when maintaining offshore wind turbine blades, interference detection is particularly important.

[0009] However, many current studies on UAV flight anti-interference are based on the flight control compensation of the UAV itself. For example, interference observers are mainly used to estimate and compensate for unknown disturbances or modeling errors in the system, but they usually cannot directly identify the specific type of interference, such as "sea breeze interference" or "water spray recoil interference". They prefer to estimate the size and direction of the disturbance and perform real-time compensation in the system. In most cases, the interference observer only estimates the sum of the disturbances, such as changes in external forces or torques of the entire system. The observer cannot directly distinguish whether a specific disturbance comes from sea breeze, water spray recoil, etc. This is because the observer does not have the ability to decode external environmental information. It only infers an unknown disturbance signal based on the error between the system output and the internal state model. Summary of the invention

[0010] In view of the deficiencies in the prior art, the present invention provides a stain following and UAV disturbance detection method for offshore wind turbine blades.

[0011] The technical solution of the present invention is: a stain following and drone disturbance detection method for offshore wind turbine blades, comprising the following steps:

[0012] Step 1: Use the drone’s onboard camera to collect wind turbine blade images at a constant frame rate;

[0013] Step 2: Preprocess the collected fan blade image to obtain the three-dimensional coordinate trajectory P(t)=[x(t), y(t), z(t)] of the fan blade stain in the world coordinate system; wherein [x(t), y(t), z(t)] represents the three-dimensional coordinate sequence that changes with time t;

[0014] Step 3: Perform time series processing on the stain trajectory P(t) = [x(t), y(t), z(t)] of the wind turbine blade, extract the time and space characteristics of the stain trajectory, decouple the interference source of UAV offshore wind turbine blade cleaning, and obtain the disturbance detection result;

[0015] Step 4: According to the results of disturbance detection, adjust the UAV flight control strategy to maintain the stability of the UAV when cleaning the stains on the fan blades.

[0016] Preferably, in step 1, the collected wind turbine blade image includes a normal color image and a depth image; the normal color image and the depth image are used to identify the plane position (x, y) and depth position z of the stain, respectively.

[0017] Preferably, in step 2, the collected fan blade image is preprocessed to obtain a three-dimensional coordinate trajectory P(t)=[x(t), y(t), z(t)] of the fan blade stain relative to the world coordinate system, which specifically includes the following steps:

[0018] S21), dividing each collected ordinary color image into k blocks of S×S grids; wherein each grid is responsible for detecting the target falling within the grid in the image;

[0019] S22), for each grid, using a convolutional neural network to extract feature information of the grid, so as to divide each image into k state feature quantities;

[0020] S23), according to the state transfer equation, the state feature quantity is converted into an intermediate state feature quantity, which is used to capture the spatial features in the image, that is:

[0021] h i+1 =Dh i +Bx i ;

[0022] In the formula, h i represents the intermediate state characteristic quantity of the i-th grid; h i+1 is the intermediate state feature quantity of the i+1th grid; D is the state transfer matrix; B is the transformation matrix of the state feature quantity; x i Represents the state characteristic quantity of the i-th grid;

[0023] S24), based on the state estimation, the observation value y of the image is obtained, and the matching degree between the current state and the target category is calculated according to the observation value y; wherein the calculation formula of the observation value y is:

[0024] y=Ch;

[0025] Wherein, C is the transformation matrix of the intermediate state feature quantity; the observation value y represents the matching degree of each intermediate state feature quantity h for the recognition target, which is used to determine the grid position of the object to be recognized in the image. The position of the recognition target in the image is determined by a multi-layer perceptron MLP, and the input layer of the multi-layer perceptron MLP is a vector composed of the image observation value y and the image position of the state feature quantity, and the output is the recognition bounding box, which includes the center coordinates (x, y) of the bounding box and the width and height (w, l) of the bounding box;

[0026] S25), aligning the depth image and the color image, obtaining the depth information of the area corresponding to the recognition bounding box to determine the three-dimensional coordinate position of the stain, and correcting the three-dimensional coordinate position of the stain;

[0027] S26), repeat steps S21)-S25) to obtain the position trajectory of the fan blade stain.

[0028] Preferably, in step S25), the three-dimensional coordinate position of the stain is corrected, which specifically includes the following steps:

[0029] S251), select multiple images with small disturbance as reference positions; and record the position of the ideal stain i and its position relative to the ideal stain j Distance Among them, the distance The calculation formula is:

[0030]

[0031] S252), construct a reference relative coordinate matrix R, wherein the reference relative coordinate matrix R records the relative position relationship of all ideal stains:

[0032]

[0033] S253), during the repeated collection and positioning process, record the actual stain position and its distance relative to the actual stain j position Calculate the relative position matrix between all actual stains detected And subtract it from the reference relative coordinate matrix R to obtain the deviation matrix ΔR;

[0034]

[0035] S254), using rigid transformation to correct the stain coordinates to ensure that the identified stains conform to the positional relationship in the reference relative coordinate matrix, and the rigid transformation is performed by the following formula:

[0036]

[0037] In the formula, x′ i , y′ i , z′ i is the position coordinate of the i-th stain after rigid transformation; x i ,y i 、z i is the position coordinate of the i-th stain before transformation; t is the translation vector, which is optimized by minimizing the coordinate deviation matrix ΔR;

[0038] S255), dynamically adjust the rotation matrix A and the translation vector t according to the deviation matrix ΔR, and gradually correct the positions of all stains until the relative coordinate error reaches a preset threshold; wherein the total error E is defined as the sum of the squares of the relative position deviations of all stains, that is:

[0039]

[0040] If the total error E is lower than a preset threshold, the correction is considered effective, and the corrected stain coordinates (x′1, y′1, z′1), (x′2, y′2, z′2), … are output as the recorded position.

[0041] Preferably, step 3 specifically includes the following steps:

[0042] S31), normalizing, removing noise and interpolating the stain trajectory P(t)=[x(t), y(t), z(t)];

[0043] S32), the three-dimensional coordinates [x(t), y(t), z(t)] of each time step are mapped to the high-dimensional feature space through linear transformation to generate the high-dimensional feature representation f of the trajectory t ;

[0044] S33), the high-dimensional feature representation after embedding f t Perform position encoding to explicitly incorporate the order information of time steps into the feature representation to ensure that the subsequent self-attention mechanism can capture the temporal order;

[0045] S34), constructing a bidirectional dynamic self-attention mechanism, and capturing the dependency relationship at different time steps in the three-dimensional trajectory data of the embedded features through the bidirectional dynamic self-attention mechanism;

[0046] S35), constructing a spatiotemporal graph convolutional network ST-GCN, capturing the spatial correlation of trajectory data through the spatiotemporal graph convolutional network ST-GCN; by treating each time step of the three-dimensional trajectory as a node in the spatiotemporal graph, and the dependency between time steps as the edge in the spatiotemporal graph, for each time step node, constructing its neighborhood; and aggregating features from neighborhood time steps through the spatiotemporal graph convolutional network ST-GCN:

[0047]

[0048] In the formula, f t (l+1) represents the node features in the time dimension t in the l+1th hidden layer; σ represents the activation function; d t is the degree of time step t; d j is the degree of time step j; W (l) is the weight matrix of the lth hidden layer; is the feature vector of node j in the lth hidden layer; is the set of neighboring time steps of time step t;

[0049] The spatiotemporal graph convolutional network ST-GCN updates the node representation of each time step layer by layer through multi-layer graph convolution operations, captures its global spatiotemporal dependencies, and obtains the spatiotemporal features of each time step;

[0050] S36), integrating the temporal and spatial features output by the bidirectional dynamic self-attention mechanism and the spatiotemporal graph convolutional network ST-GCN;

[0051] S37), input the fused spatiotemporal features into the classifier and use the Softmax activation function to predict the disturbance type and size, and output the final disturbance prediction classification result and the corresponding disturbance size Right now:

[0052]

[0053] In the formula, softmax is a nonlinear activation function; W fc is the weight of the fully connected layer; x is the fused spatiotemporal features after fusion;

[0054] The disturbance size Including the direction and magnitude of the disturbance; the predicted disturbance classification results Including wind turbine blade rotation, sea breeze interference, and nozzle impact interference.

[0055] Preferably, in step S34), the bidirectional dynamic self-attention mechanism includes forward attention and backward attention; the forward attention and backward attention respectively calculate the dependency between the current time step and the past and future time steps to generate a forward attention weight matrix, and the expression of the bidirectional dynamic self-attention mechanism is:

[0056]

[0057] Where Q = W Q f t ′, K = W K f′k 、V=W V f′ k are query matrix, key matrix and value matrix respectively; W Q , W K , W V is the learnable weight matrix, f′ k Represents the features of other time steps; softmax is a nonlinear activation function, which processes the inner product of Q and K to obtain the correlation score matrix between each feature, and performs the inner product of the result with V to obtain the attention map between the features; Represents the scaling factor. When the sequence length is long, the inner product of Q and K is scaled, which is conducive to smoothing the output of softmax and thus obtaining a more appropriate score matrix.

[0058] Preferably, in step S36), the fusion adopts a weighted addition operation to combine the two parts of features, learns the contributions of different feature sources through adaptive weights, and dynamically adjusts the importance of different time steps to obtain a more expressive feature fusion representation.

[0059] The beneficial effects of the present invention are:

[0060] 1. The present invention adopts multiple disturbance identification methods from the trajectory timing processing of visual recognition, does not rely on the drone's own sensors for identification, and directly identifies the disturbance type through the position change of the stain. In the timing signal, this change has high recognition and decoupling on the trajectory, and efficiently identifies disturbances, while completing the drone's tracking of the blade position;

[0061] 2. The present invention uses the location trajectory information of the stain as input to detect the external interference to the drone; the type of external disturbance can be identified, and more targeted adjustments can be made accordingly to ensure the tracking of the stain during flight, so as to achieve the purpose of stable stain cleaning;

[0062] 3. The present invention uses image segmentation as spatial state information and combines it with a state transition model for positioning, thereby improving the efficiency of image processing;

[0063] 4. The present invention adopts the relative position invariance principle to update and correct the stain position in real time in the stain position positioning, and dynamically adjusts the correction parameters in combination with the disturbance detection results to achieve more accurate position tracking;

[0064] 5. The present invention makes full use of the temporal and spatial characteristics of the trajectory during the time series processing, and can well detect the type, direction and size of the disturbance. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the process of the present invention;

[0066] Figure 2 A schematic diagram of stain location following of the present invention;

[0067] Figure 3 It is a flowchart of the positioning estimation algorithm of the present invention;

[0068] Figure 4 It is a schematic diagram of the flow of disturbance detection of the present invention;

[0069] Figure 5 This is a framework diagram of the spatiotemporal graph convolutional network ST-GCN of the present invention. DETAILED DESCRIPTION

[0070] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0071] like Figure 1 As shown, this embodiment provides a stain following and drone disturbance detection method for offshore wind turbine blades, comprising the following steps:

[0072] Step 1: Use the drone’s onboard camera to collect wind turbine blade images at a constant frame rate;

[0073] The fan blade images collected in this embodiment include ordinary color images and depth images; the ordinary color images and depth images are used to identify the plane position (x, y) and depth position z of the stain respectively. The frame rate collected in this embodiment can be set to 15fps; that is, 15 fan blade images are collected every 1s.

[0074] Step 2: Preprocess the collected fan blade image to obtain the three-dimensional coordinate trajectory P(t)=[x(t), y(t), z(t)] of the fan blade stain in the world coordinate system; where [x(t), y(t), z(t)] represents the three-dimensional coordinate sequence that changes with time t; Figure 2 and 3 As shown, the specific steps include:

[0075] S21), dividing each collected ordinary color image into k blocks of S×S grids; wherein each grid is responsible for detecting the target falling within the grid in the image;

[0076] S22), for each grid, using a convolutional neural network to extract feature information of the grid, so as to divide each image into k state feature quantities x;

[0077] S23), according to the state transfer equation, the state feature quantity x is converted into an intermediate state feature quantity, which is used to capture the spatial features in the image, that is:

[0078] h i+1 =Dh i+Bx i ;

[0079] In the formula, h i represents the intermediate state characteristic quantity of the i-th grid; h i+1 is the intermediate state feature quantity of the i+1th grid; D is the state transfer matrix; B is the transformation matrix of the state feature quantity; x i Represents the state characteristic quantity of the i-th grid;

[0080] S24), based on the state estimation, the observation value y of the image is obtained, and the matching degree between the current state and the target category is calculated according to the observation value y; wherein the calculation formula of the observation value y is:

[0081] y=Ch;

[0082] Wherein, C is the transformation matrix of the intermediate state feature quantity; the observation value y represents the matching degree of each intermediate state feature quantity h for the recognition target, which is used to determine the grid position of the target to be recognized in the image. The position of the recognized target in the image is determined by a multi-layer perceptron MLP, and the input layer of the multi-layer perceptron MLP is a vector composed of the image observation value y and the image position of the state feature quantity, and the output is the recognition bounding box, which includes the center coordinates (x, y) of the bounding box and the width and height (w, l) of the bounding box;

[0083] S25), aligning the depth image and the color image, obtaining the depth information of the area corresponding to the recognition bounding box, determining the three-dimensional coordinate position of the stain, and correcting the three-dimensional coordinate position of the stain;

[0084] S26), repeat steps S21)-S25) to obtain the position trajectory of the fan blade stain.

[0085] Step 3: Perform time series processing on the trajectory of the wind turbine blade stain P(t) = [x(t), y(t), z(t)], extract the time and space characteristics of the trajectory, extract the time and space characteristics of the stain trajectory, decouple the interference source of the UAV offshore wind turbine blade cleaning, and obtain the disturbance detection result; Figure 4 As shown, the specific steps include:

[0086] S31), normalizing, removing noise and interpolating the stain trajectory P(t)=[x(t), y(t), z(t)];

[0087] S32), the three-dimensional coordinates [x(t), y(t), z(t)] of each time step are mapped to the high-dimensional feature space through linear transformation to generate the high-dimensional feature representation f of the trajectory t ;

[0088] S33), the high-dimensional feature representation after embedding f tPerform position encoding to explicitly incorporate the order information of time steps into the feature representation to ensure that the subsequent self-attention mechanism can capture the temporal order;

[0089] S34), constructing a bidirectional dynamic self-attention mechanism, and capturing the dependency relationship at different time steps in the three-dimensional trajectory data of the embedded features through the bidirectional dynamic self-attention mechanism;

[0090] S35), constructing a spatiotemporal graph convolutional network ST-GCN, capturing the spatial correlation of trajectory data through the spatiotemporal graph convolutional network ST-GCN; by treating each time step of the three-dimensional trajectory as a node in the spatiotemporal graph, and the dependency between time steps as the edge in the spatiotemporal graph, for each time step node, constructing its neighborhood; and aggregating features from neighborhood time steps through the spatiotemporal graph convolutional network ST-GCN:

[0091]

[0092] In the formula, f t (l+1) represents the node features in the time dimension t in the l+1th hidden layer; σ represents the activation function; d t is the degree of time step t; d j is the degree of time step j; W (l) is the weight matrix of the lth hidden layer; f j (l) is the feature vector of node j in the lth hidden layer; is the set of neighboring time steps of time step t;

[0093] The spatiotemporal graph convolutional network ST-GCN updates the node representation of each time step layer by layer through multi-layer graph convolution operations, captures its global spatiotemporal dependencies, and obtains the spatiotemporal features of each time step;

[0094] S36), integrating the features output by the bidirectional dynamic self-attention mechanism and the spatiotemporal graph convolutional network ST-GCN;

[0095] S37) Input the fused spatiotemporal features into the classifier with regression task and use the Softmax activation function to predict the disturbance type and size, and output the final disturbance prediction classification result and the corresponding disturbance size Right now:

[0096]

[0097] In the formula, softmax is a nonlinear activation function; W fc is the weight of the fully connected layer; is the fused spatiotemporal features after fusion;

[0098] Perturbation size Including the direction and magnitude of the disturbance; the predicted disturbance classification results Including wind turbine blade rotation, sea breeze interference, and nozzle impact interference.

[0099] Step 4: According to the results of disturbance detection, adjust the UAV flight control strategy to maintain the stability of the UAV when cleaning the stains on the fan blades.

[0100] As a preferred embodiment of the present invention, in step S25), the three-dimensional coordinate position of the stain is corrected, which specifically includes the following steps:

[0101] S251), select multiple images with small disturbance as reference positions; and record the position of the ideal stain i and its position relative to the ideal stain j Distance Among them, the distance The calculation formula is:

[0102]

[0103] S252), construct a reference relative coordinate matrix R to record the relative position relationship of all ideal stains:

[0104]

[0105] S253), during repeated collection and positioning, record the stain position and its distance relative to the position of stain j Calculate the relative position matrix between all detected stains And compared with the reference relative coordinate matrix R, the deviation matrix ΔR is obtained;

[0106]

[0107] S254), using rigid transformation to correct the stain coordinates to ensure that the identified stains conform to the positional relationship in the reference relative coordinate matrix, and the rigid transformation is performed by the following formula:

[0108]

[0109] In the formula, x′ i , y′ i , z′ i is the position coordinate of the i-th stain after rigid transformation; x i ,y i 、z i is the position coordinate of the i-th stain before transformation; t is the translation vector, which is optimized by minimizing the coordinate deviation matrix ΔR;

[0110] S255), dynamically adjust the rotation matrix A and the translation vector t according to the deviation matrix ΔR, and gradually correct the positions of all stains until the relative coordinate error reaches a preset threshold; wherein the total error E is defined as the sum of the squares of the relative position deviations of all stains, that is:

[0111]

[0112] If the total error E is lower than a preset threshold, the correction is considered effective, and the corrected stain coordinates (x′1, y′1, z′1), (x′2, y′2, z′2), … are output as the recorded position.

[0113] As a preferred embodiment of the present invention, in step S31), the trajectory P(t) of the fan blade stain is

[0114] [x(t), y(t), z(t)] is normalized, denoised and interpolated; the details are as follows:

[0115] Normalize the coordinates of each stain point to the range [0,1], that is:

[0116]

[0117] in, represents the normalized fan blade stain coordinates; P(t) represents the real-time coordinates of the detected fan blade stains; P(t) max 、P(t) min They are respectively the maximum and minimum values ​​of the stain coordinates of the fan blades.

[0118] Then the moving average method is used to smooth the trajectory, which can effectively remove high-frequency noise. The formula of the moving average method is:

[0119]

[0120] Where w is the size of the moving window, is the coordinate of the fan blade stain after smoothing at time t, and i represents the i-th time step.

[0121] If there are missing data points in the stain trajectory, use linear interpolation to fill them. Assuming that the stain P(t) is missing, linear interpolation is performed based on the positions of the two stains before and after the stain P(t), that is:

[0122]

[0123] Among them, P(t-1) and P(t+1) respectively represent the positions of the two stains before and after the stain P(t).

[0124] As a preferred embodiment of this invention, in step S32), a high-dimensional feature representation f of the trajectory is generated by mapping it to a high-dimensional feature space through linear transformation. t , specifically:

[0125] The stain coordinates P(t) are regarded as a three-dimensional vector of length L, and the homology weight matrix W maps the trajectory to a high-dimensional vector, that is:

[0126] f t =P(t)W T +b

[0127] Where T represents the transpose operation; b represents the bias matrix.

[0128] As a preferred embodiment of this invention, in step S33), the embedded features are position-encoded, specifically:

[0129] Assume that the position encoding vector P t The dimension of is the same as the input feature dimension d after trajectory mapping, and then the position encoding is constructed using sine and cosine functions; for each time step t:

[0130]

[0131] Where i is the dimension index of the feature, t is the time step, and d is the dimension of the input feature;

[0132] Then the position code P t Add to the input features to get the new input feature representation:

[0133] f t ′=ft t +P t .

[0134] As a preferred embodiment of this embodiment, in step S34), the bidirectional dynamic self-attention mechanism includes forward attention and backward attention; the forward attention and backward attention respectively calculate the dependency between the current time step and the past and future time steps to generate a forward attention weight matrix. The expression of the bidirectional dynamic self-attention mechanism is:

[0135]

[0136] Where Q = W Q f t ′, K = W K f′ k 、V=W V f′ k are query matrix, key matrix and value matrix respectively; W Q , W K , W Vis the learnable weight matrix, f′ k Represents the features of other time steps; softmax is a nonlinear activation function, which processes the inner product of Q and K to obtain the correlation score matrix between each feature, and performs the inner product of the result with V to obtain the attention map between the features; Represents the scaling factor. When the sequence length is long, the inner product of Q and K is scaled, which is conducive to smoothing the output of softmax and thus obtaining a more appropriate score matrix.

[0137] As a preferred embodiment of this invention, in step S35), if Figure 5 As shown, the spatiotemporal graph convolutional network ST-GCN includes a graph convolutional network GCN and a CNN network; in this embodiment, a learnable weight matrix is ​​introduced and bitwise multiplied with the adjacency matrix to give important edges and nodes in the adjacency matrix larger weights and suppress the weights of unimportant edges and nodes; the weighted adjacency matrix A and the input f′ are sent to the graph convolutional network GCN for calculation to achieve aggregation of spatial dimension information; then the CNN network is used to achieve aggregation of temporal dimension information, and the residual structure is introduced to calculate and obtain the residual.

[0138] As a preferred embodiment of this embodiment, in step S35), the graph convolutional network GCN includes an input layer, an output layer and multiple hidden layers; the input layer receives the adjacency matrix A and the node feature matrix of the graph, normalizes the adjacency matrix A, and outputs the initial node feature representation of the graph;

[0139] Each hidden layer propagates node features according to the adjacency matrix and updates the node representation through the weight matrix and nonlinear activation function; multiple hidden layers can capture more complex local structures and feature information on the graph.

[0140] The output layer is used to generate the final node representation and perform the overall representation of the graph. The output layer generates a probability distribution for classification through an activation function.

[0141] Preferably, in step S36), the feature fusion adopts a weighted addition operation to combine the two parts of the features, learn the contribution of different feature sources through adaptive weights, and dynamically adjust the importance of different time steps to obtain a more expressive feature fusion representation. The fused spatiotemporal features It is expressed as:

[0142]

[0143] Where α1 and α2 represent the weights of features X1 and X2, respectively. X1 and X2 represent the output features of the bidirectional dynamic self-attention mechanism and the output features of ST-GCN, respectively.

[0144] Preferably, in step S4), according to the direction of the sea breeze disturbance in the disturbance detection result, the roll angle and pitch angle of the UAV are adjusted in real time to offset the influence of the wind on the attitude of the UAV; and the roll angle is increased to counteract the lateral force in crosswind, and the pitch angle is increased to reduce wind resistance in headwind;

[0145] And according to the size of the wind speed in the disturbance detection results, the thrust output of the quadcopter is adjusted in real time.

[0146] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A stain following and drone disturbance detection method for offshore wind turbine blades, characterized in that: The steps include: Step 1: Use the drone’s onboard camera to collect images of wind turbine blades; Step 2: Preprocess the collected fan blade image to obtain the three-dimensional coordinate trajectory P(t)=[x(t), y(t), z(t)] of the fan blade stain in the world coordinate system; wherein [x(t), y(t), z(t)] represents the three-dimensional coordinate sequence that changes with time t; Step 3: Perform time series processing on the stain trajectory P(t) = [x(t), y(t), z(t)] of the wind turbine blade, extract the time and space characteristics of the stain trajectory, decouple the interference source of UAV offshore wind turbine blade cleaning, and obtain the disturbance detection result; Step 4: According to the results of disturbance detection, adjust the UAV flight control strategy to maintain the stability of the UAV when cleaning the stains on the fan blades.

2. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 1 is characterized in that: In step S1), the collected wind turbine blade image includes a normal color image and a depth image; the normal color image and the depth image are used to identify the plane position (x, y) and the depth position z of the stain respectively.

3. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 1 is characterized in that: Obtaining the three-dimensional coordinate trajectory of the fan blade stain P(t)=[x(t), y(t), z(t)] specifically includes the following steps: S21), dividing each collected ordinary color image into k blocks of S×S grids; wherein each grid is used to detect the target falling within the grid in the image; S22), for each grid, using a convolutional neural network to extract feature information of the grid, so as to divide each image into k state feature quantities; S23), according to the state transfer equation, the state feature quantity is converted into an intermediate state feature quantity, which is used to capture the spatial features in the image, that is: h i+1 =Dh i +Bx i ; In the formula, h i represents the intermediate state characteristic quantity of the i-th grid; h i+1 is the intermediate state feature quantity of the i+1th grid; D is the state transfer matrix; B is the transformation matrix of the state feature quantity; x i Represents the state characteristic quantity of the i-th grid; S24), based on the state estimation, the observation value y of the image is obtained, and the matching degree between the current state and the target category is calculated according to the observation value y; wherein the calculation formula of the observation value y is: y=Ch; Wherein, C is the transformation matrix of the intermediate state feature h; the observation value y represents the matching degree of each intermediate state feature h for the identified target, which is used to determine the grid position of the target to be identified in the image. The position of the identified target in the image is determined by a multi-layer perceptron MLP, and the input layer of the multi-layer perceptron MLP is a vector composed of the image observation value y and the image position of the state feature, and the output is the recognition bounding box; S25), aligning the depth image and the color image, obtaining the depth information of the area corresponding to the recognition bounding box to determine the three-dimensional coordinate position of the stain, and correcting the three-dimensional coordinate position of the stain; S26), repeat steps S21)-S25) to obtain the trajectory of the fan blade stain.

4. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 3 is characterized in that: In step S25), the three-dimensional coordinate position of the stain is corrected, which specifically includes the following steps: S251), select multiple images with small disturbance as reference positions; and record the position of the ideal stain i and its position relative to the ideal stain j Distance Among them, the distance The calculation formula is: S252), construct a reference relative coordinate matrix R, and record the relative position relationship of all ideal stains through the reference relative coordinate matrix R: S253), record the actual stain position during repeated collection and positioning and its distance relative to the actual stain j position Calculate the relative position matrix between all actual stains detected And subtract it from the reference relative coordinate matrix R to obtain the deviation matrix ΔR; S254), using rigid transformation to correct the stain coordinates to ensure that the identified stains conform to the positional relationship in the reference relative coordinate matrix, and the rigid transformation is performed by the following formula: In the formula, x i ′ ,y i ′ 、z i ′ is the position coordinate of the i-th stain after rigid transformation; x i ,y i 、z i is the position coordinate of the i-th stain before transformation; D is the state transfer matrix; t is the translation vector, which is optimized by minimizing the coordinate deviation matrix ΔR; S255), dynamically adjust the rotation matrix A and the translation vector t according to the deviation matrix ΔR, and gradually correct the positions of all stains until the relative coordinate error reaches a preset threshold; wherein the total error E is expressed as the sum of the squares of the relative position deviations of all stains, that is: If the total error E is lower than the preset threshold, the correction is considered effective and the corrected stain coordinates (x1 ′ ,y1 ′ ,z1 ′ ),(x2 ′ ,y2 ′ ,z2 ′ ),…as the record location.

5. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 1 is characterized in that: Step S3) specifically includes the following steps: S31), normalizing, removing noise and interpolating the stain trajectory P(t)=[x(t), y(t), z(t)]; S32), the three-dimensional coordinates [x(t), y(t), z(t)] of each time step are mapped to the high-dimensional feature space through linear transformation to generate the high-dimensional feature representation f of the trajectory t ; S33), the high-dimensional feature representation after embedding f t Perform position encoding to explicitly incorporate the order information of time steps into the feature representation to ensure that the subsequent self-attention mechanism can capture the temporal order; S34), constructing a bidirectional dynamic self-attention mechanism, and capturing the dependency relationship at different time steps in the three-dimensional trajectory data of the embedded features through the bidirectional dynamic self-attention mechanism; S35), constructing a spatiotemporal graph convolutional network ST-GCN, and capturing the spatial correlation of trajectory data through the spatiotemporal graph convolutional network ST-GCN; By treating each time step of the 3D trajectory as a node in the space-time graph, and the dependencies between time steps as edges in the space-time graph, for each time step node, its neighborhood is constructed; and the features from the neighborhood time steps are aggregated through the space-time graph convolution network ST-GCN, that is: In the formula, represents the node features in the time dimension t in the l+1th hidden layer; σ represents the activation function; d t is the degree of time step t; d j is the degree of time step j; W (l) is the weight matrix of the lth hidden layer; is the feature vector of node j in the lth hidden layer; is the set of neighboring time steps of time step t; S36), integrating the temporal features and spatial features output by the bidirectional dynamic self-attention mechanism and the spatiotemporal graph convolutional network ST-GCN; S37) Input the fused spatiotemporal features into the classifier with regression task and use the Softmax activation function to predict the disturbance type and size, and output the final disturbance prediction classification result and the corresponding disturbance size Right now: In the formula, softmax is a nonlinear activation function; W fc is the weight of the fully connected layer; is the fused spatiotemporal features after fusion.

6. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 5 is characterized in that: In step S32), the high-dimensional feature representation f of the trajectory is generated by mapping it to a high-dimensional feature space through linear transformation t , specifically: The stain coordinates P(t) are regarded as a three-dimensional vector of length L, and the homology weight matrix W maps the trajectory to a high-dimensional vector, that is: f t =P(t)W T +b Where T represents the transpose operation; b represents the bias matrix.

7. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 5 is characterized in that: In step S33), the embedded features are positionally encoded, specifically: Assume that the position encoding vector P t The dimension of is the same as the input feature dimension d after trajectory mapping, and then the position encoding is constructed using sine and cosine functions; for each time step t: Where i is the dimension index of the feature, t is the time step, and d is the dimension of the input feature; Then the position code P t Add to the input features to get the new input feature representation: f t ′=f t +P t 。 8. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 5 is characterized by: The spatiotemporal graph convolutional network ST-GCN includes a graph convolutional network GCN and a CNN network; by introducing a learnable weight matrix and bitwise multiplying it with the adjacency matrix, it is used to give important edges and nodes in the adjacency matrix a larger weight and suppress the weights of unimportant edges and nodes; the weighted adjacency matrix A is combined with the input f ′ It is sent to the graph convolutional network GCN for calculation to realize the aggregation of spatial dimension information; then the CNN network is used to realize the aggregation of temporal dimension information, and the residual structure is introduced to calculate and obtain the residual.

9. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 5 is characterized by: In step S36), the feature fusion adopts weighted addition operation to combine the two parts of features, learn the contribution of different feature sources through adaptive weights, and dynamically adjust the importance of different time steps to obtain a more expressive feature fusion representation. The fused spatiotemporal features It is expressed as: Where α1 and α2 represent the weights of features X1 and X2, respectively. X1 and X2 represent the output features of the bidirectional dynamic self-attention mechanism and the output features of ST-GCN, respectively.

10. The stain following and drone disturbance detection method for offshore wind turbine blades according to claim 1, characterized in that: In step S4), the roll angle and pitch angle of the UAV are adjusted in real time according to the direction of the sea wind disturbance in the disturbance detection result; and the roll angle is increased to counteract the lateral force when the wind is crosswind, and the pitch angle is increased to reduce wind resistance when the wind is headwind; According to the wind speed in the disturbance detection result, the thrust output of the quadcopter is adjusted in real time.

Images