Wind power generation blade fault detection system

Abstract

The invention relates to the technical field of fault detection, and discloses a wind power generation blade fault detection system, which comprises a video processing module for carrying out key frame extraction on a rotation video of a wind power generation blade to obtain an image data set with N key frame images; the space-time diagram data construction module is used for constructing space-time diagram data according to the M sub-region images of the N key frame images; the blade fault detection module inputs the space-time diagram data into the trained blade fault detection model, and the output value represents the fault type of the M sub-region images of the wind driven generator blade; according to the invention, key frame extraction is carried out on the rotation video of the wind driven generator blade, each key frame image is divided into a plurality of sub-regions to construct space-time diagram data, and time and space information of the space-time diagram data is aggregated through a blade fault detection model. Therefore, the recognition precision of the fault type of the wind driven generator blade is improved, and the situation of wrong recognition is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of fault detection, and more particularly to a wind power generation blade fault detection system. Background Art

[0002] In the rapid development of the wind energy industry, it is crucial to ensure the efficient and safe operation of wind turbines. Among them, the status monitoring of wind turbine blades is particularly critical, because blade failure will not only reduce the power generation efficiency of the wind turbine, but may also cause serious safety accidents.

[0003] The announcement number is CN112950634B, and the patent name is a Chinese patent for a method for identifying wind turbine blade damage based on drone inspection. It discloses: using a drone to take images of wind turbine blades, and using a regional convolutional neural network model based on an attention mechanism to classify and regress the blade damage in the wind turbine blade images.

[0004] The announcement number is CN112233091B, and the patent name is a Chinese patent for a method for detecting and locating damage in wind turbine blade images. It discloses: a sliding window is slid along the horizontal or vertical direction of the surface image of the wind turbine blade, and a cascade strong classifier is used to determine whether the sliding window contains a damaged area. If a damaged area is contained, a convolutional neural network model is used to identify the damage type of the damaged area, including: glass fiber damage, cracks, surface damage and corrosion. The sample labels of the training samples for training the cascade strong classifier and the convolutional neural network model are obtained by manual labeling.

[0005] The above solutions all use machine vision algorithms to analyze a single image to detect damage on the surface of wind turbine blades. However, wind turbines are often located in open areas and are susceptible to complex and changeable natural environments. Various attachments may accumulate on the blade surface due to severe weather conditions, such as rain, ice and snow, dust, and even bird excrement. The attachments may be mistakenly identified as damage in a single image, resulting in possible misidentification. Summary of the invention

[0006] The purpose of the present invention is to provide a wind turbine blade fault detection system to solve the technical problems in the above-mentioned background technology.

[0007] The present invention provides a wind power blade fault detection system, comprising: A video acquisition module is used to collect the rotation video of the wind turbine blades through a high-definition camera mounted on a drone within a preset time period T; A video processing module is used to extract key frames from the rotation video of the wind turbine blades to obtain an image data set with N key frame images; A sub-region division module is used to traverse N key frame images of the image data set and divide the wind turbine blades in each key frame image into M sub-region images through a sliding window; A spatiotemporal graph data construction module, which is used to construct spatiotemporal graph data according to M sub-region images of N key frame images; The spatiotemporal graph data includes: N sub-graph data, and the N sub-graph data are respectively mapped to the N key frame images; Each subgraph data consists of M nodes and edges between nodes; A mapping relationship is established between the mth node of the nth sub-graph data and the mth sub-region image in the nth key frame image, where 1≤m≤M, 1≤n≤N; A blade fault detection model building module, which is used to build and train a blade fault detection model; A blade fault detection module, which is used to input the spatiotemporal graph data into the trained blade fault detection model, and the output value represents the fault type of the M sub-region images of the wind turbine blade; Failure types include: cracks, corrosion, coating damage, adsorption of deposits, and no damage.

[0008] Furthermore, T, N, M, the size of the sliding window, and the step size of the sliding window are all custom parameters.

[0009] Furthermore, extracting key frames from the rotation video of the wind turbine blades to obtain an image dataset with N key frame images includes the following steps: Step S201, dividing the rotation video of the wind turbine blades into frames at equal time intervals to obtain K frame images, where K is a custom parameter greater than N; Step S202, grayscale processing is performed on the K frame images to obtain a grayscale image; Step S203, converting the K grayscale images into grayscale histogram representation; The grayscale histogram is represented by a vector with 256 dimensions, and the value of each dimension represents the number of pixels with the same grayscale level as the dimension index; Step S204, normalizing the K grayscale histograms; The normalized grayscale histogram is represented by a vector with 256 dimensions, and the sum of the values ​​of all dimensions is 1; Step S205, calculating the Bhattacharyya distance between two adjacent normalized grayscale histograms, and if it is determined that the Bhattacharyya distance is greater than or equal to a preset distance threshold, the grayscale image of the next frame is used as a key frame image; The calculation formula for calculating the Bhattacharyya distance between two adjacent normalized grayscale histograms is as follows: ; Where 1≤k≤K, represents the Bhattacharyya distance between the kth normalized grayscale histogram and the k-1th normalized grayscale histogram. and Respectively represent the value of the i-th dimension of the k-th and k-1-th normalized grayscale histograms; Step S206, if it is determined that the number of key frame images is greater than N, the preset distance threshold is increased, and if it is determined that the number of key frame images is less than N, the preset distance threshold is lowered until N key frame images are obtained as the image data set.

[0010] Furthermore, the edges between nodes include: edges constructed between adjacent nodes of the same subgraph data; edges constructed between the same nodes of adjacent subgraph data.

[0011] Further, the blade fault detection model includes: a first hidden layer, N second hidden layers, M extractors, and M classifiers; The first hidden layer inputs a sub-region image that has a mapping relationship with the M nodes of the N sub-graph data of the spatiotemporal graph data, and outputs a feature map; The nth second hidden layer inputs the nth subgraph data of the spatiotemporal graph data and outputs an update matrix, the update matrix includes M row vectors, and the mth row vector of the update matrix output by the nth second hidden layer represents the update feature of the mth node of the nth subgraph data; M extractors are used to extract M row vectors of the update matrix output by the Nth second hidden layer and input them into M classifiers; The classification space of M classifiers represents the fault types of M sub-region images of the wind turbine blade.

[0012] Furthermore, the calculation formula of the first hidden layer includes: ; ; in represents the feature map of the first hidden layer output, represents the sub-region image of the first hidden layer input, A represents the intermediate feature map, and denote the first weight parameter and the second weight parameter respectively, and represent the first bias parameter and the second bias parameter respectively, ReLU represents the ReLU activation function, and softmax represents the softmax activation function.

[0013] Furthermore, the calculation formula of the nth second hidden layer includes: ; ; ; ; ; ; ; in represents the update matrix of the nth second hidden layer output, Represents the updated features of the mth node of the nth subgraph data, and Respectively represent the first intermediate feature and the second intermediate feature of the mth node of the nth subgraph data, Assign a value of 0, and Represents the feature graph of the mth node of the nth and n-1th subgraph data respectively, Assign a value of 0, Represents the set of nodes that have edges connected to the mth node in the n subgraph data. Represents the feature graph of the vth node of the nth subgraph data, , and They represent the candidate second intermediate feature, update gate and reset gate of the mth node of the nth subgraph data respectively. , , , , , , , and represent the third weight parameter, the fourth weight parameter, the fifth weight parameter, the sixth weight parameter, the seventh weight parameter, the eighth weight parameter, the ninth weight parameter, the tenth weight parameter and the eleventh weight parameter respectively, , , and represent the third bias parameter, the fourth bias parameter, the fifth bias parameter and the sixth bias parameter respectively, Indicates stacking the updated features of the M nodes of the nth subgraph data, concat indicates concatenation, and Vec indicates vectorization. represents point-by-point multiplication, sigmoid represents the sigmoid activation function, and tanh represents the hyperbolic tangent activation function.

[0014] Furthermore, sample labels of training samples used to train the blade fault detection model are obtained through manual annotation.

[0015] Furthermore, the cross entropy loss function is specified as the loss function of the blade fault detection model.

[0016] The present invention provides a storage medium storing non-temporary computer-readable instructions. When the non-temporary computer-readable instructions are executed by a computer, the above-mentioned wind power generation blade fault detection system is executed.

[0017] The beneficial effects of the present invention are as follows: the present invention extracts key frames from the rotation video of the wind turbine blades, divides each key frame image into multiple sub-regions to construct spatiotemporal graph data, and aggregates the information of the spatiotemporal graph data in time and space through a blade fault detection model, thereby improving the recognition accuracy of the fault type of the wind turbine blades and reducing the occurrence of erroneous recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a wind turbine blade fault detection system of the present invention; Figure 2 is a flow chart of extracting key frame images of the present invention; Figure 3 It is a test result diagram of the present invention.

[0019] In the figure: a video acquisition module 101, a video processing module 102, a sub-region division module 103, a spatiotemporal graph data construction module 104, a blade fault detection model construction module 105, and a blade fault detection module 106. DETAILED DESCRIPTION

[0020] like Figure 1 to Figure 3 As shown, a wind turbine blade fault detection system comprises: The video acquisition module 101 is used to collect the rotation video of the wind turbine blades through the high-definition camera carried by the drone within a preset time period T; The video processing module 102 is used to extract key frames from the rotation video of the wind turbine blades to obtain an image data set with N key frame images; A sub-region division module 103 is used to traverse N key frame images of the image data set and divide the wind turbine blades in each key frame image into M sub-region images through a sliding window; A spatiotemporal graph data construction module 104, which is used to construct spatiotemporal graph data according to M sub-region images of N key frame images; The spatiotemporal graph data includes: N sub-graph data, and the N sub-graph data are respectively mapped to the N key frame images; Each subgraph data consists of M nodes and edges between nodes; A mapping relationship is established between the mth node of the nth sub-graph data and the mth sub-region image in the nth key frame image, where 1≤m≤M, 1≤n≤N; A blade fault detection model building module 105, which is used to build and train a blade fault detection model; A blade fault detection module 106, which is used to input the spatiotemporal graph data into the trained blade fault detection model, and the output value represents the fault type of the M sub-region images of the wind turbine blade; Failure types include: cracks, corrosion, coating damage, adsorption of deposits, and no damage.

[0021] In one embodiment of the present invention, T, N, M, the size of the sliding window and the step size of the sliding window are all custom parameters. Preferably, T is set to 20 seconds, N is set to 10, M is set to 50, the length of the sliding window is set to 1 / 25 of the length of the wind turbine blade, the width of the sliding window is set to 1 / 2 of the width of the wind turbine blade, and the step size of the sliding window is set to 1 / 25 of the length of the wind turbine blade.

[0022] It should be noted that, generally, the rotation speed of a wind turbine blade is between 10 and 20 revolutions per minute. Calculated at 15 revolutions per minute, the time for the blade to rotate once is about 4 seconds. A rotation video of the blade rotating for 5 weeks is collected by a drone equipped with a high-definition camera, so T is set to 20 seconds; then 40 frame images can be obtained by dividing the frame at a time interval of 0.5 seconds, so N can be set to a positive integer less than or equal to 40, for example, N is set to 10; M is set according to the size of the wind turbine blade, the size of the sliding window and the step size of the sliding window. The length of the sliding window is set to 1 / 25 of the length of the wind turbine blade, the width of the sliding window is set to 1 / 2 of the width of the wind turbine blade, and the step size of the sliding window is set to 1 / 25 of the length of the wind turbine blade, then M=25*2=50.

[0023] In one embodiment of the present invention, Figure 2 As shown, extracting key frames from a rotating video of a wind turbine blade to obtain an image dataset with N key frame images includes the following steps: Step S201, dividing the rotation video of the wind turbine blades into frames at equal time intervals to obtain K frame images, where K is a custom parameter greater than N, and preferably, K is set to 40; Step S202, grayscale processing is performed on the K frame images to obtain a grayscale image; Step S203, converting the K grayscale images into grayscale histogram representation; The grayscale histogram is represented by a vector with 256 dimensions, and the value of each dimension represents the number of pixels with the same grayscale level as the dimension index; For example, the value of the 20th dimension is 10, that is, the dimension index is 20, which means that there are 10 pixels with a grayscale level of 20. Step S204, normalizing the K grayscale histograms; The normalized grayscale histogram is represented by a vector with 256 dimensions, and the sum of the values ​​of all dimensions is 1; Step S205, calculating the Bhattacharyya distance between two adjacent normalized grayscale histograms, and if it is determined that the Bhattacharyya distance is greater than or equal to a preset distance threshold, the grayscale image of the next frame is used as a key frame image; Step S206, if it is determined that the number of key frame images is greater than N, the preset distance threshold is increased, and if it is determined that the number of key frame images is less than N, the preset distance threshold is lowered until N key frame images are obtained as the image data set.

[0024] In one embodiment of the present invention, the calculation formula for calculating the Bhattacharyya distance between two adjacent normalized grayscale histograms is as follows: ; Where 1≤k≤K, represents the Bhattacharyya distance between the kth normalized grayscale histogram and the k-1th normalized grayscale histogram. and They represent the values ​​of the i-th dimension of the k-th and k-1-th normalized grayscale histograms respectively.

[0025] In one embodiment of the present invention, the K grayscale histograms may also be normalized using a Min-Max normalization method.

[0026] In one embodiment of the present invention, the Euclidean distance, Manhattan distance, etc. between two adjacent normalized grayscale histograms may also be calculated to extract the key frame image.

[0027] In one embodiment of the present invention, the edges between nodes include: edges constructed between adjacent nodes of the same subgraph data; edges constructed between the same nodes of adjacent subgraph data.

[0028] It should be noted that in the same subgraph data (i.e., the same key frame), the edges between adjacent nodes can capture the spatial correlation between adjacent sub-regions on the wind turbine blades, which helps to identify changes in local features, such as cracks, edges of corrosion areas, etc.; the edges of the same node in adjacent subgraph data (i.e., different key frames) can reflect changes in the time series, such as the diffusion of attachments, etc., thereby effectively reducing the occurrence of false fault identification.

[0029] In one embodiment of the present invention, the blade fault detection model comprises: a first hidden layer, N second hidden layers, M extractors and M classifiers; The first hidden layer inputs a sub-region image that has a mapping relationship with the M nodes of the N sub-graph data of the spatiotemporal graph data, and outputs a feature map; The nth second hidden layer inputs the nth subgraph data of the spatiotemporal graph data and outputs an update matrix, the update matrix includes M row vectors, and the mth row vector of the update matrix output by the nth second hidden layer represents the update feature of the mth node of the nth subgraph data; M extractors are used to extract M row vectors of the update matrix output by the Nth second hidden layer and input them into M classifiers; The classification space of M classifiers represents the fault types of M sub-region images of the wind turbine blade.

[0030] In one embodiment of the present invention, the first hidden layer may be constructed based on CNN (convolutional neural network model) or Transformer (converter model).

[0031] In one embodiment of the present invention, the calculation formula of the first hidden layer includes: ; ; in represents the feature map of the first hidden layer output, represents the sub-region image of the first hidden layer input, A represents the intermediate feature map, and denote the first weight parameter and the second weight parameter respectively, and represent the first bias parameter and the second bias parameter respectively, ReLU represents the ReLU activation function, and softmax represents the softmax activation function.

[0032] In one embodiment of the present invention, the calculation formula of the nth second hidden layer includes: ; ; ; ; ; ; ; in represents the update matrix of the nth second hidden layer output, Represents the updated features of the mth node of the nth subgraph data, and Respectively represent the first intermediate feature and the second intermediate feature of the mth node of the nth subgraph data, Assign a value of 0, and Represents the feature graph of the mth node of the nth and n-1th subgraph data respectively, Assign a value of 0, Represents the set of nodes that have edges connected to the mth node in the n subgraph data. Represents the feature graph of the vth node of the nth subgraph data, , and They represent the candidate second intermediate feature, update gate and reset gate of the mth node of the nth subgraph data respectively. , , , , , , , and represent the third weight parameter, the fourth weight parameter, the fifth weight parameter, the sixth weight parameter, the seventh weight parameter, the eighth weight parameter, the ninth weight parameter, the tenth weight parameter and the eleventh weight parameter respectively, , , and represent the third bias parameter, the fourth bias parameter, the fifth bias parameter and the sixth bias parameter respectively, Indicates stacking the updated features of the M nodes of the nth subgraph data, concat indicates concatenation, and Vec indicates vectorization. represents point-by-point multiplication, sigmoid represents the sigmoid activation function, and tanh represents the hyperbolic tangent activation function.

[0033] In one embodiment of the present invention, sample labels of training samples used to train a blade fault detection model are obtained through manual annotation.

[0034] In one embodiment of the present invention, a cross entropy loss function is specified as the loss function of the blade fault detection model.

[0035] It should be noted that the loss function is used to quantify the difference between the prediction results of the neural network model and the actual observation values. During the training process of the blade fault detection model, the loss function is used to calculate the forward propagation loss value of the blade fault detection model, and then the chain rule is used to calculate the gradient of the loss function with respect to each weight parameter and bias parameter. According to the calculated gradient and the preset learning rate, the weight parameters and bias parameters of the blade fault detection model are updated.

[0036] In one embodiment of the present invention, the calculation formula for updating the weight parameter and bias parameter of the blade fault detection model is as follows: ; in and Respectively represent the weight parameters or bias parameters after and before the update, represents the learning rate, Represents the gradient of the weight parameter or bias parameter before update.

[0037] It should be noted that if the learning rate is set too high, the step size of each parameter update will be very large, resulting in unstable model training or even failure to converge. If the learning rate is set too low, although the minimum value of the loss function can be approached more accurately, the training process will become very slow and may take more time to complete the training. The existing adaptive learning rate algorithm is used to dynamically adjust the learning rate to make the model update parameters more efficient and stable, such as Adam, Adagrad, etc., which will not be elaborated here.

[0038] In one embodiment of the present invention, the present invention provides a storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, a wind turbine blade fault detection system as described above can be executed.

[0039] like Figure 3 As shown, the blade fault detection model provided by the present invention detects the fault types of 500 sub-region images, the number of sub-region images of each fault type is 100, and the detection accuracy of the blade fault detection model provided by the present invention is 98+99+99+98+100 divided by 500, which is equal to 98.8%.

[0040] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are protected by the present embodiment.

Claims

1. A wind turbine blade fault detection system, characterized in that: include: A video acquisition module is used to collect the rotation video of the wind turbine blades through a high-definition camera mounted on a drone within a preset time period T; A video processing module is used to extract key frames from the rotation video of the wind turbine blades to obtain an image data set with N key frame images; A sub-region division module is used to traverse N key frame images of the image data set and divide the wind turbine blades in each key frame image into M sub-region images through a sliding window; A spatiotemporal graph data construction module, which is used to construct spatiotemporal graph data according to M sub-region images of N key frame images; The spatiotemporal graph data includes: N sub-graph data, and the N sub-graph data are respectively mapped to the N key frame images; Each subgraph data consists of M nodes and edges between nodes; A mapping relationship is established between the mth node of the nth sub-graph data and the mth sub-region image in the nth key frame image, where 1≤m≤M, 1≤n≤N; A blade fault detection model building module, which is used to build and train a blade fault detection model; A blade fault detection module, which is used to input the spatiotemporal graph data into the trained blade fault detection model, and the output value represents the fault type of the M sub-region images of the wind turbine blade; Failure types include: cracks, corrosion, coating damage, adsorption of deposits, and no damage.

2. A wind turbine blade fault detection system according to claim 1, characterized in that: T, N, M, the size of the sliding window, and the step size of the sliding window are all custom parameters.

3. A wind turbine blade fault detection system according to claim 1, characterized in that: Extracting key frames from a rotation video of a wind turbine blade to obtain an image dataset with N key frame images includes the following steps: Step S201, dividing the rotation video of the wind turbine blades into frames at equal time intervals to obtain K frame images, where K is a custom parameter greater than N; Step S202, grayscale processing is performed on the K frame images to obtain a grayscale image; Step S203, converting the K grayscale images into grayscale histogram representation; The grayscale histogram is represented by a vector with 256 dimensions, and the value of each dimension represents the number of pixels with the same grayscale level as the dimension index; Step S204, normalizing the K grayscale histograms; The normalized grayscale histogram is represented by a vector with 256 dimensions, and the sum of the values ​​of all dimensions is 1; Step S205, calculating the Bhattacharyya distance between two adjacent normalized grayscale histograms, and if it is determined that the Bhattacharyya distance is greater than or equal to a preset distance threshold, the grayscale image of the next frame is used as a key frame image; The calculation formula for calculating the Bhattacharyya distance between two adjacent normalized grayscale histograms is as follows: ； Where 1≤k≤K, represents the Bhattacharyya distance between the kth normalized grayscale histogram and the k-1th normalized grayscale histogram. and Respectively represent the value of the i-th dimension of the k-th and k-1-th normalized grayscale histograms; Step S206, if it is determined that the number of key frame images is greater than N, the preset distance threshold is increased, and if it is determined that the number of key frame images is less than N, the preset distance threshold is lowered until N key frame images are obtained as the image data set.

4. A wind turbine blade fault detection system according to claim 1, characterized in that: The edges between nodes include: edges built between adjacent nodes of the same subgraph data; edges built between the same nodes of adjacent subgraph data.

5. A wind turbine blade fault detection system according to claim 1, characterized in that: The blade fault detection model includes: a first hidden layer, N second hidden layers, M extractors and M classifiers; The first hidden layer inputs a sub-region image that has a mapping relationship with the M nodes of the N sub-graph data of the spatiotemporal graph data, and outputs a feature map; The nth second hidden layer inputs the nth subgraph data of the spatiotemporal graph data and outputs an update matrix, the update matrix includes M row vectors, and the mth row vector of the update matrix output by the nth second hidden layer represents the update feature of the mth node of the nth subgraph data; M extractors are used to extract M row vectors of the update matrix output by the Nth second hidden layer and input them into M classifiers; The classification space of M classifiers represents the fault types of M sub-region images of the wind turbine blade.

6. A wind turbine blade fault detection system according to claim 5, characterized in that: The calculation formula for the first hidden layer includes: ; ;in represents the feature map of the first hidden layer output, represents the sub-region image of the first hidden layer input, A represents the intermediate feature map, and denote the first weight parameter and the second weight parameter respectively, and represent the first bias parameter and the second bias parameter respectively, ReLU represents the ReLU activation function, and softmax represents the softmax activation function.

7. A wind turbine blade fault detection system according to claim 5, characterized in that: The calculation formula for the nth second hidden layer includes: ； ； ； ； ； ； ； in represents the update matrix of the nth second hidden layer output, Represents the updated features of the mth node of the nth subgraph data, and Respectively represent the first intermediate feature and the second intermediate feature of the mth node of the nth subgraph data, Assign a value of 0, and Represents the feature graph of the mth node of the nth and n-1th subgraph data respectively, Assign a value of 0, Represents the set of nodes that have edges connected to the mth node in the n subgraph data. Represents the feature graph of the vth node of the nth subgraph data, , and They represent the candidate second intermediate feature, update gate and reset gate of the mth node of the nth subgraph data respectively. , , , , , , , and represent the third weight parameter, the fourth weight parameter, the fifth weight parameter, the sixth weight parameter, the seventh weight parameter, the eighth weight parameter, the ninth weight parameter, the tenth weight parameter and the eleventh weight parameter respectively, , , and represent the third bias parameter, the fourth bias parameter, the fifth bias parameter and the sixth bias parameter respectively, Indicates stacking the updated features of the M nodes of the nth subgraph data, concat indicates concatenation, and Vec indicates vectorization. represents point-by-point multiplication, sigmoid represents the sigmoid activation function, and tanh represents the hyperbolic tangent activation function.

8. A wind turbine blade fault detection system according to claim 1, characterized in that: The sample labels of the training samples used to train the blade fault detection model are obtained through manual annotation.

9. A wind turbine blade fault detection system according to claim 1, characterized in that: Specify the cross entropy loss function as the loss function for the blade fault detection model.

10. A storage medium, characterized in that: It stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, a wind power blade fault detection system as claimed in any one of claims 1 to 9 is executed.

Images