Wind turbine generator blade state monitoring method and system based on multi-source sensing data

Abstract

The invention relates to a wind turbine generator blade state monitoring method and system based on multi-source sensing data, and the method comprises the steps: obtaining audio sensing data and video sensing data at the same time, extracting sound features and image features, fusing the features, and then carrying out the state monitoring through employing a preset neural network. Therefore, the complementarity of the multi-source data is fully utilized, the monitoring accuracy is improved, the fine characteristics of the blade state change can be captured, the early fault early warning is realized, and the method has the advantages that the complementarity of the multi-source data can be fully utilized, the monitoring accuracy is improved, the fine characteristics of the blade state change are captured, and the early fault early warning is realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of blade status detection, and in particular to a method and system for monitoring the blade status of a wind turbine based on multi-source sensor data. Background Art

[0002] Blades are key components of wind turbines, and their performance can directly affect the power generation efficiency of wind turbines. However, wind turbine blades are often exposed to harsh wind farm environments, often facing erosion from natural factors such as lightning strikes, rain, and sun exposure, which can lead to frequent blade failures. In order to detect and handle blade failures in a timely manner, effective blade status monitoring methods are particularly important.

[0003] At present, the commonly used blade fault monitoring methods mainly include vibration sensor monitoring and video monitoring. The vibration sensor can collect the audio signal generated during the operation of the blade, while the video monitoring can capture the real-time image of the blade. These two monitoring methods have their own advantages, but when used alone, it is often difficult to fully and accurately reflect the actual status of the blade.

[0004] Some existing monitoring schemes attempt to combine audio and image data for blade status monitoring. However, these schemes usually process the two data sources separately, obtain independent judgment results, and then make a comprehensive judgment. This processing method has obvious limitations: the potential correlation between audio and image data is ignored, resulting in the inability to fully utilize the complementarity of multi-source data; secondly, the results of independent processing may be contradictory, increasing the risk of misjudgment; finally, this method is difficult to capture the subtle characteristics of blade status changes, and may miss the opportunity to warn of early faults.

[0005] In addition, existing monitoring methods often lack effective data fusion mechanisms. Simply weighted averaging or performing logical operations on the results of two data sources cannot fully mine the deep information of multi-source data. This rough fusion method not only reduces the accuracy of monitoring, but may also cause some important status features to be obscured or misunderstood. Summary of the invention

[0006] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0007] The main purpose of the embodiments of the present invention is to propose a method and system for monitoring the status of wind turbine blades based on multi-source sensor data, so as to improve the monitoring accuracy, capture the subtle features of blade status changes, and achieve the advantages of early fault warning.

[0008] To achieve the above object, a first aspect of an embodiment of the present invention provides a method for monitoring the status of wind turbine blades based on multi-source sensor data, the method comprising:

[0009] Respond to requests for wind turbine blade condition monitoring;

[0010] Acquiring, according to the request, audio sensor data and video sensor data simultaneously monitored on the blades of the wind turbine generator set;

[0011] Converting the audio sensor data into temporally continuous sound data, and extracting sound features corresponding to the temporally continuous sound data;

[0012] Extracting temporally continuous picture data according to the video sensor data, and extracting image features corresponding to the temporally continuous picture data;

[0013] fusing the sound feature and the image feature to obtain a fused feature;

[0014] The fusion feature is input into a preset neural network to obtain the state of the wind turbine blade output by the neural network.

[0015] The wind turbine blade condition monitoring method based on multi-source sensor data provided in this application has at least the following beneficial effects:

[0016] By simultaneously acquiring audio sensor data and video sensor data, extracting sound features and image features, and fusing these features, and then using a preset neural network for status monitoring, the complementarity of multi-source data is fully utilized, the accuracy of monitoring is improved, and the subtle features of blade status changes can be captured to achieve early fault warning. It has the advantages of being able to fully utilize the complementarity of multi-source data, improve monitoring accuracy, capture subtle features of blade status changes, and achieve early fault warning.

[0017] In some embodiments, the fusing the sound feature and the image feature to obtain a fused feature includes:

[0018] assigning a first weight to the sound feature;

[0019] assigning a second weight to the image feature;

[0020] A weighted sum of the sound feature and the image feature is calculated to obtain the fusion feature.

[0021] In some embodiments, the loss function of the neural network includes:

[0022]

[0023] in, represents the loss function, F represents the similarity matrix based on the sound features and the image features, and h i ,h jis a set of sound data and picture data corresponding to a time node, S 1 represents a set of similar sample pairs, S 2 Represents a set of dissimilar sample pairs;

[0024] Among them, F is calculated as follows:

[0025] Dividing the temporally continuous image data and the temporally continuous sound data into multiple groups, wherein the image data at each time node and the sound data at any time node are divided into one group, and the sound data at each time node and the image data at any time node are divided into one group;

[0026] Calculate the similarity between each set of corresponding image features and sound features;

[0027] The similarity matrix is ​​constructed according to the similarities.

[0028] In some embodiments, calculating the similarity between each group of corresponding image features and sound features includes:

[0029]

[0030] Among them, f represents the similarity between a set of corresponding image features and sound features, is a set of image features x and sound features y corresponding to picture data and sound data, W represents the weight matrix, b represents the bias, and Sigmoid represents the SIGMOID function.

[0031] In some embodiments, extracting the sound features corresponding to the temporally continuous sound data includes:

[0032] The sound features are extracted from the temporally continuous sound data according to wavelet transform.

[0033] In some embodiments, the extracting image features corresponding to the temporally continuous picture data includes:

[0034] Image features are extracted from the temporally continuous picture data according to a convolution layer having a convolution kernel.

[0035] In some embodiments, the state of the wind turbine blade is intact, cracked, split, defective, and frozen.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present invention provides a wind turbine blade condition monitoring system based on multi-source sensor data, the system comprising:

[0037] A request unit, used to respond to a request for monitoring the status of blades of a wind turbine generator set;

[0038] A data acquisition unit, configured to acquire, according to the request, audio sensor data and video sensor data simultaneously monitored on the blades of the wind turbine generator set;

[0039] A first feature extraction unit, configured to convert the audio sensor data into temporally continuous sound data, and extract sound features corresponding to the temporally continuous sound data;

[0040] A second feature extraction unit, configured to extract temporally continuous picture data according to the video sensing data, and extract image features corresponding to the temporally continuous picture data;

[0041] A feature fusion unit, used for fusing the sound feature and the image feature to obtain a fusion feature;

[0042] The state monitoring unit is used to input the fusion feature into a preset neural network to obtain the state of the wind turbine blade output by the neural network.

[0043] To achieve the above-mentioned purpose, the electronic device provided in the third aspect of an embodiment of the present invention includes: at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a wind turbine blade status monitoring method based on multi-source sensor data according to the above-mentioned first aspect.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a wind turbine blade status monitoring method based on multi-source sensor data according to the first aspect above.

[0045] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a flow chart of a method for monitoring the status of wind turbine blades based on multi-source sensor data provided by an embodiment of the present application;

[0048] Figure 2 is a schematic diagram of a wind turbine blade status monitoring system based on multi-source sensor data provided by an embodiment of the present application;

[0049] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] like Figure 1 An embodiment of the present application provides a method for monitoring the status of wind turbine blades based on multi-source sensor data, the method comprising:

[0054] Step S110, responding to a request for monitoring the status of wind turbine blades;

[0055] Step S120, acquiring audio sensor data and video sensor data simultaneously monitored on the wind turbine blades according to the request;

[0056] Step S130, converting the audio sensor data into temporally continuous sound data, and extracting sound features corresponding to the temporally continuous sound data;

[0057] Step S140, extracting temporally continuous picture data according to the video sensing data, and extracting image features corresponding to the temporally continuous picture data;

[0058] Step S150, fusing the sound feature and the image feature to obtain a fused feature;

[0059] Step S160, inputting the fusion features into a preset neural network to obtain the status of the wind turbine blades output by the neural network.

[0060] Wind turbine blades are key components of wind turbines, and their performance can directly affect the power generation efficiency of wind turbines. However, blades are exposed to harsh wind environments for a long time, and will be eroded by lightning, rain, and sun, leading to frequent failures. Currently, commonly used blade fault monitoring methods include vibration sensors or video monitoring, but the existing schemes for jointly monitoring the audio collected by vibration sensors and the pictures collected by image sensors only obtain one judgment result based on the audio and another judgment result based on the picture. The two judgment results are finally judged comprehensively, and the means are independent of each other, which is prone to misjudgment.

[0061] In the process of monitoring the status of wind turbine blades, the problem of misjudgment is caused by the independent discrimination of audio and video data in the prior art. In order to solve this problem, a method for simultaneously acquiring audio sensor data and video sensor data is proposed. The audio sensor data is converted into temporally continuous sound data, and the sound features corresponding to the temporally continuous sound data are extracted; at the same time, temporally continuous picture data is extracted according to the video sensor data, and the image features corresponding to the temporally continuous picture data are extracted. By fusing the sound features and the image features, the fused features are obtained, and then the fused features are input into the preset neural network to obtain the status of the wind turbine blades. In this way, the misjudgment problem caused by the independent discrimination of audio and video data can be avoided.

[0062] In the implementation process, audio sensor data and video sensor data are collected through different sensors. Audio sensor data is collected through a microphone, and video sensor data is collected through a camera. Sound features can be extracted from temporally continuous sound data through wavelet transform, while image features can be extracted from temporally continuous image data through a convolution layer with a convolution kernel. The extracted sound features and image features are fused to obtain fused features, which are then input into a preset neural network for state monitoring.

[0063] For example, the fusion of sound features and image features can be obtained by assigning a first weight to the sound features, assigning a second weight to the image features, and calculating the weighted sum of the sound features and the image features to obtain the fusion features. Furthermore, the loss function of the neural network can be calculated by a similarity matrix based on the sound features and the image features. By dividing the temporally continuous image data and sound data into multiple groups, calculating the similarity between the image features and sound features corresponding to each group, and constructing a similarity matrix based on the similarity, the accuracy of the neural network is improved.

[0064] Compared with the prior art, the wind turbine blade status monitoring method based on multi-source sensor data of this embodiment improves the accuracy of wind turbine blade status monitoring and reduces the possibility of misjudgment by fusing audio and video data. In the prior art, independent judgment of audio and video data is prone to misjudgment, but this application overcomes this problem by comprehensively utilizing multi-source sensor data.

[0065] In some embodiments, the sound feature and the image feature are fused to obtain the fused feature, including:

[0066] assigning a first weight to the sound feature;

[0067] assigning a second weight to the image feature;

[0068] Calculate the weighted sum of the sound features and image features to obtain the fusion features.

[0069] The present application also proposes to assign a first weight to the sound feature; assign a second weight to the image feature; and calculate the weighted sum of the sound feature and the image feature to obtain a fusion feature.

[0070] In the condition monitoring of wind turbine blades, the fusion of sound features and image features is to provide more accurate monitoring results. The technical features of assigning a first weight to the sound feature and assigning a second weight to the image feature are used to assign different weight values ​​to different types of features during the fusion process. The technical feature of calculating the weighted sum of the sound feature and the image feature is used to weight the two features and fuse them into a comprehensive feature. These technical features cooperate with each other to achieve the effective fusion of sound features and image features, thereby providing more accurate input data for the monitoring of the condition of wind turbine blades. Through the above scheme, the problem of fusing sound features and image features in the condition monitoring of wind turbine blades is solved, and more accurate condition monitoring results are provided.

[0071] Specifically, assigning a first weight to the sound feature means assigning a weight value according to the importance of the sound feature during the fusion process. Assigning a second weight to the image feature means assigning another weight value according to the importance of the image feature. Calculating the weighted sum of the sound feature and the image feature is to fuse the two features through weighted calculation to obtain a comprehensive fusion feature. This fusion feature can more comprehensively reflect the status of the wind turbine blades.

[0072] Therefore, this embodiment successfully achieves the effective fusion of the two features by assigning weights to the sound features and image features respectively and calculating their weighted sum. This fusion method avoids the misjudgment that may be caused by using sound or image features alone, thereby improving the accuracy of wind turbine blade status monitoring. Compared with the prior art, the technical solution provided by this embodiment can more comprehensively utilize multi-source sensor data and improve the reliability and accuracy of the monitoring results.

[0073] In some embodiments, the loss function of the neural network includes:

[0074]

[0075] in, represents the loss function, F represents the similarity matrix based on sound features and image features, and h i ,h j is a set of sound data and picture data corresponding to a time node, S 1 represents a set of similar sample pairs, S 2 Represents a set of dissimilar sample pairs;

[0076] Among them, F is calculated as follows:

[0077] Step S210, dividing the temporally continuous picture data and the temporally continuous sound data into a plurality of groups, wherein the picture data at each time node and the sound data at any time node are divided into one group, and the sound data at each time node and the picture data at any time node are divided into one group;

[0078] Step S220, calculating the similarity between each group of corresponding image features and sound features;

[0079] Step S230: construct a similarity matrix according to the similarities.

[0080] By dividing the temporally continuous image data and sound data into multiple groups, and then calculating the similarity between the image features and sound features corresponding to each group, a similarity matrix is ​​constructed, thereby performing effective calculations in the loss function of the neural network.

[0081] These technical features can effectively solve the loss function calculation problem in wind turbine blade condition monitoring based on sound features and image features by cooperating with each other. Through the above scheme, by calculating the similarity and constructing the similarity matrix, the relationship between the sound features and the image features can be more accurately reflected, thereby improving the accuracy of the neural network in wind turbine blade condition monitoring.

[0082] In actual implementation, these steps can be further refined in the following ways. First, the division of temporally continuous image data and sound data can be achieved through sliding window technology. Specifically, the data in each time window can be taken as a group, which can ensure temporal continuity and data integrity. Second, a variety of methods can be used to calculate the similarity between image features and sound features, such as based on Euclidean distance, cosine similarity or other commonly used similarity measurement methods. The construction of the similarity matrix can be achieved through matrix operations, which can effectively integrate the similarity information of multiple groups of data.

[0083] This is because video and audio are heterogeneous information of the same object in these modalities, which increases the computational burden and affects the monitoring effect. Therefore, it is necessary to find the correlation between the features of different modal data.

[0084] Furthermore, because the amount of information in video and audio is unbalanced and unequal, it is inappropriate to directly match modality-specific representations of different modes. Therefore, this embodiment calculates the similarity between each group of corresponding image features and sound features in the following manner, including:

[0085]

[0086] Among them, f represents the similarity between a set of corresponding image features and sound features, is a set of image features x and sound features y corresponding to picture data and sound data, W represents the weight matrix, b represents the bias, and Sigmoid is the activation function.

[0087] First, the image features and sound features of each group of sample pairs are obtained, and the image feature vector and sound feature vector of each group are fused to obtain the fused features; the correlation between different modes is mined from the image features and sound features through the function f, the core commonalities of the paired relationships between a group of samples are captured, and the similarity between the vector groups of the two modes is calculated to represent a certain relationship between the vectors of the two modes; in this way, the two modal data features form a similarity matrix, which is conducive to the subsequent more accurate monitoring of the wind turbine blades.

[0088] Specifically, this application comprehensively considers the similarity relationship between sound data and image data, so that the neural network can more accurately identify the status of the wind turbine blades, thereby improving the accuracy and reliability of monitoring. Compared with the existing technology, this application has obvious advantages in processing multi-source data fusion and loss function calculation.

[0089] In some embodiments, extracting sound features corresponding to temporally continuous sound data includes:

[0090] Sound features are extracted from temporally continuous sound data based on wavelet transform.

[0091] Furthermore, the present application also proposes to extract sound features from temporally continuous sound data based on wavelet transform.

[0092] First of all, wavelet transform is a signal processing technology that can effectively extract useful feature information from time series data.

[0093] Through the solution of the present application, the problem of how to extract sound features from time-continuous sound data can be solved by using wavelet transform. Wavelet transform can capture the characteristics of sound signals at different time scales, providing an efficient and accurate sound feature extraction method, which helps to improve the accuracy and reliability of wind turbine blade status monitoring.

[0094] Specifically, this application is implemented through the following steps:

[0095] 1) Obtain temporally continuous sound data.

[0096] 2) Apply wavelet transform to process sound data.

[0097] 3) Extract the processed sound features.

[0098] Among them, wavelet transform is a commonly used signal processing method that can convert time domain signals into frequency domain, thereby extracting characteristic information from the signal. Applied in sound data processing, wavelet transform can effectively capture the changes of sound signals on different time scales, thereby extracting representative sound features.

[0099] As a preferred implementation, different wavelet basis functions (such as Daubechies wavelet, Haar wavelet, etc.) can be selected for transformation to adapt to different types of sound data. In addition, the effect of feature extraction can be further optimized by adjusting the scale parameter of the wavelet transform.

[0100] Therefore, by extracting sound features from time-continuous sound data through wavelet transform, not only the accuracy of feature extraction is improved, but also reliable data support is provided for subsequent wind turbine blade status monitoring. Compared with the prior art, the method of the present application has higher accuracy and robustness, can more effectively monitor the status of wind turbine blades, reduce the occurrence of misjudgment, and improve the operating efficiency and reliability of wind turbines.

[0101] In some embodiments, extracting image features corresponding to temporally continuous picture data includes:

[0102] Image features are extracted from temporally continuous image data using a convolutional layer with a convolutional kernel.

[0103] Image features are extracted from temporally continuous image data using a convolution layer with a convolution kernel. By using the convolution layer, useful image features can be effectively extracted from temporally continuous image data. The function of this technical means is that through the characteristics of the convolution layer, important features in the image can be automatically identified and extracted, thereby improving the accuracy and efficiency of image feature extraction. The convolution layer can process temporally continuous image data, so that the extracted image features can reflect the state changes of the wind turbine blades at different time nodes. This method reduces the error of human intervention and improves the reliability and accuracy of monitoring through the automated feature extraction process.

[0104] Specifically, the convolution layer uses convolution kernels to perform convolution operations on temporally continuous image data to extract image features at different levels. Convolution kernels can be filters of different sizes and shapes that can capture features such as edges and textures in images. The convolution operation involves sliding window processing of image data, and each operation generates a feature map. By superimposing multiple convolution layers, more complex and abstract image features can be extracted. The convolution layer can also be combined with the pooling layer for dimensionality reduction processing to further extract key information from the image.

[0105] Through the above technical solution, the present application can solve the technical problem of extracting image features from temporally continuous image data in the prior art. Compared with the prior art, the advantages of this embodiment are: the image features are automatically extracted through the convolution layer, which reduces the error caused by human intervention and improves the accuracy and efficiency of image feature extraction; the convolution layer can process temporally continuous image data, so that the extracted image features can reflect the state changes of the wind turbine blades at different time nodes, improving the reliability and accuracy of monitoring.

[0106] In some embodiments, the status of the wind turbine blade is intact, cracked, split, defective, and iced.

[0107] During long-term operation, the blades of wind turbines will be affected by various environmental factors, causing the blades to be intact, cracked, split, defective, and frozen. These conditions will directly affect the power generation efficiency and safety of the wind turbine. Therefore, by monitoring these conditions of the wind turbine blades, problems can be discovered in a timely manner and maintenance and repairs can be carried out, thereby improving the operating efficiency and safety of the wind turbine. This application defines the specific target states for monitoring by defining the states of wind turbine blades as intact, cracked, split, defective, and frozen, providing a basis for achieving comprehensive monitoring of the states of wind turbine blades.

[0108] Furthermore, in the present application, the states of the fan blades include intact, cracked, split, defective and frozen. The intact state means that the blades are in normal working condition without any damage; the cracked state means that small cracks appear on the surface or inside of the blades; the split state means that larger cracks appear on the blades; the defective state means that part of the material of the blades is missing; the frozen state means that the surface of the blades is frozen. By monitoring these states, abnormal conditions of the blades can be detected in time and handled through corresponding maintenance measures. For example, by installing audio sensors and video sensors, the sound data and image data of the blades can be obtained in real time, and these data can be analyzed through a neural network model to determine the state of the blades.

[0109] This embodiment can timely detect abnormal conditions of the blades by monitoring the status of the wind turbine blades, thereby avoiding a decrease in power generation efficiency and potential safety hazards caused by blade damage.

[0110] like Figure 2 One embodiment of the present application provides a wind turbine blade condition monitoring system based on multi-source sensor data, the system comprising:

[0111] The request unit 1000 is used to respond to a request for monitoring the status of a wind turbine blade;

[0112] The data acquisition unit 2000 is used to acquire audio sensor data and video sensor data simultaneously monitored on the blades of the wind turbine generator according to the request;

[0113] The first feature extraction unit 3000 is used to convert the audio sensor data into time-continuous sound data, and extract the sound features corresponding to the time-continuous sound data;

[0114] The second feature extraction unit 4000 is used to extract temporally continuous picture data according to the video sensing data, and extract image features corresponding to the temporally continuous picture data;

[0115] The feature fusion unit 5000 is used to fuse the sound feature and the image feature to obtain a fusion feature;

[0116] The state monitoring unit 6000 is used to input the fusion features into a preset neural network to obtain the state of the wind turbine blade output by the neural network.

[0117] It should be noted that the wind turbine blade status monitoring system based on multi-source sensor data provided in this embodiment and the above-mentioned wind turbine blade status monitoring method based on multi-source sensor data are based on the same inventive concept. Therefore, the relevant content of the above-mentioned wind turbine blade status monitoring method based on multi-source sensor data is also applicable to the content of the wind turbine blade status monitoring system based on multi-source sensor data. Therefore, it will not be repeated here.

[0118] like Figure 3 The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0119] at least one memory;

[0120] at least one processor;

[0121] at least one program;

[0122] The programs are stored in the memory, and the processor executes at least one program to implement the wind turbine blade condition monitoring method based on multi-source sensor data as described above in the present disclosure.

[0123] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0124] The electronic device according to the embodiment of the present application is described in detail below.

[0125] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0126] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the program code is stored in the memory 1700, and the processor 1600 calls and executes the wind turbine blade state monitoring method based on multi-source sensor data in the embodiment of the present invention.

[0127] Input / output interface 1800, used to implement information input and output;

[0128] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0129] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0130] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0131] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium, and which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned wind turbine blade status monitoring method based on multi-source sensor data.

[0132] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The embodiments described in the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0134] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0135] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0137] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0138] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0143] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A method for monitoring the status of wind turbine blades based on multi-source sensor data, characterized in that: The method comprises: Respond to requests for wind turbine blade condition monitoring; Acquiring, according to the request, audio sensor data and video sensor data simultaneously monitored on the blades of the wind turbine generator set; Converting the audio sensor data into temporally continuous sound data, and extracting sound features corresponding to the temporally continuous sound data; Extracting temporally continuous picture data according to the video sensor data, and extracting image features corresponding to the temporally continuous picture data; fusing the sound feature and the image feature to obtain a fused feature; The fusion feature is input into a preset neural network to obtain the state of the wind turbine blade output by the neural network.

2. The method for monitoring the status of wind turbine blades based on multi-source sensor data according to claim 1, characterized in that: The fusing the sound feature and the image feature to obtain a fused feature includes: assigning a first weight to the sound feature; assigning a second weight to the image feature; A weighted sum of the sound feature and the image feature is calculated to obtain the fusion feature.

3. The method for monitoring the status of wind turbine blades based on multi-source sensor data according to claim 1, characterized in that: The loss function of the neural network includes: in, represents the loss function, F represents the similarity matrix based on the sound features and the image features, and h i ,h j is a set of sound data and image data corresponding to a time node, S1 represents a set of similar sample pairs, and S2 represents a set of dissimilar sample pairs; Among them, F is calculated as follows: Dividing the temporally continuous image data and the temporally continuous sound data into multiple groups, wherein the image data at each time node and the sound data at any time node are divided into one group, and the sound data at each time node and the image data at any time node are divided into one group; Calculate the similarity between each set of corresponding image features and sound features; The similarity matrix is ​​constructed according to the similarities.

4. The method for monitoring the status of wind turbine blades based on multi-source sensor data according to claim 3 is characterized in that: The calculating the similarity between each group of corresponding image features and sound features includes: Among them, f represents the similarity between a set of corresponding image features and sound features, is a set of image features x and sound features y corresponding to picture data and sound data, W represents the weight matrix, b represents the bias, and Sigmoid represents the SIGMOID function.

5. The method for monitoring the status of wind turbine blades based on multi-source sensor data according to claim 1, characterized in that: The extracting of the sound features corresponding to the temporally continuous sound data includes: The sound features are extracted from the temporally continuous sound data according to wavelet transform.

6. The method for monitoring the status of wind turbine blades based on multi-source sensor data according to claim 1, characterized in that: The extracting image features corresponding to the temporally continuous picture data includes: Image features are extracted from the temporally continuous picture data according to a convolution layer having a convolution kernel.

7. The method for monitoring the status of wind turbine blades based on multi-source sensor data according to claim 1, characterized in that: The states of the wind turbine blades include intact, cracked, split, defective and frozen.

8. A wind turbine blade condition monitoring system based on multi-source sensor data, characterized in that: The system comprises: A request unit, used to respond to a request for monitoring the status of blades of a wind turbine generator set; A data acquisition unit, configured to acquire, according to the request, audio sensor data and video sensor data simultaneously monitored on the blades of the wind turbine generator set; A first feature extraction unit, configured to convert the audio sensor data into temporally continuous sound data, and extract sound features corresponding to the temporally continuous sound data; A second feature extraction unit, configured to extract temporally continuous picture data according to the video sensing data, and extract image features corresponding to the temporally continuous picture data; A feature fusion unit, used for fusing the sound feature and the image feature to obtain a fusion feature; The state monitoring unit is used to input the fusion feature into a preset neural network to obtain the state of the wind turbine blade output by the neural network.

9. An electronic device, characterized in that: include: at least one control processor and a memory for communicatively coupling with the at least one control processor; The memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the wind turbine blade condition monitoring method based on multi-source sensor data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the wind turbine blade condition monitoring method based on multi-source sensor data according to any one of claims 1 to 7.

Images