Fan health state detection method and device based on multi-modal data, equipment and medium

Abstract

The application discloses a fan health state detection method and device based on multi-modal data, equipment and medium, and relates to the technical field of intelligent fan monitoring, which comprises the following steps: collecting fan images, sound and sensor data, extracting spatial features, synchronous vibration features and time sequence features by using multi-modal processing technology, and fusing the features by a multi-modal adaptive space-time coupling decision network to generate a health score, evaluating the fan health state according to the score and giving an early warning, realizing early risk detection and active early warning, and improving the accuracy and timeliness of fan fault detection.

Description

Technical Field

[0001] The present invention relates to the field of intelligent fan monitoring technology, and in particular to a fan health status detection method, device, equipment and medium based on multimodal data. Background Art

[0002] With the increasing global demand for renewable energy, wind power has gained widespread adoption as a clean, sustainable energy solution. As the core equipment of wind power systems, wind turbines operate under high loads and in extreme weather conditions for extended periods of time. Their stability and reliability are crucial to ensuring power supply. However, due to their harsh operating environments and complex mechanical structures, wind turbines are susceptible to various forms of damage, such as blade wear and bearing cracks. These failures not only affect wind turbine efficiency but can also lead to serious safety incidents.

[0003] Traditional methods for detecting defects in wind turbines primarily include image processing, vibration monitoring, and sensor data analysis. Common image processing methods include edge detection and feature matching, which are used to identify cracks or wear on wind turbine components (such as blades and bearings). Vibration monitoring analyzes sound and vibration signals to capture changes in the status of various internal wind turbine components, and is particularly effective in detecting anomalies in bearings and motors. Furthermore, data collected by various sensors, including temperature, pressure, and vibration, can provide health information on wind turbine components, helping to monitor their operating status.

[0004] However, traditional image processing methods struggle to provide sufficient detection accuracy for small or blurred defects in complex scenes, and are prone to missed or false detections. Traditional vibration monitoring methods are susceptible to external factors in environments with strong noise interference, resulting in inaccurate detection results and an inability to effectively distinguish between normal vibration and fault vibration. Traditional sensor data analysis is unable to accurately and efficiently diagnose due to the complex interactions between wind turbine components. Therefore, an efficient and accurate method is needed to detect and provide early warnings for various wind turbine components. Summary of the Invention

[0005] The main purpose of this application is to provide a method, device, equipment and medium for detecting the health status of a wind turbine based on multimodal data, aiming to solve the technical problem of how to accurately obtain the health status of each component of the wind turbine and perform corresponding early warning operations.

[0006] To achieve the above objectives, the present application proposes a method for detecting the health status of a wind turbine based on multimodal data, the method comprising:

[0007] Acquiring data of various components of the fan, wherein the data of various components of the fan include fan image data, fan sound data, and fan sensor data;

[0008] Preprocessing the data of each component of the wind turbine to obtain processed image data, processed sound data and processed sensor data;

[0009] Performing multi-scale feature extraction processing on the processed image data to obtain spatial features;

[0010] performing acoustic-dynamic coupling synchronization feature extraction processing on the processed sound data to obtain a synchronous vibration feature;

[0011] performing hierarchical dynamic feature extraction processing on the processed sensor data to obtain time series features;

[0012] fusing the spatial features, the synchronous vibration features, and the time series features through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score;

[0013] A judgment is made based on the health score to obtain the health status of the wind turbine and issue an early warning.

[0014] In one embodiment, the step of preprocessing the data of each component of the wind turbine to obtain processed image data, processed sound data, and processed sensor data includes:

[0015] The wind turbine image data is processed using an image enhancement method to obtain processed image data, which is obtained by the following formula:

[0016]

[0017] in, represents the original image, is the Fourier transform of the image, represents the adaptive frequency filter, represents the local enhancement function in the spatial domain, is the inverse transform of Fourier transform;

[0018] The wind turbine sound data is processed using a time-frequency denoising method to obtain processed sound data, which is obtained by the following formula:

[0019]

[0020] in, represents the time domain window function, is the initial fan sound data, is the Fourier transform, is the decay function, It is noise;

[0021] The wind turbine sensor data is processed using an adaptive denoising and time series synchronization method to obtain processed sensor data, which is obtained by the following formula:

[0022]

[0023] in, is the initial fan sensor data, is the weight coefficient of the high-order cumulant, represents the dynamic weighting factor, Indicates the length of the time window, is a variable.

[0024] In one embodiment, the step of performing multi-scale feature extraction on the processed image data to obtain spatial features includes:

[0025] Dividing the processed image data into a plurality of small regions, wherein each of the small regions is represented by a spin particle;

[0026] Initializing spin particle features, wherein the spin particle features include gradient, local binary value, and grayscale standard deviation;

[0027] Constructing a spatial relationship graph of the spin particles according to the spin particle characteristics and obtaining weights;

[0028] Based on the weight, the state of the spin particle is updated through a dynamic update mechanism to obtain a particle state change amplitude;

[0029] Detecting according to the particle state change amplitude to obtain defect response values ​​of a plurality of the spin particles;

[0030] generating a defect response graph according to the defect response value;

[0031] generating a defect enhancement map based on the defect response map and the processed image data;

[0032] Feature extraction is performed on the defect enhancement image to obtain spatial features.

[0033] In one embodiment, the step of performing acoustic-dynamic coupling synchronization feature extraction processing on the processed sound data to obtain synchronized vibration features includes:

[0034] Construct nonlinear coupled vibration equations and acoustic wave propagation path models;

[0035] Inputting the processed sound data into the nonlinear coupled vibration equation and the sound wave propagation path model to obtain vibration results and sound wave synchronization effects;

[0036] Calculating based on the vibration results and the acoustic wave synchronization effect to obtain a nonlinear synchronization index and a propagation attenuation characteristic;

[0037] The nonlinear synchronization index and the propagation attenuation feature are fused to obtain the synchronization vibration feature.

[0038] In one embodiment, the step of performing hierarchical dynamic feature extraction on the processed sensor data to obtain time series features includes:

[0039] The fan structure is layered to obtain multiple levels;

[0040] constructing a kinetic model according to a plurality of said levels;

[0041] Inputting the processed sensor data into the dynamic model to obtain a plurality of hierarchical features;

[0042] A plurality of the hierarchical features are fused to obtain time series features.

[0043] In one embodiment, the step of fusing the spatial features, the synchronous vibration features, and the time series features through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score includes:

[0044] Performing dynamic weighted fusion of the spatial features, synchronous vibration features, and time series features through the spatiotemporal coupling layer of the multimodal adaptive spatiotemporal coupling decision network to generate a fused feature vector;

[0045] Obtaining the defect type and defect severity level of the wind turbine based on the fused feature vector through the decision layer of the multimodal adaptive spatiotemporal coupling decision network;

[0046] An assessment is performed based on the defect type and the defect severity level to obtain a health score of the wind turbine.

[0047] In one embodiment, the step of determining the health status of the wind turbine based on the health score and issuing an early warning includes:

[0048] When the health score is less than a first health threshold, the health state of the wind turbine is determined to be normal, and the wind turbine is monitored normally;

[0049] When the health score is greater than a first health threshold and the health score is less than a second health threshold, the health state of the wind turbine is determined to be a risk state, triggering an early warning and recommending inspection of the wind turbine;

[0050] When the health score is greater than a second health threshold, the health state of the wind turbine is determined to be a fault state, an emergency alarm is triggered, and a maintenance work order is generated, wherein the second health threshold is greater than the first health threshold.

[0051] In addition, to achieve the above-mentioned purpose, the present application also proposes a wind turbine health status detection device based on multimodal data, the wind turbine health status detection device based on multimodal data comprising:

[0052] An acquisition module is used to acquire data of various components of the fan, wherein the data of various components of the fan include fan image data, fan sound data and fan sensor data;

[0053] a processing module, configured to pre-process the data of the various components of the wind turbine to obtain processed image data, processed sound data, and processed sensor data;

[0054] An extraction module, configured to perform multi-scale feature extraction on the processed image data to obtain spatial features;

[0055] The extraction module is further used to perform acoustic-dynamic coupling synchronization feature extraction processing on the processed sound data to obtain synchronous vibration features;

[0056] The extraction module is further used to perform hierarchical dynamic feature extraction processing on the processed sensor data to obtain time series features;

[0057] a fusion module, configured to fuse the spatial features, the synchronous vibration features, and the time series features through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score;

[0058] The result module is used to make judgments based on the health score, obtain the health status of the wind turbine and issue an early warning.

[0059] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the wind turbine health status detection method based on multimodal data as described above are implemented.

[0060] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the wind turbine health status detection method based on multimodal data as described above.

[0061] This application obtains data from various components of the wind turbine, pre-processes the data from various components of the wind turbine, obtains processed image data, processed sound data and processed sensor data, and then processes the above data separately to obtain spatial features, synchronous vibration features and time series features. The spatial features, synchronous vibration features and time series features are then fused through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score. Finally, a judgment is made based on the health score to obtain the health status of the wind turbine and issue an early warning. By collecting wind turbine images, sounds and sensor data, using multimodal processing technology to extract spatial features, synchronous vibration features and time series features, and fusing these features through a multimodal adaptive spatiotemporal coupling decision network to generate a health score, the health status of the wind turbine is evaluated and an early warning is issued based on the score, thereby achieving early risk detection and active early warning, and improving the accuracy and timeliness of wind turbine fault detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 This is a flow chart of a first embodiment of a method for detecting the health status of a wind turbine based on multimodal data of the present application;

[0064] Figure 2 This is a flow chart of a second embodiment of a method for detecting a wind turbine's health status based on multimodal data according to the present application;

[0065] Figure 3 This is a flow chart of a third embodiment of a method for detecting a wind turbine health status based on multimodal data of the present application;

[0066] Figure 4 This is a schematic diagram of the module structure of a wind turbine health status detection device based on multimodal data according to an embodiment of the present application;

[0067] Figure 5 Schematic diagram of the equipment structure of the hardware operating environment involved in the wind turbine health status detection method based on multimodal data in an embodiment of the present application.

[0068] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0069] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0070] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0071] As wind power continues to increase its share in the energy mix, ensuring the long-term stable operation of wind turbine equipment has become increasingly important. Wind turbines typically operate in harsh environments, such as strong winds and extreme temperature fluctuations. These conditions can easily lead to defects such as cracks and wear in wind turbine components (such as blades, bearings, and rotors). Traditional wind turbine defect detection methods primarily rely on single-mode data analysis, such as image processing, vibration monitoring, or sensor data analysis. However, these methods have limitations when faced with complex operating environments: image processing has difficulty identifying small or blurred defects; vibration monitoring is susceptible to interference in high-noise environments, making it difficult to accurately determine complex faults; and sensor data fusion faces the challenge of interaction between different modal data.

[0072] Therefore, in order to overcome the above challenges and improve the accuracy and real-time performance of fan defect detection, the present application proposes a fan health status detection method based on multimodal data. The main solution of the embodiment of the present application is: by obtaining the data of each component of the fan, pre-processing the data of each component of the fan, obtaining processed image data, processed sound data and processed sensor data, and then processing the above data respectively to obtain spatial features, synchronous vibration features and time series features, and then fusing the spatial features, synchronous vibration features and time series features through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score, and finally making a judgment based on the health score to obtain the health status of the fan and issue an early warning.

[0073] Based on this, the embodiment of the present application provides a method for detecting the health status of a wind turbine based on multimodal data. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the wind turbine health status detection method based on multimodal data of the present application.

[0074] In this embodiment, the wind turbine health status detection method based on multimodal data includes steps S10 to S40:

[0075] Step S10: Acquire data of various components of the wind turbine.

[0076] It should be noted that the above-mentioned data of various components of the fan include fan image data, fan sound data and fan sensor data.

[0077] First, wind turbine image data is primarily collected using high-resolution cameras or thermal imaging cameras, covering key wind turbine components such as blades, bearings, and rotors. These images not only capture visible surface defects such as cracks and wear, but also enhance detection accuracy by capturing images from different angles and lighting conditions. Especially in complex environments, thermal imaging can identify localized overheating caused by friction or other factors, providing a crucial basis for early warning of faults.

[0078] Secondly, wind turbine acoustic data relies on acoustic sensors (such as microphones) to collect sound signals generated during wind turbine operation. Monitoring abnormal noise from components such as bearings and motors is particularly critical. The interactions between wind turbine components generate complex nonlinear vibrations, and the sound signals generated by these vibrations propagating through the air contain a wealth of information. Analysis of these sound signals can not only identify possible coupling and synchronization effects between components, but also proactively detect potential mechanical failures such as looseness or wear.

[0079] Finally, fan sensor data covers a variety of data types, including temperature, pressure, and vibration. Sensors installed at key locations throughout the fan monitor its operating status in real time, including but not limited to parameters such as temperature changes, vibration amplitude, and frequency. This data reflects the dynamic characteristics of the fan during operation and is crucial for understanding its performance under different operating conditions. For example, vibration sensors can detect abnormal vibrations caused by imbalance or misalignment, while temperature sensors can help monitor overheating and prevent damage caused by it. By combining these three types of fan component data, a comprehensive data foundation is provided for health assessment.

[0080] Step S20 , pre-processing the data of each component of the wind turbine to obtain processed image data, processed sound data and processed sensor data.

[0081] It’s important to note that after acquiring the raw data from each wind turbine component, the next key step is to preprocess this data to improve the accuracy and reliability of subsequent analysis. This preprocessing process primarily involves optimizing and cleaning image data, sound data, and sensor data.

[0082] First, the wind turbine image data is processed using the image enhancement method to obtain the processed image data, which is obtained by the following formula:

[0083]

[0084] in, represents the original image, is the Fourier transform of the image, represents the adaptive frequency filter, represents the local enhancement function in the spatial domain, is the inverse transform of the Fourier transform. In this embodiment, an image enhancement method based on adaptive spatial-frequency multi-scale fusion (ASF-MSFE) is used to process image data. This method combines the spatial and frequency domain information of the image and can dynamically adjust the image processing strategy, effectively improving the visibility of minor defects. By analyzing the structural characteristics of wind turbine components (such as blades and bearings) under different ambient lighting and operating conditions, the ASF-MSFE method can specifically enhance the contrast of cracks, corrosion, and wear areas, clearly presenting subtle defects that are difficult to identify using traditional methods. Furthermore, this method can eliminate artifacts caused by changes in lighting, ensuring consistent image quality.

[0085] Secondly, the wind turbine sound data is processed using the time-frequency denoising method to obtain the processed sound data, which is obtained by the following formula:

[0086]

[0087] in, represents the time domain window function, is the initial fan sound data, is the Fourier transform, is the decay function, It is noise. In this embodiment, a time-frequency denoising method based on adaptive nonlinear sound wave propagation feature extraction (ANSPF-TFD) is applied to the sound data. The sound signal generated during the operation of the fan usually contains complex nonlinear vibration modes, especially when a fault occurs, it will show abnormal noise with a specific frequency pattern. The ANSPF-TFD method uses multi-scale wavelet packet transform to perform time-frequency analysis on the sound signal, and removes background noise according to the sound wave propagation model to extract effective fault signals. This method is suitable for detecting early faults of key components such as bearings and motors, because it can not only capture the synchronous vibration characteristics between components, but also identify abnormal phenomena in the sound wave propagation path, providing important clues for fault diagnosis.

[0088] Finally, the wind turbine sensor data is processed using the adaptive denoising and time series synchronization method to obtain the processed sensor data, which is obtained by the following formula:

[0089]

[0090] in, is the initial fan sensor data, is the weight coefficient of the high-order cumulant, represents the dynamic weighting factor, Indicates the length of the time window, is a variable, Indicates Wind turbine sensor data at each moment. For this sensor data, an adaptive denoising and time series synchronization method based on high-order cumulants and dynamic time warping (HOCM-DTW) is proposed. Considering that sensors may be affected by environmental changes and inherent errors during long-term wind turbine operation, the HOCM-DTW method first calculates high-order cumulants of the sensor data to capture the non-Gaussian properties and dynamic changes in the data. Next, a dynamic time warping algorithm (DTW) is used to precisely align data from different sensors to address time deviations. This method not only effectively removes noise interference from sensor data but also ensures temporal consistency between multi-source data, laying a solid foundation for subsequent health status assessment.

[0091] Step S30: performing multi-scale feature extraction on the processed image data to obtain spatial features.

[0092] It should be noted that spatial feature extraction from image data utilizes an image feature extraction algorithm based on the Spatial Dynamic Abstract Spin Network (SDASN). The SDASN algorithm is particularly well-suited for identifying small, fuzzy, and even intermittent surface defects, such as cracks or wear on blades and bearings, from complex wind turbine images.

[0093] Furthermore, step S30 includes: dividing the processed image data into a plurality of small regions, wherein each small region is represented by a spin particle, and initializing the spin particle features, wherein the spin particle features include gradient, local binary value and grayscale standard deviation. Specifically, the pre-processed image data Divided into Non-overlapping small areas, each area With a spinning particle Indicates that the spin particle feature vector is initialized:

[0094]

[0095] in, , Represent the gradients in the horizontal and vertical directions respectively, is the texture feature of the local binary pattern, Represents the standard deviation of regional grayscale. According to the characteristics of spin particles, a spatial relationship graph of spin particles is constructed and the weight is obtained. Specifically, an undirected graph of spin particles is constructed. , where the weight is obtained as:

[0096]

[0097] in, Represents a spinning particle The characteristic mean of Represents a spinning particle The characteristic mean is used to measure the similarity between particles. Represents a spinning particle spatial location, Represents a spinning particle The spatial position of the particle is used to measure the position relationship of the particles in the image. and is a parameter that controls feature similarity and spatial distance. Based on the weight, the state of the spin particle is updated through a dynamic update mechanism to obtain the amplitude of the particle state change. Specifically, in this embodiment, the dynamic update mechanism selects the Ising model in physics. The state of each spin particle is adjusted according to the state of its neighboring particles, thereby simulating the interaction. The specific update formula is:

[0098]

[0099] in, is the feature transformation matrix, which maps the features of the domain particles to the feature space of the current particle. Represents a spinning particle Neighborhood, is the bias term, It is a nonlinear activation function. According to the amplitude of particle state change, the defect response values ​​of multiple spin particles are detected, and then the defect response map is generated based on the defect response values. In order to detect potential defect areas, we introduce the defect response value It represents the magnitude of the change in the spin particle in each update. The larger the response value, the more significant the change in the area and the presence of defects. The specific defect response value calculation formula is:

[0100]

[0101] when , it is considered that the particle at this position has changed significantly and there is a defect. represents the threshold, and finally the defect response diagram is obtained by the following formula:

[0102]

[0103] in is the indicator function, is the defect response value, if the pixel Spin particles The area represented by it is represented as 1, otherwise it is 0. Indicates the maximum number of iterations, and the defect response diagram highlights the areas where defects may exist. In order to capture defects at different scales, SDASN introduces multi-scale processing. Above, the image is downsampled and SDASN is applied for feature extraction, and finally the features of each scale are combined through weighted fusion:

[0104]

[0105] in, It is Defect response diagram at each scale, is the weighted coefficient of the scale, which determines the contribution of different scale features to the final result. Indicates the total number of scales. Through multi-scale feature fusion, SDASN can simultaneously identify defects such as large-scale corrosion and tiny cracks, thereby improving the accuracy of detection. A defect enhancement map is generated based on the defect response map and the processed image data, and feature extraction is performed on the defect enhancement map to obtain spatial features. Specifically, through the defect response map and the original image Combined, the defect enhancement map is generated. The specific formula is:

[0106]

[0107] in, To enhance the intensity, the defect areas are highlighted by embedding the response map into the original image. These enhanced images are then used by the feature extractor for further analysis of defect type, location, and severity. Multi-scale histograms of oriented gradients and LBP features are extracted from the enhanced defect image to obtain a spatial feature matrix, which in turn generates spatial features.

[0108] This approach enables SDASN to enhance its ability to detect subtle defects in complex environments, particularly gradual, non-significant damage that is difficult to identify with traditional methods, such as early-stage corrosion or fine cracks. Ultimately, the spatial features extracted through this process not only contain detailed damage information on the wind turbine component surface, but also reflect the changing trends of its internal structure, providing a solid data foundation for subsequent health assessments.

[0109] Step S40 , performing acoustic-dynamic coupling synchronization feature extraction processing on the processed sound data to obtain synchronous vibration features.

[0110] It should be noted that the processed sound data is further processed using the Sound-Motion Coupling Synchronous Feature Extraction (SOUNDSYNC) algorithm to obtain synchronized vibration signatures that reflect the health of the wind turbine. During wind turbine operation, components such as blades, bearings, and rotors generate complex nonlinear vibrations due to the external environment or internal workload. These vibrations are not limited to individual components but also form a dynamic coupled system through the propagation, reflection, attenuation, and resonance of sound waves between different components. The SOUNDSYNC algorithm is designed based on this complex mechanism of sound wave propagation and vibration coupling.

[0111] Furthermore, step S40 includes: constructing a nonlinear coupled vibration equation and an acoustic wave propagation path model, specifically, establishing a nonlinear vibration equation of wind turbine components (such as blades and bearings):

[0112]

[0113] in, For the The quality of each component, For the The damping coefficient of each component, For the The stiffness coefficient of each component, is the coupling coefficient, indicating the to The coupling strength between the components, For the The vibration displacement of each component, For the The acceleration of a component, For the The speed of each component, is the external excitation force (such as wind force, load change, etc.). Then the equation of the sound wave propagation path model is established:

[0114]

[0115] in, For the The sound pressure field at each component, is the speed of sound, is the propagation path coefficient, For the Parts and The coupling strength between the components, the sound pressure field The change over time reflects the propagation and attenuation process of the sound wave. The processed sound data is input into the nonlinear coupled vibration equation and the sound wave propagation path model to obtain the vibration results and the sound wave synchronization effect. Then, based on the vibration results and the sound wave synchronization effect, the nonlinear synchronization index and propagation attenuation characteristics are calculated. Specifically, the nonlinear synchronization index is calculated as follows:

[0116]

[0117] in, For the The mean value of the vibration of each component, For the The standard deviation of the vibration of each component, is the time length of the signal, and the nonlinear synchronization index is used to describe the degree of synchronization between the wind turbine components. When the nonlinear synchronization index is high, it indicates that the component is highly synchronized with other components, which means that the wind turbine is operating normally; conversely, it implies that the coordination between components is reduced, and the wind turbine is faulty or in an abnormal state. The propagation attenuation characteristics are calculated using the following formula:

[0118]

[0119] in, For the The initial sound pressure value of each component, For the The sound pressure value of each component is obtained by fusing the nonlinear synchronization index and propagation attenuation characteristics to obtain the synchronous vibration characteristics. is the time length of the signal.

[0120] This method can extract key synchronous vibration features from acoustic data, providing important insights for wind turbine fault diagnosis. These features not only help identify the specific fault type of the wind turbine, such as bearing wear or motor failure, but also assess the severity of the fault, supporting maintenance decisions.

[0121] Step S50 , performing hierarchical dynamic feature extraction processing on the processed sensor data to obtain time series features.

[0122] It should be noted that the processed sensor data is further analyzed using the Heterogeneous Hierarchical Adaptive Dynamical System Embedding (HHADSE) algorithm to extract time series features. During wind turbine operation, sensor data such as temperature, pressure, and vibration reflect the dynamic operating status and health of each component. However, due to the diverse and heterogeneous sources of this data, direct analysis often presents challenges. The HHADSE algorithm effectively captures the dynamic interactions and changing patterns among wind turbine components by hierarchically embedding this heterogeneous sensor data into an adaptive dynamic system.

[0123] Furthermore, step S50 also includes: dividing the fan structure into multiple layers. Specifically, the fan system is divided into the following layers: the first layer (basic component layer): core components such as the main shaft, gearbox, and motor; the second layer (subsystem layer): power transmission system, control system, cooling system, etc.; the third layer (overall system layer): the overall fan status, such as speed, power output, etc. The status characteristics of each layer are represented by the following methods:

[0124]

[0125] in, It is Parts at time No. state variables, is the number of component state variables. The dynamic model is constructed based on multiple levels, and the specific formula is:

[0126]

[0127] in, It is The state vector of each component, It is The input data vector of each component (such as sensor measurement data temperature or vibration, etc.), is a nonlinear dynamic function that represents the evolution of the component state, is a noise term, representing external disturbances or modeling errors. However, since the various components of a wind turbine do not operate independently during operation, they are often strongly coupled. For example, the rotation of the gearbox affects the operating state of the main shaft, and the load on the main shaft affects the speed of the motor. Therefore, we introduce coupling terms between components to capture these mutual influences. Specifically, the final dynamic equation can be expanded to:

[0128]

[0129] in, is the coupling coefficient, It is a component For parts In order to adapt the model to the changes in wind turbine state under different operating conditions, the HHADSE algorithm introduces an adaptive dynamic modeling method. We use adaptive algorithms (such as Kalman filtering and Bayesian optimization) to update the parameters of the system model in real time to adapt to different operating conditions such as load, temperature, and wind speed. The specific update formula is:

[0130]

[0131] in, For the The model parameters of each component, is the learning rate, which controls the step size of parameter updates, is the actual measurement data, is the predicted data, is the model parameter gradient.

[0132] The processed sensor data is input into the dynamic model to obtain multiple hierarchical features. The calculation formula of the hierarchical features is:

[0133]

[0134] in, Indicates the Parts at time Extracted Features, Indicates the number of features. Multiple hierarchical features are fused to obtain time series features. To further enhance the expressive power of features, the HHADSE algorithm uses multi-level feature fusion technology to combine the feature vectors of each level. The feature vector of each level reflects the working status of the components at that level. By fusing these features, we obtain a global, multi-dimensional feature representation that is used to comprehensively describe the health status of the wind turbine. The specific formula is:

[0135]

[0136] in, It is The hierarchical feature vector of each component is constructed, where N is the number of components. This multi-level fused feature vector contains the dynamic behavior of each component of the wind turbine and the mutual influence between them, providing comprehensive information for subsequent status assessment and fault detection.

[0137] Step S60: The spatial features, synchronous vibration features and time series features are integrated through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score.

[0138] It should be noted that in order to achieve a comprehensive assessment of the health status of the wind turbine, the spatial features, synchronous vibration features, and time series features extracted from image, sound, and sensor data need to be integrated through a multimodal adaptive spatiotemporal coupling decision network (MASCDN) to ultimately obtain a comprehensive health score.

[0139] Specifically, MASCDN independently extracts features from each modal data and combines the spatiotemporal relationships between them to construct a unified feature space. For example, spatial features in an image can reveal the specific damage to the surface of a wind turbine component; synchronous vibration features capture abnormal vibration patterns caused by interactions between components; and time series features reflect the changing trends of parameters such as temperature and pressure during wind turbine operation. MASCDN effectively combines these features by introducing a spatiotemporal coupling layer, accounting for the interactions that occur when wind turbine component failures change over time and space. Next, MASCDN uses an adaptive mechanism to dynamically adjust the weight of each modal, ensuring that the model can be automatically optimized based on real-time data. This process is accomplished through an adaptive weight adjustment formula, which appropriately reflects the effectiveness and importance of each modal feature under different operating conditions. Finally, a fusion layer combines the features of all modalities to output a health score for the wind turbine.

[0140] Step S70: determine the health status of the wind turbine based on the health score and issue an early warning.

[0141] It should be noted that the health score comprehensively considers the spatial features, synchronous vibration features and time series features extracted from images, sounds and sensor data. Through these multi-dimensional data analysis, the working conditions of various components of the wind turbine and their potential risks can be comprehensively evaluated.

[0142] Furthermore, during the initial assessment, a series of thresholds are set based on the health score to determine whether the wind turbine has defects. If the health score falls below a pre-set safety threshold, the turbine is at risk of failure or impending failure. Further detailed analysis is then initiated, including identifying the specific defect type (such as cracks, bearing wear, or motor failure) and assessing the severity of the defect. This process typically uses a multi-classification model to determine the defect type and a regression model to estimate the severity score. Once a defect is confirmed and its severity exceeds the pre-set threshold, an early warning mechanism is triggered, sending a real-time alert to maintenance personnel. The alert not only includes the fault type and severity, but also details the specific component location (e.g., blades, bearings, motors, etc.) and provides targeted remediation recommendations based on the specific situation. This precise early warning information enables maintenance teams to take swift action and schedule inspections or repairs promptly, thereby preventing potential failures from occurring or escalating, reducing unnecessary downtime and maintenance costs.

[0143] Furthermore, the early warning system features an adaptive feedback mechanism that dynamically adjusts the operating status of each module based on the latest wind turbine operating data, ensuring continuous optimization and improvement of the system. This not only improves the accuracy and timeliness of wind turbine fault detection, but also provides strong technical support for the stable operation of wind farms.

[0144] This embodiment provides a method for detecting the health status of a wind turbine based on multimodal data. The method acquires data from various wind turbine components, preprocesses the data to obtain processed image data, processed sound data, and processed sensor data. These data are then processed separately to obtain spatial features, synchronous vibration features, and time series features. These features are then fused through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score. Finally, a judgment is made based on the health score to determine the health status of the wind turbine and issue an early warning. By collecting wind turbine image, sound, and sensor data, extracting spatial features, synchronous vibration features, and time series features using multimodal processing techniques, and fusing these features through a multimodal adaptive spatiotemporal coupling decision network to generate a health score, the health status of the wind turbine is assessed and an early warning is issued based on the score. This method achieves early risk detection and proactive early warning, improving the accuracy and timeliness of wind turbine fault detection.

[0145] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 The wind turbine health status detection method based on multimodal data further includes steps S201 to S202 before step S60:

[0146] Step S201 : Dynamically weighting and fusing spatial features, synchronous vibration features, and time series features through the spatiotemporal coupling layer of the multimodal adaptive spatiotemporal coupling decision network to generate a fused feature vector.

[0147] It should be noted that the spatial features, synchronous vibration features, and time series features are dynamically weighted and fused through the spatiotemporal coupling layer of the Multimodal Adaptive Spatiotemporal Coupling Decision Network (MASCDN) to generate a fused feature vector. This process fully utilizes the inherent correlation between different modal data in the spatial and temporal dimensions, thereby providing a more accurate assessment of the health status of the wind turbine. The specific calculation formula is:

[0148]

[0149] in, represents the weight of the SDASN method, represents the spatial feature matrix, Represents the weight of the SOUNDSYNC method, represents the synchronous vibration characteristic matrix, represents the weight of the HHADSE method, represents the time series feature matrix. At this stage, the MASCDN algorithm trains data using a spatiotemporal coupling decision model. This decision model is trained based on features fused from the spatiotemporal coupling layer and ultimately outputs a prediction of the wind turbine's health status. During training, the model updates weights by minimizing the loss function, thereby learning the nonlinear relationships and spatiotemporal dynamics of different modal features. The weights are updated using the following formula:

[0150]

[0151] in, Indicates a period of time, is the loss function, is the true value, is the predicted value. The model is continuously optimized through back propagation and gradient descent to update the weight matrix 、 and , so that the network can accurately reflect the complex relationship between each modality data.

[0152] Step S202 : Based on the fused feature vector, the defect type and defect severity level of the wind turbine are obtained through the decision layer of the multimodal adaptive spatiotemporal coupling decision network.

[0153] It's important to note that, based on the fused feature vectors, MASCDN's decision layer can conduct an in-depth analysis of the wind turbine's health status to determine the specific defect type and severity level. This process is achieved by inputting the fused feature vectors into a trained model, leveraging the learned nonlinear relationships and spatiotemporal dynamics between different modal features to achieve accurate fault diagnosis.

[0154] Furthermore, in terms of identifying defect types, MASCDN uses a multi-classification model that can distinguish various types of wind turbine defects, such as blade cracks, bearing wear, motor failure, etc. By analyzing the key information in the fused feature vector, the model can identify patterns associated with specific defect types. For example, a tiny crack on a blade will appear as a local anomaly in the spatial features of the image data, and as an abnormal vibration pattern of a specific frequency in the synchronous vibration features of the sound data. By combining this information, the specific type of defect can be accurately determined. Secondly, for the assessment of defect severity, MASCDN uses a regression model to estimate the severity score of the defect. The formula is:

[0155]

[0156] in, Indicates the severity score of the defect, is the weight of the decision layer, is the bias term, Is the activation function, usually the sigmoid function is used.

[0157] Step S203 : Evaluate the defect type and defect severity level to obtain a health score of the wind turbine.

[0158] It's important to note that, first, different defect types are assigned corresponding weights. For example, severe defects such as cracks and wear in critical components like the main shaft or gearbox are given higher weights because they could potentially lead to serious failures or even downtime. Conversely, smaller surface damage, such as minor blade wear, is given lower weights because its short-term impact on turbine operation is relatively minimal. Second, the severity of the defect directly impacts the health score. A regression model is used to quantify the severity of each defect. This step goes beyond simple classification and assigns a specific numerical score based on detailed feature analysis. For example, minor wear might be scored 1-2 points, while severe cracks or mechanical looseness might be rated 8-10 points (assuming a maximum score of 10). These scores reflect the potential impact of the defect on future operational risks. Finally, all detected defects and their severity scores are aggregated, combined with historical turbine data and operating conditions, to calculate a comprehensive health score. This score provides a visual and quantitative description of the turbine's current health, enabling maintenance teams to quickly understand the equipment's status.

[0159] This embodiment generates a fused feature vector by dynamically weighting and fusing multimodal data, uses the decision layer to identify the defect type and severity, evaluates the health score of the wind turbine, improves the accuracy and timeliness of fault detection, achieves early warning, and ensures the stable operation of the wind turbine.

[0160] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction and will not be described in detail later. Figure 3 The wind turbine health status detection method based on multimodal data, step S70, further includes steps S301 to S303:

[0161] Step S301: When the health score is less than the first health threshold, the health state of the wind turbine is determined to be normal, and the wind turbine is monitored normally.

[0162] It's important to note that when the generated health score is greater than or equal to the first health threshold, the fan is in normal condition. In this case, the system continues routine monitoring of the fan to ensure continued stable operation. This regular monitoring includes not only regular data collection and analysis, but also real-time monitoring of the operating parameters of each fan component, such as temperature, vibration frequency, speed, and other key indicators.

[0163] Furthermore, for turbines in normal operation, the system focuses on preventive maintenance and early warning of failures. By accumulating and analyzing historical data, MASCDN can identify patterns in the development of potential problems and adjust monitoring strategies accordingly. For example, if the historical data for a specific component shows a slight but gradually increasing trend, more frequent inspections or preventive maintenance measures may be recommended to avoid future problems.

[0164] Step S302: When the health score is greater than the first health threshold and the health score is less than the second health threshold, the health state of the wind turbine is determined to be a risk state, triggering an early warning and recommending inspection of the wind turbine.

[0165] It should be noted that when the generated health score is greater than the first health threshold and less than the second health threshold, it indicates that the wind turbine is in a risky state, where the second health threshold is greater than the first health threshold. At this time, the early warning mechanism is automatically triggered, and further inspection of the wind turbine is recommended to prevent the occurrence or deterioration of potential faults. This early warning mechanism not only helps to promptly discover and resolve problems with the wind turbine, but also effectively avoids downtime and high repair costs caused by sudden failures. In the risk state, the early warning information will include detailed defect types, severity levels, and the specific locations of affected components. For example, if there are slight cracks on the blades or the bearings show signs of early wear, the early warning information will clearly point out these problems and help the maintenance team quickly locate the source of the fault. In addition, targeted repair suggestions will be provided based on the specific situation, such as whether it is necessary to shut down the machine for inspection immediately or take temporary measures to alleviate the current situation.

[0166] Furthermore, for turbines at risk, the system recommends actions including, but not limited to, increasing monitoring frequency, implementing more in-depth inspection procedures, and adjusting operating parameters to reduce loads. High-resolution cameras and thermal imaging technology are used to enhance monitoring of critical areas, acoustic sensors are used to closely monitor abnormal noise trends, and sensor data is used to continuously track changes in parameters such as temperature and vibration. These measures are designed to provide maintenance personnel with more time and information to make more informed decisions.

[0167] Step S303: When the health score is greater than the second health threshold, the health status of the wind turbine is determined to be a fault state, an emergency alarm is triggered, and a maintenance work order is generated.

[0168] It should be noted that when the generated health score exceeds the second health threshold, it indicates that the wind turbine is in a faulty state. At this point, the system will immediately trigger the emergency alarm mechanism and automatically create a maintenance work order to ensure timely action to prevent the fault from further deteriorating or causing more serious safety accidents. The emergency alarm will not only notify on-site operators through visual and sound signals, but will also be sent to relevant maintenance teams and technical support personnel via SMS, email or other instant messaging tools. The alarm information lists in detail the fault type, severity level and the specific location of the affected components, such as bearing wear, blade cracks or motor failure. In addition, based on the results of the fused feature vector analysis, the system will provide specific repair suggestions, such as the need to replace specific components or perform emergency shutdown maintenance.

[0169] Furthermore, once a turbine fault is confirmed, a detailed maintenance work order is automatically generated. This work order includes all necessary information, such as a description of the fault, recommended repair steps, a list of required spare parts, and an estimated repair time. It also contains historical maintenance records and solutions for similar faults, allowing technicians to quickly and effectively resolve the issue. For critical component failures, priority scheduling can be recommended to ensure that resources are deployed promptly.

[0170] This embodiment dynamically evaluates the status of the wind turbine through health scoring, implements hierarchical monitoring, early warning and emergency response, achieves early warning and rapid response, and ensures the safe and stable operation of the wind turbine.

[0171] This application also provides a wind turbine health status detection device based on multimodal data, please refer to Figure 4 , the device comprises:

[0172] The acquisition module 10 is used to acquire data of various components of the wind turbine, wherein the data of various components of the wind turbine include wind turbine image data, wind turbine sound data and wind turbine sensor data.

[0173] The processing module 20 is used to pre-process the data of each component of the wind turbine to obtain processed image data, processed sound data and processed sensor data.

[0174] The extraction module 30 is used to perform multi-scale feature extraction on the processed image data to obtain spatial features.

[0175] The extraction module 30 is further configured to perform acoustic-dynamic coupling synchronization feature extraction processing on the processed sound data to obtain synchronized vibration features.

[0176] The extraction module 30 is further configured to perform hierarchical dynamic feature extraction on the processed sensor data to obtain time series features.

[0177] The fusion module 40 is used to fuse the spatial features, the synchronous vibration features and the time series features through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score.

[0178] The result module 50 is used to make a judgment based on the health score, obtain the health status of the wind turbine and issue an early warning.

[0179] The multimodal data-based wind turbine health status detection device provided in this application utilizes the multimodal data-based wind turbine health status detection method of the aforementioned embodiment to address the technical problem of accurately acquiring the health status of each wind turbine component and executing corresponding early warning operations. Compared to the prior art, the beneficial effects of the multimodal data-based wind turbine health status detection device provided in this application are the same as those of the multimodal data-based wind turbine health status detection method provided in the aforementioned embodiment. Other technical features of the multimodal data-based wind turbine health status detection device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0180] In one embodiment, the processing module 20 is also used to process the fan image data using an image enhancement method to obtain processed image data, to process the fan sound data using a time-frequency denoising method to obtain processed sound data, and to process the fan sensor data using an adaptive denoising and timing synchronization method to obtain processed sensor data.

[0181] In one embodiment, the extraction module 30 is further used to divide the processed image data into multiple small areas, where each small area is represented by a spin particle; initialize the spin particle features, which include gradient, local binary value and grayscale standard deviation; construct a spatial relationship diagram of the spin particles based on the spin particle features and obtain weights; update the spin particle state through a dynamic update mechanism based on the weights to obtain the particle state change amplitude; perform detection based on the particle state change amplitude to obtain defect response values ​​of multiple spin particles; generate a defect response map based on the defect response values; generate a defect enhancement map based on the defect response map and the processed image data; and perform feature extraction on the defect enhancement map to obtain spatial features.

[0182] In one embodiment, the extraction module 30 is also used to construct a nonlinear coupled vibration equation and an acoustic wave propagation path model; the processed sound data is input into the nonlinear coupled vibration equation and the acoustic wave propagation path model to obtain vibration results and acoustic wave synchronization effects; calculations are performed based on the vibration results and the acoustic wave synchronization effects to obtain a nonlinear synchronization index and a propagation attenuation characteristic; and the nonlinear synchronization index and the propagation attenuation characteristic are fused to obtain a synchronous vibration characteristic.

[0183] In one embodiment, the extraction module 30 is further used to hierarchize the wind turbine structure to obtain multiple levels; construct a dynamic model based on the multiple levels; input the processed sensor data into the dynamic model to obtain multiple hierarchical features; and fuse the multiple hierarchical features to obtain time series features.

[0184] In one embodiment, the fusion module 40 is further used to dynamically weight and fuse spatial features, synchronous vibration features, and time series features through the spatiotemporal coupling layer of the multimodal adaptive spatiotemporal coupling decision network to generate a fused feature vector; based on the fused feature vector, the decision layer of the multimodal adaptive spatiotemporal coupling decision network is used to obtain the defect type and defect severity level of the wind turbine; and an evaluation is performed based on the defect type and defect severity level to obtain a health score of the wind turbine.

[0185] In one embodiment, the result module 50 is also used to, when the health score is less than a first health threshold, obtain that the health status of the fan is normal, and perform normal monitoring on the fan; when the health score is greater than the first health threshold and the health score is less than the second health threshold, obtain that the health status of the fan is a risky state, trigger an early warning and recommend that the fan be inspected; when the health score is greater than the second health threshold, obtain that the health status of the fan is a fault state, trigger an emergency alarm and generate a maintenance work order, wherein the second health threshold is greater than the first health threshold.

[0186] The present application provides a wind turbine health status detection device based on multimodal data, and the wind turbine health status detection device based on multimodal data includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the wind turbine health status detection method based on multimodal data in the above-mentioned embodiment one.

[0187] Reference below Figure 5 , which shows a schematic structural diagram of a wind turbine health status detection device based on multimodal data suitable for implementing an embodiment of the present application. The wind turbine health status detection device based on multimodal data in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5The wind turbine health status detection device based on multimodal data shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0188] like Figure 5 The wind turbine health status detection device based on multimodal data may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in ROM (Read Only Memory) 1002 or programs loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the wind turbine health status detection device based on multimodal data. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. Communication devices 1009 can allow the wind turbine health status detection device based on multimodal data to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a wind turbine health status detection device based on multimodal data with various systems, it should be understood that implementation or presence of all the illustrated systems is not required. More or fewer systems may alternatively be implemented or present.

[0189] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes program code for executing the method described in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0190] The multimodal data-based wind turbine health status detection device provided in this application, employing the multimodal data-based wind turbine health status detection method of the aforementioned embodiment, can solve the technical problem of accurately acquiring the health status of each wind turbine component and executing corresponding early warning operations. Compared to the prior art, the beneficial effects of the multimodal data-based wind turbine health status detection device provided in this application are the same as those of the multimodal data-based wind turbine health status detection method provided in the aforementioned embodiment. Other technical features of the multimodal data-based wind turbine health status detection device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0191] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0192] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0193] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer program) stored thereon for performing calculations to obtain machine-readable program instructions for executing the wind turbine health status detection method based on multimodal data in the above-mentioned embodiment.

[0194] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. The calculations performed in this embodiment show that the machine-readable storage medium may be any tangible storage medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable storage medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0195] The computer-readable storage medium may be included in the wind turbine health status detection device based on multimodal data; or may exist independently without being assembled into the wind turbine health status detection device based on multimodal data.

[0196] The computer-readable storage medium carries one or more programs. When executed by a wind turbine health status detection device based on multimodal data, the one or more programs enable the wind turbine health status detection device based on multimodal data to write computer program code for performing the operations of the present application in one or more programming languages, or a combination thereof. The programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0197] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0198] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0199] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for detecting the health status of a wind turbine based on multimodal data. This computer-readable storage medium addresses the technical problem of accurately acquiring the health status of various wind turbine components and executing corresponding early warning operations. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the method for detecting the health status of a wind turbine based on multimodal data provided in the aforementioned embodiments, and are not further elaborated here.

[0200] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for detecting the health status of a wind turbine based on multimodal data.

[0201] The computer program product provided in this application can solve the technical problem of accurately acquiring the health status of various wind turbine components and executing corresponding early warning operations. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the wind turbine health status detection method based on multimodal data provided in the above-mentioned embodiment, and will not be elaborated here.

[0202] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for detecting the health status of a wind turbine based on multimodal data, characterized in that: The method comprises: Acquiring data of various components of the fan, wherein the data of various components of the fan include fan image data, fan sound data, and fan sensor data; Preprocessing the data of each component of the wind turbine to obtain processed image data, processed sound data and processed sensor data; Performing multi-scale feature extraction processing on the processed image data to obtain spatial features; performing acoustic-dynamic coupling synchronization feature extraction processing on the processed sound data to obtain a synchronous vibration feature; performing hierarchical dynamic feature extraction processing on the processed sensor data to obtain time series features; fusing the spatial features, the synchronous vibration features, and the time series features through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score; According to the health score, the health status of the fan is judged and an early warning is issued; The step of performing acoustic-dynamic coupling synchronization feature extraction processing on the processed sound data to obtain synchronous vibration features includes: Construct nonlinear coupled vibration equations and acoustic wave propagation path models; Inputting the processed sound data into the nonlinear coupled vibration equation and the sound wave propagation path model to obtain vibration results and sound wave synchronization effects; Calculating based on the vibration results and the acoustic wave synchronization effect to obtain a nonlinear synchronization index and a propagation attenuation characteristic; A synchronous vibration feature is obtained by fusing the nonlinear synchronization index and the propagation attenuation feature; The step of performing hierarchical dynamic feature extraction on the processed sensor data to obtain time series features includes: The fan structure is layered to obtain multiple levels; constructing a kinetic model according to a plurality of said levels; Inputting the processed sensor data into the dynamic model to obtain a plurality of hierarchical features; A plurality of the hierarchical features are fused to obtain time series features.

2. The method according to claim 1, wherein The step of preprocessing the data of each component of the fan to obtain processed image data, processed sound data and processed sensor data includes: The wind turbine image data is processed using an image enhancement method to obtain processed image data, which is obtained by the following formula: in, represents the original image, is the Fourier transform of the image, represents the adaptive frequency filter, represents the local enhancement function in the spatial domain, is the inverse transform of Fourier transform; The wind turbine sound data is processed using a time-frequency denoising method to obtain processed sound data, which is obtained by the following formula: in, represents the time domain window function, is the initial fan sound data, is the Fourier transform, is the decay function, It is noise; The wind turbine sensor data is processed using an adaptive denoising and time series synchronization method to obtain processed sensor data, which is obtained by the following formula: in, is the initial fan sensor data, is the weight coefficient of the high-order cumulant, represents the dynamic weighting factor, Indicates the length of the time window, is a variable.

3. The method according to claim 1, wherein The step of performing multi-scale feature extraction processing on the processed image data to obtain spatial features includes: Dividing the processed image data into a plurality of small regions, wherein each of the small regions is represented by a spin particle; Initializing spin particle features, wherein the spin particle features include gradient, local binary value, and grayscale standard deviation; Constructing a spatial relationship graph of the spin particles according to the spin particle characteristics and obtaining weights; Based on the weight, the state of the spin particle is updated through a dynamic update mechanism to obtain a particle state change amplitude; Detecting according to the particle variation amplitude to obtain defect response values ​​of a plurality of the spin particles; generating a defect response graph according to the defect response value; generating a defect enhancement map based on the defect response map and the processed image data; Feature extraction is performed on the defect enhancement image to obtain spatial features.

4. The method according to claim 1, wherein The step of fusing the spatial features, the synchronous vibration features, and the time series features through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score includes: Performing dynamic weighted fusion of the spatial features, synchronous vibration features, and time series features through the spatiotemporal coupling layer of the multimodal adaptive spatiotemporal coupling decision network to generate a fused feature vector; Obtaining the defect type and defect severity level of the wind turbine based on the fused feature vector through the decision layer of the multimodal adaptive spatiotemporal coupling decision network; An assessment is performed based on the defect type and the defect severity level to obtain a health score of the wind turbine.

5. The method according to claim 1, wherein The step of determining the health status of the wind turbine based on the health score and issuing an early warning includes: When the health score is less than a first health threshold, the health state of the wind turbine is determined to be normal, and the wind turbine is monitored normally; When the health score is greater than a first health threshold and the health score is less than a second health threshold, the health state of the wind turbine is determined to be a risk state, triggering an early warning and recommending inspection of the wind turbine; When the health score is greater than a second health threshold, the health state of the wind turbine is determined to be a fault state, an emergency alarm is triggered, and a maintenance work order is generated, wherein the second health threshold is greater than the first health threshold.

6. A wind turbine health status detection device based on multimodal data, characterized in that: The device comprises: An acquisition module is used to acquire data of various components of the fan, wherein the data of various components of the fan include fan image data, fan sound data and fan sensor data; a processing module, configured to pre-process the data of the various components of the wind turbine to obtain processed image data, processed sound data, and processed sensor data; An extraction module, configured to perform multi-scale feature extraction on the processed image data to obtain spatial features; The extraction module is further configured to perform acoustic-dynamic coupling synchronization feature extraction processing on the processed sound data to obtain a synchronous vibration feature; further configured to construct a nonlinear coupling vibration equation and an acoustic wave propagation path model; input the processed sound data into the nonlinear coupling vibration equation and the acoustic wave propagation path model to obtain a vibration result and an acoustic wave synchronization effect; perform calculations based on the vibration result and the acoustic wave synchronization effect to obtain a nonlinear synchronization index and a propagation attenuation feature; and fuse the nonlinear synchronization index and the propagation attenuation feature to obtain a synchronous vibration feature. The extraction module is further configured to perform hierarchical dynamic feature extraction processing on the processed sensor data to obtain time series features; further configured to layer the wind turbine structure to obtain multiple layers; construct a dynamic model based on the multiple layers; input the processed sensor data into the dynamic model to obtain multiple hierarchical features; and fuse the multiple hierarchical features to obtain time series features. a fusion module, configured to fuse the spatial features, the synchronous vibration features, and the time series features through a multimodal adaptive spatiotemporal coupling decision network to obtain a health score; The result module is used to make judgments based on the health score, obtain the health status of the wind turbine and issue an early warning.

7. A wind turbine health status detection device based on multimodal data, characterized in that: The device includes: a memory, a processor, and a wind turbine health status detection program based on multimodal data stored in the memory and running on the processor, wherein the wind turbine health status detection program based on multimodal data is configured to implement the steps of the wind turbine health status detection method based on multimodal data as described in any one of claims 1-5.

8. A storage medium, characterized in that: The storage medium stores a wind turbine health status detection program based on multimodal data. When the wind turbine health status detection program based on multimodal data is executed by the processor, the steps of the wind turbine health status detection method based on multimodal data as described in any one of claims 1 to 5 are implemented.

Images