Deep learning fault diagnosis based on multi-modal and multi-source data fusion

Abstract

The invention relates to the technical field of offshore wind field operation and maintenance, and provides multi-modal and multi-source data fused deep learning fault diagnosis, which is characterized in that multi-modal data and multi-source data are acquired, cleaning work, unit conversion and format conversion work are carried out on the data, the data are integrated, and the fault diagnosis efficiency is improved. The time domain features, the frequency domain features, the time frequency features and the text features are taken as basic features to perform data feature extraction work, under the assistance of reducing feature dimension work, the calculation complexity is reduced, useful information is kept, an original model is generated through fusion, the original model is trained and optimized by using a deep learning model, and the calculation efficiency is improved. And fault diagnosis of offshore wind field operation and maintenance is completed for the fault diagnosis model. Multi-modal and multi-source heterogeneous data are combined, various information sources in a wind driven generator system are fully utilized, more accurate and comprehensive diagnosis services are provided, the accuracy and reliability of fault diagnosis are improved, the robustness of the system is enhanced, and therefore stable operation and efficient maintenance of the wind driven generator are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of offshore wind farm operation and maintenance, and specifically to a deep learning fault diagnosis based on multi-modal and multi-source data fusion. Background Art

[0002] With the transformation of the global energy structure and the advancement of the sustainable development strategy, offshore wind power, as a clean and renewable energy, is gradually becoming an important part of the development of new energy. As a key link to ensure the long-term and stable operation of wind farms, offshore wind farm operation and maintenance faces complex marine environment and high cost challenges. The marine climate is changeable, equipment maintenance is difficult, and the operation and maintenance costs are high. Therefore, efficient operation and maintenance management is crucial to reducing wind farm operating costs and improving power generation efficiency. At the same time, with the advancement of technology and the accumulation of experience, offshore wind power operation and maintenance is gradually developing towards intelligence and automation, in order to achieve more efficient and economical operation and maintenance services.

[0003] Fault diagnosis is one of the key technologies to ensure the safe and efficient operation of key equipment in the intelligent manufacturing process. The multimodal fusion deep learning intelligent fault diagnosis method is a method that uses multiple different types of data sources (such as sound, vibration signals, images, text data, etc.) to improve the accuracy of power equipment or system fault diagnosis. Deep learning fault diagnosis with multi-source heterogeneous data fusion is a method that uses data from multiple heterogeneous sources and fuses and analyzes these data through deep learning technology to achieve accurate diagnosis of system or equipment faults. Its core is to effectively integrate data from different sources and types (such as sensor data, audio, images, text, etc.) so as to extract relevant fault information from them.

[0004] This application combines multi-modality with multi-source heterogeneous data, fully utilizes various information sources in the wind turbine system, provides more accurate and comprehensive diagnostic services, improves the accuracy and reliability of fault diagnosis, and enhances the robustness of the system, thereby ensuring the stable operation and efficient maintenance of wind turbines. In this way, a deep learning fault diagnosis based on multi-modality and multi-source data fusion is provided. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a deep learning fault diagnosis based on the fusion of multimodal and multi-source data, aiming to combine multimodal and multi-source heterogeneous data, make full use of various information sources in the wind turbine system, provide more accurate and comprehensive diagnostic services, improve the accuracy and reliability of fault diagnosis, enhance the robustness of the system, and thus ensure the stable operation and efficient maintenance of wind turbines.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A deep learning fault diagnosis method based on multi-modal and multi-source data fusion includes the following steps:

[0008] Step 1: Collect multi-modal data and multi-source data. According to the characteristics of different sensors with different sampling rates, establish a time frame for data synchronization and alignment, and align all collected data at the same time so that the aligned data should have consistent time resolution;

[0009] Step 2: Perform data cleaning and detect outliers on the data. If abnormal data is detected, delete the abnormal values, replace the abnormal values, or perform data interpolation to process the abnormal data to prevent the abnormal data from affecting the analysis results.

[0010] Step 3: Use any of the noise elimination methods such as filter, wavelet denoising, sliding average filtering and frequency domain filtering to denoise the data and improve the signal quality;

[0011] Step 4: Convert the data into units and formats, unify the units of the same type of data, and convert the data into appropriate formats, such as extracting text data into structured data and converting image data into feature vectors;

[0012] Step 5: Perform data fusion and extract features based on time domain features, frequency domain features, time-frequency features, and text features. After the extraction is completed, use methods such as principal component analysis (PCA), linear discriminant analysis (LDA), and t-SNE to reduce feature dimensions and computational complexity while maintaining useful information.

[0013] Step 6: Perform multi-modal and multi-source data fusion, perform feature-level fusion, model-level fusion and decision-level fusion in sequence, and generate the original model;

[0014] Step 7: Use the deep learning model to train and optimize the original model and improve it into a fault diagnosis model;

[0015] Step 8: Deploy the trained fault diagnosis model to the actual wind farm monitoring system, analyze the real-time collected data, and monitor potential faults in a timely manner;

[0016] Step 9: Based on trend analysis and anomaly detection, predict impending failures and provide early warnings and maintenance recommendations;

[0017] Step 10: Automatically generate a diagnostic report for the device and automatically match the corresponding solution in the expert database according to the fault type and device model.

[0018] Preferably, the multimodal data collected in step 1 includes:

[0019] Visual data, surveillance camera images or videos: used to monitor the appearance, structural integrity, changes in the surrounding environment, etc. of wind turbines. Image or video data is used to detect external physical damage or erosion;

[0020] Acoustic data: Captures the noise generated by wind turbines during operation to analyze the health of mechanical components and detect abnormal sounds such as noise or vibration to help diagnose potential mechanical problems;

[0021] Text data: Operation and maintenance logs: record the operation status, fault events, maintenance activities, etc. of wind turbines. Text data includes reports, problem descriptions, and maintenance suggestions in natural language format.

[0022] Time series data: Sensor data: including wind speed, wind direction, temperature, humidity, power generation, rotation speed, vibration, etc. The data is recorded by various sensors in the form of continuous time series to provide detailed information about the operating status of the generator and external environmental conditions;

[0023] Graphic data: Power system topology diagram: describes the topological structure and power flow of wind turbines connected to the power grid to assist in understanding the power transmission and distribution, and the changes in power load under different conditions.

[0024] Preferably, the multi-source data collected in step 1 includes:

[0025] Vibration signal: Use acceleration sensors to collect vibration data of key components of the wind turbine, such as gearboxes, bearings, main shafts, and generators, and set appropriate sampling frequencies;

[0026] Temperature data: Monitor the temperature of the generator and gearbox through temperature sensors such as thermocouples or thermal resistor sensors to detect overheating and record temperature data at certain time intervals;

[0027] Electrical parameters: By installing current transformers and voltage transformers on the power transmission lines, the electrical data of the motor, including current, voltage, power factor and other data, is obtained to monitor the status of the electrical system. Data is collected at an appropriate sampling frequency. The data is matched with the frequency and other parameters of the power grid to accurately reflect the operating status of the motor.

[0028] Structural health monitoring data: strain gauges and other equipment used to detect the strain and stress state of towers and blades;

[0029] Environmental data: such as wind speed, wind direction, temperature and humidity, which affect the operating environment of the fan.

[0030] Preferably, the specific implementation method of data synchronization and alignment in step 1 is:

[0031] When the data sampling rates of all sensors are the same or simply aligned, direct time alignment is used to convert the timestamps of all sensors into a common time base and directly align the timestamps of all data points to this common time base.

[0032] When different sensors have different sampling frequencies or the time stamps differ greatly, an interpolation algorithm is used for alignment. An appropriate interpolation method is selected according to the characteristics of the data. Any of the common interpolation methods such as linear interpolation, polynomial interpolation, and spline interpolation can be selected.

[0033] When data needs to be aligned at a uniform time interval, data resampling is adopted to define a uniform time interval for up and down sampling;

[0034] When multiple data sources need to be aligned within the same time window, time window alignment is used to define the time window to aggregate and align the data within the time window and process missing data at the time window boundary.

[0035] Preferably, the specific steps of outlier detection in step 2 are:

[0036] S1. When the deviation of a data point from the mean value exceeds three standard deviations, it is considered an outlier and these outliers are replaced with reasonable values, such as the average value of adjacent data points;

[0037] S2. Perform regression analysis and time series analysis, predict the expected value of the data through the regression model, and compare the actual value with the predicted value, mark the data points with large prediction errors as outliers, apply models such as ARIMA to time series data, and detect outliers in the model prediction errors;

[0038] S3. The isolation forest method is used to "isolate" data points by randomly selecting features and split points. Abnormal points usually require fewer splits to be isolated. The support vector machine (SVM) is used to identify the boundaries of normal data points using the one-class classification (One-Class SVM) method in the support vector machine. Data points that deviate from the boundaries are considered outliers.

[0039] Preferably, the specific steps of feature level fusion in step six are:

[0040] S1, obtain feature vectors X1, X2, X3...X from multiple different modalities and data sources N , where if the feature is an image feature, CNN is used to extract the feature vector;

[0041] S2. Set the dimensions of multiple different modalities and data sources as D1, D2, D3, ... N ;

[0042] S3, concatenating and integrating the feature vectors in S1, and integrating the feature vector dimensions in S2;

[0043] S4, generate large eigenvector: X = [X1, X 2， X3……X N ], whose dimension is D1+D 2+ D3……+D N .

[0044] Preferably, the specific steps of model-level fusion in step six are:

[0045] S1. Train models for each modality or data source, such as convolutional neural networks (CNNs) for image data and long short-term memory networks (LSTMs) for time series data.

[0046] S2. Fuse the outputs of multiple models through ensemble learning methods, such as random forest and XGBoost, to improve the robustness and accuracy of predictions.

[0047] Preferably, the decision-level fusion in step six adopts either voting method or weighted average method as the fusion method. The voting method combines the prediction results of each model to make a voting decision and selects the most common prediction category as the final result; the weighted average method assigns weights to each model according to the performance and reliability of the model, and calculates the weighted average to generate the final decision.

[0048] Preferably, the model for deep learning model training in step seven is specifically selected from:

[0049] When processing the fused high-dimensional feature vector, a deep neural network (DNN) is used to capture complex nonlinear relationships;

[0050] When used to extract spatial features from images and time-frequency data, convolutional neural networks (CNNs) are used to process image data and time-frequency representations;

[0051] When used for time series analysis of wind turbine operation data, recurrent neural networks (RNN) and their variants (LSTM / GRU) are used to process time series data and are able to remember long-term dependencies.

[0052] Preferably, the specific selection for optimization in step seven is:

[0053] When adjusting model parameters, use any optimization algorithm such as Adam, RMSprop, etc. to perform optimization to improve training efficiency and accuracy;

[0054] When used to evaluate the generalization ability of the model, cross-validation and hyperparameter tuning are used to select the best model parameters to prevent overfitting. Hyperparameters are adjusted through grid search or Bayesian optimization to select the best model configuration.

[0055] In order to improve the stability of the model, L1 / L2 regularization, Dropout and other techniques are used to prevent overfitting.

[0056] The present invention provides a deep learning fault diagnosis method based on multi-modal and multi-source data fusion. It has the following beneficial effects:

[0057] 1. The present invention combines multi-modality with multi-source heterogeneous data, fully utilizes various information sources in the wind turbine system, provides more accurate and comprehensive diagnostic services, improves the accuracy and reliability of fault diagnosis, and enhances the robustness of the system, thereby ensuring the stable operation and efficient maintenance of the wind turbine.

[0058] 2. The present invention establishes a variety of fault diagnosis models based on multi-modal and multi-source heterogeneous data, and adopts deep learning model training and optimization during the establishment process to improve the fault diagnosis model, so that the fault diagnosis model can be continuously upgraded and iterated. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Example:

[0062] Please see attached Figure 1 , an embodiment of the present invention provides a multi-modal and multi-source data fusion deep learning fault diagnosis, comprising the following steps:

[0063] Step 1: Collect multi-modal data and multi-source data. According to the characteristics of different sensors with different sampling rates, establish a time frame for data synchronization and alignment, and align all collected data at the same time so that the aligned data should have consistent time resolution;

[0064] The multimodal data collected include:

[0065] Visual data, surveillance camera images or videos: used to monitor the appearance, structural integrity, changes in the surrounding environment, etc. of wind turbines. Image or video data is used to detect external physical damage or erosion;

[0066] Acoustic data: Captures the noise generated by wind turbines during operation to analyze the health of mechanical components and detect abnormal sounds such as noise or vibration to help diagnose potential mechanical problems;

[0067] Text data: Operation and maintenance logs: record the operation status, fault events, maintenance activities, etc. of wind turbines. Text data includes reports, problem descriptions, and maintenance suggestions in natural language format.

[0068] Time series data: Sensor data: including wind speed, wind direction, temperature, humidity, power generation, rotation speed, vibration, etc. The data is recorded by various sensors in the form of continuous time series to provide detailed information about the operating status of the generator and external environmental conditions;

[0069] Graphic data: Power system topology diagram: describes the topological structure and power flow of wind turbines connected to the power grid to assist in understanding the power transmission and distribution, and the changes in power load under different conditions.

[0070] The collected multi-source data include:

[0071] Vibration signal: Use acceleration sensors to collect vibration data of key components of the wind turbine, such as gearboxes, bearings, main shafts, and generators, and set appropriate sampling frequencies;

[0072] Temperature data: Monitor the temperature of the generator and gearbox through temperature sensors such as thermocouples or thermal resistor sensors to detect overheating and record temperature data at certain time intervals;

[0073] Electrical parameters: By installing current transformers and voltage transformers on the power transmission lines, the electrical data of the motor, including current, voltage, power factor and other data, is obtained to monitor the status of the electrical system. Data is collected at an appropriate sampling frequency. The data is matched with the frequency and other parameters of the power grid to accurately reflect the operating status of the motor.

[0074] Structural health monitoring data: strain gauges and other equipment used to detect the strain and stress state of towers and blades;

[0075] Environmental data: such as wind speed, wind direction, temperature and humidity, which affect the operating environment of the fan.

[0076] The specific implementation methods of data synchronization and alignment are as follows:

[0077] When the data sampling rates of all sensors are the same or simply aligned, direct time alignment is used to convert the timestamps of all sensors into a common time base and directly align the timestamps of all data points to this common time base.

[0078] When different sensors have different sampling frequencies or the time stamps differ greatly, an interpolation algorithm is used for alignment. An appropriate interpolation method is selected according to the characteristics of the data. Any of the common interpolation methods such as linear interpolation, polynomial interpolation, and spline interpolation can be selected.

[0079] When data needs to be aligned at a uniform time interval, data resampling is adopted to define a uniform time interval for up and down sampling;

[0080] When multiple data sources need to be aligned within the same time window, time window alignment is used to define the time window to aggregate and align the data within the time window and process missing data at the time window boundary.

[0081] Step 2: Perform data cleaning and detect outliers on the data. If abnormal data is detected, delete the abnormal values, replace the abnormal values, or perform data interpolation to process the abnormal data to prevent the abnormal data from affecting the analysis results.

[0082] Among them, the specific steps of outlier detection are:

[0083] S1. When the deviation of a data point from the mean value exceeds three standard deviations, it is considered an outlier and these outliers are replaced with reasonable values, such as the average value of adjacent data points;

[0084] S2. Perform regression analysis and time series analysis, predict the expected value of the data through the regression model, and compare the actual value with the predicted value, mark the data points with large prediction errors as outliers, apply models such as ARIMA to time series data, and detect outliers in the model prediction errors;

[0085] S3. The isolation forest method is used to "isolate" data points by randomly selecting features and split points. Abnormal points usually require fewer splits to be isolated. The support vector machine (SVM) is used to identify the boundaries of normal data points using the one-class classification (One-Class SVM) method in the support vector machine. Data points that deviate from the boundaries are considered outliers.

[0086] Step 3: Use any of the noise elimination methods such as filter, wavelet denoising, sliding average filtering and frequency domain filtering to denoise the data and improve the signal quality;

[0087] Step 4: Convert the data into units and formats, unify the units of the same type of data, and convert the data into appropriate formats, such as extracting text data into structured data and converting image data into feature vectors;

[0088] Step 5: Perform data fusion and extract features based on time domain features, frequency domain features, time-frequency features, and text features. After the extraction is completed, use methods such as principal component analysis (PCA), linear discriminant analysis (LDA), and t-SNE to reduce feature dimensions and computational complexity while maintaining useful information.

[0089] Step 6: Perform multi-modal and multi-source data fusion, perform feature-level fusion, model-level fusion and decision-level fusion in sequence, and generate the original model;

[0090] Among them, the specific steps of feature level fusion are:

[0091] S1, obtain feature vectors X1, X2, X3...X from multiple different modalities and data sources N , where if the feature is an image feature, CNN is used to extract the feature vector;

[0092] S2. Set the dimensions of multiple different modalities and data sources as D1, D2, D3, ... N ;

[0093] S3, concatenate and integrate the feature vectors in S1, and integrate the feature vector dimensions in S2;

[0094] S4, generate large eigenvector: X = [X1, X 2， X3……X N ], whose dimension is D1+D 2+ D3……+D N .

[0095] Among them, the specific steps of model-level fusion are:

[0096] S1. Train models for each modality or data source, such as convolutional neural networks (CNNs) for image data and long short-term memory networks (LSTMs) for time series data.

[0097] S2. Fuse the outputs of multiple models through ensemble learning methods, such as random forest and XGBoost, to improve the robustness and accuracy of predictions.

[0098] Among them, decision-level fusion adopts either voting method or weighted average method as the fusion method. The voting method combines the prediction results of each model to make a voting decision and selects the most common prediction category as the final result; the weighted average method assigns weights to each model according to the performance and reliability of the model, and calculates the weighted average to generate the final decision.

[0099] Step 7: Use the deep learning model to train and optimize the original model and improve it into a fault diagnosis model;

[0100] Among them, the specific model selection for deep learning model training is:

[0101] When processing the fused high-dimensional feature vector, a deep neural network (DNN) is used to capture complex nonlinear relationships;

[0102] When used to extract spatial features from images and time-frequency data, convolutional neural networks (CNNs) are used to process image data and time-frequency representations;

[0103] When used for time series analysis of wind turbine operation data, recurrent neural networks (RNN) and their variants (LSTM / GRU) are used to process time series data and are able to remember long-term dependencies.

[0104] Among them, the specific options for optimization are:

[0105] When adjusting model parameters, use any optimization algorithm such as Adam, RMSprop, etc. to perform optimization to improve training efficiency and accuracy;

[0106] When used to evaluate the generalization ability of the model, cross-validation and hyperparameter tuning are used to select the best model parameters to prevent overfitting. Hyperparameters are adjusted through grid search or Bayesian optimization to select the best model configuration.

[0107] In order to improve the stability of the model, L1 / L2 regularization, Dropout and other techniques are used to prevent overfitting.

[0108] Step 8: Deploy the trained fault diagnosis model to the actual wind farm monitoring system, analyze the real-time collected data, and monitor potential faults in a timely manner;

[0109] Among them, the fault diagnosis model includes:

[0110] Wind turbine blade damage diagnosis model: applied to lightning damage / leading edge corrosion / blade aerodynamic imbalance / bolt breakage / blade body cracks / blade overload and life prediction fields;

[0111] Gearbox and main bearing damage diagnosis model: Applicable to the fields of main bearing wear / gearbox bearing wear / gear tooth surface peeling / gear tooth breakage / generator bearing wear / generator support loosening / gearbox oil temperature abnormality;

[0112] Generator damage diagnosis model: Applied to bearing vibration fault / winding temperature rise fault / bearing temperature fault / cooler fault / generator stator-rotor air gap fault / insulation thermal life assessment / grease life assessment fields;

[0113] Electrical and control system damage diagnosis model: Applicable to the fields of yaw misalignment with wind / abnormal power curve / abnormal pitch execution / abnormal yaw drive / abnormal IGBT temperature / abnormal control cabinet temperature / abnormal pitch motor temperature / abnormal converter temperature;

[0114] Tower and foundation damage diagnosis model: Applicable to the fields of tower top displacement exceeding limit / tower tilting and overturning / abnormal tower vibration / dangerous speed range monitoring / tower welding / tower bolt fracture / foundation settlement / foundation loosening.

[0115] Step 9: Based on trend analysis and anomaly detection, predict impending failures and provide early warnings and maintenance recommendations;

[0116] Step 10: Automatically generate a diagnostic report for the device and automatically match the corresponding solution in the expert database according to the fault type and device model.

[0117] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A deep learning fault diagnosis based on multi-modal and multi-source data fusion, characterized in that: The following steps are involved: Step 1: Collect multi-modal data and multi-source data. According to the characteristics of different sensors with different sampling rates, establish a time frame for data synchronization and alignment, and align all collected data at the same time so that the aligned data should have consistent time resolution; Step 2: Perform data cleaning and detect outliers on the data. If abnormal data is detected, delete the abnormal values, replace the abnormal values, or perform data interpolation to process the abnormal data to prevent the abnormal data from affecting the analysis results. Step 3: Use any of the noise elimination methods such as filter, wavelet denoising, sliding average filtering and frequency domain filtering to denoise the data and improve the signal quality; Step 4: Convert the data into units and formats, unify the units of the same type of data, and convert the data into appropriate formats, such as extracting text data into structured data and converting image data into feature vectors; Step 5: Perform data fusion and extract features based on time domain features, frequency domain features, time-frequency features, and text features. After the extraction is completed, use methods such as principal component analysis (PCA), linear discriminant analysis (LDA), and t-SNE to reduce feature dimensions and computational complexity while maintaining useful information. Step 6: Perform multi-modal and multi-source data fusion, perform feature-level fusion, model-level fusion and decision-level fusion in sequence, and generate the original model; Step 7: Use the deep learning model to train and optimize the original model and improve it into a fault diagnosis model; Step 8: Deploy the trained fault diagnosis model to the actual wind farm monitoring system, analyze the real-time collected data, and monitor potential faults in a timely manner; Step 9: Based on trend analysis and anomaly detection, predict impending failures and provide early warnings and maintenance recommendations; Step 10: Automatically generate a diagnostic report for the device and automatically match the corresponding solution in the expert database according to the fault type and device model.

2. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: The multimodal data collected in step 1 includes: Visual data, surveillance camera images or videos: used to monitor the appearance, structural integrity, changes in the surrounding environment, etc. of wind turbines. Image or video data is used to detect external physical damage or erosion; Acoustic data: Captures the noise generated by wind turbines during operation to analyze the health of mechanical components and detect abnormal sounds such as noise or vibration to help diagnose potential mechanical problems; Text data: Operation and maintenance logs: record the operation status, fault events, maintenance activities, etc. of wind turbines. Text data includes reports, problem descriptions, and maintenance suggestions in natural language format. Time series data: Sensor data: including wind speed, wind direction, temperature, humidity, power generation, rotation speed, vibration, etc. The data is recorded by various sensors in the form of continuous time series to provide detailed information about the operating status of the generator and external environmental conditions; Graphic data: Power system topology diagram: describes the topological structure and power flow of wind turbines connected to the power grid to assist in understanding the power transmission and distribution, and the changes in power load under different conditions.

3. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: The multi-source data collected in step 1 includes: Vibration signal: Use acceleration sensors to collect vibration data of key components of the wind turbine, such as gearboxes, bearings, main shafts, and generators, and set appropriate sampling frequencies; Temperature data: Monitor the temperature of the generator and gearbox through temperature sensors such as thermocouples or thermal resistor sensors to detect overheating and record temperature data at certain time intervals; Electrical parameters: By installing current transformers and voltage transformers on the power transmission lines, the electrical data of the motor, including current, voltage, power factor and other data, is obtained to monitor the status of the electrical system. Data is collected at an appropriate sampling frequency. The data is matched with the frequency and other parameters of the power grid to accurately reflect the operating status of the motor. Structural health monitoring data: strain gauges and other equipment used to detect the strain and stress state of towers and blades; Environmental data: such as wind speed, wind direction, temperature and humidity, which affect the operating environment of the fan.

4. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: The specific implementation method of data synchronization and alignment in step 1 is as follows: When the data sampling rates of all sensors are the same or simply aligned, direct time alignment is used to convert the timestamps of all sensors into a common time base and directly align the timestamps of all data points to this common time base. When different sensors have different sampling frequencies or the time stamps differ greatly, an interpolation algorithm is used for alignment. An appropriate interpolation method is selected according to the characteristics of the data. Any of the common interpolation methods such as linear interpolation, polynomial interpolation, and spline interpolation can be selected. When data needs to be aligned at a uniform time interval, data resampling is adopted to define a uniform time interval for up and down sampling; When multiple data sources need to be aligned within the same time window, time window alignment is used to define the time window to aggregate and align the data within the time window and process missing data at the time window boundary.

5. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: The specific steps of outlier detection in step 2 are: S1. When the deviation of a data point from the mean value exceeds three standard deviations, it is considered an outlier and these outliers are replaced with reasonable values, such as the average value of adjacent data points; S2. Perform regression analysis and time series analysis, predict the expected value of the data through the regression model, and compare the actual value with the predicted value, mark the data points with large prediction errors as outliers, apply models such as ARIMA to time series data, and detect outliers in the model prediction errors; S3. Use the isolation forest method to "isolate" data points by randomly selecting features and split points. Abnormal points usually require fewer splits to be isolated. Use the support vector machine (SVM) and use the one-class classification (One-ClassSVM) method in the support vector machine to identify the boundaries of normal data points. Data points that deviate from the boundaries are considered outliers.

6. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: The specific steps of feature level fusion in step 6 are: S1, obtain feature vectors X1, X2, X3...X from multiple different modalities and data sources N , where if the feature is an image feature, CNN is used to extract the feature vector; S2. Set the dimensions of multiple different modalities and data sources as D1, D2, D3, ... N ; S3, concatenating and integrating the feature vectors in S1, and integrating the feature vector dimensions in S2; S4, generate large eigenvector: X = [X1, X 2， X3……X N ], whose dimension is D1+D 2+ D3……+D N .

7. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: The specific steps of model-level fusion in step 6 are: S1. Train models for each modality or data source, such as convolutional neural networks (CNNs) for image data and long short-term memory networks (LSTMs) for time series data. S2. Fuse the outputs of multiple models through ensemble learning methods, such as random forest and XGBoost, to improve the robustness and accuracy of predictions.

8. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: In the step 6, the decision-level fusion adopts either a voting method or a weighted average method as a fusion method. The voting method combines the prediction results of each model to make a voting decision and selects the most common prediction category as the final result. The weighted average method assigns weights to each model based on its performance and reliability, and calculates the weighted average to generate the final decision.

9. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: The specific model selection for deep learning model training in step 7 is: When processing the fused high-dimensional feature vector, a deep neural network (DNN) is used to capture complex nonlinear relationships; When used to extract spatial features from images and time-frequency data, convolutional neural networks (CNNs) are used to process image data and time-frequency representations; When used for time series analysis of wind turbine operation data, recurrent neural networks (RNN) and their variants (LSTM / GRU) are used to process time series data and are able to remember long-term dependencies.

10. The multi-modal and multi-source data fusion deep learning fault diagnosis according to claim 1, characterized in that: The specific selections optimized in step seven are: When adjusting model parameters, use any optimization algorithm such as Adam, RMSprop, etc. to perform optimization to improve training efficiency and accuracy; When used to evaluate the generalization ability of the model, cross-validation and hyperparameter tuning are used to select the best model parameters to prevent overfitting. Hyperparameters are adjusted through grid search or Bayesian optimization to select the best model configuration. In order to improve the stability of the model, L1 / L2 regularization, Dropout and other techniques are used to prevent overfitting.

Images