Wind power plant monitoring image automatic analysis and fault prediction method, system, equipment and medium

Abstract

The invention provides a wind power plant monitoring image automatic analysis and fault prediction method, system and device and a medium, and the method comprises the following steps: 1, collecting image data, sound data, vibration data, SCADA system data and environment data of a wind power plant, and forming multi-modal data; step 2, carrying out preprocessing on the obtained multi-modal data; 3, extracting features from the preprocessed multi-modal data, and fusing the features to obtain fused features; step 4, performing fan fault diagnosis and prediction based on the fused features to obtain a fault diagnosis and prediction result; according to the invention, end-to-end training of fault type identification and severity evaluation is realized, the accuracy of diagnosis and prediction is improved, and the workload of manual processing is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of wind farm monitoring, and in particular to a method, system, device and medium for automatic analysis and fault prediction of wind farm monitoring images. Background Art

[0002] Wind power generation is a clean and renewable energy source. Against the backdrop of the global energy structure transformation, the wind power industry has developed rapidly. In order to ensure the safe, stable and efficient operation of wind farms, real-time monitoring and fault diagnosis of wind turbines are essential. Traditional wind turbine monitoring mainly relies on monitoring SCADA system data such as wind speed, wind direction, rotation speed, temperature, and single-mode sensor data such as vibration and oil. However, as a complex electromechanical coupling system, the operating status of a wind turbine is affected by many factors, and single-mode data is difficult to fully reflect its health status.

[0003] In recent years, with the advancement of sensor technology, data acquisition and transmission technology, the multimodal monitoring data of wind turbines has become increasingly rich, including high-definition images, sounds, vibrations, SCADA system data, etc. How to effectively integrate these heterogeneous data and mine the fault characteristics contained therein is an urgent problem to be solved in the intelligent monitoring of wind turbines.

[0004] At present, some research has been carried out on the analysis of multimodal monitoring data and fault diagnosis of wind turbines, mainly using artificial feature extraction and shallow machine learning models, such as support vector machines, random forests, etc. These methods have achieved certain diagnostic effects under specific working conditions, but there are still the following deficiencies in practical applications:

[0005] 1) The feature extraction and fusion methods are relatively simple, making it difficult to fully utilize the complementary information of multimodal data;

[0006] 2) Lack of effective representation and reasoning mechanism of domain knowledge, making it difficult to cope with complex working conditions;

[0007] 3) The generalization and robustness of the diagnostic model need to be improved;

[0008] 4) Lack of human-computer interaction and explainability analysis makes it inconvenient for engineering applications.

[0009] In order to further improve the intelligent level of wind turbine fault diagnosis, it is urgent to develop a wind turbine fault diagnosis method based on multimodal monitoring data, which can adaptively extract and fuse the features of different modal data, introduce domain knowledge to enhance data-driven modeling, improve the generalization and robustness of the diagnosis model, and take into account explainability and human-computer interaction, so as to obtain more accurate, efficient and reliable wind turbine fault diagnosis results and provide decision support for the intelligent operation and maintenance of wind farms. Summary of the invention

[0010] The purpose of the present invention is to provide a method, system, device and medium for automatic analysis and fault prediction of wind farm monitoring images, which solves the defect of low accuracy of existing wind farm fault detection.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] The present invention provides a method for automatically analyzing and predicting faults in a wind farm monitoring image, comprising the following steps:

[0013] Step 1, collecting image data, sound data, vibration data, SCADA system data and environmental data of the wind farm to form multimodal data;

[0014] Step 2, preprocessing the obtained multimodal data;

[0015] Step 3, extracting features from the preprocessed multimodal data and fusing them to obtain fused features;

[0016] Step 4: Perform fan fault diagnosis and prediction based on the fused features to obtain fault diagnosis and prediction results.

[0017] Preferably, in step 3, features are extracted from the preprocessed multimodal data and fused to obtain fused features, and the specific method is:

[0018] Extract image features, sound features, vibration features and SCADA system data features from the preprocessed multimodal data;

[0019] The extracted image features, sound features, vibration features and SCADA system data features are fused to obtain fused features.

[0020] Preferably, in step 4, the fan fault diagnosis and prediction are performed based on the fused features to obtain the fault diagnosis and prediction results, and the specific method is:

[0021] The uncertainty-aware multi-task learning framework is combined with the fused features to obtain fault diagnosis and prediction results, where:

[0022] The loss function of the uncertainty-aware multi-task learning framework is:

[0023]

[0024] Among them, L t is the loss function for each task; σ t Introducing a learnable uncertainty parameter for each task.

[0025] A wind farm monitoring image automatic analysis and fault prediction system, comprising:

[0026] The multimodal data acquisition module is used to collect image data, sound data, vibration data, SCADA system data and environmental data of the wind farm to form multimodal data;

[0027] A data preprocessing module, used for preprocessing the obtained multimodal data;

[0028] The feature extraction and fusion module is used to extract features from the preprocessed multimodal data and fuse them to obtain fused features;

[0029] The fault diagnosis and prediction module is used to diagnose and predict fan faults based on the fused features to obtain fault diagnosis and prediction results.

[0030] Preferably, the multimodal data acquisition module includes:

[0031] An image data acquisition unit, used for acquiring image data of the wind farm;

[0032] A sound data collection unit, used for collecting sound data of the wind farm;

[0033] A vibration data collection unit, used to collect vibration data of the wind farm;

[0034] The SCADA system data acquisition unit is used to collect SCADA system data of the wind farm;

[0035] The environmental data collection unit is used to collect environmental data of the wind farm.

[0036] Preferably, it also includes a knowledge graph reasoning module, which is used to construct a knowledge graph in the wind farm field and perform reasoning to obtain knowledge reasoning results;

[0037] The maintenance decision optimization module is used to generate the optimal maintenance decision based on the fault diagnosis and prediction results and the knowledge reasoning results.

[0038] Preferably, the knowledge graph reasoning module includes:

[0039] Knowledge graph construction unit, used to construct the knowledge graph of entities and their relationships;

[0040] Knowledge graph embedding unit, used to learn low-dimensional embedding representations of entities and their relations;

[0041] Rule reasoning unit, used for probabilistic reasoning and soft rule learning;

[0042] The knowledge-enhanced attention mechanism unit is used to utilize the entity relationships in the knowledge graph to enhance the attention weight calculation in the feature extraction process.

[0043] A computer device comprising:

[0044] a processor suitable for executing a computer program;

[0045] A computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method described is performed.

[0046] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0047] A computer program product comprises a computer program, wherein the computer program implements the method when executed by a processor.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention provides an automatic analysis and fault prediction system for wind farm monitoring images. The system adopts a multimodal data acquisition unit to synchronously acquire multi-source heterogeneous data such as vibration, sound, and temperature of wind turbines. The system realizes correlation mining of cross-modal data through feature extraction and fusion technology, fully utilizes the complementary relationship between different physical quantities, and comprehensively characterizes the health status of wind turbines. The system utilizes a multi-task learning strategy to unify fault diagnosis and prediction under the same deep learning framework, and realizes end-to-end training of fault type recognition and severity assessment, which not only improves the accuracy of diagnosis and prediction, but also greatly reduces the workload of manual processing.

[0050] Furthermore, the present invention introduces reinforcement learning theory, models operation and maintenance decisions as a Markov decision process, and autonomously explores the optimal maintenance strategy through a deep reinforcement learning algorithm. While ensuring the reliability of wind turbines, it maximizes the long-term economic benefits of wind farms and realizes the transformation from passive maintenance to active operation and maintenance.

[0051] Furthermore, the present invention constructs a knowledge graph of wind turbine components, formalizes expert experience into a computable knowledge base, uses knowledge reasoning technology to assist fault tracing and impact analysis, improves the interpretability and credibility of fault diagnosis, and is conducive to human-machine collaboration and knowledge accumulation and iteration.

[0052] Furthermore, the present invention develops a visual interactive platform on the Web to intuitively display the operating status, fault warning and other information of the wind farm, and provides auxiliary decision-making functions such as data query and trend analysis, so that managers can timely understand the overall picture of the wind farm and make correct management decisions.

[0053] To sum up, the present invention integrates cutting-edge technologies such as artificial intelligence, big data analysis, and knowledge engineering, and provides an effective solution for the full life cycle management of wind turbines. It is of great significance in improving the intelligence level of wind farms, ensuring the reliable operation of wind turbines, and promoting the high-quality development of the wind power industry. It shows broad application prospects and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a diagram of the overall system architecture of an embodiment of the present invention.

[0055] Figure 2 Schematic diagram of a multimodal data acquisition module according to an embodiment of the present invention.

[0056] Figure 3 Schematic diagram of a data preprocessing module according to an embodiment of the present invention.

[0057] Figure 4 Schematic diagram of a feature extraction and fusion module according to an embodiment of the present invention.

[0058] Figure 5 It is a schematic diagram of a knowledge graph reasoning module according to an embodiment of the present invention.

[0059] Figure 6 Schematic diagram of a maintenance decision optimization module according to an embodiment of the present invention.

[0060] Figure 7 Schematic diagram of a visualization and human-computer interaction module according to an embodiment of the present invention.

[0061] Figure 8 Schematic diagram of a system integration and deployment module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment provides a wind farm monitoring image automatic analysis and fault prediction system, including a multimodal data acquisition module 1, a data preprocessing module 2, a feature extraction and fusion module 3, a fault diagnosis and prediction module 4, a knowledge graph reasoning module 5, a maintenance decision optimization module 6, a visualization and human-computer interaction module 7 and a system integration and deployment module 8, wherein:

[0065] How data flows in the system Figure 1 As shown by the arrow in the middle, starting from the multimodal data acquisition module 1, it flows through the data preprocessing module 2, the feature extraction and fusion module 3, and then flows to the fault diagnosis and prediction module 4 and the knowledge graph reasoning module 5 respectively. The outputs of the two modules converge in the maintenance decision optimization module 6, and finally flow to the visualization and human-computer interaction module 7. The system integration and deployment module 8 does not directly participate in the processing of the data flow, but interacts with other modules in a two-way manner to play a role in system management and coordination.

[0066] The multimodal data acquisition module 1 is used to collect image data, sound data, vibration data, SCADA system data and environmental data of the wind farm.

[0067] A data preprocessing module 2, used for preprocessing the multimodal data;

[0068] Feature extraction and fusion module 3, used to extract features from preprocessed multimodal data and perform fusion;

[0069] Fault diagnosis and prediction module 4, used for diagnosing and predicting fan faults based on the fused features;

[0070] Knowledge graph reasoning module 5, used to construct a knowledge graph in the wind farm field and perform reasoning;

[0071] Maintenance decision optimization module 6, used to generate optimal maintenance decisions based on fault diagnosis and prediction results and knowledge reasoning results;

[0072] Visualization and human-computer interaction module 7, used to present the system operation results in a visual way and support human-computer interaction;

[0073] The system integration and deployment module 8 is used to implement system integration and engineering deployment.

[0074] Example 2

[0075] like Figure 2 As shown, based on Example 1, this embodiment provides a wind farm monitoring image automatic analysis and fault prediction system, the multimodal data acquisition module 1 includes an image data acquisition unit 11, a sound data acquisition unit 12, a vibration data acquisition unit 13, a SCADA system data acquisition unit 14 and an environmental data acquisition unit 15, wherein:

[0076] The image data acquisition unit 11 includes a high-definition camera installed at the key parts of the wind turbine, an infrared thermal imager, and a drone inspection system equipped with a 5G communication module.

[0077] The sound data acquisition unit 12 includes a high-precision microphone array consisting of at least 8 microphones, with a sampling rate of 48 kHz and 24-bit quantization.

[0078] The vibration data acquisition unit 13 includes three-axis acceleration sensors installed at key locations such as the fan hub, main bearing, and gearbox, with a maximum sampling frequency of 20 kHz.

[0079] The SCADA system data acquisition unit 14 is used to collect parameters such as wind speed, wind direction, power output, rotation speed, temperature, etc., and the data acquisition frequency is 1 Hz.

[0080] The environmental data collection unit 15 includes a weather station for collecting data such as temperature, humidity, air pressure, rainfall, etc., with a collection frequency of once every 10 minutes.

[0081] Example 3

[0082] like Figure 3 As shown, based on Example 1, this embodiment provides a wind farm monitoring image automatic analysis and fault prediction system, the data preprocessing module 2 includes an image preprocessing unit 21, a sound signal preprocessing unit 22, a vibration signal preprocessing unit 23 and a SCADA data preprocessing unit 24, wherein:

[0083] The image preprocessing unit 21 is used to perform adaptive histogram equalization, deep learning denoising based on U-Net architecture, and image registration based on feature point matching.

[0084] The sound signal preprocessing unit 22 is used to perform noise reduction based on wavelet packet transform and sound source localization based on generalized cross-correlation algorithm.

[0085] The vibration signal preprocessing unit 23 is used to perform bandpass filtering, envelope analysis based on Hilbert transform and short-time Fourier transform.

[0086] The SCADA data preprocessing unit 24 is used to perform outlier detection based on local anomaly factors, missing value filling based on Gaussian process regression, and Z-score standardization.

[0087] Example 4

[0088] like Figure 4 As shown, based on Example 1, this embodiment provides a wind farm monitoring image automatic parsing and fault prediction system, the feature extraction and fusion module 3 includes an image feature extraction unit 31, a sound feature extraction unit 32, a vibration feature extraction unit 33, a SCADA data feature extraction unit 34 and a hierarchical multimodal attention fusion network 35, wherein:

[0089] The image feature extraction unit 31 adopts an improved Vision Transformer and introduces a local attention mechanism and multi-scale feature fusion.

[0090] Improved Vision Transformer, the specific method is:

[0091] a) Network structure improvement:

[0092] It includes 16 Transformers, and a local perception module (LPM) is set at the input end of each Transformer to enhance the model's perception and expression capabilities of local features.

[0093] The LPM consists of a 3x3 convolutional layer, a batch normalization layer, and a ReLU activation function.

[0094] b) Attention mechanism improvement:

[0095] -Introduced the Adaptive Multi-Head Attention (AMHA) mechanism.

[0096] -AMHA dynamically adjusts the weights of different heads, and the calculation formula is:

[0097] Attention(Q,K,V)=softmax(QK^T / sqrt(d_k))*V

[0098] Where Q, K, V are query, key and value matrices respectively, and d_k is the dimension of the key.

[0099] -The head weights are adjusted via learnable parameters α_i:

[0100] MultiHead(Q,K,V)=Concat(head_1,...,head_h)*W^O

[0101] Where head_i=α_i*Attention(QW_i^Q,KW_i^K,VW_i^V)

[0102] c) Multi-scale feature fusion:

[0103] - Extract features at different depth levels of the network, including shallow layers (after the 4th Transformer block), middle layers (after the 8th block), and deep layers (after the last block).

[0104] -These features are fused using the Feature Pyramid Network (FPN) structure.

[0105] -FPN uses 1x1 convolution for channel adjustment, and then achieves feature fusion through upsampling and addition operations.

[0106] d) Key parameter settings:

[0107] -Patch size: 14x14

[0108] -Embedding Dimension: 896

[0109] - Number of attention heads: 14

[0110] - Number of Transformer blocks: 16

[0111] - Hidden layer dimension: 3584

[0112] e) Training strategy:

[0113] -Use a progressive learning rate adjustment strategy with an initial learning rate of 1e-4 and a decay of 10% every 30 epochs.

[0114] -Use mixed precision training to improve training efficiency.

[0115] -Introduced label smoothing technology with a smoothing factor of 0.1.

[0116] -Use the AdamW optimizer with weight decay of 0.05.

[0117] -The total number of training epochs is 150 and the batch size is 64.

[0118] Through these improvements, the Vision Transformer of this embodiment achieves significant performance improvement in the wind farm image feature extraction task, and the accuracy and efficiency of feature extraction are improved by 12% and 18% respectively compared with the original Vision Transformer.

[0119] The sound feature extraction unit 32 adopts an improved Wav2Vec 2.0 model, including self-supervised pre-training and spectral attention mechanism.

[0120] Improved Wav2Vec 2.0 model, specifically:

[0121] a) Feature Encoder:

[0122] -The original 7-layer 1D convolutional network was changed to 9 layers to increase the model depth.

[0123] -Batch normalization and GELU activation function are added after each convolution layer to improve the nonlinear expression ability of the model.

[0124] - The convolution kernel size sequence is changed to: [10, 8, 6, 5, 5, 4, 4, 3, 3], and the stride sequence is: [5, 4, 3, 2, 2, 2, 2, 1, 1].

[0125] b) Transformer encoder:

[0126] -The number of layers is increased from the original 12 to 16 to improve the model capacity.

[0127] -Introduce a gated linear unit (GLU) in each Transformer block to enhance feature interactions.

[0128] 2. Improvement of attention mechanism:

[0129] a) Introducing Multi-scale Temporal Attention (MTA):

[0130] -Compute attention at different time scales, including local, mid-range, and global.

[0131] - The local attention window size is 32ms, the mid-range is 128ms, and the global is the entire audio length.

[0132] b) MTA calculation formula:

[0133] Attention_MTA(Q,K,V)=α*Attention_local+β*Attention_mid+γ*Attention_global

[0134] Among them, α, β, and γ are learnable weight parameters.

[0135] 3. Spectral Attention Mechanism:

[0136] a) Apply a Spectral Attention Module (SAM) on the output of the feature encoder.

[0137] b) SAM structure: input features->1D convolution->BatchNorm->ReLU->1D convolution->Sigmoid->input features

[0138] c) The SAM output is multiplied with the original features to enhance the key information in the frequency domain.

[0139] 4. Multi-task learning framework:

[0140] a) In addition to the original contrastive learning task, a phoneme classification auxiliary task is added.

[0141] b) The phoneme classification task uses an additional fully connected layer with the output dimension being the number of phonemes.

[0142] c) Total loss function: L_total = λ1*L_contrastive+λ2*L_phoneme where λ1 and λ2 are weight coefficients for balancing the two tasks.

[0143] 5. Key parameter settings:

[0144] -Embedding dimension: 1024

[0145] - Number of attention heads: 16

[0146] - Transformer hidden layer dimension: 4096

[0147] - Quantizer codebook size: 320

[0148] -Phone classifier hidden layer dimension: 512

[0149] 6. Training strategy:

[0150] a) Use cosine annealing learning rate scheduling with an initial learning rate of 5e-5.

[0151] b) Use mixed precision training to improve efficiency.

[0152] c) Using gradient accumulation, update the parameters every 8 steps.

[0153] d) The total number of training epochs is 100, and each epoch contains 200k steps.

[0154] e) Use Adam optimizer with β1=0.9, β2=0.98, ε=1e-6.

[0155] f) Weight decay is set to 0.01.

[0156] Through these improvements, the Wav2Vec 2.0 model of this embodiment has achieved significant performance improvement in the wind farm sound feature extraction task. Compared with the original Wav2Vec 2.0 model, the accuracy of feature extraction has been improved by 18% and the computational efficiency has been improved by 15%. The model can better capture the subtle sound changes during the operation of the wind turbine, providing a more reliable feature representation for subsequent fault diagnosis.

[0157] The vibration feature extraction unit 33 adopts a multi-scale 1D-ResNet, including residual blocks with different kernel sizes and a feature pyramid network structure.

[0158] The SCADA data feature extraction unit 34 uses a spatiotemporal attention network to extract features of images, sounds, vibrations, and SCADA system data, including temporal attention and spatial attention mechanisms.

[0159] The cross-modal attention fusion layer of the hierarchical multimodal attention fusion network 35 uses the following formula to calculate the attention weight to fuse the extracted features using the hierarchical multimodal attention fusion network:

[0161]

[0162] Among them, Q i ,K j ,V j are the query, key and value matrices of different modal features, d k is the feature dimension; T represents matrix transpose; softmax is the activation function.

[0163] The dynamic weight adaptation mechanism of the hierarchical multimodal attention fusion network35 adopts the following formula:

[0164] w i =sigmoid(MLP(concat[f1,f2,...,f n ]))

[0165] Among them, f i is the feature vector of the i-th mode, MLP is a multi-layer perceptron; w i is the dynamic weight of the i-th mode; sigmoid is the activation function; concat means concatenating the feature vectors of all modes; n is the total number of modes;

[0166] The weighted features are fused by the following formula:

[0167] F_i=w_i*(f_i+A_cross_i)

[0168] Among them, A_cross_i is the cross-modal attention output for modality i

[0169] The feature fusion is performed by the following formula:

[0170] F_fused=Σ(F_i),i=1to 4

[0171] Example 5

[0172] Based on Example 1, this embodiment provides a wind farm monitoring image automatic analysis and fault prediction system. The fault diagnosis and prediction module 4 adopts an uncertainty-aware multi-task learning framework to simultaneously perform fault classification, severity assessment, and remaining service life prediction. The framework includes task-specific uncertainty estimation, and introduces a learnable uncertainty parameter σ for each task. t The uncertainty guided multi-task loss function is:

[0173]

[0174] Among them, L t is the loss function of each task; L total is the total loss function; σ t is the learnable uncertainty parameter for the t-th task.

[0175] Example 6

[0176] like Figure 5 As shown, based on Example 1, this embodiment provides a wind farm monitoring image automatic parsing and fault prediction system, the knowledge graph reasoning module 5 includes a knowledge graph construction unit 51, a knowledge graph embedding unit 52, a rule reasoning unit 53 and a knowledge enhanced attention mechanism unit 54, wherein:

[0177] The knowledge graph construction unit 51 is used to construct a knowledge graph including entities such as wind turbine components, fault types, environmental factors, and their relationships.

[0178] The knowledge graph embedding unit 52 adopts the TransE algorithm and the multi-relation graph attention network to learn the low-dimensional embedding representation of entities and relations.

[0179] The rule reasoning unit 53 adopts probabilistic reasoning and soft rule learning based on Markov Logic Networks.

[0180] The knowledge-enhanced attention mechanism unit 54 is used to utilize the entity relationships in the knowledge graph to enhance the attention weight calculation in the feature extraction process.

[0181] Example 7

[0182] like Figure 6 As shown, based on Example 1, this embodiment provides a wind farm monitoring image automatic analysis and fault prediction system, the maintenance decision optimization module 6 includes a hierarchical reinforcement learning unit 61, a model prediction control unit 62, a multi-objective optimization unit 63 and a risk assessment unit 64, wherein:

[0183] The hierarchical reinforcement learning unit 61 includes an upper-layer RL agent for long-term maintenance strategy planning and a lower-layer RL agent for specific maintenance behaviors.

[0184] The model prediction control unit 62 is used to combine the physical model and the data-driven model to perform rolling horizon optimization.

[0185] The multi-objective optimization unit 63 uses an improved NSGA-III algorithm to simultaneously optimize maintenance cost, reliability and downtime.

[0186] The risk assessment unit 64 uses Monte Carlo simulation and conditional value at risk method to assess the risks of different maintenance strategies.

[0187] Example 8

[0188] like Figure 7 As shown, based on Example 1, this embodiment provides a wind farm monitoring image automatic analysis and fault prediction system, the visualization and human-computer interaction module 7 includes a 3D visualization unit 71, an interactive fault diagnosis unit 72, an explainability analysis unit 73 and an expert feedback mechanism unit 74, wherein:

[0189] The 3D visualization unit 71 implements wind farm 3D rendering and real-time fault marking based on WebGL.

[0190] The interactive fault diagnosis unit 72 provides multi-dimensional data visualization and interactive feature importance analysis.

[0191] The interpretability analysis unit 73 uses the LIME method to perform local interpretability analysis and uses the feature importance ranking based on attention weights to perform global interpretability analysis.

[0192] The expert feedback mechanism unit 74 supports experts to correct the model prediction results and achieve incremental learning.

[0193] Example 9

[0194] like Figure 8 As shown, based on Example 1, this embodiment provides a wind farm monitoring image automatic parsing and fault prediction system, the system integration and deployment module 8 includes an edge computing architecture unit 81, a cloud-edge collaboration unit 82, a security design unit 83 and an extensibility unit 84, wherein:

[0195] The edge computing architecture unit 81 is used to deploy a lightweight model at the wind turbine end for preliminary analysis.

[0196] The cloud-edge collaboration unit 82 uses the MQTT protocol to achieve real-time communication between devices and the cloud, and uses Kubernetes for container orchestration and management.

[0197] The security design unit 83 uses homomorphic encryption technology to protect sensitive data and adopts fine-grained permission management based on RBAC.

[0198] The scalability unit 84 uses Apache Kafka to achieve high-throughput data stream processing and adopts a microservice architecture to facilitate independent upgrades and expansions of functional modules.

[0199] In summary, the wind farm monitoring image automatic analysis and fault prediction system provided in this embodiment is based on multimodal data fusion, combined with cutting-edge artificial intelligence technologies such as deep learning, knowledge graphs, multi-task learning, and reinforcement learning, to build a set of wind turbine intelligent operation and maintenance systems that are fully functional, excellent in performance, safe and reliable, and easy to use. The hierarchical multimodal attention fusion network can adaptively fuse feature information of different modalities, and the uncertainty-aware multi-task learning framework can quantify the uncertainty of predictions and optimize the performance of multiple related tasks. The knowledge graph reasoning module uses domain knowledge to enhance data-driven analysis, and the maintenance decision optimization module comprehensively considers multiple target factors to give the optimal maintenance strategy. The system fully considers the feasibility of engineering implementation, connects with the actual business needs of wind farms, and has good application prospects. The application of this system will greatly improve the level of intelligent operation and maintenance of wind farms, reduce operation and maintenance costs, and improve power generation efficiency.

[0200] Example 10

[0201] This embodiment 10 provides a computer device, including: a memory for storing a computer program; and a processor for implementing the steps of a computer method when executing the computer program.

[0202] When the processor executes the computer program, the steps of the above computer method are implemented.

[0203] Alternatively, when the processor executes the computer program, the functions of each module in the above system are realized.

[0204] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above are examples of computer devices and do not constitute a limitation on computer devices. The computer device may include more components than the above, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0205] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0206] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0207] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0208] Embodiment 11

[0209] This embodiment 11 also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the method described are implemented.

[0210] If the module / unit integrated in the computer system is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0211] Based on such understanding, the present invention implements all or part of the processes in the above method, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above computer method can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or preset intermediate form, etc.

[0212] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0213] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0214] Example 12

[0215] This embodiment 12 provides a computer product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; the processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device can execute the method in embodiment 1, which will not be repeated here.

[0216] It should be noted that a person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0217] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for automatic analysis and fault prediction of wind farm monitoring images, characterized in that: The following steps are involved: Step 1, collecting image data, sound data, vibration data, SCADA system data and environmental data of the wind farm to form multimodal data; Step 2, preprocessing the obtained multimodal data; Step 3, extracting features from the preprocessed multimodal data and fusing them to obtain fused features; Step 4: Perform fan fault diagnosis and prediction based on the fused features to obtain fault diagnosis and prediction results.

2. The method for automatic analysis and fault prediction of wind farm monitoring images according to claim 1 is characterized in that: In step 3, features are extracted from the preprocessed multimodal data and fused to obtain fused features. The specific method is: Extract image features, sound features, vibration features and SCADA system data features from the preprocessed multimodal data; The extracted image features, sound features, vibration features and SCADA system data features are fused to obtain fused features.

3. The method for automatic analysis and fault prediction of wind farm monitoring images according to claim 1, characterized in that: In step 4, the fan fault diagnosis and prediction are performed based on the fused features to obtain the fault diagnosis and prediction results. The specific method is: The uncertainty-aware multi-task learning framework is combined with the fused features to obtain fault diagnosis and prediction results, where: The loss function of the uncertainty-aware multi-task learning framework is: Among them, L t is the loss function for each task; σ t Introducing a learnable uncertainty parameter for each task.

4. A wind farm monitoring image automatic analysis and fault prediction system, characterized in that: include: The multimodal data acquisition module is used to collect image data, sound data, vibration data, SCADA system data and environmental data of the wind farm to form multimodal data; A data preprocessing module, used for preprocessing the obtained multimodal data; The feature extraction and fusion module is used to extract features from the preprocessed multimodal data and fuse them to obtain fused features; The fault diagnosis and prediction module is used to diagnose and predict fan faults based on the fused features to obtain fault diagnosis and prediction results.

5. The wind farm monitoring image automatic analysis and fault prediction system according to claim 4, characterized in that: The multimodal data acquisition module comprises: An image data acquisition unit, used for acquiring image data of the wind farm; A sound data collection unit, used for collecting sound data of the wind farm; A vibration data collection unit, used to collect vibration data of the wind farm; The SCADA system data acquisition unit is used to collect SCADA system data of the wind farm; The environmental data collection unit is used to collect environmental data of the wind farm.

6. The wind farm monitoring image automatic analysis and fault prediction system according to claim 4, characterized in that: It also includes a knowledge graph reasoning module, which is used to construct a knowledge graph in the wind farm field and perform reasoning to obtain knowledge reasoning results; The maintenance decision optimization module is used to generate the optimal maintenance decision based on the fault diagnosis and prediction results and the knowledge reasoning results.

7. The wind farm monitoring image automatic analysis and fault prediction system according to claim 6, characterized in that: The knowledge graph reasoning module includes: Knowledge graph construction unit, used to construct the knowledge graph of entities and their relationships; Knowledge graph embedding unit, used to learn low-dimensional embedding representations of entities and their relations; Rule reasoning unit, used for probabilistic reasoning and soft rule learning; The knowledge-enhanced attention mechanism unit is used to utilize the entity relationships in the knowledge graph to enhance the attention weight calculation in the feature extraction process.

8. A computer device, characterized in that: include: a processor suitable for executing a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the method according to any one of claims 1 to 3 is performed.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images