Offshore wind turbine generator blade fault detection method based on multi-modal data and heterogeneous graph

Abstract

The invention provides an offshore wind turbine generator blade fault detection method based on multi-modal data and a heterogeneous graph. The method comprises the steps of obtaining multi-modal data of n modals of an offshore wind turbine generator blade; carrying out fusion processing on the multi-modal data through joint embedded learning; the high-order feature information of the wind turbine generator blades is extracted, and the relevance and consistency among the modal data are improved through a joint loss function; constructing a heterogeneous graph of the wind turbine generator blades; representing nodes and edges of the heterogeneous graph in a triple form; the triple of the heterogeneous graph is scored by using an improved Complex model; enhancing the relevance between the weak similarity modal data and the regional blades through a small model; and identifying a potential fault of the blade in the region based on the region incidence matrix and carrying out early warning. According to the method, the accuracy of wind turbine generator blade fault detection can be improved, multi-modal data can be efficiently processed by optimizing joint embedded representation and using a Complex model for scoring, and the method has adaptability and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of wind turbine fault detection, and in particular to an offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graphs. Background Art

[0002] Offshore wind turbines are the core equipment of offshore wind farms, and their stable and efficient operation is crucial to the economy and reliability of wind farms. However, the harsh conditions of the offshore environment, including extreme weather, high humidity, salt spray corrosion, and complex marine ecosystems, pose severe challenges to the normal operation of offshore wind turbines. These environmental factors not only affect the performance of offshore wind turbines, but may also cause equipment failures, increase maintenance costs, and even cause safety accidents.

[0003] As a key component of offshore wind turbines, the health of wind turbine blades directly affects the energy conversion efficiency. Problems such as blade damage, corrosion, pollution and structural fatigue may lead to reduced power generation efficiency and even equipment failure. Therefore, real-time monitoring and abnormality detection of wind turbine blades are of great significance to ensure the stable operation of offshore wind turbines and extend their service life.

[0004] Existing methods for detecting blade faults in offshore wind turbines mainly include those based on graph convolutional networks (GCN), heterogeneous graph attention networks (HAN), knowledge graphs, and autoencoders. Among them, the scheme based on graph convolutional networks constructs a graph structure of wind turbines and uses the powerful feature extraction capability of GCN to identify abnormal states of blades. The scheme based on heterogeneous graph attention networks uses the attention mechanism to enhance the model's ability to distinguish different types of nodes and edges, so as to better capture the complex interactive relationships in wind turbines. The scheme based on knowledge graphs constructs a knowledge graph containing wind turbine components, fault modes, and maintenance operations, and uses graph embedding technology to assist fault diagnosis. The scheme based on autoencoders uses graph autoencoders to learn the normal operating mode of wind turbines and identify potential faults of blades through anomaly detection.

[0005] Although these technical solutions have made some progress, there are still some obvious limitations. First, GCN and HAN-based solutions may be insufficient in processing multimodal data and capturing complex interactive relationships. They mainly rely on graph structure information and do not fully utilize the intrinsic connections between different data modalities. Although knowledge graph-based solutions can integrate rich semantic information, building and maintaining an accurate knowledge graph requires a lot of expertise and resources. Autoencoder-based solutions may lack generalization capabilities when faced with new or unseen fault modes, because they are usually based on historical data learning and may have difficulty in effectively identifying newly emerging abnormal behaviors. These solutions perform poorly when dealing with similar relationships and have difficulty coping with interactions between offshore wind turbine entities.

[0006] Therefore, the existing fault detection methods still have much room for improvement in terms of accuracy, real-time performance, adaptability and economy. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides an offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graphs.

[0008] The technical solution of the present invention is: a method for detecting blade faults of offshore wind turbines based on multimodal data and heterogeneous graphs, comprising the following steps:

[0009] S1), obtaining multi-modal data of n modes of offshore wind turbine blades;

[0010] S2) fusing n multimodal data through joint embedding learning to extract high-order feature information of wind turbine blades, and improving the correlation and consistency between each modal data through a joint loss function;

[0011] S3), constructing a heterogeneous graph of wind turbine blades;

[0012] S4), representing the nodes and edges of the heterogeneous graph as triples;

[0013] S5), using the improved Complex model to score the triples of the heterogeneous graph to determine the authenticity of the triples;

[0014] S6), enhancing the correlation between weakly similar modal data and regional leaves through small models;

[0015] S7), based on the regional association matrix, identify potential failures of blades in the region and issue an early warning.

[0016] Preferably, in step S2), the multimodal data {M 1 ,M 2 ,…,M n}, the multimodal data with a similarity score greater than the first similarity threshold is marked as "normal" and used as high-similarity modal data; the multimodal data with a similarity score greater than the second similarity threshold and less than or equal to the first similarity threshold is marked as "potential abnormal" and used as weak-similarity modal data; the multimodal data with a similarity score less than the second similarity threshold is marked as "abnormal" and used as non-similar modal data;

[0017] Then the labeled information is used as the new modal data M n+1 , and the new modal data M n+1 Added to the original multimodal data, the final multimodal data is expressed as:

[0018] M={M 1 ,M 2 ,…,M n ,M n+1}.

[0019] Preferably, in step S2), the calculation expression of the similarity score is:

[0020] S i =sim(m i,j ,m i,k );

[0021] In the formula, S i represents the similarity score of the sample of the i-th modality; m i,j 、m i,k They represent the j-th sample and the k-th sample data in the ith mode respectively; sim represents the similarity function.

[0022] Preferably, in step S2), all multimodal data are mapped to a unified low-dimensional embedding space through joint embedding learning to generate a unified embedding representation Z, namely:

[0023] Z=(Z 1 ,Z 2 ,……,Z n+1) ;

[0024] In the formula, Z i =M i ×d is the embedding representation corresponding to the i-th modality, and d is the dimension of the embedding space; Z n+1 =M n+1 ×d represents modal data M n+1 The embedded representation of .

[0025] Preferably, in step S2), the joint loss function It is expressed as:

[0026]

[0027] In the formula, is the intra-modal consistency loss; is the inter-modal correlation loss, is the similarity labeling loss, which is used to strengthen the distinction between labeled data, and λ 1 ,λ 2 is the weight hyperparameter of the loss function, λ 3 To control the weight of similarity loss;

[0028] Based on the joint loss function The optimized embedding representation Z is expressed as Z = {E 1 ,E 2 ,…,E n ,E n+1}.

[0029] As a preference, in step S2), for the modal data M i , the intra-modal consistency loss It is expressed as:

[0030]

[0031] In the formula, is the intra-modal consistency loss; f enc (M i ) is the mode M i The feature encoding function of M i The i-th modal data; Z i is the embedding representation corresponding to the i-th modality data; α represents the weight factor, which is used to balance the consistency goal between the original modality and the labeled modality; ∥·∥ F is the Frobenius norm, which is used to measure the difference between matrices; f sym (M n+1 ) is the embedding representation of the similarity mark. By converting the labels “normal”, “potential abnormality” and “abnormality” into low-dimensional vector representations, the vectors corresponding to the similarity marks in the embedding space can reflect the similarities or differences between different data categories. i+1 The modal data M representing the similarity mark n+1 Embedding representation; the optimization goal is to constrain Z i+1 and f sym (M n+1 ), ensuring that the modality representation in the embedding space can reflect the influence of similarity markers.

[0032] Preferably, in step S2), the inter-modal correlation loss It is expressed as:

[0033]

[0034] Among them, α 1 , α 2 are the weight hyperparameters of high similarity modal data, weak similarity modal data, and non-similar modal data respectively; Indicates the loss of high similarity modal data; is the loss of weakly similar modal data and non-similar modal data.

[0035] Preferably, the loss of high similarity modal data It is expressed as:

[0036]

[0037] The loss of weakly similar modal data and non-similar modal data It is expressed as:

[0038]

[0039] In the formula, Z i is the embedding representation of modality i, Z j is the embedding representation of modality j, sim(Z i ,Z j ) is the similarity between the embeddings of modality i and modality j. The optimization goal is to maximize the similarity between modality data with high similarity and reduce the similarity between modality data with weak similarity and non-similar modality data, so as to distinguish data categories.

[0040] Preferably, in step S3), the isomeric graph G of the wind turbine blades is expressed as:

[0041] G = (V, E);

[0042] V={v b ,v t ,v e ,v r ,v s ,v f ,v a};

[0043] E={(v i ,v j )|relationship type};

[0044] Where V represents the node set; E represents the edge set; v b represents a leaf node; v t Wind turbine node; v e is the environment node; v r is a regional node; v s is the sensor node; vf is the fault-damaged node, indicating that the fault damage is related; v a It is an early warning node, which indicates early warning information from system monitoring and data analysis.

[0045] Preferably, in step S4), the nodes and edges of the heterogeneous graph are represented as triples (h, r, t), where h is the head node, t is the tail node, and r is the relationship.

[0046] The head node h represents the complex embedding of blades, fans, environment, and regions; the complex embedding representation dimension of the head node h is M h ×2d; that is:

[0047] h=h real +ih imag

[0048] In the formula, h real 、h imag are the real part embedding of the head node and the imaginary part embedding of the head node respectively; i represents the unit of the imaginary part;

[0049] The tail node t is represented by the complex embedding of objects such as fault type, overall status of the wind turbine, and environmental factors. The complex embedding representation dimension of the tail node t is M t ×2d, the complex embedding of the tail node t is:

[0050] t=t real +it imag ;

[0051] Among them, t real ,t imag They are the real part embedding of the tail node and the imaginary part embedding of the tail node respectively;

[0052] The relation r represents the complex embedding of edge types:

[0053] r=r real +ir imag

[0054] Among them, r real and r imag are the real and imaginary embeddings of relation r, respectively.

[0055] Preferably, in step S5), the scoring expression of the improved Complex model is:

[0056]

[0057] Among them, f(h,r,t) represents the scoring function of the triple (h,r,t), Re is the real part of the result, is the complex conjugate embedding of the tail node; is the complex inner product, is the value of the complex vector in the kth dimension.

[0058] Preferably, in step S5), the negative log-likelihood loss is used Optimize the triple score f(h,r,t), that is:

[0059]

[0060] Where τ is the set of positive triplets, τ′ is the set of negative triplets generated by negative sampling, and (h′, r′, t′) is a negative triplet.

[0061] Preferably, in step S6), the correlation score of the leaves in the region is expressed as:

[0062]

[0063] In the formula, represents the association score; Represents leaf node b respectively i and b j The embedded representation of h i ,h j is the head node of nodes i and j; w is the weight of the small model; represents the transposition operation; F is the activation function;

[0064] Preferably, in step S6), the negative log-likelihood loss of the improved Complex model is used. Adding regional and environmental association constraints, we get the extended total loss function

[0065]

[0066] Among them, λ is the weight coefficient, which is used to control the contribution ratio of the small model. is the regional association loss;

[0067] Smaller expanded total loss function It means that the small model can not only correctly capture the similarity and dissimilarity relationship between leaves, but also better model the correlation characteristics between leaves and regional environment; the larger extended total loss function This indicates that the small model fails to fit the data effectively, possibly because the embedding representation is inaccurate or the regional associations between leaves are not captured correctly.

[0068] Preferably, in step S6), the regional correlation loss It is expressed as:

[0069]

[0070] In the formula, represents a set of leaf node pairs; y i,j Indicates the true label of nodes i and j, used to represent node b i and b j Similarity relationship between

[0071] when Value and true label y i,j When the difference is small, it means that the small model models the relationship between leaves in the region better; if the difference is large, it means that the small model fails to effectively capture the regional relationship between leaves.

[0072] Preferably, in step S7), for multiple wind turbine blades in the entire region, the regional association matrix is ​​expressed as:

[0073] Preferably, in step S7), the expression for detecting potential fault anomaly of blades in the region is:

[0074]

[0075] In the formula, Indicates leaf b i Fault warning result, h i It is leaf b i The hidden feature vector of , σ is the activation function, w is the weight of the model, Represents a transpose operation.

[0076] Preferably, in step S7), the fault warning determination formula is as follows:

[0077]

[0078] In the formula, θ fault is the fault threshold. When the prediction score exceeds this threshold, it indicates that the blade is faulty. normal It is the threshold of normal state, indicating that the blade is in normal working condition.

[0079] The beneficial effects of the present invention are:

[0080] 1. The present invention can significantly improve the accuracy of wind turbine blade fault detection. Through joint embedding learning, it can effectively integrate data from multiple sensors and capture the relationship between different modal data, so as to accurately identify the health status of the blades. By using complex regional correlation loss and similarity / dissimilarity relationship modeling, the spatial and physical similarity between blades is enhanced, thereby improving the accuracy and robustness of fault detection.

[0081] 2. The present invention optimizes the joint embedding representation and uses the Complex model for scoring, which can efficiently process multimodal data and quickly identify abnormal behaviors between blades in the embedding space. The computational complexity is further reduced by targeting the small model associated with the blades, ensuring the real-time response capability of fault detection in the operation of wind turbines, which is particularly suitable for scenarios with high requirements for health monitoring of wind turbines.

[0082] 3. The present invention is adaptable and robust. By considering the similarity and dissimilarity relationship between blades, the present invention can adapt to the differences between different wind turbines and cope with complex situations in different wind turbines and different regional environments. Even when environmental conditions change greatly or sensor data is noisy, it can maintain high detection performance, providing strong support for the long-term stable operation of wind turbines. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Schematic diagram of the process of the present invention;

[0084] Figure 2 A schematic diagram of an isomorphic graph of a wind turbine blade constructed according to the present invention;

[0085] Figure 3 Flow chart of cross validation in an embodiment of the present invention. DETAILED DESCRIPTION

[0086] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0087] like Figure 1 As shown, the present invention provides a method for detecting blade faults of offshore wind turbines based on multimodal data and heterogeneous graphs, comprising the following steps:

[0088] S1), obtaining multi-modal data of n modes of offshore wind turbine blades;

[0089] In this embodiment, the multimodal data includes fan blade images, blade vibration data, blade temperature data, ambient humidity and salt spray concentration data; wherein the fan blade image is obtained by a drone camera, the blade vibration data is obtained by a vibration sensor, and the blade temperature data is obtained by a temperature sensor; the ambient humidity data is obtained by a humidity sensor, and the salt spray concentration data is obtained by a salt spray sensor. The multimodal data is represented as: {M 1 ,M 2 ,…,M n}.

[0090] S2) fusing the multimodal data of n modes through joint embedding learning to extract high-order feature information of wind turbine blades, and improving the correlation and consistency between the modal data through a joint loss function;

[0091] In this embodiment, the joint embedding learning technology is used to map the data of multiple modalities into a unified embedding space, and ensure the intra-modal consistency and inter-modal correlation, that is, to ensure that the data of each modality can retain its original characteristics and internal structure and optimize the objective function so that the relationship between different modalities can be reflected in the embedding space.

[0092] By jointly embedding learning from n modal data {M 1 ,M 2 ,…,M n} to extract and learn their intrinsic relations and construct a low-dimensional universal embedding space Z so as to represent the intra-modal consistency and inter-modal correlation, thereby better characterizing the structural information of multimodal data.

[0093] In this embodiment, the multimodal data {M 1 ,M 2 ,…,M n}, and the multimodal data with a similarity score greater than the first similarity threshold μ+ησ is marked as “normal” and used as high-similarity modal data; that is, S i >μ+ησ;

[0094] Multimodal data with a similarity score greater than the second similarity threshold μ-ησ and less than or equal to the first similarity threshold μ+ησ are marked as "potential anomalies" and used as weak similarity modal data; that is, μ-ησ i ≤μ+ησ;

[0095] The multimodal data with similarity scores less than the second similarity threshold μ-ησ are marked as “abnormal” and regarded as non-similar modal data; that is, S i <μ-ησ;

[0096] Among them, μ is the mean of the similarity score, σ is the standard deviation of the similarity score, and η represents a preset constant;

[0097] Among them, the similarity score S i The calculation expression is:

[0098]

[0099] In the formula, S i represents the similarity score of the sample of the i-th modality; m i,j 、m i,k ​They represent the j-th sample and the k-th sample data in the ith mode respectively; sim represents the similarity function.

[0100] Then the labeled information is used as the new modal data M n+1 , and the new modal data M n+1 Added to the original multimodal data, the final multimodal data is expressed as:

[0101] M={M 1 ,M 2 ,…,M n ,M n+1}.

[0102] Through joint embedding learning, all multimodal data are mapped to a unified low-dimensional embedding space to generate a unified embedding representation Z. The low-dimensional embedding space can remove redundant features in the original data, so that the model can focus on the most important features, improve the generalization ability of the model, and finally input the low-dimensional embedding space into the improved Complex model, that is:

[0103] Z=(Z 1 ,Z 2 ,……,Z n+1) ;

[0104] In the formula, Z i =M i ×d is the embedding representation corresponding to the i-th modality, and d represents the dimension of the embedding space; Z n+1 =M n+1 ×d represents modal data M n+1 The embedded representation of .

[0105] In this embodiment, the joint loss function It is expressed as:

[0106]

[0107] In the formula, is the intra-modal consistency loss; is the inter-modal correlation loss, is the similarity labeling loss, which is used to strengthen the distinction between labeled data, and λ 1 ,λ 2 is the weight hyperparameter of the loss function, λ 3 To control the weight of similarity loss;

[0108] For the modal data M i , the intra-modal consistency loss It is expressed as:

[0109]

[0110] In the formula, is the intra-modal consistency loss; f enc (M i ) is the mode M i The feature encoding function of M i The i-th modal data; Z i is the embedding representation corresponding to the i-th modality data; α represents the weight factor, which is used to balance the consistency goal between the original modality and the labeled modality; ∥·∥ F is the Frobenius norm, which is used to measure the difference between matrices; f sym (M n+1 ) is the embedding representation of the similarity mark. By converting the labels “normal”, “potential abnormality” and “abnormality” into low-dimensional vector representations, the vectors corresponding to the similarity marks in the embedding space can reflect the similarities or differences between different data categories. i+1 The modal data M representing the similarity mark n+1 Embedding representation; the optimization goal is to constrain Z i+1 and f sym (M n+1 ), ensuring that the modality representation in the embedding space can reflect the influence of similarity markers.

[0111] The loss of inter-modal correlation It is expressed as:

[0112]

[0113] Among them, α 1 , α 2 are the weight hyperparameters of high similarity modal data, weak similarity modal data, and non-similar modal data respectively; Indicates the loss of high similarity modal data; is the loss of weakly similar modal data and non-similar modal data.

[0114] The loss of high similarity modal data It is expressed as:

[0115]

[0116] The loss of weakly similar modal data and non-similar modal data It is expressed as:

[0117]

[0118] In the formula, Z i is the embedding representation of modality i, Z j is the embedding representation of modality j, sim(Z i,Z j ) is the similarity between the embeddings of modality i and modality j. The optimization goal is to maximize the similarity between modality data with high similarity and reduce the similarity between modality data with weak similarity and non-similar modality data, so as to distinguish data categories.

[0119] Among them, the similarity sim(Z i ,Z j ) is expressed as:

[0120]

[0121] By maximizing the inter-modality similarity sim(Z i ,Z j ), ensuring that the embedding space can reflect the correlation and synergy between modalities.

[0122] Based on the joint loss function The optimized embedding representation Z is expressed as Z = {E 1 ,E 2 ,…,E n ,E n+1}.

[0123] Offshore wind turbines may present different operating characteristics under the influence of different geographical environments, climatic conditions and operating conditions (such as wind speed, waves and other factors). In order to enable the joint embedding learning method to dynamically adjust according to the changes in these environmental factors and enhance the adaptability of the joint embedding learning, this embodiment also introduces a similarity threshold Dynamically adjust the joint embedding learning to ensure that the joint embedding learning can adapt to different wind turbine operating environments, and the similarity threshold The dynamic adjustment of can enable joint embedding learning to stably learn effective features in the face of these changes, while avoiding premature response to data changes, which leads to overfitting of joint embedding learning. The similarity threshold τ is expressed as:

[0124]

[0125] in, is the initial similarity threshold, γ is the threshold decay rate, and t is the number of training iterations; Dynamically adjust τ In the early stage of training, the similarity threshold is high, and joint embedding learning has strict requirements on the similarity of data, which helps prevent the model from "relaxing" the similarity judgment too early, thereby ensuring that the joint embedding learning technology learns more "reliable" features. As training progresses, the similarity threshold gradually decreases, allowing the model to adapt to different changes more flexibly, especially when the data changes greatly or new types of faults appear, it can detect potential anomalies more sensitively.

[0126] S3), constructing a heterogeneous graph of wind turbine blades; Figure 2 As shown: the heterogeneous graph G of the wind turbine blade is expressed as:

[0127] G = (V, E);

[0128] V={v b ,v t ,v e ,v r ,v s ,v f ,v a};

[0129] E={(v i ,v j )|relationship type};

[0130] Where V represents the node set; E represents the edge set; v b represents a leaf node; v t Wind turbine node; v e is the environment node; v r is a regional node; v s is a sensor node. Among them, the environment node v e Including temperature, humidity, salt spray and other characteristics. The sensor node v s is an embedded representation of a specific sensor, used to supplement additional modal associations; the regional node v r The geographical distribution characteristics of wind turbines are shown. Each regional node connects wind turbines in the same region. f is the fault damage node, indicating fault damage related, such as fault type (crack, corrosion, etc.), blade surface damage, crack, etc., v a It is an early warning node, which indicates early warning information from system monitoring and data analysis, such as "fault warning", "abnormal equipment status", etc.

[0131] S4), representing the nodes and edges of the heterogeneous graph as triples;

[0132] The nodes and edges of the heterogeneous graph are represented as triples (h, r, t), where h is the head node, t is the tail node, and r is the relationship.

[0133] The head node h represents the complex embedding of fan blades, fan, environment, and region; the complex embedding representation dimension of the head node h is M h ×2d; that is:

[0134] h=h real +ih imag

[0135] In the formula, h real 、h imagare the real part embedding of the head node and the imaginary part embedding of the head node respectively; i represents the unit of the imaginary part;

[0136] The tail node t is represented by the complex embedding of objects such as fault type, overall status of the wind turbine, and environmental factors: The complex embedding representation dimension of the tail node t is M t ×2d, the complex embedding of the tail node t is:

[0137] t=t real +it imag ;

[0138] Among them, t real ,t imag They are the real part embedding of the tail node and the imaginary part embedding of the tail node respectively;

[0139] The relation r represents the complex embedding of edge types:

[0140] r=r real +ir imag

[0141] Among them, r real and r imag are the real and imaginary embeddings of relation r, respectively.

[0142] S5) Use the improved Complex model to score the triples of the heterogeneous graph to determine the authenticity of the triples; the higher the score, the more authentic the triple is: it means that the relationship between the head node (such as blade), relationship (such as fault type) and tail node (such as crack) is in line with reality, and the triple is considered to be authentic. The low score means that the triple is not authentic: it means that the relationship of this triple does not hold in the model and may be wrong, and the triple is judged to be unauthentic. Figure 3 As shown, the scoring threshold is selected through subsequent cross-validation to determine the authenticity.

[0143] The scoring expression of the improved Complex model is:

[0144]

[0145] Among them, f(h,r,t) represents the scoring function of the triple (h,r,t), Re is the real part of the result, is the complex conjugate embedding of the tail node; is the complex inner product, is the value of the complex vector in the kth dimension.

[0146] By expanding the complex inner product, the above formula can be expressed as:

[0147]

[0148] For normal data, the triple of similarity relations ( resemblance, ) will have a higher f(h,r,t) value, and the dissimilarity relationship ( Not similar, ) will have a lower value.

[0149] For weak similarity modal data, the f(h,r,t) value is close to zero, indicating that the similarity between the node pairs is weak.

[0150] For non-similar modal data, the f(h,r,t) value deviates significantly from the expected range.

[0151] For the mismatch between the corresponding relationship embedding r and the node embedding h, t caused by non-similar modal data, which leads to significant deviation in the calculation of the score f(h, r, t), the negative log-likelihood loss is used. Optimize the triple score f(h,r,t), that is:

[0152]

[0153] Where τ is the set of positive triplets, τ′ is the set of negative triplets generated by negative sampling, and (h′, r′, t′) is a negative triplet.

[0154] S6), enhancing the correlation between weakly similar modal data and regional leaves through small models;

[0155] The main purpose of small model scoring is to further enhance the accuracy of fault prediction and detection in the context of regional associations and environmental factors. Small models are simpler and can focus more on fault detection in certain specific areas or small ranges, reducing computational complexity. Compared with the Complex model, it can simultaneously capture details and grasp the overall system relationship.

[0156] For the complex association between a single fan blade and the fan blades in a region, the scoring can be extended by a small model. The association score of the blades in the region is expressed as:

[0157]

[0158] In the formula, represents the association score; Represents leaf node b respectively i and b j The embedded representation of h i ,h j is the head node of nodes i and j; w is the weight of the small model; represents the transposition operation; F is the activation function;

[0159] Through the negative log-likelihood loss of the improved Complex model Adding regional and environmental association constraints, we get the extended total loss function

[0160]

[0161] Among them, λ is the weight coefficient, which is used to control the contribution ratio of the small model. is the regional association loss;

[0162] Smaller expanded total loss function It means that the small model can not only correctly capture the similarity and dissimilarity relationship between leaves, but also better model the correlation characteristics between leaves and regional environment; the larger extended total loss function This indicates that the small model fails to fit the data effectively, possibly because the embedding representation is inaccurate or the regional associations between leaves are not captured correctly.

[0163] The regional correlation loss It is expressed as:

[0164]

[0165] Where, t i,j Indicates the true label of nodes i and j; used to represent node b i and b j The similarity relationship between Represents a set of leaf node pairs, including the indexes of all leaf node pairs that need to be calculated;

[0166] when Value and true label t i,j When the difference is small, it means that the small model models the relationship between leaves in the region better; if the difference is large, it means that the small model fails to effectively capture the regional relationship between leaves.

[0167] S7), based on the regional association matrix, identify potential failures of blades in the region and issue an early warning.

[0168] For multiple fan blades in the entire area, the regional association matrix is ​​expressed as:

[0169]

[0170] The expression for detecting potential fault anomaly of blades in the region is:

[0171]

[0172] In the formula, Indicates leaf b i Fault warning result, hi It is leaf b i The hidden feature vector of , σ is the activation function, w is the weight of the model, and T represents the transposition operation.

[0173] The fault warning judgment formula is as follows:

[0174]

[0175] In the formula, θ fault is the fault threshold. When the prediction score exceeds this threshold, it indicates that the blade is faulty. normal It is the threshold of normal state, indicating that the blade is in normal working condition.

[0176] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A method for detecting blade faults of offshore wind turbines based on multimodal data and heterogeneous graphs, characterized in that: The steps include: S1), obtaining multi-modal data of n modes of offshore wind turbine blades; S2) fusing n multimodal data through joint embedding learning to extract high-order feature information of wind turbine blades, and improving the correlation and consistency between each modal data through a joint loss function; And introduce the similarity threshold τ to dynamically adjust the joint embedding learning; S3), constructing a heterogeneous graph of wind turbine blades; S4) Represent the nodes and edges of the heterogeneous graph as triples (h, r, t), where h is the head node, t is the tail node, and r is the relationship; S5), using the improved Complex model to score the triples of the heterogeneous graph to determine the authenticity of the triples; S6), enhancing the correlation between weakly similar modal data and regional leaves through small models; S7), based on the regional association matrix, identify potential failures of blades in the region and issue an early warning.

2. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1 is characterized in that: In step S2), the multimodal data {M1, M2, ..., M n }, marking the multimodal data whose similarity score is greater than the first similarity threshold as "normal" and as high-similarity modal data; The multimodal data whose similarity score is greater than the second similarity threshold and less than or equal to the first similarity threshold is marked as "potential anomaly" and used as weak similarity modal data; Marking the multimodal data with a similarity score less than a second similarity threshold as "abnormal" and treating it as non-similar modal data; Then the labeled information is used as the new modal data M n+1 , and the new modal data M n+1 Added to the original multimodal data, the final multimodal data is expressed as: M={M1,M2,…,M n ,M n+1 }; Through joint embedding learning, all multimodal data are mapped into a unified low-dimensional embedding space to generate a unified embedding representation Z, namely: Z=(Z1,Z2,……,Z n+1) ; In the formula, Z i =M i ×d is the embedding representation corresponding to the i-th modality, and d is the dimension of the embedding space; Z n+1 =M n+1 ×d represents modal data M n+1 The embedded representation of .

3. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1 is characterized in that: In step S2), the joint loss function It is expressed as: In the formula, is the intra-modal consistency loss; is the inter-modal correlation loss, is the similarity labeling loss, which is used to strengthen the distinction of labeled data, λ1,λ2 are the weight hyperparameters of the loss function, and λ3 is the weight for controlling the similarity loss; Based on the joint loss function The optimized embedding representation Z is represented as Z = {E1, E2, …, E n ,E n+1 }.

4. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 3 is characterized in that: In step S2), for the modal data M i , the intra-modal consistency loss It is expressed as: In the formula, is the intra-modal consistency loss; f enc (M i ) is the mode M i The feature encoding function of M i The i-th modal data; Z i is the embedding representation corresponding to the i-th modality data; α represents the weight factor, which is used to balance the consistency goal between the original modality and the labeled modality; ||·|| F is the Frobenius norm, which is used to measure the difference between matrices; f sym (M n+1 ) is the embedding representation of the similarity mark. By converting the labels "normal", "potential abnormality", and "abnormality" into low-dimensional vector representations, the vectors corresponding to the similarity marks in the embedding space can reflect the similarities or differences between different data categories; Z i+1 The modal data M representing the similarity mark n+1 Embedding representation; the optimization goal is to constrain Z i+1 and f sym (M n+1 ), ensuring that the modality representation in the embedding space can reflect the influence of similarity markers.

5. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 3 is characterized in that: In step S2), the inter-modal correlation loss It is expressed as: Among them, α1 and α2 are the weight hyperparameters of high similarity modal data, weak similarity modal data and non-similar modal data respectively; Indicates the loss of high similarity modal data; is the loss of weakly similar modal data and non-similar modal data.

6. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 5 is characterized in that: The loss of high similarity modal data It is expressed as: The loss of weakly similar modal data and non-similar modal data It is expressed as: In the formula, Z i is the embedding representation of modality i, Z j is the embedding representation of modality j, sim(Z i ,Z j ) is the similarity between the embeddings of modality i and modality j. The optimization goal is to maximize the similarity between modality data with high similarity and reduce the similarity between modality data with weak similarity and non-similar modality data, so as to distinguish data categories.

7. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1, characterized in that: In step S3), the heterogeneous graph G of the wind turbine blades is expressed as: G = (V, E); V={v b ,v t ,v e ,v r ,v s ,v f ,v a }; E={(v i ,v j )|relationship type}; Where V represents the node set; E represents the edge set; v b represents a leaf node; v t Wind turbine node; v e is the environment node; v r is a regional node; v s is the sensor node, v f is the fault-damaged node, indicating that the fault damage is related; v a It is an early warning node, which indicates early warning information from system monitoring and data analysis.

8. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1, characterized in that: In step S5), the scoring expression of the improved Complex model is: Among them, f(h,r,t) represents the scoring function of the triple (h,r,t), Re is the real part of the result, is the complex conjugate embedding of the tail node; is the complex inner product, is the value of the complex vector in the kth dimension; And through the negative log-likelihood loss Optimize the triple score f(h,r,t), that is: Where τ is the set of positive triplets, τ′ is the set of negative triplets generated by negative sampling, and (h′, r′, t′) is a negative triplet.

9. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1, characterized in that: In step S6), the correlation score of the leaves in the region is expressed as: In the formula, represents the association score; Represents leaf node b respectively i and b j Embedding representation of ; h i ,h j is the head node of nodes i and j; w is the weight of the small model; Represents a transpose operation; F is the activation function; Through the negative log-likelihood loss of the improved Complex model Adding regional and environmental association constraints, we get the extended total loss function Among them, λ is the weight coefficient, which is used to control the contribution ratio of the small model. is the regional association loss; The regional correlation loss It is expressed as: In the formula, Represents a group leaf node pair; y i,j represents the true label of nodes i and j, and represents node b i and b j The similarity relationship between them.

10. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1, characterized in that: In step S7), the expression for detecting potential fault anomaly of blades in the region is: In the formula, Indicates leaf b i Fault warning result, h i It is leaf b i The hidden feature vector of , σ is the activation function, w is the weight of the model, Represents a transpose operation; The fault warning judgment formula is as follows: In the formula, θ fault is the fault threshold. When the prediction score exceeds this threshold, it indicates that the blade is faulty. normal It is the threshold of normal state, indicating that the blade is in normal working condition.

Images