Offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph

Abstract

The present invention provides a method for detecting offshore wind turbine blade faults based on multimodal data and heterogeneous graphs. The method comprises obtaining multimodal data of n modes of offshore wind turbine blades; fusing the multimodal data through joint embedding learning to extract high-order feature information of the wind turbine blades and improving the correlation and consistency between the modal data using a joint loss function; constructing a heterogeneous graph of the wind turbine blades; representing the nodes and edges of the heterogeneous graph as triples; scoring the triples of the heterogeneous graph using an improved Complex model; enhancing the correlation between weakly similar modal data and regional blades using a small model; and identifying potential blade faults within the region based on a regional correlation matrix and issuing a warning. The method can improve the accuracy of wind turbine blade fault detection. By optimizing the joint embedding representation and using the Complex model for scoring, the method can efficiently process multimodal data and exhibits adaptability and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of wind turbine fault detection, and in particular to a method for detecting blade faults of offshore wind turbines based on multimodal data and heterogeneous graphs. Background Art

[0002] Offshore wind turbines are the core equipment of offshore wind farms, and their stable and efficient operation is crucial to the economic viability and reliability of wind farms. However, the harsh conditions of the offshore environment, including extreme weather, high humidity, salt spray corrosion, and complex marine ecosystems, pose a severe challenge to the normal operation of offshore wind turbines. These environmental factors not only affect the performance of offshore wind turbines but can also cause equipment failures, increase maintenance costs, and even lead to safety accidents.

[0003] As a key component of offshore wind turbines, the health of wind turbine blades directly impacts energy conversion efficiency. Blade damage, corrosion, contamination, and structural fatigue can lead to reduced power generation efficiency and even equipment failure. Therefore, real-time monitoring and anomaly detection of wind turbine blades are crucial for ensuring stable operation and extending the service life of offshore wind turbines.

[0004] Existing methods for detecting blade faults in offshore wind turbines mainly include those based on graph convolutional networks (GCNs), heterogeneous graph attention networks (HANs), knowledge graphs, and autoencoders. The graph convolutional network-based approach constructs a graph structure for the wind turbine and leverages the powerful feature extraction capabilities of GCNs to identify abnormal blade states. The heterogeneous graph attention network-based approach uses an attention mechanism to enhance the model's ability to distinguish between different types of nodes and edges, thereby better capturing the complex interactions within the wind turbine. The knowledge graph-based approach constructs a knowledge graph encompassing wind turbine components, fault modes, and maintenance operations, and utilizes graph embedding technology to assist in fault diagnosis. The autoencoder-based approach uses a graph autoencoder to learn the normal operating mode of the wind turbine and identify potential blade faults through anomaly detection.

[0005] Although these technical solutions have made some progress, there are still some obvious limitations. First, solutions based on GCN and HAN may be insufficient in processing multimodal data and capturing complex interactive relationships. They mainly rely on graph structure information and do not fully utilize the intrinsic connections between different data modalities. Although solutions based on knowledge graphs can integrate rich semantic information, building and maintaining an accurate knowledge graph requires a lot of expertise and resources. Solutions based on autoencoders may lack generalization capabilities when faced with new or unseen fault modes because they are usually based on historical data learning and may have difficulty in effectively identifying newly emerging abnormal behaviors. These solutions perform poorly in dealing with similar relationships and have difficulty coping with interactions between offshore wind turbine entities.

[0006] Therefore, the existing fault detection methods still have much room for improvement in terms of accuracy, real-time performance, adaptability and economy. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the present invention provides an offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graphs.

[0008] The technical solution of the present invention is: a method for detecting blade faults of offshore wind turbines based on multimodal data and heterogeneous graphs, comprising the following steps:

[0009] S1), obtaining multimodal data of n modes of offshore wind turbine blades;

[0010] S2) fusing n multimodal data through joint embedding learning to extract high-order feature information of wind turbine blades and improving the correlation and consistency between modal data through a joint loss function;

[0011] S3), constructing a heterogeneous graph of wind turbine blades;

[0012] S4), representing the nodes and edges of the heterogeneous graph as triples;

[0013] S5), using the improved Complex model to score the triples of the heterogeneous graph to determine the authenticity of the triples;

[0014] S6), enhancing the correlation between weakly similar modal data and regional leaves through small models;

[0015] S7) Identify potential failures of blades in the region based on the regional association matrix and issue an early warning.

[0016] As a preference, in step S2), the multimodal data {M1, M2, ..., M n} are marked, and multimodal data with a similarity score greater than the first similarity threshold is marked as "normal" and used as high-similarity modal data; multimodal data with a similarity score greater than the second similarity threshold and less than or equal to the first similarity threshold is marked as "potential abnormal" and used as weak-similarity modal data; multimodal data with a similarity score less than the second similarity threshold is marked as "abnormal" and used as non-similar modal data;

[0017] Then the labeled information is used as the new modal data M n+1 , and the new modal data M n+1 Added to the original multimodal data, the final multimodal data is expressed as:

[0018] M={M1,M2,…,M n ,M n+1}.

[0019] Preferably, in step S2), the calculation expression of the similarity score is:

[0020] S i =sim(m i,j ,m i,k );

[0021] Where S i represents the similarity score of the sample of the i-th modality; m i,j 、m i,k They represent the j-th sample and the k-th sample data in the i-th mode respectively; sim represents the similarity function.

[0022] Preferably, in step S2), all multimodal data are mapped to a unified low-dimensional embedding space through joint embedding learning to generate a unified embedding representation Z, namely:

[0023] Z=(Z1,Z2,……,Z n+1) ;

[0024] Where Z i =M i ×d is the embedding representation corresponding to the i-th modality, and d is the dimension of the embedding space; Z n+1 =M n+1 ×d represents modal data M n+1 Embedding representation of .

[0025] As a preference, in step S2), the joint loss function Expressed as:

[0026]

[0027] Where, is the intra-modal consistency loss; is the inter-modal correlation loss, is the similarity labeling loss, which is used to strengthen the distinction of labeled data, λ1 and λ2 are the weight hyperparameters of the loss function, and λ3 is the weight for controlling the similarity loss;

[0028] Based on the joint loss function The optimized embedding representation Z is represented as Z={E1,E2,…,E n ,E n+1}.

[0029] As a preference, in step S2), for the modal data M i , the intra-modal consistency loss Expressed as:

[0030]

[0031] Where, is the intra-modal consistency loss; f enc (M i ) is the mode M i The feature encoding function of M i The i-th modal data; Z i is the embedding representation corresponding to the i-th modality data; α is the weight factor used to balance the consistency goal between the original modality and the labeled modality; ∥·∥ F is the Frobenius norm, which is used to measure the difference between matrices; f sym (M n+1 ) is the embedding representation of the similarity mark. By converting the labels "normal", "potential abnormality", and "abnormal" into low-dimensional vector representations, the vectors corresponding to the similarity marks in the embedding space can reflect the similarity or difference between different data categories; Z i+1 Modal data M representing similarity tags n+1 Embedding representation; the optimization goal is to constrain Z i+1 and f sym (M n+1 ) distance, ensuring that the modality representation in the embedding space can reflect the influence of similarity markers.

[0032] As a preference, in step S2), the inter-modal correlation loss Expressed as:

[0033]

[0034] Among them, α1 and α2 are the weight hyperparameters of high similarity modal data, weak similarity modal data and non-similar modal data respectively; Indicates the loss of high similarity modal data; is the loss of weakly similar modal data and non-similar modal data.

[0035] Preferably, the loss of high similarity modal data Expressed as:

[0036]

[0037] The loss of weak similarity modal data and non-similar modal data Expressed as:

[0038]

[0039] Where Z i is the embedding representation of modality i, Z j is the embedding representation of modality j, sim(Z i ,Z j ) is the similarity between the embeddings of modality i and modality j. The optimization goal is to maximize the similarity between modality data with high similarity and reduce the similarity between modality data with weak similarity and non-similar modality data, so as to distinguish data categories.

[0040] Preferably, in step S3), the heterogeneous graph G of the wind turbine blades is represented as:

[0041] G=(V,E);

[0042] V={v b ,v t ,v e ,v r ,v s ,v f ,v a};

[0043] E={(v i ,v j )|relationship type};

[0044] Where V represents the node set; E represents the edge set; v b represents a leaf node; v t Wind turbine node; v e is the environment node; v r is a regional node; v s is the sensor node; v f is the fault-damaged node, indicating fault-damage correlation; v a It is an early warning node, which represents early warning information from system monitoring and data analysis.

[0045] Preferably, in step S4), the nodes and edges of the heterogeneous graph are represented as triples (h, r, t), where h is the head node, t is the tail node, and r is the relationship.

[0046] The head node h represents the complex embedding of blades, fans, environment, and regions; the complex embedding of the head node h represents the dimension M h ×2d; that is:

[0047] h=h real +ih imag

[0048] Where h real 、h imag are the real part embedding of the head node and the imaginary part embedding of the head node respectively; i represents the unit of the imaginary part;

[0049] The tail node t represents the complex embedding of objects such as fault type, overall status of the wind turbine, and environmental factors. The complex embedding representation dimension of the tail node t is M t ×2d, the complex embedding of the tail node t is:

[0050] t=t real +it imag ;

[0051] Among them, t real , t imag They are the real part embedding of the tail node and the imaginary part embedding of the tail node respectively;

[0052] The relation r represents the complex embedding of edge types:

[0053] r=r real +ir imag

[0054] Among them, r real and r imag are the real and imaginary embeddings of relation r, respectively.

[0055] Preferably, in step S5), the scoring expression of the improved Complex model is:

[0056]

[0057] Among them, f(h,r,t) represents the scoring function of the triple (h,r,t), Re is the real part of the result, is the complex conjugate embedding of the tail node; is the complex inner product, is the value of the complex vector in the kth dimension.

[0058] Preferably, in step S5), the negative log-likelihood loss Optimize the triple score f(h,r,t), that is:

[0059]

[0060] Where τ is the set of positive triplets, τ′ is the set of negative triplets generated by negative sampling, and (h′, r′, t′) is a negative triplet.

[0061] Preferably, in step S6), the correlation score of the leaves in the region is expressed as:

[0062]

[0063] Where, represents the association score; Represents leaf node b respectively i and b j Embedded representation of h i ,h j is the head node of nodes i and j; w is the weight of the small model; represents the transposition operation; F is the activation function;

[0064] Preferably, in step S6), the negative log-likelihood loss of the improved Complex model is used Adding regional and environmental association constraints, we get the extended total loss function

[0065]

[0066] Among them, λ is the weight coefficient, which is used to control the contribution ratio of the small model. is the regional association loss;

[0067] Smaller expanded total loss function It means that the small model can not only correctly capture the similarity and dissimilarity relationship between leaves, but also better model the correlation characteristics between leaves and regional environment; the larger extended total loss function This indicates that the small model fails to fit the data effectively, possibly because the embedding representation is inaccurate or the regional correlation between leaves is not correctly captured.

[0068] As a preference, in step S6), the regional correlation loss Expressed as:

[0069]

[0070] Where, represents a set of leaf node pairs; y i,j Indicates the true label of nodes i and j, used to represent node b i and bj similarity relationship between them;

[0071] when Value and true label y i,j When the difference is small, it means that the small model models the relationship between leaves in the region better; if the difference is large, it means that the small model fails to effectively capture the regional relationship between leaves.

[0072] Preferably, in step S7), for the multiple wind turbine blades in the entire region, the regional correlation matrix is ​​expressed as:

[0073] Preferably, in step S7), the expression for detecting potential fault anomalies of blades in the region is:

[0074]

[0075] Where, Indicates leaf b i Fault warning result, h i It is leaf b i The implicit feature vector of , σ is the activation function, w is the weight of the model, Represents a transpose operation.

[0076] Preferably, in step S7), the fault warning determination formula is as follows:

[0077]

[0078] Where θ fault is the fault threshold. When the prediction score exceeds this threshold, it means that the blade is faulty. normal It is the threshold of normal state, indicating that the blade is in normal working condition.

[0079] The beneficial effects of the present invention are:

[0080] 1. This invention can significantly improve the accuracy of wind turbine blade fault detection. Through joint embedding learning, it can effectively integrate data from multiple sensors and capture the relationship between different modal data, thereby accurately identifying the health status of the blades. By utilizing complex regional correlation loss and similarity / dissimilarity relationship modeling, it enhances the spatial and physical similarity between blades, thereby improving the accuracy and robustness of fault detection.

[0081] 2. This invention optimizes the joint embedding representation and uses the Complex model for scoring, which can efficiently process multimodal data and quickly identify abnormal behavior between blades in the embedding space. By using a small model targeting the association between blades, the computational complexity is further reduced, ensuring the real-time response capability of fault detection during wind turbine operation. It is particularly suitable for scenarios with high requirements for wind turbine health monitoring.

[0082] 3. The present invention is adaptable and robust. By considering the similarity and dissimilarity relationship between blades, the present invention can adapt to the differences between different wind turbines and cope with complex situations under different wind turbines and different regional environments. Even when the environmental conditions change greatly or the sensor data is noisy, it can maintain high detection performance, providing strong support for the long-term stable operation of wind turbines. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Schematic diagram of the process of the present invention;

[0084] Figure 2 A schematic diagram of an isomeric graph of a wind turbine blade constructed according to the present invention;

[0085] Figure 3 Flowchart of cross validation in an embodiment of the present invention. DETAILED DESCRIPTION

[0086] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0087] like Figure 1 As shown, the present invention provides a method for detecting blade faults of offshore wind turbines based on multimodal data and heterogeneous graphs, comprising the following steps:

[0088] S1), obtaining multimodal data of n modes of offshore wind turbine blades;

[0089] In this embodiment, the multimodal data includes fan blade images, blade vibration data, blade temperature data, ambient humidity, and salt spray concentration data; wherein the fan blade images are acquired by a drone camera, the blade vibration data are acquired by a vibration sensor, and the blade temperature data are acquired by a temperature sensor; the ambient humidity data are acquired by a humidity sensor, and the salt spray concentration data are acquired by a salt spray sensor. The multimodal data is represented as: {M1, M2, ..., M n}.

[0090] S2) fusing the multimodal data of n modes through joint embedding learning to extract high-order feature information of wind turbine blades and improving the correlation and consistency between the modal data through a joint loss function;

[0091] In this embodiment, the joint embedding learning technology is used to map data of multiple modalities into a unified embedding space, and ensure intra-modal consistency and inter-modal correlation, that is, to ensure that the data of each modality can retain its original characteristics and internal structure and optimize the objective function so that the relationship between different modalities can be reflected in the embedding space.

[0092] By joint embedding learning from n modal data {M1,M2,…,M n} to extract and learn their intrinsic relations and construct a low-dimensional universal embedding space Z so as to represent the intra-modal consistency and inter-modal correlation, thereby better characterizing the structural information of multimodal data.

[0093] In this embodiment, the multimodal data {M1, M2, ..., M n} is marked, and the multimodal data with a similarity score greater than the first similarity threshold μ+ησ is marked as “normal” and regarded as high similarity modal data; that is, S i >μ+ησ;

[0094] Multimodal data with a similarity score greater than the second similarity threshold μ-ησ and less than or equal to the first similarity threshold μ+ησ are marked as "potential anomalies" and used as weak similarity modal data; that is, μ-ησ i ≤μ+ησ;

[0095] Multimodal data with similarity scores less than the second similarity threshold μ-ησ are marked as “abnormal” and regarded as non-similar modal data; that is, S i <μ-ησ;

[0096] Where μ is the mean of the similarity score, σ is the standard deviation of the similarity score, and η represents a preset constant;

[0097] Among them, the similarity score S i The calculation expression is:

[0098]

[0099] Where S i represents the similarity score of the sample of the i-th modality; m i,j 、m i,k They represent the j-th sample and the k-th sample data in the i-th mode respectively; sim represents the similarity function.

[0100] Then the labeled information is used as the new modal data M n+1 , and the new modal data M n+1 Added to the original multimodal data, the final multimodal data is expressed as:

[0101] M={M1,M2,…,M n ,M n+1}.

[0102] ​Through joint embedding learning, all multimodal data are mapped to a unified low-dimensional embedding space to generate a unified embedding representation Z. The low-dimensional embedding space can remove redundant features in the original data, so that the model can focus on the most important features and improve the generalization ability of the model. Finally, the low-dimensional embedding space is input into the improved Complex model, that is:

[0103] Z=(Z1,Z2,……,Z n+1) ;

[0104] Where Z i =M i ×d is the embedding representation corresponding to the i-th modality, and d represents the dimension of the embedding space; Z n+1 =M n+1 ×d represents modal data M n+1 Embedding representation of .

[0105] In this embodiment, the joint loss function Expressed as:

[0106]

[0107] Where, is the intra-modal consistency loss; is the inter-modal correlation loss, is the similarity labeling loss, which is used to strengthen the distinction of labeled data, λ1 and λ2 are the weight hyperparameters of the loss function, and λ3 is the weight for controlling the similarity loss;

[0108] For the modal data M i , the intra-modal consistency loss Expressed as:

[0109]

[0110] Where, is the intra-modal consistency loss; f enc (M i ) is the mode M i The feature encoding function of M i The i-th modal data; Z i is the embedding representation corresponding to the i-th modality data; α is the weight factor used to balance the consistency goal between the original modality and the labeled modality; ∥·∥ F is the Frobenius norm, which is used to measure the difference between matrices; f sym (M n+1) is the embedding representation of the similarity mark. By converting the labels "normal", "potential abnormality", and "abnormal" into low-dimensional vector representations, the vectors corresponding to the similarity marks in the embedding space can reflect the similarity or difference between different data categories; Z i+1 Modal data M representing similarity tags n+1 Embedding representation; the optimization goal is to constrain Z i+1 and f sym (M n+1 ) distance, ensuring that the modality representation in the embedding space can reflect the influence of similarity markers.

[0111] The loss of intermodal correlation Expressed as:

[0112]

[0113] Among them, α1 and α2 are the weight hyperparameters of high similarity modal data, weak similarity modal data and non-similar modal data respectively; Indicates the loss of high similarity modal data; is the loss of weakly similar modal data and non-similar modal data.

[0114] The loss of high similarity modal data Expressed as:

[0115]

[0116] The loss of weak similarity modal data and non-similar modal data Expressed as:

[0117]

[0118] Where Z i is the embedding representation of modality i, Z j is the embedding representation of modality j, sim(Z i ,Z j ) is the similarity between the embeddings of modality i and modality j. The optimization goal is to maximize the similarity between modality data with high similarity and reduce the similarity between modality data with weak similarity and non-similar modality data, so as to distinguish data categories.

[0119] Among them, the similarity sim(Z i ,Z j ) is expressed as:

[0120]

[0121] By maximizing the inter-modality similarity sim(Z i ,Zj ), ensuring that the embedding space can reflect the correlation and synergy between modalities.

[0122] Based on the joint loss function The optimized embedding representation Z is represented as Z={E1,E2,…,E n ,E n+1}.

[0123] Offshore wind turbines may exhibit different operating characteristics under the influence of different geographical environments, climatic conditions, and operating conditions (such as wind speed, waves, etc.). In order to enable the joint embedding learning method to dynamically adjust according to changes in these environmental factors and enhance the adaptability of the joint embedding learning, this embodiment also introduces a similarity threshold Dynamically adjust the joint embedding learning to ensure that the joint embedding learning adapts to different wind turbine operating environments, and the similarity threshold The dynamic adjustment of can enable the joint embedding learning to stably learn effective features in the face of these changes, while avoiding premature response to data changes, which leads to overfitting of the joint embedding learning. The similarity threshold τ is expressed as:

[0124]

[0125] in, The dynamic adjustment of τ is based on the initial similarity threshold, γ is the threshold decay rate, and t is the number of training iterations. Initially, the similarity threshold is high, and joint embedding learning places strict demands on data similarity. This helps prevent the model from prematurely relaxing similarity judgments, thereby ensuring that joint embedding learning learns more reliable features. As training progresses, the similarity threshold gradually decreases, allowing the model to more flexibly adapt to changes. This allows for more sensitive detection of potential anomalies, particularly when the data fluctuates significantly or new types of faults emerge.

[0126] S3), constructing a heterogeneous graph of wind turbine blades; Figure 2 As shown: the heterogeneous graph G of the wind turbine blade is expressed as:

[0127] G=(V,E);

[0128] V={v b ,v t ,v e ,v r ,v s ,v f ,v a};

[0129] E={(v i ,v j )|relationship type};

[0130] Where V represents the node set; E represents the edge set; v b represents a leaf node; v t Wind turbine node; v e is the environment node; v r is a regional node; v s is the sensor node. Among them, the environment node v e Including temperature, humidity, salt spray and other characteristics. The sensor node v s is the embedding representation of a specific sensor, which is used to supplement the additional modality association; the regional node v r The geographical distribution characteristics of wind turbines are shown. Each regional node is connected to wind turbines in the same region. f is the fault damage node, indicating fault damage related, such as fault type (crack, corrosion, etc.), blade surface damage, crack, etc., v a It is an early warning node, which indicates early warning information from system monitoring and data analysis, such as "fault warning" and "abnormal equipment status".

[0131] S4), representing the nodes and edges of the heterogeneous graph as triples;

[0132] The nodes and edges of the heterogeneous graph are represented as triples (h, r, t), where h is the head node, t is the tail node, and r is the relationship.

[0133] The head node h represents the complex embedding of fan blades, fan, environment, and region; the complex embedding representation dimension of the head node h is M h ×2d; that is:

[0134] h=h real +ih imag

[0135] Where h real 、h imag are the real part embedding of the head node and the imaginary part embedding of the head node respectively; i represents the unit of the imaginary part;

[0136] The tail node t represents the complex embedding of objects such as fault type, overall status of the wind turbine, and environmental factors: the complex embedding representation dimension of the tail node t is M t ×2d, the complex embedding of the tail node t is:

[0137] t=t real +it imag ;

[0138] Among them, t real , t imag They are the real part embedding of the tail node and the imaginary part embedding of the tail node respectively;

[0139] The relation r represents the complex embedding of edge types:

[0140] r=r real +ir imag

[0141] Among them, r real and r imag are the real and imaginary embeddings of relation r, respectively.

[0142] S5) Use the improved Complex model to score the triples of the heterogeneous graph to determine the authenticity of the triples; the higher the score, the more authentic the triple: it means that the relationship between the head node (such as blade), relationship (such as fault type) and tail node (such as crack) is in line with reality, and the triple is considered to be authentic. The lower the score, the unauthentic the triple: it means that the relationship of the triple does not hold in the model and may be wrong, and the triple is judged to be unauthentic. Figure 3 As shown, the scoring threshold is selected through subsequent cross-validation to judge the authenticity.

[0143] The scoring expression of the improved Complex model is:

[0144]

[0145] Among them, f(h,r,t) represents the scoring function of the triple (h,r,t), Re is the real part of the result, is the complex conjugate embedding of the tail node; is the complex inner product, is the value of the complex vector in the kth dimension.

[0146] By expanding the complex inner product, the above formula can be expressed as:

[0147]

[0148] For normal data, the triple of similarity relations ( resemblance, ) will have a higher f(h,r,t) value, and the dissimilarity relationship ( Not similar, ) will have a lower value.

[0149] For weak similarity modal data, the f(h,r,t) value is close to zero, indicating that the similarity between the node pairs is weak.

[0150] For non-similar modal data, the f(h,r,t) value deviates significantly from the expected range.

[0151] For the mismatch between the corresponding relationship embedding r and the node embedding h, t caused by non-similar modal data, which leads to significant deviation in the calculation of the score f(h, r, t), the negative log-likelihood loss is used. Optimize the triple score f(h,r,t), that is:

[0152]

[0153] Where τ is the set of positive triplets, τ′ is the set of negative triplets generated by negative sampling, and (h′, r′, t′) is a negative triplet.

[0154] S6), enhancing the correlation between weakly similar modal data and regional leaves through small models;

[0155] The primary purpose of small model scoring is to further enhance the accuracy of fault prediction and detection within the context of regional correlations and environmental factors. Small models are more simplified, enabling greater focus on fault detection within specific areas or small regions, reducing computational complexity. Compared to complex models, they can capture both detailed information and overall system relationships.

[0156] For the complex association between a single fan blade and the fan blades in a region, the scoring can be extended by a small model. The association score of the blades in the region is expressed as:

[0157]

[0158] Where, represents the association score; Represents leaf node b respectively i and b j Embedded representation of h i ,h j is the head node of nodes i and j; w is the weight of the small model; represents the transposition operation; F is the activation function;

[0159] By using the negative log-likelihood loss of the improved Complex model Adding regional and environmental association constraints, we get the extended total loss function

[0160]

[0161] Among them, λ is the weight coefficient, which is used to control the contribution ratio of the small model. is the regional association loss;

[0162] Smaller expanded total loss function It means that the small model can not only correctly capture the similarity and dissimilarity relationship between leaves, but also better model the correlation characteristics between leaves and regional environment; the larger extended total loss function This indicates that the small model fails to fit the data effectively, possibly because the embedding representation is inaccurate or the regional correlation between leaves is not correctly captured.

[0163] The regional correlation loss Expressed as:

[0164]

[0165] Where, t i,j Indicates the true label of nodes i and j; used to represent node b i and b j The similarity relationship between Represents a set of leaf node pairs, including the indexes of all leaf node pairs that need to be calculated;

[0166] when Value and true label t i,j When the difference is small, it means that the small model models the relationship between leaves in the region better; if the difference is large, it means that the small model fails to effectively capture the regional relationship between leaves.

[0167] S7) Identify potential failures of blades in the region based on the regional association matrix and issue an early warning.

[0168] For multiple wind turbine blades in the entire area, the regional correlation matrix is ​​expressed as:

[0169]

[0170] The expression for detecting potential fault anomaly of blades in the region is:

[0171]

[0172] Where, Indicates leaf b i Fault warning result, h i It is leaf b i The implicit feature vector of , σ is the activation function, w is the weight of the model, and T represents the transposition operation.

[0173] The fault warning judgment formula is as follows:

[0174]

[0175] Where θ fault is the fault threshold. When the prediction score exceeds this threshold, it means that the blade is faulty. normalIt is the threshold of normal state, indicating that the blade is in normal working condition.

[0176] The above embodiments and descriptions are only for explaining the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, which shall fall within the scope of the invention to be protected.

Claims

1. A method for detecting blade faults in offshore wind turbines based on multimodal data and heterogeneous graphs, characterized in that: The steps include: S1) Obtain the blades of offshore wind turbines Multimodal data of multiple modalities; S2), by jointly embedding learning Multimodal data are fused and processed; To extract high-order feature information of wind turbine blades and improve the correlation and consistency between modal data through a joint loss function; And introduce similarity threshold Dynamically adjust joint embedding learning; Multimodal data analysis through similarity scoring Marking: multimodal data with a similarity score greater than a first similarity threshold is marked as "normal" and used as high-similarity modal data; Multimodal data with a similarity score greater than the second similarity threshold and less than or equal to the first similarity threshold are marked as "potential anomalies" and used as weakly similar modal data; Marking multimodal data with a similarity score less than a second similarity threshold as "abnormal" and treating it as non-similar modal data; Then the marked information is used as the new modal data , and the new modal data Added to the original multimodal data, the final multimodal data is expressed as: ； Through joint embedding learning, all multimodal data are mapped into a unified low-dimensional embedding space to generate a unified embedding representation. ,Right now: ； Where, It is The embedding representation corresponding to each modality is The dimensionality of the embedding space; Represents modal data Embedded representation of S3), constructing a heterogeneous graph of wind turbine blades; S4) Represent the nodes and edges of the heterogeneous graph as triples ,in, is the head node, Represents the tail node; To express a relationship; S5) Using the improved Complex model to score the triples of heterogeneous graphs to determine the authenticity of the triples; S6), enhancing the correlation between weakly similar modal data and regional leaves through small models; S7) Identify potential failures of blades in the region based on the regional association matrix and issue an early warning.

2. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1 is characterized in that: In step S2), the joint loss function Expressed as: Where, is the intra-modal consistency loss; is the inter-modal correlation loss, is the similarity labeling loss, which is used to strengthen the distinction between labeled data. is the weight hyperparameter of the loss function, To control the weight of similarity loss; Based on the joint loss function Optimized embedding representation Expressed as .

3. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 2 is characterized in that: In step S2), for the modal data , the intra-modal consistency loss Expressed as: Where, is the intra-modal consistency loss; For modal The feature encoding function of No. modal data; It is The embedding representation corresponding to each modality data; represents the weight factor used to balance the consistency goal between the original modality and the labeled modality; is the Frobenius norm, which is used to measure the difference between matrices; For the embedding representation of similarity tags, by converting the labels "normal", "potential abnormality", and "abnormal" into low-dimensional vector representations, the vectors corresponding to the similarity tags in the embedding space can reflect the similarity or difference between different data categories; Modality data representing similarity tags Embedding representation; the optimization goal is to constrain and distance, ensuring that the modality representation in the embedding space can reflect the influence of similarity markers.

4. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graphs according to claim 2 is characterized in that: In step S2), the inter-modal correlation loss Expressed as: in, are the weight hyperparameters for high similarity modal data, weak similarity modal data, and non-similar modal data respectively; Indicates the loss of high similarity modal data; is the loss of weakly similar modal data and non-similar modal data.

5. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 4 is characterized in that: The loss of high similarity modal data Expressed as: The loss of weak similarity modal data and non-similar modal data Expressed as: Where, For modal The embedding representation of For modal The embedding representation of For modal and modal The similarity between embeddings,the optimization goal is to maximize the similarity between high-similarity modal data and reduce the similarity between weakly similar modal data and non-similar modal data,,distinguishing data categories.

6. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1, characterized in that: In step S3), the heterogeneous graph of the wind turbine blades Expressed as: ； ； ； in, Represents a collection of nodes; represents an edge set; Represents a leaf node; Wind turbine node; for Environment node; is a regional node; For sensor nodes, , indicating that the fault is damage related; It is an early warning node, which represents early warning information from system monitoring and data analysis.

7. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graph according to claim 1, characterized in that: In step S5), the scoring expression of the improved Complex model is: in, Represents a triple The scoring function, To take the real part of the result, is the complex conjugate embedding of the tail node; is the complex inner product, is the complex vector in The value of the dimension; And through the negative log-likelihood loss Optimizing triple scoring ,Right now: Where, is a set of positive triples, The set of negative triplets generated by negative sampling, is a negative triplet.

8. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graphs according to claim 1, characterized in that: In step S6), the correlation score of the leaves in the region is expressed as: ； Where S represents the association score; 、 Represents leaf nodes and Embedded representation of For nodes and The head node of is the small model weight; Represents a transpose operation; is the activation function; By using the negative log-likelihood loss of the improved Complex model Adding regional and environmental association constraints, we get the extended total loss function ; ； in, is the weight coefficient, which is used to control the contribution ratio of the small model. is the regional correlation loss; The regional correlation loss Expressed as: ； Where, Represents a group leaf node pair; Representation node The true label of the node and The similarity relationship between them.

9. The offshore wind turbine blade fault detection method based on multimodal data and heterogeneous graphs according to claim 1, characterized in that: In step S7), the expression for detecting potential fault anomalies of blades in the region is: Where, Indicates leaves Fault warning results, It's a leaf The hidden eigenvector of is the activation function, is the weight of the model, Represents a transpose operation; The fault warning judgment formula is as follows: Where, is the fault threshold. When the prediction score exceeds this threshold, it indicates that the blade is faulty. It is the threshold of normal state, indicating that the blade is in normal working condition.

Images