Offshore wind turbine monitoring data recovery method and device based on multi-modal structure

Abstract

The invention provides an offshore wind turbine monitoring data recovery method and device based on a multi-modal structure, and the method comprises the steps: obtaining to-be-recovered multi-modal monitoring data of an offshore wind turbine, and enabling the to-be-recovered multi-modal monitoring data to have local continuous loss of one or more modal data; according to the time duration of local continuous missing data in the to-be-recovered multi-modal monitoring data, calling one or more data filling models corresponding to periods to carry out data filling on the multi-modal data by utilizing a long-short period aggregation strategy to obtain a complete filling result of the multi-modal data; wherein each data filling model is obtained by training based on sample multi-modal monitoring data containing data missing of different time lengths, actual complete data corresponding to each sample multi-modal monitoring data and predicted complete data. According to the method, feature fusion is carried out on the multi-modal data, the correlation of the features in space and time is extracted, data filling with variable length and long time sequence is carried out by using a long-short period aggregation strategy, and the problem of large error of long time sequence filling is solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent prediction technology, and in particular to a method and apparatus for recovering offshore wind turbine monitoring data based on a multimodal structure. Background Technology

[0002] Offshore wind power systems are supported by a lower tower and foundation structure, with the health of the supporting structure determining the safe and efficient operation of the wind turbine. Offshore wind power systems and their supporting structures are complex and subject to static and dynamic loads from weather, waves, ocean currents, and turbine operation over extended periods. This results in high construction and maintenance costs and low safety. Therefore, an economical, feasible, and accurate wind turbine monitoring system and data analysis and evaluation method are crucial. However, due to the long-term impact of marine environmental factors such as weather, waves, and currents on offshore wind power systems, supporting structures, and health monitoring systems, problems such as sensor and data acquisition instrument malfunctions and data loss are difficult to avoid. Establishing a technical method for repairing and predicting abnormal or missing data based on normal data from other periods can effectively solve the problems of data reliability and integrity in the health status assessment process.

[0003] Currently, the methods used are mainly based on numerical prediction and fitting models. Large computers are used to perform numerical interpolation, fitting and other calculations to complete the repair and prediction of abnormal or missing data, and then assess the health status of the wind power generation system and its supporting structure.

[0004] However, when faced with the problem of long-term data gaps, this method fails to accurately fit long-term missing data due to issues such as inaccurate physical simulation equations and difficulty in numerically solving nonlinear equations. Summary of the Invention

[0005] This invention provides a method and apparatus for recovering offshore wind turbine monitoring data based on a multimodal structure, which solves the problem of inaccurate fitting of long-term missing data in the prior art and achieves accurate filling of long-term missing data.

[0006] This invention provides a method for recovering monitoring data of offshore wind turbines based on a multimodal structure, comprising the following steps:

[0007] Acquire multimodal monitoring data of offshore wind turbines to be restored, wherein the multimodal monitoring data to be restored has local continuous missing data in one or more modes;

[0008] Based on the duration of locally continuous missing data in the multimodal monitoring data to be recovered, a long-short cycle aggregation strategy is used to call one or more data imputation models corresponding to the corresponding cycles to impute the multimodal data, thereby obtaining the complete imputation result of the multimodal data;

[0009] The data imputation models are trained based on multimodal monitoring data of samples with missing data of different time lengths, the actual complete data corresponding to the multimodal monitoring data of each sample, and the predicted complete data.

[0010] The data imputation model is used to extract features from each modal monitoring data to obtain the single-modal feature vector corresponding to each modal monitoring data. The model then performs spatial and temporal correlation feature fusion on each single-modal feature vector to obtain multimodal fusion features. Finally, the multimodal fusion features are used to perform temporal imputation processing on the offshore wind turbine data structure to generate a complete imputation result for the multimodal monitoring data.

[0011] According to the offshore wind turbine monitoring data recovery method based on multimodal structure provided by the present invention, the multimodal monitoring data includes two or more of wind speed, vibration, stress, and deformation, which are obtained by collecting signals from sensors distributed at different locations on the offshore wind turbine.

[0012] According to the offshore wind turbine monitoring data recovery method based on multimodal structure provided by the present invention, the data imputation model includes multiple single-modal feature extraction networks, multimodal feature fusion networks and data imputation networks, wherein the single-modal feature extraction networks and the multimodal feature fusion networks are all trained using the Transformer architecture.

[0013] According to the offshore wind turbine monitoring data recovery method based on multimodal structure provided by the present invention, the training process of the single-modal feature extraction network specifically includes:

[0014] Obtain a multimodal complete dataset of offshore wind turbines, wherein the multimodal complete dataset includes a specific number of single-modal complete data;

[0015] Each of the single-modal complete data is time-position encoded, and the data points in the single-modal complete data are concatenated with the corresponding time-position encoding results to obtain the single-modal time-coded data sequence corresponding to each single-modal complete data.

[0016] Each of the aforementioned single-modal time-coded data sequences is masked using a sliding window of different time lengths k to obtain a single-modal sample dataset containing missing data of different time lengths; wherein, the value of time length k is determined based on the frequency of occurrence of missing data of each time length in the historical single-modal data of offshore wind turbines;

[0017] The Transformer architecture of the unimodal feature extraction network is trained using the unimodal sample dataset and the complete unimodal data to obtain the trained unimodal feature extraction network.

[0018] According to the offshore wind turbine monitoring data recovery method based on multimodal structure provided by the present invention, the step of temporal and position encoding of each single-modal complete data specifically includes:

[0019] Each complete single-modal data point is time-encoded for location. Based on the length of the time-series data within the single-modal data, the absolute location is represented using sine and cosine functions. The relative location is obtained by multiplying the sine and cosine functions.

[0020]

[0021] Where PE represents positional encoding; pos represents the position of a feature in the data sequence; the positional encoding of each feature is a vector embedding, where i represents the index of each element in the vector; d model The dimension of the time location encoding vector;

[0022] The time location matrix corresponding to each complete single-modal data is represented as follows:

[0023]

[0024] in, Each positional encoding vector alternates between sin and cos.

[0025] According to the offshore wind turbine monitoring data recovery method based on a multimodal structure provided by the present invention, the training process of the multimodal feature fusion network and the data imputation network specifically includes:

[0026] The trained BERT network is used to extract features from each single-modal sample data in the single-modal sample dataset to obtain the feature extraction results of each single-modal sample data.

[0027] The spatial position of each sensor is encoded to obtain the spatial position encoding result of each sensor.

[0028] The feature extraction results of each single-modal sample data are connected with the spatial location encoding results of the sensor to obtain the single-modal time-space encoded data sequence corresponding to each single-modal sample data;

[0029] The single-modal time-space encoded data sequences corresponding to each single-modal sample data are input into the Transformer architecture of the multimodal feature fusion network for feature fusion to obtain fused features;

[0030] The fused features are input into the data filling network to obtain the filling result for time length k;

[0031] Based on the complete multimodal dataset and the imputation results of k time length, the multimodal feature fusion network and the data imputation network are trained.

[0032] The offshore wind turbine monitoring data recovery method based on multimodal structure provided by the present invention utilizes a long-short cycle aggregation strategy to call one or more data imputation models corresponding to the cycle to impute the multimodal data, specifically including:

[0033] Based on the duration of local continuous data loss in the multimodal data to be recovered and the pre-trained data imputation models of different types, an objective function is constructed with the goal of minimizing the number of iterations required to predict data loss at a specific duration. This objective function determines the type and number of data imputation models to be invoked. The type of data imputation model indicates the duration period of the missing data targeted during the training process.

[0034] The objective function is solved using a greedy strategy to determine the optimal combination of different types of data imputation models;

[0035] The data imputation of the multimodal data to be recovered is performed in the order of long-period data imputation model to short-period data imputation model, according to the time series from front to back.

[0036] The present invention also provides a data recovery device for offshore wind turbine monitoring based on a multimodal structure, comprising the following modules:

[0037] The acquisition module is used to acquire multimodal monitoring data of offshore wind turbines to be restored, wherein the multimodal monitoring data to be restored has local continuous missing data of one or more modes;

[0038] The data filling module is used to fill in the multimodal data by calling one or more corresponding period data filling models based on the time length of locally continuous missing data in the multimodal monitoring data to be recovered, using a long and short period aggregation strategy, so as to obtain the complete filling result of the multimodal data.

[0039] The data imputation models are trained based on multimodal monitoring data of samples with missing data of different time lengths, the actual complete data corresponding to the multimodal monitoring data of each sample, and the predicted complete data.

[0040] The data imputation model is used to extract features from each modal monitoring data to obtain the single-modal feature vector corresponding to each modal monitoring data. The model then performs spatial and temporal correlation feature fusion on each single-modal feature vector to obtain multimodal fusion features. Finally, the multimodal fusion features are used to perform temporal imputation processing on the offshore wind turbine data structure to generate a complete imputation result for the multimodal monitoring data.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the offshore wind turbine monitoring data recovery method based on the multimodal structure described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the offshore wind turbine monitoring data recovery method based on a multimodal structure as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the offshore wind turbine monitoring data recovery method based on a multimodal structure as described above.

[0044] The present invention provides a method and apparatus for recovering offshore wind turbine monitoring data based on a multimodal structure. This method involves acquiring multimodal monitoring data of an offshore wind turbine to be recovered, wherein one or more modal data have locally continuous missing values. Based on the time length of the locally continuous missing data in the multimodal monitoring data, a long-short cycle aggregation strategy is used to call one or more corresponding period data imputation models to impute the multimodal data, resulting in a complete imputation result. Each data imputation model is trained based on sample multimodal monitoring data containing missing data of different time lengths, the actual complete data corresponding to each sample multimodal monitoring data, and the predicted complete data. The data imputation model is used to extract features from each modal monitoring data to obtain single-modal feature vectors corresponding to each modal monitoring data. Spatial and temporal correlation features are fused from each single-modal feature vector to obtain multimodal fused features. These multimodal fused features are then used for temporal imputation processing of the offshore wind turbine data structure to generate a complete imputation result for the multimodal monitoring data. This invention uses multimodal sensors for feature fusion to solve the problem of multimodal data fusion from different sources; it applies spatial and temporal dimensions for feature alignment to solve the problem of feature correlation in space and time; and it uses a long-short cycle aggregation strategy to perform data imputation of indefinite and long time series to solve the problem of large imputation error in long time series. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1This is a flowchart illustrating the offshore wind turbine monitoring data recovery method based on a multimodal structure provided by the present invention.

[0047] Figure 2 This is a schematic diagram of the data completion model provided by the present invention.

[0048] Figure 3 This is a schematic diagram of the structure of the offshore wind turbine monitoring data recovery device based on a multimodal structure provided by the present invention.

[0049] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] The present invention will now be described in detail with reference to the accompanying drawings. The specific operation methods in the method embodiments can also be applied to the device embodiments or system embodiments. In the description of the present invention, unless otherwise stated, "at least one" includes one or more. "Multiple" refers to two or more. For example, at least one of A, B, and C includes: A existing alone, B existing alone, A and B existing simultaneously, A and C existing simultaneously, B and C existing simultaneously, and A, B, and C existing simultaneously. In the present invention, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0052] The present invention will now be described in detail with reference to specific embodiments.

[0053] In some specific embodiments of the present invention, such as Figure 1 As shown, this solution provides a method for recovering offshore wind turbine monitoring data based on a multimodal structure, including the following steps:

[0054] Step 100: Obtain the multimodal monitoring data of the offshore wind turbine to be restored, wherein the multimodal monitoring data to be restored has local continuous missing data of one or more modes;

[0055] Step 200: Based on the time length of locally continuous missing data in the multimodal monitoring data to be recovered, use a long-short cycle aggregation strategy to call one or more data imputation models corresponding to the corresponding cycle to impute the multimodal data and obtain the complete imputation result of the multimodal data;

[0056] The data imputation models are trained based on multimodal monitoring data of samples with missing data of different time lengths, the actual complete data corresponding to the multimodal monitoring data of each sample, and the predicted complete data.

[0057] The data imputation model is used to extract features from each modal monitoring data to obtain the single-modal feature vector corresponding to each modal monitoring data. The model then performs spatial and temporal correlation feature fusion on each single-modal feature vector to obtain multimodal fusion features. Finally, the multimodal fusion features are used to perform temporal imputation processing on the offshore wind turbine data structure to generate a complete imputation result for the multimodal monitoring data.

[0058] It should be noted that existing data imputation solutions can only use large computers to perform numerical interpolation, fitting and other calculations to complete the repair and prediction of abnormal or missing data. The fitting results are inaccurate for long-term missing data. In addition, the existing data imputation algorithms are relatively simple and cannot be executed in stages, which makes it difficult to meet the requirements of accuracy and high efficiency in the data recovery process.

[0059] Therefore, this invention acquires multimodal monitoring data of offshore wind turbines to be restored, and determines the optimal combination of corresponding data filling models based on the time length of locally continuous missing data in the multimodal monitoring data to be restored, using a long and short period aggregation strategy, to perform data filling of indefinite and long time series, thus solving the problem of large errors in long time series filling.

[0060] The above steps will be explained in detail below through specific embodiments.

[0061] Step 100: Obtain the multimodal monitoring data of the offshore wind turbine to be restored, wherein the multimodal monitoring data to be restored has local continuous missing data of one or more modes;

[0062] In some possible embodiments of the present invention, the multimodal monitoring data includes two or more of wind speed, vibration, stress, and deformation, which are obtained by collecting signals from sensors distributed at different locations on the offshore wind turbine.

[0063] Specifically, this embodiment provides an implementation method for multimodal monitoring data, which includes monitoring data such as wind speed, vibration, stress, and deformation.

[0064] It is worth noting that the multimodal data filling method provided in this embodiment of the invention can be applied to any multimodal data, and is not limited to the monitoring data of the above-mentioned modes such as wind speed, vibration, stress, and deformation. Monitoring data of other modes can be referred to in this embodiment of the invention, and will not be elaborated here.

[0065] In summary, the purpose of this invention is to utilize the Transformer structure to establish correlations between information from multiple sensors. By combining information from different modalities, such as wind speed, vibration, stress, and deformation, information complementarity and context enhancement can be achieved, reducing potential missing or incorrect data from single modalities and thus improving the accuracy of numerical prediction. The Transformer model achieves unified processing of multimodal data by converting data from different modalities into token embeddings in a shared embedding space. This invention's Transformer, through a multi-head attention mechanism, can simultaneously capture information in different representation subspaces, enabling the model to focus on both global and local features.

[0066] Therefore, this embodiment provides multimodal monitoring data of offshore wind turbines, including wind speed, vibration, stress, and deformation. By performing feature fusion on multimodal sensors, the problem of multimodal data fusion from different sources is solved.

[0067] Step 200: Based on the time length of locally continuous missing data in the multimodal monitoring data to be recovered, use a long-short cycle aggregation strategy to call one or more data imputation models corresponding to the corresponding cycle to impute the multimodal data and obtain the complete imputation result of the multimodal data;

[0068] The data imputation models are trained based on multimodal monitoring data of samples with missing data of different time lengths, the actual complete data corresponding to the multimodal monitoring data of each sample, and the predicted complete data.

[0069] The data imputation model is used to extract features from each modal monitoring data to obtain the single-modal feature vector corresponding to each modal monitoring data. The model then performs spatial and temporal correlation feature fusion on each single-modal feature vector to obtain multimodal fusion features. Finally, the multimodal fusion features are used to perform temporal imputation processing on the offshore wind turbine data structure to generate a complete imputation result for the multimodal monitoring data.

[0070] In some possible embodiments of the present invention, such as Figure 2 As shown, the data imputation model includes multiple unimodal feature extraction networks, multimodal feature fusion networks, and data imputation networks. The unimodal feature extraction networks and the multimodal feature fusion networks are all trained using the Transformer architecture.

[0071] Specifically, this embodiment provides an implementation of a data imputation model. By using a Transformer architecture to extract single-modal features, the function of a single-modal feature extraction network is realized. By using a Transformer architecture to fuse multi-modal features, the function of a multi-modal feature fusion network is realized. The data imputation network can be implemented using a Transformer architecture, a fully connected layer, or other data imputation networks.

[0072] In some possible embodiments of the present invention, the training process of the single-modal feature extraction network specifically includes:

[0073] Obtain a multimodal complete dataset of offshore wind turbines, wherein the multimodal complete dataset includes a specific number of single-modal complete data;

[0074] Each of the single-modal complete data is time-position encoded, and the data points in the single-modal complete data are concatenated with the corresponding time-position encoding results to obtain the single-modal time-coded data sequence corresponding to each single-modal complete data.

[0075] Each of the aforementioned single-modal time-coded data sequences is masked using a sliding window of different time lengths k to obtain a single-modal sample dataset containing missing data of different time lengths; wherein, the value of time length k is determined based on the frequency of occurrence of missing data of each time length in the historical single-modal data of offshore wind turbines;

[0076] The Transformer architecture of the unimodal feature extraction network is trained using the unimodal sample dataset and the complete unimodal data to obtain the trained unimodal feature extraction network.

[0077] Specifically, this embodiment provides an implementation method for training a single-modal feature extraction network. By acquiring complete multimodal data, a multimodal sample dataset containing missing data of different time lengths is obtained through masking. Based on the multimodal sample dataset and the complete multimodal data, the Transformer architecture of the single-modal feature extraction network is trained, specifically the BERT network in the Transformer architecture, to obtain the trained single-modal feature extraction network.

[0078] Specifically, a fixed-size window is applied to the data, and some form of masking operation is performed on the data within the window. The window slides across the data sequence, usually continuously, with each window covering k time units in the sequence.

[0079] In a possible implementation, a framework for a self-supervised learning method for multimodal data embedding space is proposed for a single-modal feature extraction network, specifically including the following steps:

[0080] Step 1: Normalization process:

[0081] In this step, unsupervised training models for each modality are trained to extract features from each modality. That is, unsupervised pre-training is based on the transformer model. Given an unlabeled sequence, the data are normalized by dividing the original input data by the maximum and minimum values ​​and processing them to [0,1].

[0082] Step 2, Time Location Encoding:

[0083] In this step, each sequence data is encoded according to its time position.

[0084] Specifically, in certain models, especially when dealing with sequential data, positional encoding is information added to the input data to help the model understand the positional relationships between elements in the sequence. Positional encoding can be a simple numerical sequence (such as 1, 2, 3...) or a more complex encoding method, such as a combination of sine and cosine functions.

[0085] Furthermore, "concatting" the input data and positional encoding together means linking them sequentially. This involves stacking different vectors together. For example, in this embodiment, the embedding vector of a feature might be concatenated with a positional encoding vector so that the model not only knows the meaning of each feature but also its position within the entire data, thus helping the model better understand the structure and context of the data.

[0086] In a possible embodiment, the step of temporal location encoding of each of the single-modal complete data specifically includes:

[0087] Each complete single-modal data point is time-encoded for location. Based on the length of the time-series data within the single-modal data, the absolute location is represented using sine and cosine functions. The relative location is obtained by multiplying the sine and cosine functions.

[0088]

[0089] Where PE represents positional encoding; pos represents the position of a feature in the data sequence; the positional encoding of each feature is a vector embedding, where i represents the index of each element in the vector; d model The dimension of the time location encoding vector;

[0090] The time location matrix corresponding to each complete single-modal data is represented as follows:

[0091]

[0092] in, Each positional encoding vector alternates between sin and cos.

[0093] Step 3: Connect via concat:

[0094] In this step, the input data and location codes are concatenated using a concat operation.

[0095] Step 4: Masking operation:

[0096] In this step, the input data needs to be masked using a sliding window of fixed length k (e.g., 1 hour, 3 hours, 6 hours, 24 hours) to simulate data loss, thus obtaining the final input embedding of the model.

[0097] Step 5, Training:

[0098] The sample dataset used during training is the input embedding of the final model, while the self-supervised labels used during training are the raw data without masks. Both are fed into the BERT network to learn the correlations within the data and complete the training of the feature extraction model.

[0099] In some possible embodiments of the present invention, the training process of the multimodal feature fusion network and the data imputation network specifically includes:

[0100] The trained BERT network is used to extract features from each single-modal sample data in the single-modal sample dataset to obtain the feature extraction results of each single-modal sample data.

[0101] The spatial position of each sensor is encoded to obtain the spatial position encoding result of each sensor.

[0102] The feature extraction results of each single-modal sample data are connected with the spatial location encoding results of the sensor to obtain the single-modal time-space encoded data sequence corresponding to each single-modal sample data;

[0103] The single-modal time-space encoded data sequences corresponding to each single-modal sample data are input into the Transformer architecture of the multimodal feature fusion network for feature fusion to obtain fused features;

[0104] The fused features are input into the filling network to obtain a filling result of time length k;

[0105] Based on the complete multimodal dataset and the imputation results of k time length, the multimodal feature fusion network and the data imputation network are trained.

[0106] Specifically, this embodiment provides an implementation method for training the multimodal feature fusion network and the data imputation network. The multimodal feature fusion network is used to fuse the single-modal feature extraction results, and the data imputation network is used to fill in the missing data according to the fused features.

[0107] Specifically, Transformer uses position-time encoding to preserve the order information of data in sensor sequences at different locations. Through a self-attention mechanism, it allows the model to calculate the correlation between the data and other time points in the sequence, as well as spatial correlation. This is helpful for predicting long-term, variable-length data. Therefore, this embodiment applies Transformer to the spatial and temporal dimensions to perform feature alignment and address the issue of feature correlation in space and time.

[0108] In a possible implementation, feature extraction networks for each modality are trained, and correlation learning between multimodalities is achieved using a transformer and spatial location encoding. The transformer can handle any combination of input modalities, which are then projected into a joint multimodal embedding space. Internally, the transformer allows each modality to interact with each other. Considering every possible combination of input modalities, the proposed architecture is trained using a combined contrastive loss.

[0109] Furthermore, since wind speed, deformation, and vibration levels vary depending on the sensor's location, a spatial location encoding is needed to incorporate prior knowledge such as the sensor's specific location into the deep learning network. Similar to temporal location encoding, we obtain the location encoding vector PE based on the sensor's location coordinates (x, y, z). i PE i =[x i ,y i ,z i The length of the feature vector is the same. The multimodal feature extraction network is trained by combining the features extracted by the feature extraction networks of each modality through the concat operation.

[0110] In summary, the goal of multimodal networks is to learn projection functions from multiple modalities into a joint embedding space, such that semantically similar inputs are close to each other. For example, the projection of wind speed data should approximate the projection of deformation data. This paper primarily considers three modalities: wind speed, deformation, and vibration, but it's important to note that the proposed method can be extended to more modalities. This method jointly processes any number of modalities and allows modalities to interact with each other. This embodiment proposes that the design and training of the fusion Transformer should consider three aspects of multimodal learning:

[0111] Firstly, it allows different modalities to interact with each other and learn multimodal correlations;

[0112] Secondly, it is modally independent and handles any possible combination of modal inputs;

[0113] Thirdly, it can handle inputs of any length, although the lengths of different patterns and samples may vary.

[0114] To enable the fused Transformer to solve all these tasks, the general self-attention idea in the Transformer block is followed, and key, query, and value weights are shared across all tokens, independent of their input modalities. In this way, self-attention learns which input tokens to focus on from a single modality and any combination of modalities. To train the model, a combination loss function is proposed that considers the contrastive loss between all possible and available input combinations. For example, in the cases of wind speed, deformation, and vibration, the loss is based not only on each modal embedding but also on pairs of wind speed-deformation, deformation-vibration, and wind speed-vibration combinations.

[0115] See also Figure 2 The model aims to learn representations for any number and combination of input modalities, hoping that the projection function f will learn how to fuse information from multiple modalities to improve the joint embedding representation. Therefore, a multimodal, modality-independent Transformer is proposed, where the input token's key, queries, values, and all further transformations are computationally independent of the modality. The multimodal fusion transformer applies conventional transformer blocks, each consisting of a multi-head self-attention mechanism, an MLP, two preceding LayerNorm transformations, and two residual connections, thus obtaining a feature extraction network for multimodal data correlation.

[0116] In a possible implementation, using a multimodal feature fusion network, we train models with four sliding windows of different lengths, i.e., models with four different time lengths of missing data, namely 1 hour, 3 hours, 6 hours, and 24 hours.

[0117] In some possible embodiments of the present invention, a long-short cycle aggregation strategy is used to call one or more data imputation models corresponding to the corresponding cycles to impute the multimodal data, specifically including:

[0118] Based on the duration of local continuous data loss in the multimodal data to be recovered and the pre-trained data imputation models of different types, an objective function is constructed with the goal of minimizing the number of iterations required to predict data loss at a specific duration. This objective function determines the type and number of data imputation models to be invoked. The type of data imputation model indicates the duration period of the missing data targeted during the training process.

[0119] The objective function is solved using a greedy strategy to determine the optimal combination of different types of data imputation models;

[0120] The data imputation of the multimodal data to be recovered is performed in the order of long-period data imputation model to short-period data imputation model, according to the time series from front to back.

[0121] Specifically, this embodiment provides an implementation method for calling a data imputation model using a long-short cycle aggregation strategy. Based on the length of time the data is missing, an optimal combination of data imputation models is selected through a greedy algorithm, and the multimodal monitoring data is imputed using the optimal combination of data imputation models.

[0122] Specifically, for time series data prediction of indefinite and medium-to-long term, the error of the AI ​​model increases rapidly with the number of iterations as the amount of missing data increases. For example, if 7 days of data are missing, and we predict a time period of 6 hours, the model needs to iterate 7 × 24 / 6 = 28 times. This high number of iterations is time-consuming and leads to significant error accumulation. The accuracy of the 7-day result obtained by iterating 28 times with a 6-hour model is significantly lower than that obtained by iterating 7 times with a 24-hour model. Furthermore, if a 1-hour model is used for 168 iterations, the error exhibits a superlinear upward trend. Therefore, in this embodiment of the invention, a long-short cycle aggregation strategy is used for data prediction of indefinite and long time series to address the problem of large prediction errors in long time series.

[0123] More specifically, to mitigate iteration errors, this invention proposes an effective strategy: sorting the selected data imputation models according to their period length, and then imputing data sequentially from the longest to the shortest period. Under this premise, a greedy algorithm is used to call these models, minimizing the number of iterations required to predict data conditions at a specific time. For example, for a 24-hour prediction, only one call to the 24-hour interval model is needed; while for a 23-hour prediction, three calls to the 6-hour data imputation model, one call to the 3-hour data imputation model, and two calls to the 1-hour data imputation model are required. By using multiple data imputation models with different time intervals (i.e., different periods) to capture different temporal relationships, this strategy not only reduces iteration errors but also avoids the training resource consumption caused by recursive training. During training, the model only needs to use data conditions at a single time point to supervise model training, while existing methods (such as FourCastNet) often supervise data conditions at multiple time points to reduce iteration errors, thus multiplying memory consumption and training time, and reducing the stability of the training process.

[0124] Furthermore, given a defined type and number of data imputation models and an appropriate imputation order, any missing modal data can be filled using fused features output from a multimodal feature fusion network. For example, if missing deformation data from offshore wind turbines exists, features such as wind speed, stress, and deformation can be extracted using a single-modal feature extraction network. The extracted features are then concatenated with the spatial location encoding of the sensors and input into the multimodal feature fusion network to obtain fused features. Finally, these fused features are used to imput the deformation data.

[0125] The above-described configuration in this embodiment utilizes the Transformer structure to establish the correlation between information from multiple sensors. By combining information from different modalities, information complementarity and context enhancement can be achieved, reducing the potential loss or error caused by single-modal data, thereby improving the accuracy of data completion.

[0126] The offshore wind turbine monitoring data recovery method based on multimodal structure provided by this invention performs feature fusion on multimodal sensors to solve the problem of multimodal data fusion from different sources; applies transforer to the spatial and temporal dimensions to perform feature alignment to solve the problem of feature correlation in space and time; and uses a long-short period aggregation strategy to perform data prediction of indefinite and long time series to solve the problem of large prediction errors in long time series.

[0127] The following describes the offshore wind turbine monitoring data recovery device based on a multimodal structure provided by the present invention. The offshore wind turbine monitoring data recovery device based on a multimodal structure described below and the offshore wind turbine monitoring data recovery method based on a multimodal structure described above can be referred to in correspondence.

[0128] In some specific embodiments of the present invention, such as Figure 3 As shown, this solution provides a data recovery device for offshore wind turbine monitoring based on a multimodal structure, comprising:

[0129] The acquisition module 31 is used to acquire the multimodal monitoring data of the offshore wind turbine to be restored, wherein the multimodal monitoring data to be restored has local continuous missing data of one or more modes;

[0130] The data filling module 32 is used to fill in the multimodal data by calling one or more corresponding period data filling models according to the time length of locally continuous missing data in the multimodal monitoring data to be recovered, using a long and short period aggregation strategy, so as to obtain the complete filling result of the multimodal data.

[0131] The data imputation models are trained based on multimodal monitoring data of samples with missing data of different time lengths, the actual complete data corresponding to the multimodal monitoring data of each sample, and the predicted complete data.

[0132] The data imputation model is used to extract features from each modal monitoring data to obtain the single-modal feature vector corresponding to each modal monitoring data. The model then performs spatial and temporal correlation feature fusion on each single-modal feature vector to obtain multimodal fusion features. Finally, the multimodal fusion features are used to perform temporal imputation processing on the offshore wind turbine data structure to generate a complete imputation result for the multimodal monitoring data.

[0133] The offshore wind turbine monitoring data recovery device based on multimodal structure provided in this embodiment of the invention has a similar implementation principle and beneficial effects to the offshore wind turbine monitoring data recovery method based on multimodal structure shown in the above embodiment. Please refer to the implementation principle and beneficial effects of the offshore wind turbine monitoring data recovery method based on multimodal structure shown in the above embodiment, which will not be repeated here.

[0134] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logic instructions in the memory 430 to execute a method for recovering offshore wind turbine monitoring data based on a multimodal structure. This method includes: acquiring multimodal monitoring data of an offshore wind turbine to be recovered, wherein the multimodal monitoring data to be recovered has local continuous missing data in one or more modalities; based on the time length of the locally continuous missing data in the multimodal monitoring data to be recovered, using a long-short cycle aggregation strategy, calling one or more corresponding period data imputation models to impute the multimodal data, obtaining a complete imputation result for the multimodal data; wherein each data imputation model is trained based on sample multimodal monitoring data containing data missing at different time lengths, the actual complete data corresponding to each sample multimodal monitoring data, and the predicted complete data; the data imputation model is used to extract features from each modal monitoring data to obtain single-modal feature vectors corresponding to each modal monitoring data, perform spatial and temporal correlation feature fusion on each single-modal feature vector to obtain multimodal fusion features, and perform temporal imputation processing on the multimodal fusion features to generate a complete imputation result for the multimodal monitoring data.

[0135] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the offshore wind turbine monitoring data recovery method based on multimodal structure provided by the above methods. The method includes: acquiring multimodal monitoring data of offshore wind turbines to be recovered, wherein the multimodal monitoring data to be recovered has local continuous missing data in one or more modalities; according to the time length of the locally continuous missing data in the multimodal monitoring data to be recovered, using a long-short cycle aggregation strategy, calling one or more data imputation models corresponding to the corresponding cycles to impute the multimodal data. Modal data is imputed to obtain the complete imputation result of the multimodal data; wherein, each of the data imputation models is trained based on sample multimodal monitoring data with missing data of different time lengths, the actual complete data corresponding to each sample multimodal monitoring data, and the predicted complete data; the data imputation model is used to extract features from each modal monitoring data to obtain the single modal feature vector corresponding to each modal monitoring data, perform spatial and temporal correlation feature fusion on each single modal feature vector to obtain multimodal fusion features, and perform temporal imputation processing on the offshore wind turbine data structure of the multimodal monitoring data to generate the complete imputation result of the multimodal monitoring data.

[0137] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for recovering offshore wind turbine monitoring data based on a multimodal structure provided by the methods described above. This method includes: acquiring multimodal monitoring data of an offshore wind turbine to be recovered, wherein the multimodal monitoring data to be recovered has local continuous missing data in one or more modalities; based on the time length of the locally continuous missing data in the multimodal monitoring data to be recovered, using a long-short cycle aggregation strategy, calling one or more corresponding cycle data imputation models to impute the multimodal data, thereby obtaining the recovered data. The complete incomplete result of multimodal data; wherein, each of the data incomplete models is trained based on multimodal monitoring data of samples with missing data of different time lengths, the actual complete data corresponding to each of the sample multimodal monitoring data, and the predicted complete data; the data incomplete model is used to extract features from each of the modal monitoring data to obtain the single-modal feature vectors corresponding to each modal monitoring data, perform spatial and temporal correlation feature fusion on each single-modal feature vector to obtain multimodal fusion features, and perform temporal incomplete processing on the offshore wind turbine data structure of the multimodal fusion features to generate the complete incomplete result of the multimodal monitoring data.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recovering offshore wind turbine monitoring data based on a multimodal structure, characterized in that: include: Acquire multimodal monitoring data of an offshore wind turbine to be restored, wherein the multimodal monitoring data to be restored has local continuous missing of one or more modal data; According to the time length of local continuous missing data in the multimodal monitoring data to be restored, using the long-short cycle aggregation strategy, calling one or more data filling models of corresponding cycles to fill the multimodal data, and obtaining a complete filling result of the multimodal data; Wherein, each of the data filling models is obtained by training based on sample multimodal monitoring data containing data missing of different time lengths, actual complete data corresponding to each of the sample multimodal monitoring data, and predicted complete data; The data filling model is used to extract features from each modal monitoring data respectively, obtain the single modal feature vector corresponding to each modal monitoring data, perform feature fusion of spatial and temporal correlation on each single modal feature vector, obtain multimodal fusion features, perform time series filling processing on the offshore wind turbine data structure with the multimodal fusion features, and generate a complete filling result of the multimodal monitoring data.

2. The offshore wind turbine monitoring data recovery method based on multi-modal structure according to claim 1 is characterized in that: The multimodal monitoring data includes two or more of wind speed, vibration, stress, and deformation, and the wind speed, vibration, stress, and deformation are obtained by collecting signals from sensors distributed at different positions of offshore wind turbines.

3. The offshore wind turbine monitoring data recovery method based on multi-modal structure according to claim 1 is characterized in that: The data filling model includes multiple unimodal feature extraction networks, a multimodal feature fusion network and a data filling network. The unimodal feature extraction network and the multimodal feature fusion network are both trained using the Transformer architecture.

4. The offshore wind turbine monitoring data recovery method based on multi-modal structure according to claim 3 is characterized in that: The training process of the unimodal feature extraction network specifically includes: Acquire a multimodal complete data set of an offshore wind turbine, wherein the multimodal complete data set includes a specific number of single-modal complete data; Performing time position coding on each of the unimodal complete data, connecting the data points in the unimodal complete data with the corresponding time position coding results, to obtain a unimodal time coded data sequence corresponding to each of the unimodal complete data; Masking each of the unimodal time-coded data sequences according to sliding windows of different time lengths k to obtain unimodal sample data sets containing missing data of different time lengths; wherein the value of the time length k is determined according to the occurrence frequency of missing data of each time length in the historical unimodal data of offshore wind turbines; The Transformer architecture of the unimodal feature extraction network is trained using the unimodal sample data set and each of the unimodal complete data to obtain a trained unimodal feature extraction network.

5. The offshore wind turbine monitoring data recovery method based on multi-modal structure according to claim 4 is characterized in that: The step of performing temporal position encoding on each of the single-modal complete data specifically includes: Each unimodal complete data is position-encoded by time. According to the length of the time series data in the unimodal data, the sine function and cosine function are used to represent the absolute position, and the relative position is obtained by multiplying the sine function and the cosine function: Among them, PE represents position encoding; pos represents the position of the feature in the data sequence; the position encoding of each feature is a vector embedding, i represents the index of each element in the vector; d model Represents the dimension of the temporal position encoding vector; The time position matrix corresponding to each single-modal complete data is expressed as: in, Each position encoding vector is an alternation of sin and cos.

6. The offshore wind turbine monitoring data recovery method based on multi-modal structure according to claim 4 is characterized in that: The training process of the multimodal feature fusion network and the data filling network specifically includes: Performing feature extraction on each unimodal sample data in the unimodal sample data set through the trained BERT network to obtain a feature extraction result of each unimodal sample data; Encoding the spatial position of each sensor to obtain a spatial position encoding result of each sensor; Connecting the feature extraction results of each unimodal sample data with the spatial position encoding results of the sensor to obtain a unimodal time-space encoding data sequence corresponding to each unimodal sample data; The unimodal time-space encoding data sequence corresponding to each unimodal sample data is input into the Transformer architecture of the multimodal feature fusion network for feature fusion to obtain fused features; Input the fused features into the data filling network to obtain a filling result of k time length; Based on the multimodal complete data set and the filling results of k time lengths, the multimodal feature fusion network and the data filling network are trained.

7. The offshore wind turbine monitoring data recovery method based on multi-modal structure according to any one of claims 1 to 6, characterized in that: Using the long-short cycle aggregation strategy, calling one or more corresponding cycle data filling models to fill the multimodal data, specifically including: According to the time length of local continuous missing data in the multimodal data to be restored and different types of pre-trained data filling models, an objective function is constructed with the goal of minimizing the number of iterations for predicting missing data of a specific time length, so as to determine the type and number of data filling models to be called; the type of the data filling model indicates the time length period of the missing data targeted during the training process; The objective function is solved by using a greedy strategy to determine the optimal combination of different types of data filling models; The multimodal data to be restored are filled in the order from the long-period data filling model to the short-period data filling model, and data are filled in from the front to the back according to the time series.

8. An offshore wind turbine monitoring data recovery device based on a multi-modal structure, characterized in that: include: An acquisition module is used to acquire multimodal monitoring data of offshore wind turbines to be restored, wherein the multimodal monitoring data to be restored has local continuous missing of one or more modal data; A filling module is used to fill in the multimodal data according to the time length of the local continuous missing data in the multimodal monitoring data to be restored, using a long-short cycle aggregation strategy, calling one or more data filling models of corresponding cycles to fill in the multimodal data, and obtain a complete filling result of the multimodal data; Wherein, each of the data filling models is obtained by training based on sample multimodal monitoring data containing data missing of different time lengths, actual complete data corresponding to each of the sample multimodal monitoring data, and predicted complete data; The data filling model is used to extract features from each modal monitoring data respectively, obtain the single modal feature vector corresponding to each modal monitoring data, perform feature fusion of spatial and temporal correlation on each single modal feature vector, obtain multimodal fusion features, perform time series filling processing on the offshore wind turbine data structure with the multimodal fusion features, and generate a complete filling result of the multimodal monitoring data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for recovering offshore wind turbine monitoring data based on a multimodal structure according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for recovering offshore wind turbine monitoring data based on a multimodal structure as claimed in any one of claims 1 to 7 is implemented.

Images