A method and system for filling missing data in a marine weather forecasting system

Abstract

The application provides a missing data filling method and system in a marine weather forecast system, and the method comprises the following steps: step 1: obtaining a real-time data set; step 2: selecting an existing data set of the same kind as the real-time data set as training data of a meta-learning process to obtain initial parameters; step 3: establishing an OSELM model by using the real-time data set to obtain a trained a model and initial data flow generated by the a model; step 4: concept drift detection, establishing a b model by using data blocks in a drift detection window d based on the a model, calculating a drift value by using the output weight difference of the a model and the b model after obtaining the b model, comparing the drift value with a specified threshold value, judging whether a concept drift occurs, and outputting a final data flow; and step 5: based on the final data flow filled in step 4, obtaining a complete data flow. The technical scheme of the application solves the problem that the filling method in the prior art cannot adapt to high-speed dynamic data flow.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more particularly to a method and system for filling in missing data in a marine weather forecasting system. Background Technology

[0002] With the continuous upgrading of hardware and the rapid development of the Internet of Things, many big data bureaus have been established in various regions in recent years, and related research has also flourished. Research on marine meteorological data is increasingly data-driven, and the real-time, massive flow of marine meteorological information transmitted from wireless sensors is currently the primary research method. Marine meteorological forecasting can be divided into weather forecasting and climate prediction. Weather forecasting typically predicts changes in atmospheric conditions over a short period, while climate prediction projects possible trends in climate development over a future period. The foundation of marine meteorological forecasting is marine meteorological observation and analysis. By utilizing marine meteorological observation equipment such as weather stations, radar, and satellites, numerical values ​​of marine meteorological elements such as temperature, humidity, air pressure, wind direction and speed, and precipitation are collected and recorded. Then, through the analysis of these observation records, the evolution and changing patterns of the atmospheric system can be understood and grasped. This data can be used to analyze the changing patterns of marine meteorological phenomena, thereby predicting future weather conditions. Furthermore, marine meteorological forecasting also requires the utilization of marine meteorological knowledge, including marine meteorological observation techniques and marine meteorological data analysis methods, to more accurately predict future weather conditions.

[0003] Unlike traditional static data, marine meteorological data streams have the following four characteristics: 1) Spatiotemporal nature: Marine meteorological data is temporal and spatial, and the data changes with time and space. This makes the visualization of marine meteorological data uniquely challenging; 2) Data distribution, i.e., the correlation between data, changes dynamically over time; 3) Large data volume: Marine meteorological data involves a large amount of data, and the maximum value cannot be predicted; 4) Once the data is processed, it cannot be retrieved and processed again unless it is specifically saved.

[0004] Currently, existing missing value imputation algorithms have the following limitations when processing data streams: 1) Many missing value imputation algorithms require a certain amount of time to analyze and impute the data. When processing real-time data streams, this may lead to data delays or inability to process data in a timely manner; 2) Existing algorithms rely heavily on historical data, but when facing dynamic data streams, old data no longer exists when new data arrives; 3) Most existing work is based on the assumption that the correlation between data attributes changes relatively smoothly over time, utilizing the data distribution relationship to impute the data. However, in marine meteorological data, the data distribution and the relationships between attributes change dynamically with different times and environments, even causing changes in the data model, leading to a deterioration in imputation effectiveness. Summary of the Invention

[0005] To address the problems existing in the current technology, this paper provides a method and system for filling missing data in marine meteorological forecasting systems. Due to the rich and diverse nature of marine meteorological data, and considering the sparsity, heterogeneity and flow characteristics of the data, an efficient incremental update method for filling the model is proposed. This solves the problem that existing data filling methods cannot adapt to real-time marine meteorological data streams, realizes the filling of missing values ​​in marine meteorological data streams, and provides a good data foundation for upper-level data analysis.

[0006] The technical means employed in this invention are as follows:

[0007] A method for filling missing data in a marine weather forecasting system includes the following steps:

[0008] Step 1: Real-time data acquisition, given n data sources within the detection area of ​​the sensor network. Based on the selected data source, the data acquired by the sensor is transmitted to the server to obtain a real-time dataset;

[0009] Step 2: Initialize parameters for meta-learning training. Set the network in the meta-learning algorithm to the same settings as the neural network in the OSELM model. Query the complete data tuples in the similarity regression task. Select the existing dataset of the same type as the real-time dataset as the training data for the meta-learning process to obtain the initialization parameters w1, b1.

[0010] Step 3: Build an OSELM model using the real-time dataset. The OSELM model calculates missing values ​​using the complete attribute values ​​in the missing dataset. Incrementally update the OSELM model using the complete data in the real-time dataset at each time step, and impute the missing arrays in the real-time dataset at each time step to obtain the trained model a and the initial data stream generated by model a.

[0011] Step 4: Concept drift detection. Based on model a, model b is built using the data blocks within the drift detection window d. After obtaining model b, the drift value is calculated by the difference in output weights between models a and b and compared with a specified threshold to determine whether concept drift has occurred. If concept drift has occurred, the initialization phase is restarted, step 3 is executed again, and model a is initialized using the data at the first time t after concept drift. After detection, the specified threshold is updated under specified conditions. The updated threshold is used for detection in step 3, and the final data stream is output.

[0012] Step 5: Query the complete data stream. Based on the final data stream completed in Step 4, obtain the complete data stream.

[0013] Furthermore, step 1 specifically includes the following steps:

[0014] Step 1-1: At time t, for data from n data sources... Historical data is integrated and used as the dataset for the current moment;

[0015] Step 1-2: Based on the dataset S obtained in Step 1-1 t Perform attribute partitioning, attribute A m For the missing attribute, the remaining attributes F = {A1, A2, ..., A...} m-1} represents the complete attribute;

[0016] Steps 1-3: Transfer dataset S t Classified as missing dataset and the complete dataset Two parts, missing dataset The complete dataset consists of the missing data tuples arriving at time t. It consists of complete data tuples arriving at time t.

[0017] Furthermore, step 2 specifically includes the following steps:

[0018] Step 2-1: Given a dataset S of the same type as the real-time data stream, randomly select 4 million data blocks from dataset S, each containing 100 data items, to form the training set S. train ={S1,S2,....S n}(0 <n≤4×10 6 Each data item represents an observation at a certain moment, representing a task, and together they form task p(Τ).

[0019] Step 2-2: Randomly initialize the parameters θ of the meta-learning algorithm;

[0020] Steps 2-3: Take task T1 from p(T) as the support set for the inner optimization of the meta-learning algorithm, enter the inner optimization, evaluate the gradient change based on the currently initialized θ value, and then perform a one-step update to obtain the updated θ' value;

[0021] Steps 2-4: Take other tasks T2 from p(T) as the query set for the outer optimization of meta-learning, then calculate the loss value in the query set according to the updated θ', and finally update the θ value according to the sum of the loss values ​​of all tasks.

[0022] Step 2-5: Save the updated training parameters θ from Step 2-4 to obtain the input parameters for training parameters θ, i.e., the initialization parameters w1, b1.

[0023] Furthermore, step 3 specifically includes the following steps:

[0024] Step 3-1: For missing datasets and the complete dataset Perform normalization to reduce the data to the range of (0,1);

[0025] Step 3-2: Establish the OSELM model by initializing the OSELM model using the data block read in at the initial time t;

[0026] Step 3-3: Incrementally update the OSELM model. At each time t, use the new data from time t to update the OSELM model based on the model from time t-1.

[0027] Furthermore, the OSELM model includes an input layer, a hidden layer, and an output layer; step 3-2 specifically includes the following steps:

[0028] Step 3-2-1: Calculate the hidden layer output matrix H0, using the following formula:

[0029]

[0030] Where N0 is the number of data items in the data block;

[0031] Step 3-2-2: Calculate the output weight β (0) The formula is as follows:

[0032]

[0033] β (0) =(K0) -1 H0 T T0 (3)

[0034] Wherein, the input layer matrix is ​​X i The input weight is w i The bias is b i The hidden layer has L nodes, the activation function is g, the regularization coefficient is C, and the target matrix is ​​T. i ;

[0035] Step 3-2-3: Initialize stage parameters:

[0036] X i The input data is an N×d matrix, where N represents the number of samples and d represents the number of features. The hidden layer output matrix H is an N×L matrix. The input weights are a d×L matrix, and the input bias is b. i , a one-dimensional array of size L.

[0037] Furthermore, step 3-3 specifically includes the following steps:

[0038] For t i Calculate the hidden layer output matrix H based on time-sample calculation i ;

[0039] The output weight at time t+1 is calculated recursively using the output weight at time t:

[0040]

[0041] After the online learning phase, in t i At each time step, an output weight β is obtained. (i) Because the data is acquired in real time, the data is divided into missing attribute Am and complete attribute F = {A1, A2, ..., A...}. m-1 Substituting the complete attribute value X of the missing tuple into the OSELM model network structure, we obtain the estimated value Y for the missing attribute. i Let Y be the value at time i;

[0042] Y i =g(X) i )×β (i) (5)

[0043] The missing attribute imputation process and the output weight calculation process are performed simultaneously. When the output weight calculation is completed, the missing attribute values ​​are filled with the complete attribute values ​​from the missing tuples.

[0044] Furthermore, step 4 specifically includes the following steps:

[0045] Step 4-1: Establish model a, using the data S at time t. t Initialize to obtain model a, and obtain the output weights β. a ;

[0046] Step 4-2: Establish model b. The first model b is based on model a at time t, and is incrementally updated using data blocks from time t+d to obtain the output weights β of model b. b The second model b is based on model a at time t+1 and incrementally updated using the data block at time t+d+1. This continues until the d-th model b is based on model a at time t+d-1 ​​and incrementally updated using the data block read in at time t+2d-1. At this point, a detection window is filled, and the sequence B = {b1, b2...b} of model b in detection window d is... d};

[0047] Step 4-3: Calculate the drift value and record the output weight β of model a at time t. a The output weights β of model b are obtained by incrementally updating the data block at time t+d based on model a. b To calculate the difference between the two output weights, the absolute values ​​of the differences between corresponding elements at each position in the matrix are averaged. φ is the weight difference function, and drifted is the drift value. The formula is as follows:

[0048] drifted=φ(βa -β b (6)

[0049] Step 4-4: Dual trigger mechanism. After the first b-model is established, the drift value is calculated as `drifted`. If the drift value is greater than the trigger threshold σ1, counting begins. When a detection window is full, the drift values ​​D = {d1, d2, ..., d} of the d b-models within the counting period are calculated. d} Calculate the 50% quantile. If it is greater than the concept drift approval threshold σ2, then it is determined that the data at time t+d has undergone concept drift.

[0050] Steps 4-5: Threshold adaptive update.

[0051] Furthermore, in steps 4-5, after concept drift occurs, a cooling-off period mechanism is employed to prevent frequent threshold updates, including the following steps:

[0052] At time t, a cooling period of 4 window sizes d has passed, denoted as cooling = 4d. The 80th percentile of 1.5 times the drift value of each data block at historical time [t-3d, td] is used as the new threshold for subsequent detection.

[0053] This invention also provides a missing data imputation system for a marine weather forecasting system, used to implement any of the missing data imputation methods in the above-mentioned marine weather forecasting system, comprising:

[0054] Meta-learning learning layer, data acquisition layer, data imputation layer, and concept drift detection layer;

[0055] The meta-learning learning layer is used to learn the initialization parameter w. i ,b i ;

[0056] The data acquisition layer is used to classify real-time marine meteorological data into complete data tuples and missing data tuples, as well as complete attributes and missing attributes.

[0057] The data imputation layer is used for data analysis of marine meteorological data, including a data preprocessing module, a model initialization module, and an incremental update module;

[0058] The data preprocessing module is used to classify dynamic data as missing or complete, and to perform data normalization operations.

[0059] The model initialization module is used to initialize the OSELM model, calculate the hidden layer matrix H0 at time t0, and output the weights β. (0) ;

[0060] The incremental update module is used to update the model based on the old model at time t. i The output weight β is updated continuously using new data increments. (i);

[0061] The concept drift detection layer is used to judge the model's prediction performance in real-time dynamic data. If concept drift occurs, the model's imputation performance will decrease. To better detect true concept drift, a dual-trigger mechanism is adopted, utilizing the output weights β of models a and b. a ,β b The drift value is calculated based on the difference between the two. If both triggering mechanisms are met, it is determined that a concept drift phenomenon has occurred at that moment. In this case, the model needs to be re-initialized using the data block from the next moment and step 3 is executed.

[0062] The data imputation module classifies the data as missing or complete based on the data acquisition layer, uses the complete data to train and obtain the output weights, and then uses the complete attribute values ​​in the missing tuples as the model input to fill in the missing attribute values.

[0063] Compared with the prior art, the present invention has the following advantages:

[0064] The missing data imputation method and system provided by this invention solve the problem that existing missing value imputation algorithms cannot be applied to marine meteorological data streams. Due to the diversity and large volume of marine meteorological data, considering the data dynamics, heterogeneity, and sparsity, an efficient incremental update method for the imputation model is proposed. In response to the dynamic changes of data streams in different environments and at different times, a concept drift detection and update strategy is proposed to complete the imputation of missing values ​​in marine meteorological data streams. This enables further mining and analysis of stream data, completes data imputation under various marine meteorological environments, and constructs accurate, effective, and high-quality air forecasts.

[0065] Based on the above reasons, this invention can be widely applied in the field of data processing. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of the data filling system structure described in this invention.

[0068] Figure 2 This is a diagram illustrating the missing data tuple filling process in the missing data filling method of the present invention.

[0069] Figure 3 This is a drift detection diagram based on the concept of this invention.

[0070] Figure 4 is a comparison chart before and after data stream imputation using the data imputation method described in this invention. Figure 4a To fill in the previous data plot, Figure 4b This is a data plot after filling in the missing data. Detailed Implementation

[0071] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0072] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0073] like Figure 1-3 As shown, this invention provides a method for filling in missing data in a marine weather forecasting system, comprising the following steps:

[0074] Step 1: Real-time data acquisition, given n data sources within the detection area of ​​the sensor network. Based on the selected data source, the data acquired by the sensor is transmitted to the server to achieve real-time acquisition of the data stream.

[0075] Step 1-1: At time t, for data from n data sources... Historical data is integrated and used as the dataset for the current moment;

[0076] Step 1-2: Based on the dataset S obtained in Step 1-1 t Perform attribute partitioning, attribute A m For the missing attribute, the remaining attributes F = {A1, A2, ..., A...} m-1} represents the complete attribute;

[0077] Steps 1-3: Transfer dataset St Classified as missing dataset and the complete dataset The data consists of two parts, namely the missing data tuple that arrives at time t and the complete data tuple.

[0078] Step 2: Initialize parameters for meta-learning training. Set the meta-learning network to be the same as the neural network settings of the OSELM model. Query complete data tuples in the similarity regression task and select high-quality data as training data for the meta-learning process to obtain the initialization parameters w1, b1.

[0079] Step 2-1: For the collected high-quality dataset S, randomly select 4 million data blocks from dataset S, with each data block containing 100 data items, to form the training set S. train ={S1,S2,....S n}(0 <n≤4×10 6 Each data item represents an observation at a certain moment, representing a task, and together they form task p(Τ).

[0080] Step 2-2: Randomly initialize parameter θ;

[0081] Steps 2-3: Take task T1 from p(T) as the support set for the inner optimization of meta-learning, enter the inner optimization, evaluate the gradient change based on the currently initialized θ value, and then perform a one-step update to obtain the updated θ' value;

[0082] Steps 2-4: Take other tasks T2 from p(T) as the query set for the outer optimization of meta-learning, then calculate the loss value in the query set according to the updated θ', and finally update the θ value according to the sum of the loss values ​​of all tasks.

[0083] Steps 2-5: Fine-tune the parameter θ on the dataset acquired in real time by the present invention to adapt it to the specific task of the present invention.

[0084] Steps 2-6: Save the training parameters θ so that the initialization parameters w1 and b1 can be set during the model initialization process later.

[0085] Step 3: Model initialization: For the acquired real-time data stream containing a large number of missing values, perform imputation model initialization, parameter initialization, incremental update imputation model and related parameter updates;

[0086] Step 3-1: For missing datasets and the complete dataset Perform normalization to reduce the data to the range of (0,1);

[0087] Step 3-2: Initialization phase: Initialize using the data block read in at time t;

[0088] The model used is the OSELM model, which has three layers: an input layer, a hidden layer, and an output layer; the input layer matrix X... i Input weight w i Bias b i Number of hidden layer nodes L, activation function g, regularization coefficient C, target matrix T i (Training session i)

[0089] (1) Calculate the hidden layer output matrix H0, where N0 is the number of data items in the data block.

[0090]

[0091] (2) Calculate the output weight β (0)

[0092]

[0093] β (0) =(K0) -1 H0 T T0 (3)

[0094] (3) Initialization phase parameters

[0095] X i The input data is an N×d matrix, where N represents the number of samples and d represents the number of features. The hidden layer output matrix H is an N×L matrix, and the input weights are w. i Given a matrix of size d×L, with input bias b i , a one-dimensional array of size L.

[0096] Step 3-3: Incremental update of the model. At each time t, the model at time t-1 is incrementally updated using the new data from that time.

[0097] Incremental update phase:

[0098] (1) For t i Calculate the hidden layer output matrix H based on time-sample calculation i (As shown in formula (1) in step 3-2)

[0099] (2) Calculate the output weight at time t+1 by recursively using the output weight at time t:

[0100]

[0101] After the online learning phase, you can... i At each time step, an output weight β is obtained. (i)Because the data is acquired in real time, the data is divided into missing attribute Am and complete attribute F = {A1, A2, ..., A...}. m-1 By substituting the complete attribute value X of the missing tuple into the OSELM model network structure, the estimated value Y for the missing attribute can be obtained.

[0102] Y = g(X) i )×β (i) (5)

[0103] The missing attribute imputation process and the output weight calculation process are performed simultaneously. When the output weight calculation is completed, the missing attribute values ​​are filled with the complete attribute values ​​from the missing tuples.

[0104] Step 4: Concept drift detection. The model established in Step 3 is called model a. Then, model b is established based on model a using the data blocks within the drift detection window d. After obtaining model b, the drift value is calculated by the difference in output weights between models a and b and compared with the threshold to determine whether concept drift has occurred. If concept drift has occurred, the initialization phase is restarted, Step 3 is executed again, and the model is initialized using the data at the first time t after concept drift. After detection, the threshold is updated when certain conditions are met, and the new threshold is used for subsequent detection.

[0105] Step 4-1: Establish model a, using the data S at time t. t Initialize to obtain model a, and obtain the output weights β. a ;

[0106] Step 4-2: Establish model b. The first model b is based on model a at time t, and is incrementally updated using data blocks from time t+d to obtain the output weights β of model b. b The second model b is based on model a at time t+1 and incrementally updated using the data block at time t+d+1. This continues until the d-th model b is based on model a at time t+d-1 ​​and incrementally updated using the data block read in at time t+2d-1. At this point, a detection window is filled, and the sequence B = {b1, b2...b} of model b in detection window d is... d};

[0107] Step 4-3: Calculate the drift value and record the output weight β of model a at time t. a The output weights β of model b are obtained by incrementally updating the data block at time t+d based on model a. b The difference between the two output weights is calculated by averaging the absolute values ​​of the differences between corresponding elements at each position in the matrix. φ is the weight difference function, and drifted is the drift value.

[0108] drifted=φ(β a -βb (6)

[0109] Step 4-4: Dual trigger mechanism. After the first b-model is established, the drift value is calculated as `drifted`. If the drift value is greater than the trigger threshold σ1, counting begins. When a detection window is full, the drift values ​​D = {d1, d2, ..., d} of the d b-models within the counting period are calculated. d} Calculate the 50% quantile. If it is greater than the concept drift approval threshold σ2, then the data at time t+d is determined to have experienced concept drift.

[0110] As shown in Figure 4, at time t=45, with a detection window d=5, the drift value is greater than the trigger threshold σ1=2, and the counting period begins. The drift values ​​from time 45 to 50 are recorded. By calculating the values ​​from time 45 to 50, it is found that the drift value within the detection window is greater than the approval threshold σ2=0.4. At this point, it is determined that the data at time 50 has experienced concept drift. 43 0.2953 44 0.0704 45 4.2590 46 6.7278 47 5.6565 48 4.8165 49 3.8029 51 0.3303 52 0.4448

[0120] This figure shows the drift value changes from time 43 to 52. The drift value changed significantly from time 45 to 49, but returned to its original level after time 50. This indicates that the model detected the concept drift phenomenon at time 50 and initialized the model, making the model stable.

[0121] Steps 4-5: Threshold adaptive update. The setting of the threshold will also affect the detection effect. A lower threshold will trigger more frequently, while a higher threshold will ignore some concept drift points. Therefore, this invention needs to design an adaptive threshold to find the most suitable threshold.

[0122] This invention proposes a "cooling period" denoted as "cooling". At time t, a cooling period of 4 window sizes d has passed (cooling = 4d). The 80th percentile of 1.5 times the drift value of each data block at the historical time [t-3d, td] is used as a new threshold for subsequent detection.

[0123] Step 5: Query the complete data stream. Based on the complete data stream filled in in Step 4, it is used to record real-time marine weather conditions to facilitate subsequent marine weather forecast analysis.

[0124] The present invention also provides a missing data imputation system in a marine weather forecasting system, comprising a meta-learning learning layer, a data acquisition layer, a data imputation layer and a concept drift detection layer;

[0125] The meta-learning learning layer is used to learn the initialization parameter w. i ,b i ;

[0126] The data acquisition layer is used to classify real-time marine meteorological data into complete data tuples and missing data tuples, as well as complete attributes and missing attributes.

[0127] The data imputation layer is used for data analysis of marine meteorological data, including a data preprocessing module, a model initialization module, and an incremental update module;

[0128] The data preprocessing module is used to classify dynamic data as missing or complete, and to perform data normalization operations.

[0129] The model initialization module is used to initialize the OSELM model, calculate the hidden layer matrix H0 at time t0, and output the weights β. (0) ;

[0130] The incremental update module is used to update the model based on the old model at time t. i The output weight β is updated continuously using new data increments. (i) ;

[0131] The concept drift detection layer is used to judge the model's prediction performance in real-time dynamic data. If concept drift occurs, the model's imputation performance will decrease. To better detect true concept drift, a dual-trigger mechanism is adopted, utilizing the output weights β of models a and b. a ,β b The drift value is calculated based on the difference between the two. If both triggering mechanisms are met, it is determined that a concept drift phenomenon has occurred at that moment. In this case, the model needs to be re-initialized using the data block from the next moment and step 3 is executed.

[0132] The data imputation module classifies the data as missing or complete based on the data acquisition layer, uses the complete data to train and obtain the output weights, and then uses the complete attribute values ​​in the missing tuples as the model input to fill in the missing attribute values.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for filling missing data in a marine weather forecasting system, characterized in that, Includes the following steps: Step 1: Real-time data acquisition, given n data sources within the detection area of ​​the sensor network. Based on the selected data source, the data acquired by the sensor is transmitted to the server to obtain a real-time dataset; Step 2: Initialize parameters for meta-learning training. Set the network in the meta-learning algorithm to the same settings as the neural network in the OSELM model. Query complete data tuples in the similarity regression task, and select existing datasets of the same type as the real-time dataset as the training data for the meta-learning process to obtain the initialization parameters. ; Step 3: Build an OSELM model using the real-time dataset. The OSELM model calculates missing values ​​using the complete attribute values ​​in the missing dataset. Incrementally update the OSELM model using the complete data from the real-time dataset at each time step, imputing the missing arrays in the real-time dataset at each time step, to obtain the trained OSELM model. Model and The initial data stream generated by the model; Step 4: Concept drift detection, based on The model uses a drift detection window Data block creation within The model was obtained. After modeling, through The model's output weight differences are used to calculate drift values ​​and compared with a specified threshold to determine if concept drift has occurred. If concept drift has occurred, the initialization phase is restarted, step 3 is executed again, and the first value after concept drift is used. Time data pairs The model is initialized, and after detection, the specified threshold is updated under specified conditions. The updated threshold is used for detection in step 3, and the final data stream is output. Step 4-1: Establish Model, using Time data S t Initialize to obtain The model yields the output weights. ; Step 4-2: Establish Model, the first The model is based on Moment Based on the model, using The data blocks at each time point are incrementally updated to obtain... Model output weights The second one The model is based on Moment Model-based use The data blocks at each time point are incrementally updated until the [time point is reached]. indivual The model is based on Moment Model-based use The data blocks read in continuously are incrementally updated. At this point, the count fills one detection window. The sequence of the model in the detection window d ; Step 4-3: Calculate the drift value and record it. time Model output weights Records Based on the model, using Incremental updates of time data blocks are obtained Model output weights Calculate the difference between the two output weights by averaging the absolute values ​​of the differences between corresponding elements at each position in the matrix. For the weighted difference function, The drift value is given by the following formula: （6） Step 4-4: Dual trigger mechanism, when the first After the model is built, the drift value is calculated. If the drift value is greater than the trigger threshold Then, counting begins. When a detection window is full, the counting period is... indivual Model drift value Calculate the 50% quantile; if it exceeds the concept drift approval threshold... Then determine The concept of time data has shifted; Steps 4-5: Threshold adaptive update; Step 5: Query the complete data stream. Based on the final data stream completed in Step 4, obtain the complete data stream.

2. The method for filling missing data in a marine weather forecasting system according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 1-1: At any moment, for those from Data sources Historical data is integrated and used as the dataset for the current moment; Step 1-2: Based on the dataset obtained in Step 1-1 Perform attribute division, attributes For missing attributes, the remaining attributes For complete attributes; Steps 1-3: Transfer the dataset Classified as missing dataset and the complete dataset Two parts, missing dataset Depend on The dataset consists of missing data tuples arriving at specific times, and the complete dataset. Depend on The complete data tuples that arrive at the time.

3. The method for filling missing data in a marine weather forecasting system according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2-1: There is an existing dataset of the same type as the real-time data stream. From the dataset The training set is composed of 4 million randomly selected data blocks, each containing 100 data items. Each data item represents an observation at a certain moment, representing a task, and together they form a task. ; Step 2-2: Randomly initialize the parameters of the meta-learning algorithm ; Steps 2-3: From Take the task out of the middle As a support set for the inner optimization of the meta-learning algorithm, it enters the inner optimization process based on the currently initialized... The value is evaluated, the gradient change is assessed, and then an update is performed to obtain the updated value. value; Steps 2-4: From Remove other tasks from the middle This serves as the query set for the outer optimization of meta-learning, and then, based on the updated... Calculate in the query set The value is ultimately determined based on all tasks. Update by sum of values Value status; Step 2-5: Update the training parameters from Step 2-4 Save and obtain training parameters The input parameters, i.e., the initialization parameters. .

4. The method for filling missing data in a marine weather forecasting system according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3-1: For missing datasets and the complete dataset Perform normalization to reduce the data to the range of (0,1); Step 3-2: Build the OSELM model, using the initial... The OSELM model is initialized by reading in data blocks at any time. Step 3-3: Incrementally update the OSELM model, each At that moment, The OSELM model is incrementally updated using the new data at time t.

5. The method for filling missing data in a marine weather forecasting system according to claim 4, characterized in that, The OSELM model includes an input layer, a hidden layer, and an output layer; step 3-2 specifically includes the following steps: Step 3-2-1: Calculate the hidden layer output matrix The formula is as follows: （1） in, The number of data items in the data block; Step 3-2-2: Calculate the output weights The formula is as follows: （2） （3） Wherein, the input layer matrix is The input weights are , bias is The number of hidden layer nodes is The activation function is The regularization coefficient is The target matrix is ; Step 3-2-3: Initialize stage parameters: The input data is of size . The matrix, The representative is the sample size. The representative is the characteristic number. The hidden layer output matrix has a size of The matrix has input weights. Size is The matrix, input bias Size is A one-dimensional array.

6. The method for filling missing data in a marine weather forecasting system according to claim 4, characterized in that, Step 3-3 specifically includes the following steps: right Calculate the hidden layer output matrix using time-series samples ; Recursive utilization Time-based output weight calculation Output weights at each time step: （4） After the online learning phase, Obtain an output weight at any time Because the data is acquired in real time, the data is divided into missing attributes. and complete attributes Complete attribute values ​​of missing tuples Substitution The model network structure is used to obtain estimates for the missing attributes. Y , Y i for i Moment Y value; （5） The missing attribute imputation process and the output weight calculation process are performed simultaneously. When the output weight calculation is completed, the missing attribute values ​​are filled with the complete attribute values ​​from the missing tuples.

7. The method for filling missing data in a marine weather forecasting system according to claim 1, characterized in that, In steps 4-5, after concept drift occurs, a cooling-off period mechanism is used to prevent frequent threshold updates, including the following steps: exist The time has passed the size of 4 windows. The cooldown period is denoted as The 80th percentile of 1.5 times the drift value of each data block at the historical time [t-3d, td] is used as a new threshold for subsequent detection.

8. A missing data imputation system for a marine weather forecasting system, used to implement the missing data imputation method in the marine weather forecasting system according to any one of claims 1-7, characterized in that, include: Meta-learning learning layer, data acquisition layer, data imputation layer, and concept drift detection layer; Meta-learning learning layers are used to learn initialization parameters. ; The data acquisition layer is used to classify real-time marine meteorological data into complete data tuples and missing data tuples, as well as complete attributes and missing attributes. The data imputation layer is used for data analysis of marine meteorological data, including a data preprocessing module, a model initialization module, and an incremental update module; The data preprocessing module is used to classify dynamic data as missing or complete, and to perform data normalization operations. The model initialization module is used to initialize the OSELM model and perform calculations. Hidden layer matrix at time step Output weights ; The incremental update module is used to update the model based on the old model. The output weights are constantly updated using new data increments. ; The concept drift detection layer is used to judge the model's prediction performance in real-time dynamic data. If concept drift occurs, the model's imputation performance will decrease. To better detect true concept drift, a dual-trigger mechanism is adopted, utilizing... Model output weights The difference between them is used to calculate the drift value. If both triggering mechanisms are satisfied, it is determined that a concept drift phenomenon has occurred at that moment, and the model needs to be re-initialized using the data block of the next moment, and step 3 is executed. The data imputation module classifies the data as missing or complete based on the data acquisition layer, uses the complete data to train and obtain the output weights, and then uses the complete attribute values ​​in the missing tuples as the model input to fill in the missing attribute values.

Images