A multi-model fusion method for ocean waves based on reinforcement learning

Abstract

The present invention belongs to the technical field of model fusion in marine environment, and proposes a wave multi-model fusion method based on reinforcement learning, including: S1, defining the state, action and reward function of the reinforcement learning model; S2, designing and initializing the reinforcement learning agent; S3, using the experience replay buffer to collect data; S4, sampling data from the experience replay buffer to update the network; S5, dynamically adjusting the model fusion weight to achieve adaptive weighting. The present invention is based on the reinforcement learning algorithm, and dynamically adjusts the weight coefficients of each basic model through interaction and feedback with the environment, so as to achieve the optimal model fusion effect in different marine environments. By defining reasonable states, actions and reward functions, the present invention enables the model to adaptively learn how to optimally combine the outputs of the basic models, ensuring that high prediction accuracy and reliability are always maintained in a complex and dynamic marine environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of model fusion in marine environment, and in particular relates to a wave multi-model fusion method based on reinforcement learning. Background Art

[0002] In the field of marine science and engineering, accurate wave prediction has important application value for offshore operations, coastal disaster prevention and mitigation, and marine engineering design. At present, wave simulation mainly relies on physical models and numerical models, such as the SWAN model, the WAVEWATCH III model, and the WAM model. These models simulate ocean dynamic processes such as wind waves and surges based on different physical parameterization schemes, and are often used for wave forecasting and research in oceans and coastal areas. However, the prediction results of a single model are affected by multiple uncertainties such as the parameterization process of the model itself, boundary conditions, and driving wind fields. Especially in complex coastal and inland sea areas, the errors are more significant. Therefore, multi-model fusion has become an effective way to solve this problem.

[0003] Most of the existing multi-model fusion methods adopt a fixed-weight integration strategy, that is, the weights of each model are determined by multivariate regression analysis or least squares method based on historical data, and then new fusion prediction results are generated. The multi-model super-integration method was first proposed by Krishnamurti et al. in 1999. It is an effective technology to improve prediction accuracy by fusing multiple models. The core idea of ​​this method is to calculate the weights of different models based on the output of multiple numerical models through least squares regression analysis of historical data, so as to make more accurate predictions for the future. The implementation steps are as follows:

[0004] 1. Model selection and data collection: The multi-model hyper-ensemble method first needs to select multiple numerical models. These models can be prediction models based on different physical processes, such as numerical models in the fields of meteorology, ocean waves or hydrology. Each model can make predictions for the same target variable and generate prediction results from multiple sources. Then, the historical data of these models are collected as the basis for weight calculation and learning.

[0005] 2. Weight calculation: For each basic model, the historical performance of the model is evaluated by comparing it with the real observed data. The historical data is divided into a training data set and a prediction data set. During the training phase, the multi-model super-ensemble method uses the least squares method to analyze the errors of different models in historical data, and then calculates the weight of each model. The weight of each basic model is determined based on the accuracy of its historical performance. Models with better performance will be given higher weights, while models with relatively poor performance will be given smaller weights.

[0006] 3. Model ensemble prediction: In the prediction stage, the calculated optimal weights are applied to the prediction outputs of each basic model to generate a multi-model ensemble prediction result.

[0007] This method analyzes the performance of different models in historical data, uses least squares fitting to obtain the optimal weights, and uses these weights for model fusion in the prediction stage. Although this method improves the prediction accuracy compared to the traditional simple average integration, its weights remain fixed throughout the prediction process and lack the ability to adapt to environmental changes. Especially in the case of complex and changeable marine environments, the prediction accuracy and reliability of the model will be significantly reduced.

[0008] In order to overcome the limitations of fixed weights, researchers have tried a variety of improvement strategies, such as using Kalman filtering to dynamically update model weights, or improving the predictive ability of each model through bias correction. However, the Kalman filtering method relies on real-time evaluation of model performance. For ocean models with large computational workloads, real-time calculations and weight updates have high computational costs. In addition, the Kalman filtering method often exhibits insufficient adaptability when dealing with rapidly changing nonlinear dynamic environments.

[0009] In recent years, the application of deep learning methods has gradually gained attention in the field of multi-model fusion. Unlike traditional fixed-weight integration methods, deep learning models can automatically learn the characteristics of each model in different scenarios through training with a large amount of historical data, thereby generating more accurate fusion prediction results. However, the application of deep learning in multi-model fusion also has certain limitations. First, deep learning models are usually trained based on static data sets. Once the learning is completed, their fusion strategy remains unchanged throughout the prediction process, which makes it difficult to dynamically adapt to changes in data distribution. Secondly, when performing multi-model fusion, deep learning models have limited understanding of the dynamic changes of time series data and lack an effective mechanism to adjust the weights of each base model in real time, resulting in their shortcomings in dealing with the non-stationarity and spatiotemporal complexity of the marine environment. Summary of the invention

[0010] In order to solve the problems of insufficient adaptability and prediction error caused by static weights in traditional ensemble methods, the present invention proposes a wave multi-model fusion prediction method based on reinforcement learning, which adopts the following technical solutions:

[0011] A wave multi-model fusion method based on reinforcement learning includes the following steps:

[0012] S1. Define the state, action, and reward functions of the reinforcement learning model

[0013] m Separate models were used F 1,F 2, ..., F m express;

[0014] state s ( t ) is defined as containing the past w The basic model prediction results at each moment and t − w +1 to t Model weight at time:

[0015] ;

[0016] action Defined as the weight distribution of each basic model:

[0017] ;

[0018] Define a reward function based on the relative error between the fusion model and the standard dataset ;

[0019] S2. Design and initialize the reinforcement learning agent

[0020] Initialize the policy network π θ ( a | s ), two value networks Q φ1 ( s , a )and Q φ2 ( s , a ), and two target value networks Q φT1 ( s , a )and Q φT2 ( s , a );

[0021] S3. Collect data using the experience playback buffer

[0022] For each time step, according to the current strategy π θ ( a | s ) generates a series of empirical data ( s , a , r , s '), and store it in the experience replay buffer, where sis the state of the current time step, a and r are the corresponding actions and rewards, s 'For the next state;

[0023] S4, sampling data from the experience replay buffer to update the network;

[0024] S41, randomly extract a batch of data from the experience playback buffer ( s , a , r , s '), used to update the policy network and the value network, by minimizing the following loss function to update the j ( j = 1, 2) Parameters of the value network:

[0025] ;

[0026] in, N is the batch size, Q targeti is the output value of the target value network;

[0027] S42. Update the parameters of the policy network by minimizing the following objective function:

[0028] ;

[0029] S43. Adjust the value of α by minimizing the following loss function:

[0030] ;

[0031] In the formula, represents the target entropy value;

[0032] S44. The parameters of the value network are smoothly transferred to the target value network through the following formula:

[0033] ;

[0034] In the formula, or is the smoothing parameter, f j are the parameters of the value network, f Tj are the parameters of the target value network;

[0035] S45, repeat S41-S42 until the strategy converges;

[0036] S5. Dynamically adjust model fusion weights to achieve adaptive weighting

[0037] The policy network learns how to dynamically adjust the weights of each basic model under different conditions, and weights each basic model to obtain the final integrated prediction model.

[0038] Furthermore, in step S1:

[0039] ;

[0040] in: rErr ( t ) represents the fusion model error ( Err ( t )) and the best model error before fusion ( Err ( t )) is calculated as:

[0041] ;

[0042] in, Err ( t )express t The minimum basic model error at time, norbErr ( t ) is used to standardize the error difference and is defined as:

[0043] ;

[0044] in, β Yes Err ( t ) is the scaling factor for scaling, e is a constant.

[0045] Furthermore, in step S41:

[0046] ;

[0047] in, c is the discount factor, α is the entropy loss weight, α lnπ( a ′∣ s ′) is the entropy term.

[0048] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The present invention is based on a reinforcement learning algorithm, and through interaction and feedback with the environment, the weighting coefficients of each basic model are dynamically adjusted, thereby achieving the optimal model fusion effect in different marine environments. The present invention defines reasonable states, actions, and reward functions, so that the model can adaptively learn how to optimally combine the outputs of the basic models to ensure that high prediction accuracy and reliability are always maintained in a complex and dynamic marine environment. The present invention not only overcomes the limitations of traditional model weighting methods, but also provides a more flexible and intelligent integrated prediction solution, which significantly improves the adaptability and prediction performance to complex marine environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a distribution diagram of the RMSE of the effective wave height of the integrated model according to an embodiment of the present invention;

[0050] Figure 2 This is the RMSE distribution diagram of the SWAN-1 significant wave height according to the embodiment of the present invention;

[0051] Figure 3 This is the RMSE distribution diagram of the SWAN-2 significant wave height according to the embodiment of the present invention;

[0052] Figure 4 It is the relative change rate of RMSE in the embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0054] 1. This embodiment proposes a wave multi-model fusion prediction method based on reinforcement learning, including:

[0055] Step 1: To build a reinforcement learning framework, first define the state, action, and reward function. The state represents the key information related to the task goal in the current environment. In this invention, the state consists of the prediction results of the basic model at several moments in the past and the corresponding model weights. By including historical information, the state can reflect the temporal variation characteristics of the marine environment, which helps the agent make reasonable decisions based on the state.

[0056] Assume that m Independent models F 1, F 2, ..., F m ,exist tAt this moment, the goal of model fusion is to obtain the final integrated prediction by weighting each base model. F '( t ):

[0057] ;

[0058] In the formula, oh k ( t ) represents the model F k ( t )’s weight;

[0059] At the moment t , the state of the reinforcement learning algorithm s ( t ) is defined as containing the past w The basic model prediction results at each moment and t − w +1 to t Model weight at time:

[0060] ;

[0061] Actions are decisions made by the reinforcement learning agent at each time step. The actions in this invention are weight assignments to each base model:

[0062] ;

[0063] The reward function is used to evaluate the quality of each action to guide the agent on how to adjust its strategy. In this invention, the reward function is defined based on the relative error between the fusion model and the standard data set, with the goal of maximizing prediction accuracy. In order to improve the stability of the reward and avoid reward fluctuations caused by errors that are too large or too small, this invention uses a standardized process for the error difference and smoothes the reward through logarithmic transformation to ensure a stable training process:

[0064] ;

[0065] In the formula, Err ( t ) is the fusion model error, Err ( t ) is the minimum error of the basic model, β Yes Err ( t ) is the scaling factor for scaling, e is a small constant used to avoid Err ( t ) When the value is too small, the value may become unstable.

[0066] In the present invention, the reward function is defined in a way that the error difference is standardized to ensure the stability and accuracy of the model training. Its main advantages are:

[0067] 1. The reward function uses logarithmic transformation, the formula is:

[0068] ;

[0069] By performing a logarithmic transformation on the error difference, the error fluctuation can be effectively smoothed, making the reward function change more smoothly and avoiding training instability caused by an error value that is too large or too small.

[0070] 2. When defining the reward function, standardization is introduced:

[0071] ;

[0072] This standardization process ensures that the error measurement in different scenarios is consistent, avoiding the problem of uneven reward scales caused by different scenarios, thereby improving the stability of the training process;

[0073] 3. The goal of the defined reward function is to make the error of the fusion model as small as possible from the minimum error in the base model. By penalizing the error difference in logarithmic form, when the error of the fusion model is large, the agent's reward will be significantly reduced, prompting it to adjust its strategy, thereby gradually reducing the prediction error. This penalty mechanism makes the reward moderately sensitive to the error, which can not only push the model to develop in the direction of reducing the error, but also prevent the training from being unstable due to the instantaneous large fluctuations in the error value.

[0074] Step 2: Initialize the policy network π θ ( a | s ), two value networks Q φ1 ( s , a )and Q φ2 ( s , a ), and two target value networks Q φT1 ( s , a )and Q φT2 ( s , a ).

[0075] Step 3: Generate experience data by interacting with the environment and store it in the experience replay buffer. For each time step, according to the current policyπ θ ( a | s ) generates a series of empirical data ( s , a , r , s ') and stores it in the experience replay buffer. Here s represents the state of the current time step, a and r For the corresponding actions and rewards, s ' is the next state.

[0076] Step 4: Sample data from the buffer to update the network.

[0077] In order to continuously optimize and improve the model, this step updates the parameters of the strategy network and value network based on the SAC algorithm, as follows:

[0078] ① Randomly extract a batch of data from the experience replay buffer ( s , a , r , s '), used to update the policy network and value network. Update the first j ( j = 1, 2) Parameters of the value network:

[0079] ;

[0080] in, N is the batch size, Q targeti The output value of the target value network is used to estimate the value of the current action-state pair. It is calculated by the target value network and used as a target to update the current value network parameters, so that the model can estimate future rewards more accurately. In reinforcement learning, the target value network is a delayed update network, which is mainly used to reduce the estimation bias during training. Q targeti Calculated through the target value network:

[0081] ;

[0082] in, c is the discount factor, α is the entropy loss weight. Entropy term α lnπ( a ′∣ s ′) is to encourage the strategy to be random in its decision making.

[0083] ② Update the parameters of the policy network by minimizing the following objective function:

[0084] ;

[0085] ③Adjust by minimizing the following loss function α Values:

[0086] ;

[0087] In the formula, Represents the target entropy value.

[0088] ④ Smoothly transfer the parameters of the value network to the target value network in the following way:

[0089] ;

[0090] In the formula, or is the smoothing parameter.

[0091] ⑤Repeat ①~④ until the strategy converges.

[0092] Step 5: Dynamically adjust the model fusion weights to achieve adaptive weighting.

[0093] Through the above steps, the policy network learns how to dynamically adjust the weights of each basic model in different states to maximize the accuracy of the fusion prediction. Based on the reinforcement learning framework, the agent can automatically adjust the weights when the environment changes, so that the fusion model can maintain high prediction accuracy under different ocean conditions.

[0094] In the present invention, the reward function is designed to be based on the relative error between the fusion model error and the optimal basic model error. This reward function is not a simple error accumulation or static indicator, but by dynamically comparing the performance of the fusion model and the basic model, combined with the logarithmic transformation of the error, it ensures that the reward can better reflect the actual prediction effect of each time step. The present invention introduces a reinforcement learning strategy in the fusion model, and realizes the dynamic management of the contribution of each basic model by adaptively learning and adjusting the weight of the basic model. Different from the fixed weight method in traditional model fusion, the present invention uses the action in reinforcement learning to dynamically predict the weight of each basic model at a given time step. The distribution of weights is predicted by the policy network in reinforcement learning, and the weights will change in real time as time goes by and the model continues to learn. This mechanism enables the model to flexibly adjust the contribution of each basic model under different marine environmental conditions, thereby improving the overall prediction accuracy. The above design solves the problem that the traditional fixed weight fusion method cannot effectively cope with dynamic marine environmental changes, thereby achieving the effectiveness and accuracy of multi-model fusion in complex environments.

[0095] 2. The following is a detailed introduction with examples.

[0096] The significant wave height data simulated by the SWAN models established by Yang et al. (2022) and Gong et al. (2022) (denoted as SWAN-1 and SWAN-2, respectively) are used as two basic models, and the significant wave height data in the ERA5 reanalysis dataset are used as benchmark data.

[0097] SWAN-1 used measured data to conduct a sensitivity analysis of the model parameters, focusing on wind energy input, white cap dissipation and energy dissipation caused by bottom friction, and thus optimized and corrected the model. The model uses an unstructured grid, with a calculation time step of 10 minutes, and outputs node wind and wave parameters hourly.

[0098] SWAN-2 takes the influence of typhoons into consideration during the simulation process. By superimposing and coupling the wind field generated by the typhoon model with the CFSR / CFSv2 reanalysis wind field according to a certain weight coefficient, the reconstructed wind field can accurately describe the wind conditions near the typhoon center, and can also fully reflect the wind conditions in the periphery of the typhoon and in normal weather. The calculation area of ​​the model adopts a two-layer nested structure, the time resolution of the model is 20 minutes, and the simulation results are output every 3 hours.

[0099] 1. Adjust the temporal and spatial consistency of basic model data and benchmark data.

[0100] In order to ensure the consistency of the three data, the data of the two SWAN models were first interpolated in spatial scale to match the grid of ERA5 data. In terms of temporal scale, the data at 0:00, 3:00, 6:00, 12:00, 18:00 and 21:00 every day were selected for subsequent research. The significant wave height data from 2011 to 2016 were selected, of which the data from 2011 to 2015 were used as the training data set, and the data from 2016 were used as the validation data set.

[0101] 2. Define state, action, and reward functions.

[0102] Two independent SWAN models are used F 1 and F 2 means that at time t , the state of the reinforcement learning algorithm s ( t ) is defined as containing the past w The basic model prediction results at each moment and t − w +1 to t Model weight at time:

[0103] (1)

[0104] In the formula, oh k ( t ) represents the model F k ( t )’s weight.

[0105] The action here is the weight assignment to each base model:

[0106] (2)

[0107] The reward function is defined based on the relative error between the fusion model and the standard dataset:

[0108] (3)

[0109] In the formula, Err ( t ) is the fusion model error, Err ( t ) is the minimum error of the basic model, β Yes Err ( t ) is the scaling factor for scaling, e is a small constant used to avoid Err ( t ) When the value is too small, the value may become unstable.

[0110] 3. Initialize the network.

[0111] Policy Network π θ ( a | s ), two value networks Q φ1 ( s , a )and Q φ2 ( s , a ), and two target value networks Q φT1 ( s , a )and Q φT2 ( s , a ) to initialize.

[0112] 4. Interact with the environment to generate experience data and store it in the experience replay buffer.

[0113] For each time step, according to the current strategy π θ ( a |s ) generates a series of empirical data ( s , a , r , s ') and stores it in the experience replay buffer. Here s represents the state of the current time step, a and r For the corresponding actions and rewards, s ' is the next state.

[0114] 5. Sample data from the buffer to update the network.

[0115] ① Randomly extract a batch of data from the experience replay buffer ( s , a , r , s '), minimize the following loss function to update the j ( j = 1, 2) Parameters of the value network:

[0116] (4)

[0117] in, N is the batch size. Q targeti Calculated through the target value network:

[0118] (5)

[0119] in, c is the discount factor, α is the entropy loss weight.

[0120] ② Minimize the following objective function to update the parameters of the policy network:

[0121] (6)

[0122] ③Minimize the following loss function to adjust α Values:

[0123] (7)

[0124] In the formula, Represents the target entropy value.

[0125] ④ Smoothly transfer the parameters of the value network to the target value network:

[0126] (8)

[0127] In the formula, or is the smoothing parameter.

[0128] ⑤Repeat ①~④ until the strategy converges.

[0129] 6. Dynamically adjust model fusion weights to achieve adaptive weighting.

[0130] Through the above steps, the policy network learns how to dynamically adjust the weights of each basic model in different states to maximize the accuracy of the fusion prediction. The final integrated prediction is obtained by weighting each basic model. F '( t ):

[0131] (9)

[0132] 7. Integrated model performance evaluation.

[0133] According to formula (9), the integrated prediction results can be obtained. In order to reflect the advantages of the reinforcement learning method, the root mean square error (RMSE) is used to quantify the differences between the integrated prediction results of 2016 and the prediction results of the two SWAN models and the ERA5 data. The results are respectively Figure 1 , Figure 2 and Figure 3 Each rectangular cell in the figure represents the prediction result of a specific spatial point. By comparison, it can be found that the RMSE value of the integrated prediction result in the entire target area is generally lower than that of the two basic models, which shows that the integrated model can better handle the prediction of effective wave height under various complex sea conditions and meteorological conditions. In addition, the error reduction percentage ((basic model error - integrated model error) / basic model error × 100%) was calculated, and the results are shown in Figure 4 As shown in the figure, it can be observed that the integrated model has a significant RMSE reduction in most areas compared with the two basic models. In some sea areas, the RMSE reduction rate exceeds 40%, which further shows that the integrated model can combine the advantages of the two basic models and effectively reduce the prediction error.

[0134] The above-described embodiments of the present invention do not constitute a limitation on the protection scope of the present invention. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A wave multi-model fusion method based on reinforcement learning, characterized in that: The steps include: S1. Define the state, action, and reward functions of the reinforcement learning model m Separate models were used F 1, F 2, ..., F m express; state s ( t ) is defined as containing the past w The basic model prediction results at each moment and t − w +1 to t Model weight at time: ； action Defined as the weight distribution of each basic model: ； Define a reward function based on the relative error between the fusion model and the standard dataset ; ； in: rErr ( t ) represents the fusion model error Err ( t ) and the best model error before fusion ib ( t ), the calculation formula is: ； in, ib ( t )express t The minimum basic model error at time, norbErr ( t ) is used to standardize the error difference and is defined as: ； in, β Yes ib ( t ) is the scaling factor for scaling, ε is a constant; S2. Design and initialize the reinforcement learning agent Initialize the policy network π ( a | s ), two value networks Q φ1 ( s , a )and Q φ2 ( s , a ), and two target value networks Q φT1 ( s , a )and Q φT2 ( s , a ); S3. Collect data using the experience playback buffer For each time step, according to the current strategy π ( a | s ) generates a series of empirical data ( s , a , r , s '), and store it in the experience replay buffer, where s is the state of the current time step, a and r are the corresponding actions and rewards, s 'For the next state; S4, sampling data from the experience replay buffer to update the network; S41, randomly extract a batch of data from the experience playback buffer ( s , a , r , s '), used to update the policy network and the value network, by minimizing the following loss function to update the j The parameters of the value network, j is 1 or 2: ； in, N is the batch size, Q targeti is the output value of the target value network; ； in, γ is the discount factor, α is the entropy loss weight, α lnπ( a ′∣ s ′) is the entropy term; S42. Update the parameters of the policy network by minimizing the following objective function: ； S43. Adjust the value of α by minimizing the following loss function: ； In the formula, represents the target entropy value; S44. The parameters of the value network are smoothly transferred to the target value network through the following formula: ； In the formula, η is the smoothing parameter, φ j are the parameters of the value network, φ Tj are the parameters of the target value network; S45, repeat S41-S42 until the strategy converges; S5. Dynamically adjust model fusion weights to achieve adaptive weighting The policy network learns how to dynamically adjust the weights of each basic model under different conditions, and weights each basic model to obtain the final integrated prediction model.

Images