A Multi-Innovation OIF Elman Network Modeling Method Based on Measurement Data

Abstract

The application belongs to the technical field of system modeling, and relates to a multi-innovation OIF Elman network modeling method based on measurement data, comprising the following steps: constructing a mathematical model of the OIF Elman network, and obtaining a network to-be-estimated parameter matrix through defining a parameter matrix; introducing a sliding window mechanism, defining a multi-innovation criterion function by using the to-be-estimated parameter matrix, and constructing a parameter estimation sub-algorithm for estimating the weight of the OIF Elman network; adopting an exponential decay and minimum limit mode to establish a dynamic factor adjustment strategy; introducing the dynamic factor adjustment strategy in the parameter estimation sub-algorithm, and constructing a gradient parameter estimation algorithm; estimating the parameter matrix of the OIF Elman network model based on the collected input and output measurement data through the gradient parameter estimation algorithm; and the application is suitable for online modeling and is superior to the existing method in modeling accuracy.

Description

Technical Field

[0001] This invention belongs to the field of system modeling technology, specifically relating to a multi-innovation OIF Elman network modeling method based on measurement data. Background Technology

[0002] Accurately capturing and modeling system dynamics from sequence measurement data is crucial for applications such as state monitoring, soft measurement modeling, and system identification. This necessitates constructing mathematical models that reflect time dependencies and effectively utilize historical information to improve the quality of decision-making based on measurement data. Unlike feedforward neural networks, recurrent neural networks (RNNs) preserve internal states by introducing recurrent connections, facilitating the storage and utilization of historical information when processing sequence measurement data. Among numerous recurrent networks, the Elman network is a representative model. Its main characteristic lies in achieving recurrent feedback of hidden layers through successor layers, thereby capturing time-dependent characteristics. This structure enhances the model's ability to characterize dynamic patterns and is therefore widely used in tasks requiring the processing of time-series measurement data and characterization of system dynamic behavior.

[0003] The classic Elman network has served as the foundation for numerous subsequent structural variations aimed at further enhancing the network's dynamic modeling capabilities. Among these variations, the Output-Input Feedback (OIF) Elman network stands out, evolving from the improved Elman network. The improved Elman network distinguishes itself by introducing self-feedback connections with fixed gains in context nodes, but its feedback paths are limited to hidden layer nodes. Given that feedback information from each layer in a neural network can affect its ability to process dynamic information, the OIF Elman network further develops this structure by introducing feedback information from the output layer, effectively improving the model's sensitivity to historical states. Specifically, the OIF Elman network adds a second receiving layer to store and feed back information from the output layer, forming a dual feedback structure that further enhances the network's ability to process dynamic information.

[0004] In machine learning, training the weights of a neural network is essentially a fitting process to observed samples. In system identification, this process can be viewed as a parameter estimation problem based on measurement data, with the core objective of building a high-precision dynamic model. Parameter estimation is a key factor affecting network modeling performance; accurate parameter estimation not only improves the model's adaptability to noise and uncertainty but also significantly enhances its generalization ability.

[0005] For Elman-type networks, the most widely used parameter estimation method is the gradient method, which typically updates parameter estimates iteratively or recursively. In addition, swarm intelligence algorithms have been used in recent years to solve parameter estimation problems, including the Cuckoo Search algorithm, Sparrow Search algorithm, Particle Swarm Optimization algorithm, and Seagull Optimization algorithm. These methods optimize parameters through a collaborative search mechanism, effectively exploring the solution space and providing new parameter estimation strategies for Elman-type networks.

[0006] While various parameter estimation algorithms have been proposed for Elman-type networks, methods specifically for OIF Elman networks remain scarce. Most existing methods rely on the Stochastic Gradient (SG) algorithm, widely used in neural networks, but this algorithm does not fully consider the unique structural characteristics of OIF Elman networks. Notably, Zhang et al. proposed a Chicken Swarm Optimization (CSO) algorithm for OIF Elman networks. This method simulates the hierarchical division of labor in chicken foraging behavior to perform a global optimization search for network parameters. However, the CSO algorithm is essentially an offline estimation method, relying on batch data processing, which can lead to high computational costs. Furthermore, its ability to dynamically adapt is limited, making it unsuitable for online modeling scenarios requiring frequent parameter updates.

[0007] A method for online estimation of State of Harm (SOH) of power batteries for new energy vehicles (CN104035037A) addresses the following issues: In terms of model structure, the OIF Elman network in this method differs from the traditional Elman network only by adding a support layer. However, the self-connected feedback gain factor (referred to as the self-feedback factor) of the support layer is a fixed value and does not update in real time. In other words, the feedback mechanism of the network in this method has a fixed structure and lacks adaptive adjustment capabilities. Regarding the algorithm, this method employs an adaptive learning rate momentum gradient descent backpropagation algorithm for parameter estimation, i.e., weight training. This algorithm is based on the traditional quadratic error criterion function and fails to fully reflect the dynamic changes in model error, easily getting trapped in local minima, resulting in low network parameter update efficiency or unstable training results.

[0008] Research on System Identification Method Based on Exponential Criterion Function (Lu Lei, Master's Thesis, Hubei University of Technology, 2021). In terms of model structure, this method studies the parameter estimation problem of two linear systems: controlled autoregressive system and finite impulse response moving average system. Its modeling form is a linear difference equation. In terms of algorithm, the algorithm in this method is derived based on the exponential criterion function, and the proposed algorithm is only applicable to linear systems. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-innovation OIF Elman network modeling method based on measurement data.

[0010] To achieve the objectives of this invention, the following technical solutions are adopted.

[0011] A multi-innovation OIF Elman network modeling method based on measurement data includes the following steps:

[0012] S1. Construct a mathematical model of the OIF Elman network based on the OIF Elman network structure, and use the mapping relationship between the input and output of the mathematical model to identify the system and determine the parameters to be estimated in the mathematical model.

[0013] S2. Using the parameters to be estimated as elements, define the parameter matrix and vector to obtain the parameter matrix to be estimated for the OIF Elman network.

[0014] S3. Introduce a sliding window mechanism, use the parameter matrix to be estimated, define a multi-innovation criterion function based on measurement data and weight decay factor, and construct a parameter estimation sub-algorithm for estimating the weights of the OIF Elman network based on the multi-innovation criterion function.

[0015] S4. By adopting the methods of exponential decay and minimum value constraint, a dynamic factor adjustment strategy that can adaptively adjust the self-feedback factor and the weight decay factor is established.

[0016] S5. In the parameter estimation sub-algorithm, a dynamic factor adjustment strategy is introduced to construct a gradient parameter estimation algorithm;

[0017] S6. Based on the collected input and output measurement data, the parameter matrix of the OIF Elman network model is estimated using the gradient parameter estimation algorithm.

[0018] Furthermore, the mathematical model is described as follows:

[0019] Hidden layer:

[0020] (1)

[0021] (2)

[0022] in: It is network input; , , These are the outputs of the hidden layer, the first acceptor layer, and the second acceptor layer, respectively. , , These are the weights connecting the first receiving layer and the hidden layer, the weights connecting the input layer and the hidden layer, and the weights connecting the second receiving layer and the output layer, respectively. It is the intermediate vector, defined as

[0023] The activation function for the hidden layer nodes is the tanh function:

[0024] (3)

[0025] First supporting layer:

[0026] (4)

[0027] in: It is the self-connection feedback gain factor of the first receiving layer, or simply the self-feedback factor;

[0028] Second receiving layer and output layer:

[0029] (5)

[0030] (6)

[0031] in: It is network output; It is the self-feedback factor of the second receiving layer; These are the weights that connect the hidden layer and the output layer.

[0032] Furthermore, the parameter matrix to be estimated is:

[0033] (7)

[0034] In the formula: , , , These are intermediate parameter matrices and vectors.

[0035] Furthermore, the multiple innovation criterion function is:

[0036]

[0037] In the formula: ; It is a weight decay factor, which controls parameter shrinkage through L2 regularization; It is the network at all times The error between the output and the expected output, This is the output vector of the hidden layer of the network.

[0038] Furthermore, the dynamic factor adjustment strategy is expressed as follows:

[0039] (28)

[0040] (29)

[0041] (30)

[0042] in: for The new information of the moment; It is an adjustment factor; It ensures the minimum value of the lower bound of the factor; It is the initial value; It is the decay rate that controls the adaptation speed.

[0043] Furthermore, the gradient parameter estimation algorithm is expressed as follows:

[0044] (38)

[0045] in: By solving optimization problems The obtained step size; It is by The stacked innovation matrix is ​​composed of parameter estimates and observation data obtained at time t.

[0046] Beneficial effects

[0047] This invention constructs a multi-innovation criterion function based on measurement data and weight decay factors; proposes a dynamic factor adjustment strategy to achieve adaptive adjustment of weight decay factors and network self-feedback factors; and proposes a gradient parameter estimation algorithm based on dynamic factors and multi-innovation, which has recursive properties and can realize real-time updates of network parameters during data acquisition. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the structure of the OIF Elman network described in this invention;

[0049] Figure 2 This is a graph showing the change of power load data over time according to the present invention;

[0050] Figure 3 For the purposes of this invention The following is a MAPEs plot on the validation data using the method described below;

[0051] Figure 4 This is a comparison chart of actual and predicted power loads according to the present invention. Detailed Implementation

[0052] The present invention will be further described in conjunction with the embodiments and accompanying drawings.

[0053] Example 1, such as Figure 1 The above describes a method for estimating parameters of a multi-innovation OIF Elman network based on measurement data:

[0054] Step 1, Problem Description:

[0055] Consider a multi-input single-output (OIF) Elman network (output-input feedback Elman network), which consists of... Each input layer node It consists of a hidden layer node and an output node, such as Figure 1 As shown. Its mathematical model is described as follows:

[0056] Hidden layer:

[0057] (1)

[0058] (2)

[0059] in: It is network input; , , These are the outputs of the hidden layer, the first acceptor layer, and the second acceptor layer, respectively. , , These are the weights connecting the first receiving layer and the hidden layer, the weights connecting the input layer and the hidden layer, and the weights connecting the second receiving layer and the output layer, respectively. It is the intermediate vector, defined as

[0060] Here, the tanh function is used as the activation function for the hidden layer nodes.

[0061] (3)

[0062] The tanh function is chosen as the activation function here because it maps the input to the interval (−1, 1), which helps to speed up the model's convergence and improve training stability. Compared to the sigmoid function, it has a larger gradient when the input is close to 0, which can more effectively transmit error signals and mitigate the gradient vanishing problem.

[0063] First supporting layer:

[0064] (4)

[0065] in: It is the self-connection feedback gain factor of the first receiving layer, or simply the self-feedback factor.

[0066] Second receiving layer and output layer:

[0067] (5)

[0068] (6)

[0069] in: It is network output; It is the self-feedback factor of the second receiving layer; These are the weights that connect the hidden layer and the output layer.

[0070] Equations (1)-(6) constitute the OIF Elman network. In machine learning, equations (1)-(6) are considered expressions of the network, where and Indicates the number of nodes. , , , These are the network parameters to be trained. Unlike machine learning, in system identification, we treat the mapping relationship between the input and output of a mathematical model as an expression of the model, where... and Indicates the model order. , , , These are the parameters to be estimated in the model. The purpose of parameter estimation is to estimate unknown parameters using measured input and output data.

[0071] Define parameter matrices and vectors:

[0072]

[0073] The parameters to be estimated in the OIF Elman network can be written as a parameter matrix:

[0074] (7)

[0075] Step 2: Modeling Methods

[0076] The objective of this invention is to propose a novel parameter estimation algorithm for estimating the parameter matrix of an OIF Elman network model. The algorithm utilizes dynamic measurement data. To model.

[0077] Define the output vector of the hidden layer of the network as:

[0078] (8)

[0079] OIF Elman network at time The error between the output and the expected output is:

[0080] (9)

[0081] The stochastic gradient algorithm, widely used in neural networks, minimizes the criterion function. To determine the optimal parameter estimates. Let Representative parameter vector At any moment The parameter estimation recursive update formula for the stochastic gradient algorithm of the OIF Elman network can then be expressed as:

[0082] (10)

[0083] (11)

[0084] in: This represents an information matrix that integrates measurement data with parameter estimates from the previous time step.

[0085] Equations (9)-(11) use only single information To update the parameter estimates of the OIF Elman network, where Representative moment The hidden layer output estimate. This type of method can be classified as a single-innovation correction algorithm because it only relies on the latest innovation term for parameter updates. However, this type of algorithm fails to fully utilize historical information, which may lead to suboptimal parameter estimation results. Furthermore, the self-feedback factor in the network... and The fixed values ​​further limit the network's modeling capabilities. To overcome these limitations and improve the accuracy of parameter estimation, this invention derives a gradient parameter estimation algorithm for OIF Elman networks based on dynamic factors and multiple innovations, starting from the perspective of innovation correction and dynamic penalty.

[0086] To improve the accuracy of parameter estimation, a dynamic window mechanism is introduced, utilizing the nearest window... One measurement data ( (Also representing the length of the innovation), the multi-innovation criterion function with a weighted decay factor is defined as follows:

[0087]

[0088] in: It is a weight decay factor that controls parameter shrinkage through L2 regularization, thereby improving the generalization ability and stability of the model in the gradient-based optimization process.

[0089] Define network input vector and the output vector of layer 1 :

[0090] (12)

[0091] (13)

[0092] Define the stacked error matrix and the stacked output matrix:

[0093] (14)

[0094] (15)

[0095] Introducing a stacked matrix:

[0096] (16)

[0097] in: It is the intermediate vector matrix and

[0098] (17)

[0099] (18)

[0100] beg about The gradient matrix can be obtained by taking the first derivative and using equations (14)-(16).

[0101] (19)

[0102] Similarly, we can obtain information about Gradient vector and matrix:

[0103] (20)

[0104] (twenty one)

[0105] (twenty two)

[0106] in: These are the stacked input matrix of the network, the stacked output vector of the hidden layers, and the stacked output vector of the second successor layer:

[0107] (twenty three)

[0108] (twenty four)

[0109] (25)

[0110] Based on equations (19)-(22), a stacked innovation matrix is ​​introduced.

[0111] (26)

[0112]

[0113]

[0114] Then regarding The gradient matrix can be calculated by the following formula:

[0115] (27)

[0116] Furthermore, this invention also develops a dynamic factor adjustment strategy that utilizes exponential decay and minimum value constraints to ensure the self-feedback factor. and and weight decay factor Adaptive adjustment. For simplicity, let's assume... for The update formulas for the self-feedback factor and the weight decay factor are designed as follows:

[0117] (28)

[0118] (29)

[0119] (30)

[0120] in: It is an adjustment factor; It ensures the minimum value of the lower bound of the factor; It is the initial value; It is the decay rate that controls the adaptation speed.

[0121] make In expression (1) At any moment The estimation. Based on the dynamic self-feedback factors in equations (28) and (29), in equation (4) In formula (5) The estimate can be calculated using the following formula:

[0122] (31)

[0123] (32)

[0124] Therefore, we can obtain the following from equation (24): In formula (15) In equation (25) In equation (8) In equation (13) The estimated value:

[0125] (33)

[0126] (34)

[0127] (35)

[0128] (36)

[0129] (37)

[0130] Based on the above derivation, the minimization is searched using the negative gradient. Using equations (27) and (30), we obtain the parameter estimation algorithm for the OIFElman network based on Dynamic Factor and Multi-Innovation-based Gradient (DFMIG):

[0131] (38)

[0132] in: By solving optimization problems The obtained step size; It is by The stacked innovation matrix is ​​composed of parameter estimates and observation data obtained at time t.

[0133] Example 2, as Figures 2 to 4 The image shows an application example of the method in power load forecasting.

[0134] Electricity load forecasting plays a crucial role in ensuring the safe and stable operation of the power system, and its accuracy directly impacts generation planning, grid dispatch optimization, and electricity market trading decisions. Simultaneously, accurate load forecasting can promote the integration of renewable energy, reduce system operating costs, and is an important technical support for realizing smart grids and the digital transformation of energy. In this example, the performance of the proposed method is evaluated using electricity load data from Wuxi, China, collected between August 1st and 10th, 2018. Figure 2As shown, the dataset is sampled at 15-minute intervals, with 96 data points collected per day, in megawatts (MW).

[0135] A sliding window mechanism is employed, using data from the five consecutive days prior to the prediction date and the time step preceding the current moment on the prediction date as input for prediction. The training, validation, and test sets are dynamically updated daily: in each iteration, the data from the first six days is used for training (with the first five days as input and the sixth day as output), the data from the seventh day is used for validation, and the data from the eighth day is used for testing. Specifically, the load data from August 8th, 9th, and 10th (i.e., days 8, 9, and 10) are used as the test set to evaluate prediction performance. For each day's prediction, the data from the previous five days is used as input to generate the prediction result, thereby verifying the algorithm's ability to handle sequence prediction tasks.

[0136] Because data from the previous five consecutive days is used as the prediction input, the network order is... Fixed at 5. To determine the order. and the length of the new information To determine the optimal value, this study uses electricity load data from August 1st to 8th and evaluates it using the DFMIG algorithm. Tests were conducted... and The predictive performance is evaluated based on all possible combinations within the range of 1 to 15, using the Mean Absolute Percentage Error (MAPE) calculated from validation data (i.e., load data from August 7th). MAPE is defined as follows:

[0137]

[0138] in, Indicates the number of data points. and These represent actual power load and predicted power load, respectively. By comparing different... By combining the corresponding MAPE values, the optimal parameter set is ultimately determined as the parameter combination that minimizes MAPE on the validation data. For example... Figure 3 As shown, when The configuration that yields the minimum MAPE value is determined to be the optimal parameter combination.

[0139] The proposed DFMIG algorithm was used for power load forecasting, and comparative experiments were conducted using the SG and CSO algorithms. The CSO algorithm is an iterative algorithm, while the SG algorithm, like the proposed algorithm, is a recursive algorithm. In addition to MAPE, the evaluation metrics also include Root Mean Square Error (RMSE) and R².

[0140]

[0141]

[0142] Table 1 summarizes the RMSE, MAPE, and R² values ​​of the three algorithms on days 8, 9, and 10. Figure 4 The comparison of prediction results from the three algorithms is shown.

[0143] Table 1. Comparison of RMSE, MAPE, and R² performance indicators for electricity load forecasting

[0144]

[0145] from Figure 4 It can be observed that the proposed DFMIG algorithm closely tracks the actual load curve in the predictions over three consecutive days, significantly outperforming the SG and CSO algorithms. Particularly on day 10, the CSO algorithm exhibits a significant prediction bias. Table 1 shows that the proposed algorithm demonstrates optimal performance across all three evaluation metrics and different test dates. In contrast, the CSO algorithm performs reasonably well on days 8 and 9, but its accuracy drops significantly on day 10, specifically with a sharp increase in RMSE and MAPE values ​​and a significant decrease in R² value. It should be noted that the CSO algorithm is specifically designed for the OIF-Elman network model and is an offline estimation method; its performance is highly dependent on the number of iterations. While increasing the number of iterations can improve prediction accuracy, it incurs a significant computational overhead. In this simulation experiment, the average training time for a single stage of the DFMIG algorithm is 0.0117 seconds, while the SG algorithm is 0.0027 seconds and the CSO algorithm is 2.2544 seconds. Although the SG algorithm and the proposed algorithm both use a recursive mechanism and exhibit relatively stable performance under the sliding window mechanism, their prediction accuracy is significantly lower than that of the DFMIG algorithm on all test days. Overall, the proposed DFMIG algorithm demonstrates a significant advantage in balancing prediction accuracy and computational efficiency.

[0146] This invention addresses the parameter estimation problem of OIF Elman networks within a dynamic system modeling framework based on measurement data, proposing a DFMIG algorithm. This algorithm extends the innovation correction framework by integrating multiple innovation concepts and a penalty mechanism, while also introducing dynamic adjustment strategies for weight decay factors and self-feedback factors, effectively improving the network's dynamic performance. Simulation results verify the superior performance of the DFMIG-based OIF Elman network in measurement data modeling.

[0147] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A multi-innovation OIF Elman network modeling method based on measurement data, characterized in that, Applied to the field of power system load forecasting, it includes the following steps: S1. Based on the OIF Elman network structure, construct an OIF Elman network model, and use the mapping relationship between the input and output of the network model to identify the system and determine the parameters to be estimated; wherein: the network model includes a hidden layer, a first receiving layer, a second receiving layer, and an output layer, the first receiving layer has a first self-feedback factor, and the second receiving layer has a second self-feedback factor; the input is the power load data of the previous five consecutive days; the output is the power load data of the sixth day; S2. Using the parameters to be estimated as elements, define a parameter matrix and a vector to obtain the parameter matrix to be estimated for the network model; S3. Introduce a sliding window mechanism, use the parameter matrix to be estimated to define a multi-innovation criterion function based on power load measurement data and weight decay factor, and construct a parameter estimation sub-algorithm for estimating the weight of the network model based on the multi-innovation criterion function; S4. Employing exponential decay and minimum value constraints, a dynamic factor adjustment strategy is established that can adaptively adjust the first and second self-feedback factors and the weight decay factor; wherein: the dynamic factor adjustment strategy is expressed as: in: for The new information of the moment It is an adjustment factor; It ensures the minimum value of the lower bound of the factor; It is the initial value; It is the decay rate that controls the adaptation speed; S5. In the parameter estimation sub-algorithm, a dynamic factor adjustment strategy is introduced to construct a gradient parameter estimation algorithm; wherein: the gradient parameter estimation algorithm is expressed as: , in: By solving optimization problems The obtained step size; It is by The stacked information matrix is ​​composed of parameter estimates obtained at each time point and the power load measurement data. It is the length of the new information; S6. Based on the collected input and output measurement data, the parameter matrix of the network model is estimated through a gradient parameter estimation algorithm, which is used for the formulation of power generation plans, the optimization of grid dispatch, and the consumption of renewable energy.

2. The multi-innovation OIF Elman network modeling method based on measurement data according to claim 1, characterized in that: The network model is described as follows: Hidden layer: (1) (2) in: It is a network input, which consists of electricity load data from the previous five consecutive days; , , These are the outputs of the hidden layer, the first acceptor layer, and the second acceptor layer, respectively. and It is the model order. , , These are the weights connecting the first receiving layer and the hidden layer, the weights connecting the input layer and the hidden layer, and the weights connecting the second receiving layer and the output layer, respectively. It is the intermediate vector, defined as ; The activation function for the hidden layer nodes is the tanh function: (3) First supporting layer: (4) in: It is the self-connection feedback gain factor of the first receiving layer, or simply the self-feedback factor; Second receiving layer and output layer: (5) (6) in: This is network output; the network output is the power load data for the sixth day. It is the self-feedback factor of the second receiving layer; These are the weights that connect the hidden layer and the output layer.

3. The multi-innovation OIF Elman network modeling method based on measurement data according to claim 2, characterized in that: The parameter matrix to be estimated is: (7) In the formula: , , , These are intermediate parameter matrices and vectors.

4. The multi-innovation OIF Elman network modeling method based on measurement data according to claim 3, characterized in that: The multiple innovation criterion function is: ； In the formula: ; It is a weight decay factor, which controls parameter shrinkage through L2 regularization; It is the network at all times The error between the output and the expected output, This is the output vector of the hidden layer of the network.

Images