A data-driven nonlinear system state estimation method

Abstract

The present disclosure relates to the technical field of optimal estimation, and comprises a data-driven nonlinear system state estimation method. History information of a system state at time t and a to-be-trained policy function model are obtained to determine history information at time t+1; based on a to-be-trained value function model, a first estimated evaluation value corresponding to the history information at time t and a second estimated evaluation value corresponding to the history information at time t+1 are determined; the to-be-trained value function model is iteratively trained based on the first estimated evaluation value, the second estimated evaluation value, a current estimated value at time t, and an actual state value at time t; the to-be-trained policy function model is iteratively trained based on a steady-state estimated evaluation value of the trained value function model corresponding to time t, to obtain a trained policy function model corresponding to time t; and a state estimation value of the system state is determined based on the trained policy function model. The state estimation difficulty can be reduced, and the state estimation accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of optimal estimation, and particularly relates to a data-driven nonlinear system state estimation method. BACKGROUND

[0002] State estimation of dynamic systems, which is used to extract accurate information of system states from observation data possibly contaminated by noise, i.e. filtering, plays a vital role in various engineering fields such as autonomous driving, robotics and aerospace, and is a basic prerequisite for control work. Common filters are usually based on Bayesian filtering, i.e. updating the posterior distribution of system states according to the Bayesian formula, and then obtaining point estimates of system states by applying specific criteria. The pioneering Kalman filter (KF) is a special case of Bayesian filtering, which is optimal under the least mean square error criterion in linear systems with Gaussian noise. For nonlinear systems, the extended Kalman filter (EKF) is one of the most commonly used filtering techniques. This method approximates the nonlinear stochastic state space model by first-order Taylor series expansion to obtain a suboptimal filter. Since the error of the second and higher order terms is ignored, this algorithm may diverge when facing strong nonlinear systems. Another widely used method is the unscented Kalman filter (UKF), which uses a set of sigma points to parameterize the mean and covariance of the posterior distribution, avoiding direct model linearization. However, when there is a significant difference between the system model and the actual dynamics and the noise is asymmetric, the UKF is prone to divergence. Overall, the accuracy of these two nonlinear filtering methods will decrease significantly when facing strong nonlinearities or non-Gaussian noise. Compared to the variants of Kalman filter, the particle filter (PF) based on the principle of Monte Carlo sampling shows better accuracy in the estimation problem of nonlinear non-Gaussian systems. However, this method needs to propagate and resample a large number of particles at each time step, resulting in high computational complexity and limiting its real-time performance. Unlike using particles to approximate the entire posterior distribution, moving horizon estimation (MHE) achieves maximum a posteriori estimation by recursively minimizing an estimation cost function defined on a sliding time window. Similar to PF, MHE improves the estimation accuracy, but its computational burden also increases accordingly, and in the case of high-dimensional state space, the real-time performance is severely limited.

[0003] In recent years, the rapid development of deep learning technology has led to a surge in learning-based filtering methods. One typical approach combines recurrent neural networks (RNNs) with Kalman filters, achieving good performance in numerical experiments. However, these methods often lack rigorous theoretical guarantees, making it difficult to establish the equivalence or approximate equivalence between the obtained filter and the optimal filter. Variational Bayesian filters (VBFFs) propose another deep learning-based approach, utilizing deep neural networks nested in recursive structures to model the variational distribution of process noise conditional on the current observations and previously estimated noise. While this type of method provides a convincing theoretical foundation through variational inference and demonstrates accurate estimation capabilities in practical experiments, its reliance on the Gaussian noise assumption and precise modeling limits its applicability in a wider range of contexts. Another class of methods approximates the Mode-Heterogeneous Hypothesis (MHE) using deep neural networks. However, because these methods primarily focus on estimating the modalities of the posterior state distribution rather than the entire distribution, their accuracy is limited, and solving for the arrival loss during training is difficult to implement in practical applications.

[0004] A common problem with the aforementioned estimation methods is their dependence on the accuracy of the system model and parameters, or at least on a structured model of the system, which is often a characteristic of model-driven estimation. However, the need for explicit and accurate models significantly limits the applicability of such estimation methods to complex systems. In the field of industrial control, there are many systems with highly complex dynamic characteristics. Accurate modeling of such systems is difficult, which leads to the problem of high difficulty and low accuracy in estimating nonlinear systems. Summary of the Invention

[0005] In view of this, this disclosure proposes a data-driven nonlinear system state estimation method, which can solve the problems of high difficulty and low accuracy in traditional nonlinear system estimation.

[0006] According to one aspect of this disclosure, a data-driven method for estimating the state of a nonlinear system is provided, the method comprising:

[0007] The historical information of the system state to be estimated in the nonlinear system at time t is obtained. The historical information includes the state estimates of N historical times before time t and the historical observations of the next time after each historical time. t is a positive integer starting from 0, and N is a preset positive integer.

[0008] Based on the policy function model to be trained, the historical information at time t is used to determine the action at time t, and the current estimated value at time t is obtained;

[0009] Based on the current estimated value of time t and the system observation value of time t+1, determine the historical information of time t+1;

[0010] determine a first estimated evaluation value corresponding to the history information of the time t and a second estimated evaluation value corresponding to the history information of the time t+1 based on the value function model to be trained;

[0011] train the value function model to be trained based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value of the time t, and the actual state value of the time t using a reinforcement learning algorithm to obtain a trained value function model corresponding to the time t;

[0012] train the policy function model to be trained based on the steady-state estimated evaluation value of the trained value function model corresponding to the time t using a reinforcement learning algorithm to obtain a trained policy function model corresponding to the time t;

[0013] After the training of the policy function model is completed, determine the state estimated value of the system state based on the trained policy function model and the system observation value of the nonlinear system.

[0014] In a possible implementation, the training of the value function model to be trained based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value of the time t, and the actual state value of the time t using a reinforcement learning algorithm to obtain a trained value function model corresponding to the time t includes:

[0015] obtain a first loss function corresponding to the value function model to be trained;

[0016] input the first estimated evaluation value, the second estimated evaluation value, the current estimated value of the time t, and the actual state value of the time t into the first loss function to obtain a loss function value of the jth iteration; j is a positive integer starting from 1;

[0017] if the jth iteration does not reach an iteration stop condition, update the model parameters of the value function model to be trained based on a gradient descent method;

[0018] after the parameter update, set the value of j to j+1, and execute again the step of inputting the first estimated evaluation value, the second estimated evaluation value, the history information of the time t, and the actual state value corresponding to the history information of the time t into the first loss function to obtain the loss function value of the jth iteration and the subsequent steps until the jth iteration reaches the iteration stop condition, to obtain the trained value function model corresponding to the time t.

[0019] In a possible implementation, the first loss function is represented by the following formula:

[0020]

[0021] Among them, J critic This represents the first loss function of the value function model "critic" to be trained; This represents the function for calculating the expected value. This represents the historical information corresponding to the time t. The steady-state distribution that the policy function model to be trained can achieve under the current policy π is obeyed; This represents the historical information corresponding to time t+1. The steady-state distribution that the policy function model to be trained can achieve under the current policy π is obeyed; the l t The loss equation is... This represents the historical information corresponding to time t. Under the given conditions, the current policy π of the policy function model to be trained is based on historical information h. t Choose the current estimate at time t as the action π(h) at time t. t The resulting losses; e t The actual state value x at time t is represented by the given value. t The current estimated value at time t The error between; γ represents the preset discount factor; V represents the value function model to be trained; w represents the current model parameters of the value function model to be trained.

[0022] In one possible implementation, the model parameters of the value function model to be trained, updated based on gradient descent, are expressed by the following formula:

[0023]

[0024] in, The first loss function J represents critic Regarding the gradient of the model parameter w, it is calculated by using the first loss function J. critic The partial derivatives with respect to the model parameter w are obtained; α represents the preset learning rate.

[0025] In one possible implementation, where the nonlinear system can be modeled based on a state transition model, the step of iteratively training the policy function model to be trained using a reinforcement learning algorithm based on the steady-state estimate evaluation value of the trained value function model at time t to obtain the trained policy function model at time t includes:

[0026] obtaining a second loss function corresponding to the to-be-trained policy function model, the second loss function being based on minimizing a sum of an error of a current estimation value at the time t and a steady-state estimation evaluation value output by a trained value function model corresponding to the time t;

[0027] inputting the historical information corresponding to the time t+1 into the trained value function model corresponding to the time t to obtain the steady-state estimation evaluation value;

[0028] inputting the current estimation value at the time t, the actual state value at the time t, and the steady-state estimation evaluation value into the second loss function to obtain a loss function value of the kth iteration; k is a positive integer starting from 1;

[0029] updating model parameters of the to-be-trained policy function model based on a gradient descent method in a case where the kth iteration does not reach an iteration stop condition;

[0030] after the parameter updating, setting the value of k to k+1, and performing again the step of inputting the current estimation value at the time t, the actual state value at the time t, and the steady-state estimation evaluation value into the second loss function to obtain the loss function value of the kth iteration and the following steps until the kth iteration reaches the iteration stop condition, to obtain a trained policy function model corresponding to the time t.

[0031] In a possible implementation, the second loss function is represented by the following formula:

[0032]

[0033] wherein J actor represents the second loss function of the to-be-trained policy function model actor; represents an expected value calculation function; represents historical information corresponding to the time t subject to a steady-state distribution that can be reached under a current policy π of the to-be-trained policy function model; represents historical information corresponding to the time t+1 subject to a steady-state distribution that can be reached under the current policy π of the to-be-trained policy function model; l t is a loss equation, represents historical information corresponding to the time t under the condition that the current policy π of the to-be-trained policy function model is used to select a current estimation value at the time t as an action π(h t at the time t according to the historical information h t , resulting in a loss; e tan actual state value x representing the time t t an error between the current estimated value of the time t and γ represents a preset discount factor; a historical information corresponding to the time t+1 a steady-state estimated evaluation value obtained after inputting the trained value function model corresponding to the time t into the model parameter w.

[0034] In a possible implementation, the model parameter of the to-be-trained policy function model is updated based on the gradient descent method, and is represented by the following formula:

[0035]

[0036] wherein, the second loss function J actor the gradient of the model parameter of the to-be-trained policy function model is obtained by calculating the partial derivative of the second loss function J actor with respect to the model parameter ; and β represents a preset learning rate.

[0037] In a possible implementation, when the nonlinear system cannot be modeled based on a state transition model, the steady-state estimated evaluation value obtained by the trained value function model corresponding to the time t is used to iteratively train the to-be-trained policy function model by using a reinforcement learning algorithm to obtain a trained policy function model corresponding to the time t, and the method comprises the following steps:

[0038] obtaining a third loss function corresponding to the to-be-trained policy function model, the third loss function being established based on minimizing the steady-state estimated evaluation value output by the trained value function model corresponding to the time t;

[0039] inputting the historical information corresponding to the time t into the trained value function model corresponding to the time t to obtain the steady-state estimated evaluation value;

[0040] inputting the steady-state estimated evaluation value into the third loss function to obtain a loss function value of the kth iteration; k is a positive integer starting from 1;

[0041] when the kth iteration does not reach an iteration stop condition, updating the model parameter of the to-be-trained policy function model based on the gradient descent method;

[0042] After the parameter is updated, the value of k is equal to k+1, the step of inputting the steady-state estimated evaluation value of the third loss function is executed again, and the subsequent steps are executed until the kth iteration reaches the iteration stopping condition, and the trained policy function model corresponding to the time t is obtained.

[0043] In a possible implementation, the third loss function is represented by the following formula:

[0044]

[0045] wherein J actor represents the third loss function of the policy function model actor to be trained; represents an expected value calculation function; represents the historical information corresponding to the time t obeys the steady-state distribution that can be reached under the current policy π of the policy function model to be trained; represents the historical information corresponding to the time t and the current estimated value at the time t is input into the trained value function model corresponding to the time t with the model parameter w, and the steady-state estimated evaluation value obtained is input into the third loss function;

[0046] Correspondingly, the model parameter of the policy function model to be trained is updated based on the gradient descent method, and is represented by the following formula:

[0047]

[0048] wherein, represents the third loss function J actor the gradient of the model parameter of the policy function model to be trained is obtained by calculating the partial derivative of the third loss function J actor with respect to the model parameter ; and β represents a preset learning rate.

[0049] According to another aspect of the present disclosure, a data-driven nonlinear system state estimation device is provided, comprising a processor, a memory for storing processor-executable instructions, wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0050] According to another aspect of the present disclosure, a non-volatile computer readable storage medium having computer program instructions stored thereon is provided, wherein the computer program instructions are executed by a processor to implement the above method.

[0051] According to another aspect of the present disclosure, there is provided a computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying computer readable code, which when run in a processor of an electronic device, the processor in the electronic device performs the above method.

[0052] By obtaining historical information of a system state to be estimated of a nonlinear system at time t; determining an action at time t using the historical information at time t based on a policy function model to be trained, to obtain a current estimation value at time t; determining historical information at time t+1 based on the current estimation value at time t and a system observation value at time t+1; determining a first estimated evaluation value corresponding to the historical information at time t and a second estimated evaluation value corresponding to the historical information at time t+1 based on the value function model to be trained; iteratively training the value function model to be trained based on the first estimated evaluation value, the second estimated evaluation value, the current estimation value at time t, and an actual state value at time t using a reinforcement learning algorithm, to obtain a trained value function model corresponding to time t; iteratively training the policy function model to be trained based on a steady-state estimated evaluation value of the trained value function model corresponding to time t using the reinforcement learning algorithm, to obtain a trained policy function model corresponding to time t; and determining a state estimation value of the system state based on the trained policy function model using the system observation value of the nonlinear system after the training of the policy function model is completed. The policy function model can be trained offline based on the historical information using the reinforcement learning algorithm, and the obtained policy function model can be used to perform state estimation on the nonlinear system with non-Gaussian noise online, without relying on accurate system modeling, thereby reducing the difficulty of system estimation of the nonlinear system and improving the accuracy of system estimation.

[0053] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0055] Figure 1 A flowchart of a data-driven nonlinear system state estimation method according to an embodiment of the present disclosure is shown;

[0056] Figure 2 A flowchart of a data-driven nonlinear system state estimation method according to another embodiment of the present disclosure is shown;

[0057] Figure 3 A schematic diagram showing the training process of a value function model and a policy function model according to an embodiment of the present disclosure is shown;

[0058] Figure 4 a schematic diagram showing a model training code implementation of an embodiment of the present disclosure;

[0059] Figure 5 a schematic diagram showing a RMSE result comparison of data-driven nonlinear system state estimation of the present application and existing state estimation methods of an embodiment of the present disclosure;

[0060] Figure 6 a block diagram showing a data-driven nonlinear system state estimation apparatus according to an embodiment of the present disclosure;

[0061] Figure 7 a block diagram showing a data-driven nonlinear system state estimation apparatus according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] Various exemplary embodiments, features and aspects of the present disclosure will be explained in detail below with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements or features. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0063] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0064] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated in the present disclosure can be omitted. It will be appreciated by those skilled in the art that the present disclosure can be practiced without these specific details. In some instances, well-known methods, procedures, components, and circuits have not been described in detail since it can obscure the pertinent aspects of the subject matter.

[0065] First, several terms related to the present application are introduced.

[0066] Reinforcement Learning (RL) is a machine learning method that learns decision-making and behavior optimization through interaction between an agent and the environment. In reinforcement learning, the agent explores the environment by trying different actions and adjusts the policy according to the feedback (reward) from the environment to maximize the long-term cumulative reward. Among them, the agent (Agent) and the environment (Environment) are two core components in reinforcement learning.

[0067] Agent: is the decision-maker in the reinforcement learning system, which achieves a certain goal by perceiving the environment and performing actions. The agent can be a robot, a software program, or any entity that can interact with the environment and take actions.

[0068] Environment: The external world in which the agent exists, including everything that the agent can perceive and affect. The environment provides state information to the agent and gives feedback according to the agent's actions. In this application, the agent is a software program, the environment is constructed based on the historical information at each time, and the action indicates the state estimate value at each time. At this time, the agent interacts with the environment through the strategy to determine the state estimate value, and the environment provides the estimated feedback to the agent according to the state estimate value.

[0069] Interaction relationship: The interaction between the agent and the environment is carried out through a cycle process. The agent perceives the current state (s) at each time step, such as the current state is the historical information at time t is the state estimate value at historical time k, y k is the observation value of the system at historical time k. Based on the strategy π selected, an action (a) is selected, such as the action is the current estimate value of the system state at time t ; then execute the action, the environment updates the state and gives the agent an evaluation feedback (r), such as updating the current state to the historical information at time t+1, and feeding back the estimated evaluation value of the action corresponding to the historical information at time t+1. This process is repeated continuously, so that the agent learns the optimal strategy through trial and error to maximize the cumulative reward.

[0070] In reinforcement learning, state, action and reward are the basic elements to describe the interaction between agent and environment.

[0071] The goal of reinforcement learning is to find a strategy that makes the agent get the maximum cumulative reward in the interaction with the environment.

[0072] Policy (π): The rule or function that the agent uses to select actions at each state, which can be deterministic (selecting a fixed action at each state) or stochastic (selecting different actions with certain probabilities at each state). The policy can be represented as π(s) = a, indicating selecting action a at state s, or π(a|s), indicating the probability of selecting action a at state s.

[0073] Illustratively, one algorithmic framework of reinforcement learning includes an actor-critic model, which is a reinforcement learning algorithm based on value function and policy function. The actor (i.e., the policy function model) is responsible for learning the policy function to select actions according to the current state, and the critic (i.e., the value function model) is responsible for learning the value function to evaluate the value of the state estimation of the current state. The actor-critic model continuously optimizes the policy through two steps: policy evaluation (PEV) and policy improvement (PIM).

[0074] In the policy evaluation step, the critic updates the value function according to the value of the current state and action. In the policy improvement step, the actor updates the policy function according to the evaluation result of the critic. The two steps are alternately performed until the model converges.

[0075] Specifically, in the policy evaluation step, the critic uses the value function to evaluate the value of the current state. The value function can be a state value function (V function) or an action value function (Q function). In the policy improvement step, the actor updates the policy function according to the evaluation result of the critic. A commonly used method is to use a policy gradient algorithm to update the policy parameters by maximizing the expected value of cumulative rewards. The policy gradient algorithm can use gradient ascent or gradient descent to update the policy parameters so that the agent selects better actions.

[0076] Since the traditional system estimation method relies on the accuracy of the system model and parameters, or at least relies on the structured model of the system. Based on this, state estimation needs to be transformed from model-driven estimation to data-driven estimation, in which the system dynamics are represented by data samples rather than explicit system models, thereby reducing the dependence on system models. Reinforcement learning (RL) is becoming a typical method for data-driven control problems, in addition, RL can handle high-dimensional problems by using approximation functions with strong fitting ability such as deep neural networks. Reinforcement learning has some application attempts in optimal estimation, but these methods are limited by the assumption of zero-mean Gaussian noise and explicit modeling of the system.

[0077] Based on this, the application proposes a data-driven nonlinear system state estimation method, called data-driven approximate optimal filter (DAOF). The DAOF is represented by a deep neural network and trained using a reinforcement learning algorithm. This method is suitable for nonlinear systems with non-Gaussian noise and can estimate the state of complex systems that cannot be explicitly constructed as state space equations.

[0078] Next, the data-driven nonlinear system state estimation method proposed in the application will be described in detail.

[0079] Figure 1 A flowchart of the data-driven nonlinear system state estimation method according to an embodiment of the present disclosure is shown. This embodiment takes the method used in a server or a user terminal or other electronic device with computing power as an example, where the user terminal includes but is not limited to a mobile phone, a computer, a tablet computer, etc. This embodiment does not limit the device type of the electronic device. As shown in Figure 1 The method includes:

[0080] In step 101, the historical information of the system state to be estimated of the nonlinear system at time t is obtained, which includes the state estimation value at N historical time points before time t and the historical observation value at the next time point of each historical time point.

[0081] Where t is a positive integer starting from 0, and N is a predetermined positive integer.

[0082] In this embodiment, the nonlinear system refers to a physical system to be estimated, which is a dynamic system in reality, such as a vehicle control system, a power system, etc. The system state of the nonlinear system changes with time, and this embodiment does not limit the implementation of the nonlinear system. In this embodiment, the nonlinear system can be a nonlinear industrial object with complex variables and / or variable environmental disturbances. The system state refers to the system variable in the nonlinear system, such as the position of the vehicle and / or the speed of the vehicle, etc. Estimating the system state can achieve monitoring, early warning, etc. of the nonlinear system, and this embodiment does not limit the application scenario of the estimation result of the system state.

[0083] In one example, N is all historical time points before time t, at this time, the historical information at time t is: is the state estimation value at historical time point k, y k is the observation value of the system at historical time point k. In the following, parameters with the same symbol label have the same meaning, and this embodiment will not be described again in the following. At this time, h tThe historical information at time t includes all historical state estimation values before time t, i.e., all state estimation values from time 0 to time t-1, and historical observation values at the next time of each historical time, i.e., from time 1 to time t. The historical observation value refers to the system observation value at the next time of the historical time, and the system observation value is data that can be directly obtained from the nonlinear system, which usually contains some noise. The system observation value is an external manifestation of the system state, but is not directly equivalent to the system state, and the system observation value can be used to update the estimation of the system state.

[0084] In another example, N is a fixed value, and at this time, the historical information at time t is approximated by a sliding window of a fixed length N to express all historical information at time t. Illustratively, the expression of the historical information at time t is as follows:

[0085]

[0086] The value of N is determined based on the real-time requirement and the accuracy requirement, and by selecting a suitable length N of the sliding window, a balance between the real-time requirement and the accuracy requirement can be achieved.

[0087] Optionally, the historical information in the embodiment is historical information after position encoding, so that the relative position relationship between the data corresponding to different historical times can be obtained through subsequent reinforcement learning, and the accuracy of state estimation is improved. The encoding method of the historical information includes but is not limited to a sine-cosine position encoding method, a learning type encoding method, a sequential index encoding method, or a Fourier encoding method, and the embodiment does not limit the encoding method of the historical information.

[0088] Due to the periodicity of the sine-cosine position encoding method, this encoding method can be seamlessly extended to any length sequence; compared with the learning type encoding, no additional training is required, and the encoding complexity is reduced; compared with the sequential index encoding, continuous position information is provided instead of discrete index, which is beneficial to the model to better understand the relative position in the sequence; compared with the Fourier encoding, the implementation is simpler and the frequency control is more intuitive. Based on this, the embodiment takes the sine-cosine position encoding method as an example to illustrate the position encoding of the historical data corresponding to the historical times to obtain the historical information.

[0089] The sine-cosine position encoding method is expressed by the following formula:

[0090] PE(pos, 2i) = sin(pos / 10000 2i / d ),

[0091] PE(pos, 2i+1) = cos(pos / 10000 2i / d ),

[0092] PE (pos, 2i) represents the position encoding at the 2i index position at the time step pos (i.e. the time in this embodiment); PE (pos, 2i+1) represents the position encoding at the 2i+1 index position at the time step pos (i.e. the time in this embodiment); d represents the data dimension of the position encoding, which is consistent with the data dimension of the historical data to be encoded; i represents the index position in the position encoding, i is an integer starting from 0 and sequentially taking values.

[0093] In step 102, the historical information at time t is used to determine the action at time t based on the strategy function model to be trained, and the current estimated value at time t is obtained.

[0094] In this embodiment, the strategy function model and the value function model in the following are determined through a pre-constructed reinforced estimation problem (REP). The reinforced estimation problem refers to an optimal estimation problem constructed by minimizing the cumulative error with a discount factor. The association between the reinforced estimation problem and reinforcement learning is established to obtain the strategy function model and the value function model.

[0095] The reinforced estimation problem can be represented by the following formula:

[0096]

[0097] wherein, represents the optimal state estimation sequence from time k to infinity; represents the expected value of the weighted loss function value of all future time steps under the condition that the historical information h t at a given time t is given; x k is the state value to be estimated at time k, or the actual state value of the system state at time k; is the state estimation value at time k, and the closer the state estimation value is to the actual state value, the better; h t is the historical information at time t, y k is the system observation value at time k, and φ(·,·) is a loss function measuring the gap between the state estimation value and the actual state value, and γ∈[0,1) is a discount factor; represents finding an optimal state estimation sequence from time t to infinity, so that the expected value is minimized.

[0098] The association between the reinforced estimation problem and reinforcement learning is established, including: taking the historical information h t at time t as the state in reinforcement learning, taking the current estimation value at time t as the action in reinforcement learning, and taking the estimation error square is defined as a loss function, and the state transition probability is defined as represents the history information h at a given time t t and the current estimate value at time t , the probability distribution of the next state (i.e., the history information at time t+1) h t+1 , the value function obtained can be expressed by the following formula:

[0099]

[0100] where V represents the state value function, represents the weighted sum of the loss function values from time t to the future time, given that the state is the history information h t at time t; Q represents the action value function, represents the weighted sum of the loss function values from time t to the future time, given that the state is the history information h t at time t, and the action is the current estimate value . The current estimate value at time t refers to the state estimate value at time t determined using the current policy function.

[0101] In this embodiment, a parameterized neural network is used to approximate the value function, and a value function model, i.e., a critic, is obtained. The value function model can be expressed as:

[0102]

[0103] where, represents the evaluation value output by the value function given that the state is the history information h at time t, represents the evaluation value output by the value function model given that the model parameter of the neural network is w and the state is the history information h t at time t. The V here includes the above-mentioned state value function and the above-mentioned action value function.

[0104] According to the above reinforcement estimation problem, the optimal state estimation sequence is obtained by minimizing the value function, i.e., which can be expressed by the following formula:

[0105]

[0106] where V(h t ) represents the state value function given that the state is the history information h t at time t, represents the state value function given that the state is the history information h t at time t, and the action is the current estimate value Action value function in the following cases.

[0107] By solving The resulting filter is actually the reinforcement learning policy π. For reinforcement learning, the optimal policy can be derived using the Bellman Equation. The basic idea of ​​the Bellman Equation is that the value function of an optimal policy (i.e., the maximum cumulative reward obtainable by following this policy starting from a certain state) can be represented by considering the optimal value function of subsequent states after taking the optimal action in the current state. This equation is a necessary and sufficient condition for optimality. Based on this, to obtain the optimal filter (or optimal policy), the Bellman equation for the reinforcement estimation problem is derived as follows:

[0108]

[0109] Among them, V * (h t ) represents state h t The optimal state value function, e t The actual state value x at time t is represented by the given value. t The current estimated value at time t The error between; This indicates that the state distribution of historical information at time t+1 follows a certain pattern. In this case, calculate the loss function value of the current state and the next state h. t+1 The expected value of the weighted sum of the optimal state value functions, correspondingly, To find the current state value that minimizes the expected value of the weighted sum. Thus, the state h is obtained. t The optimal state value function. Represents state h t and actions The optimal action value function; This indicates that the state distribution of historical information at time t+1 follows a certain pattern. In the case of calculating the next state h t+1 Take the best action The expected value of the optimal action-value function. This represents the optimal action that minimizes the expected value of the optimal action-value function. This indicates that the state distribution of historical information at time t+1 follows a certain pattern. In the case of [condition], calculate the expected value of the loss function for the current state value.

[0110] As can be seen from the above equation, in this embodiment, the Bellman equation is used to explore one step forward to determine the optimal value function. This avoids the high computational complexity caused by searching for estimates at all future times when directly minimizing the value function, thus improving computational efficiency. In other embodiments, the Bellman equation can also explore at least two steps forward. This embodiment does not limit the implementation method of the Bellman equation for the enhanced estimation problem.

[0111] In this embodiment, different filter structures are used for state estimation with and without a model. The following sections will introduce the state estimation with and without a model.

[0112] 1. For state estimation with a model, i.e., when the nonlinear system can be modeled based on a state transition model, the state estimation can be achieved based on that state transition model. Accordingly, the filter structure is as follows:

[0113]

[0114] Where f(·) is the state transition function in system modeling, referred to as the prediction part in this embodiment, and π(·) is the filtering policy or policy function, referred to as the update part in this embodiment. Under this policy structure, in order to solve for the optimal policy, the policy iteration method in reinforcement learning is adopted, and the iterative processes of policy evaluation and policy improvement are derived separately, expressed by the following formula:

[0115]

[0116] Where j is the number of iterations (or the number of iterations) in the policy evaluation. This indicates that the state-value function uses the current policy π and historical information h from time t after the (j+1)th iteration. t Determined evaluation value, This indicates that the state-value function, after the j-th iteration, uses the current policy π and the historical information h from time t+1. t+1 A definite evaluation value. p(h) t+1 |h t ,π(h t )) indicates that given the current state h t and by the policy function π in state h t The next action selected π(h) t In the case of ), the next state h t+1 The probability distribution. k represents the number of iterations in policy improvement, π k+1 (h t ) indicates that after the (k+1)th iteration, the policy function π is in state h t The action to choose from, Given a strategy πk The steady-state value function reached after policy evaluation (i.e., the policy function π in the k-th iteration), or the function that converges when the state value function is iterated a certain number of times.

[0117] 2. For model-free state estimation, i.e., when a nonlinear system cannot be modeled using a state transition model, state estimation cannot be achieved based on the state transition model. Accordingly, the filter structure is as follows:

[0118]

[0119] Under this policy structure, in order to solve for the optimal policy, the policy iteration method in reinforcement learning is adopted to derive the iterative processes of policy evaluation and policy improvement, which are expressed by the following equation:

[0120]

[0121] in, This indicates that after the (j+1)th iteration, the action-value function uses the current strategy π to process the historical information h at time t. t and current estimate Determined evaluation value, This indicates that the action-value function uses the current strategy π and historical information h from time t+1 after the j-th iteration. t+1 and historical information h at time t+1 t+1 Action π(h) t+1 The determined evaluation value. Given a strategy π k The steady-state action value function reached after policy evaluation, or the action value function that converges after a certain number of iterations. π k+1 (h t ) indicates that after the (k+1)th iteration, the policy function π is in state h t The action to choose from, Indicates the search for something that can make Strategies to reach the minimum value.

[0122] Optionally, for state estimation with a model, the iterative process in state estimation without a model can also be used; for state estimation without a model, the iterative process in state estimation with a model can also be used. In this case, since the state transition model in state estimation without a model cannot be determined, the steady-state distribution of the state can be approximated by multiple samplings, and the state transition probability can be determined.

[0123] In this embodiment, a parameterized neural network is used to approximate the policy function, resulting in a policy function model, i.e., the actor. The policy function model can be expressed as:

[0124]

[0125] wherein, denotes the action taken by the policy function given the history information at time t, denotes the action output by the neural network given the model parameters of the neural network, the history information h t at time t.

[0126] After the policy function model is constructed, the policy function model is a to-be-trained policy function model. In this embodiment, the to-be-trained policy model refers to a policy function model that is not trained completely, that is, the policy function model in the training process also belongs to the to-be-trained policy model.

[0127] At this time, the history information at time t is used to determine the action at time t based on the to-be-trained policy function model, to obtain the current estimated value at time t, including: inputting the history information at time t into the to-be-trained policy function model to obtain the action at time t output by the to-be-trained policy function model, and the action at time t is the current estimated value at time t

[0128] In step 103, the history information at time t+1 is determined based on the current estimated value at time t and the system observation value at time t+1.

[0129] If the history information is all historical data of a certain time in the past, the history information h t+1 at time t+1 can be represented by the following formula:

[0130]

[0131] wherein, denotes the current estimated value at time t, y t+1 denotes the system observation value at time t+1.

[0132] If the history information is the historical data of the past N times of a certain time, the history information h t+1 at time t+1 can be represented by the following formula:

[0133]

[0134] In step 104, the first estimated evaluation value corresponding to the history information at time t and the second estimated evaluation value corresponding to the history information at time t+1 are determined based on the to-be-trained value function model.

[0135] The to-be-trained value function model refers to a value function model that is not trained completely, that is, the value function model in the training process also belongs to the to-be-trained policy model. ​

[0136] Based on the value function model to be trained, the first estimated evaluation value corresponding to the historical information at time t and the second estimated evaluation value corresponding to the historical information at time t+1 are determined, including: inputting the historical information at time t into the value function model to be trained to obtain the first estimated evaluation value output by the value function model; inputting the historical information at time t+1 into the value function model to be trained to obtain the second estimated evaluation value output by the value function model.

[0137] Step 105, based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t, the value function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained value function model corresponding to time t.

[0138] In this embodiment, based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t, the value function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained value function model corresponding to time t, including steps 1051-1054:

[0139] Step 1051, obtain the first loss function corresponding to the value function model to be trained.

[0140] In this embodiment, the first loss function is determined based on the policy evaluation process in reinforcement learning. For the above model-based state estimation and model-free state estimation, the first loss function obtained based on the corresponding policy evaluation process can be represented by the following formula:

[0141]

[0142] Wherein, J critic represents the first loss function of the value function model critic to be trained; represents the expected value calculation function; represents the historical information corresponding to time t obeys the steady-state distribution that can be reached under the current policy π of the policy function model to be trained; represents the historical information corresponding to time t+1 obeys the steady-state distribution that can be reached under the current policy π of the policy function model to be trained; t is the loss equation, represents the historical information corresponding to time t under the condition of, using the current policy π of the policy function model to be trained to select the current estimated value at time t as the action π(h t ) at time t, resulting in the loss; t e t ​The actual state value x at time t t The current estimate at time t The error between; γ represents the preset discount factor; V represents the value function model to be trained, where V includes the value function models corresponding to the action value function and the state value function; w represents the current model parameters of the value function model to be trained.

[0143] The steady-state distribution refers to the distribution of states whose mean changes tend to zero after a certain number of time steps, given a specific policy and state transition probabilities, starting from an initial state. For state estimation with a model, since the state transition probability p(h... t+1 |h t ,π(h t The state can be explicitly obtained through system modeling (as mentioned above), therefore, given the current policy π, the state can be derived. and The steady-state distribution of the state. For model-free state estimation, since both the state transition probabilities and the policy are uncertain and probabilistic, the state obtained after each certain time step may be different. In this case, the steady-state distribution of the state can be approximated by sampling. That is, starting from the initial state, running according to the given policy for a certain number of time steps, sampling the state corresponding to that time step, and after multiple runs, using the results of each sampling to approximate the steady-state distribution of the state.

[0144] Step 1052: Input the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t into the first loss function to obtain the loss function value for the j-th iteration. Here, j is a positive integer starting from 1.

[0145] According to the first loss function mentioned above, it is... This is the first estimated evaluation value; This is the second estimated evaluation value. x is the current estimate. t Given the actual state value at time t, we obtain the loss function value J for the j-th iteration. critic .

[0146] Step 1053: If the iteration stopping condition is not met in the j-th iteration, update the model parameters of the value function model to be trained based on gradient descent.

[0147] The model parameters of the value function model to be trained are updated based on gradient descent, as expressed by the following formula:

[0148]

[0149]

[0150] wherein, denotes the first loss function J critic The gradient with respect to the model parameter w is calculated by computing the first loss function J critic The partial derivative with respect to the model parameter w is obtained; a represents a preset learning rate.

[0151] Step 1054, after the parameter is updated, the value of j is equal to j+1, and the step of inputting the first estimated evaluation value, the second estimated evaluation value, the historical information at time t, and the actual state value corresponding to the historical information at time t into the first loss function to obtain the loss function value of the jth iteration (i.e., step 1052) and the subsequent steps are executed again until the jth iteration reaches the iteration stopping condition, thereby obtaining the trained value function model corresponding to time t.

[0152] The iteration stopping condition includes but is not limited to: the loss function value of the jth iteration indicates that the value function model converges; or the value of j reaches a first preset iteration number. At this time, after step 1052, it further includes determining whether the current iteration reaches the iteration stopping condition, if the iteration stopping condition is reached, the model parameter corresponding to the current iteration is taken as the model parameter of the trained value function model corresponding to time t, and the trained value function model corresponding to time t is obtained; if the iteration stopping condition is not reached, step 1053 is executed.

[0153] Step 106, based on the steady-state estimated evaluation value of the trained value function model corresponding to time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to time t.

[0154] In this embodiment, because the filter structures of the model-based state estimation and the model-free state estimation are different, the loss functions for training the policy function model are also different.

[0155] 1. In the case of modeling based on a state transition model of a nonlinear system, based on the steady-state estimated evaluation value of the trained value function model corresponding to time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to time t, including steps 1061-1065:

[0156] Step 1061, a second loss function corresponding to the policy function model to be trained is obtained, wherein the second loss function is established based on minimizing the sum of the error of the current estimated value at time t and the steady-state estimated evaluation value output by the trained value function model corresponding to time t.

[0157] The second loss function is represented by the following formula:

[0158]

[0159] Among them, J actor This represents the second loss function of the policy function model actor to be trained; This represents the function for calculating the expected value. This represents the historical information corresponding to time t. The steady-state distribution that the policy function model to be trained can achieve under the current policy π; This represents the historical information corresponding to time t+1. The steady-state distribution that the policy function model to be trained can achieve under the current policy π; t The loss equation is... This represents the historical information corresponding to time t. Under the given conditions, the current policy π is modeled using the policy function to be trained, based on historical information h. t Choose the current estimate at time t as the action π(h) at time t. t The losses resulting from this; e t The actual state value x at time t t The current estimate at time t The error between them; γ represents the preset discount factor; This indicates the historical information corresponding to time t+1. The steady-state estimate is obtained by inputting the trained value function model with input model parameters w at time t.

[0160] Step 1062: Input the historical information corresponding to time t+1 into the trained value function model corresponding to time t to obtain the steady-state estimation evaluation value.

[0161] The steady-state estimation evaluation value is as described above.

[0162] Step 1063: Input the current estimated value at time t, the actual state value at time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value for the kth iteration.

[0163] Where k is a positive integer starting from 1.

[0164] The current estimate at time t is The actual state value at time t is x. t .

[0165] Step 1064: If the iteration stopping condition is not met in the k-th iteration, update the model parameters of the policy function model to be trained based on gradient descent.

[0166] The model parameters of the to-be-trained strategy function model are updated based on the gradient descent method, and are represented by the following formula:

[0167]

[0168] wherein, represents the second loss function J actor The gradient of the model parameters of the to-be-trained strategy function model is obtained by calculating the partial derivative of the second loss function J actor with respect to the model parameters ; and β represents a preset learning rate.

[0169] In step 1065, after the parameter update, the value of k is set to k+1, and the steps of inputting the current estimated value at time t, the actual state value at time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value of the kth iteration, and the subsequent steps are executed again until the kth iteration reaches the iteration stopping condition, so as to obtain the trained strategy function model corresponding to time t.

[0170] The iteration stopping condition includes but is not limited to: the loss function value of the kth iteration indicates that the strategy function model converges; or the value of k reaches a second preset iteration number. At this time, after step 1063, it further includes determining whether the current iteration reaches the iteration stopping condition. If the iteration stopping condition is reached, the model parameters corresponding to the current iteration are used as the model parameters of the trained strategy function model corresponding to time t, so as to obtain the trained strategy function model corresponding to time t. If the iteration stopping condition is not reached, step 1064 is executed.

[0171] 2. In the case where the nonlinear system cannot be modeled based on the state transition model, the steady-state estimated evaluation value corresponding to the trained value function model at time t is used to iteratively train the to-be-trained strategy function model based on the reinforcement learning algorithm, so as to obtain the trained strategy function model corresponding to time t, including the following steps 1066-10691:

[0172] In step 1066, a third loss function corresponding to the to-be-trained strategy function model is obtained, and the third loss function is established based on minimizing the steady-state estimated evaluation value output by the trained value function model at time t.

[0173] The third loss function is represented by the following formula:

[0174]

[0175] wherein, J actor represents the third loss function of the to-be-trained strategy function model actor; represents an expected value calculation function; denotes the history information corresponding to time t subject to the steady-state distribution that can be reached by the current policy p under the policy function model to be trained; denotes the history information corresponding to time t and the current estimate value of time t , the steady-state estimated evaluation value obtained after inputting the trained value function model corresponding to time t with model parameters w.

[0176] Step 1067, inputting the history information corresponding to time t into the trained value function model corresponding to time t to obtain the steady-state estimated evaluation value.

[0177] Step 1068, inputting the steady-state estimated evaluation value into the third loss function to obtain the loss function value of the kth iteration. k is a positive integer starting from 1.

[0178] The steady-state estimated evaluation value is

[0179] Step 1069, in the case where the kth iteration does not reach the iteration stopping condition, updating the model parameters of the policy function model to be trained based on the gradient descent method.

[0180] Updating the model parameters of the policy function model to be trained based on the gradient descent method is represented by the following formula:

[0181]

[0182]

[0183] wherein, denotes the third loss function J actor the gradient of the model parameters of the policy function model to be trained is obtained by calculating the partial derivative of the third loss function J actor with respect to the model parameters ; and b denotes a preset learning rate.

[0184] Step 10691, after updating the parameters, setting the value of k equal to k+1, and executing again the step of inputting the steady-state estimated evaluation value into the third loss function to obtain the loss function value of the kth iteration and the subsequent steps until the kth iteration reaches the iteration stopping condition, to obtain the trained policy function model corresponding to time t.

[0185] The iteration stop condition in this step is the same as or different from the iteration stop condition in step 1065, and the iteration stop condition in this step includes but is not limited to: the loss function value of the kth iteration indicates that the policy function model converges; or, the value of k reaches a third preset iteration number, and the embodiment does not limit the implementation manner of the iteration stop condition. At this time, after step 1068, it further includes determining whether the current iteration reaches the iteration stop condition, if the iteration stop condition is reached, the model parameter corresponding to the current iteration is used as the model parameter of the trained policy function model corresponding to time t, and the trained policy function model corresponding to time t is obtained; if the iteration stop condition is not reached, step 1069 is executed.

[0186] Step 107, after the training of the policy function model is completed, the state estimation value of the system state is determined based on the system observation value of the nonlinear system using the trained policy function model.

[0187] Optionally, after step 106, it is determined whether the training stop condition is reached; if the stop condition is reached, the training is stopped, and the policy function model is obtained; if the training stop condition is not reached, t is set to t+1, and step 101 is executed again until the training stop condition is reached.

[0188] Illustratively, the training stop condition includes but is not limited to: the value of t reaches a preset step number, such as 500; or, the model parameter variation amount of the policy function model of adjacent two time points is less than a preset variation threshold, and the embodiment does not limit the setting manner of the training stop condition.

[0189] After the training of the policy function model is completed, the policy function model is obtained . Then, for the nonlinear system, the system observation value y1 of time 1 is input into the policy function model to obtain the state estimation value of time 1; then, the system observation value y2 of time 2 is input into the policy function to make the policy function obtain the state estimation value of time 2 based on the state estimation value of time 1 and the system observation value y2, and the cycle is repeated until the state estimation of each time point is completed.

[0190] Reference Figure 2 According to the above content, it can be known that the data-driven nonlinear system state estimation method in the embodiment at least includes the following working processes:

[0191] 1. Construction of reinforcement estimation problem: in the embodiment, the optimal estimation problem is constructed by minimizing the cumulative error with a discount factor;

[0192] 2. Reinforced estimation strategy structure design: in the embodiment, the reinforcement estimation strategies based on the model and the model-free are designed according to the reinforcement estimation problem form, that is, the policy function model in the above.

[0193] 3. Offline training of network parameters: In this embodiment, the network model is designed to approximate the problem value function and estimation strategy, and the model parameters are solved by gradient descent.

[0194] refer to Figure 3 The offline training process shown obtains the actual state value x of the system at time t. t and system observation y t ; Historical information at time t Input the policy function model to be trained and the value function model to be trained, respectively. y represents the state estimate from time tN to time t. t-N+1:t+1 This represents the system observations from time t-N+1 to time t+1. The policy function model to be trained outputs the current estimate at time t. The value function model to be trained outputs the first estimated evaluation value of the current estimate. or ; Current estimate based on time t Compared with the actual state value x t The error e between t The first estimated evaluation value is used to update the model parameters of the value function model and the policy function model through the policy evaluation iteration process and the policy improvement iteration process, and the gradient descent algorithm is used to update the model parameters of the value function model and the policy function model to obtain the trained policy function model corresponding to time t.

[0195] refer to Figure 4 The iterative update process of the value function model and policy function model shown is based on... Figure 4 It can be seen that, assuming the model parameters w of the initialized value function model to be trained are the initialization parameters w0, the model parameters of the initialized policy function model to be trained are... Initialization parameters The learning rate is initialized to α and β. Initializing the historical information h0∈S at time 0 means that the historical information h0 belongs to the state space S, which includes the set of all possible states of the system. Repeat the following process: [The process continues with the historical information at time t]. Input the policy function model π to be trained, obtain and store the historical information at time t+1. The gradient of the value function model (critic) to be trained is calculated using the first loss function, and the model parameters of the value function model to be trained are updated accordingly. After the value function model to be trained satisfies the iteration stopping condition, the gradient of the policy function model to be trained is calculated using the second or third loss function, and the model parameters of the policy function model (i.e., actor) to be trained are updated. until the trained policy function model corresponding to the time t is obtained when the trained policy function model meets the iteration stopping condition; until the final policy function model is obtained when the model training meets the training stopping condition.

[0196] 4. Estimating network online application: inputting the system observation value of the nonlinear system into the trained policy function model to output the state estimation value of the system state online.

[0197] In summary, the data-driven nonlinear system state estimation method in the embodiment obtains the historical information of the system state to be estimated of the nonlinear system at time t; determines the action at time t based on the historical information at time t using the trained policy function model to obtain the current estimation value at time t; determines the historical information at time t+1 based on the current estimation value at time t and the system observation value at time t+1; determines the first estimation evaluation value corresponding to the historical information at time t and the second estimation evaluation value corresponding to the historical information at time t+1 based on the trained value function model; iteratively trains the trained value function model using the reinforcement learning algorithm based on the first estimation evaluation value, the second estimation evaluation value, the current estimation value at time t, and the actual state value at time t to obtain the trained value function model corresponding to time t; iteratively trains the trained policy function model using the reinforcement learning algorithm based on the steady-state estimation evaluation value of the trained value function model corresponding to time t to obtain the trained policy function model corresponding to time t; determines the state estimation value of the system state based on the trained policy function model after the training of the policy function model is completed using the system observation value of the nonlinear system; the policy function model can be trained offline based on the historical information using the reinforcement learning algorithm, the obtained policy function model can be used for state estimation of the nonlinear system with non-Gaussian noise online, and the state estimation of the system state can be realized without relying on accurate system modeling, reducing the difficulty of system estimation of the nonlinear system and improving the accuracy of system estimation.

[0198] The data-driven nonlinear system state estimation method provided in the present application will be described below with an example. In this example, the nonlinear system is a vehicle centroid side slip angle model, and the following equation is established according to tire dynamics:

[0199]

[0200] where β is the centroid side slip angle, and correspondingly, denotes the derivative of the centroid side slip angle; ω is the vehicle yaw rate, and correspondingly, denotes the derivative of the vehicle yaw rate; δ fThe steering wheel angle is used as the control input of the model. The tire dynamics model is based on the Pacejka model, or Magic Formula. The meanings of the model parameters in the system model and the values of the parameters are shown in Table 1 below.

[0201] Table 1:

[0202]

[0203] The state to be estimated in the system model is X = [β, ω], and the process noise ξ is considered. The state equation is obtained by discretization using the forward Euler method:

[0204]

[0205] where X t represents the state estimation value of the system state at time t; X t+1 represents the state estimation value of the system state at time t+1; ΔT represents the length of time elapsed from time t to time t+1, i.e., the time interval; represents the rate of change of the system state, i.e., the average rate of change in the time interval ΔT; ξ t represents the process noise at time t.

[0206] The observation of the system model is Y = [a y , ω] T , where a y is the longitudinal acceleration of the vehicle, and ζ is the observation noise. The observation equation is as follows:

[0207]

[0208] where y1 represents the system observation value of the longitudinal acceleration a y of the vehicle; and y2 represents the system observation value of the yaw rate ω of the vehicle.

[0209] In this example, the process noise ξ t is a Gaussian mixed noise, and the observation noise ζ t is a Laplace noise. The noises ξ t and ζ t satisfy:

[0210]

[0211] where μ represents a Gaussian distribution (or normal distribution), and represent the mean vectors of the two Gaussian distributions, respectively, represents the covariance matrices of the two Gaussian distributions, and is a 2x2 identity matrix. Laplace(·) denotes the Laplace distribution, is the mean vector of the Laplace distribution, is the scale parameter matrix of the Laplace distribution.

[0212] To verify the performance of the data-driven approximate-optimal filtering algorithm proposed in this application, the algorithm is compared with the particle filter and the supervised learning method below. Through 100 Monte Carlo experiments, each experiment includes 500 time steps (or time points), to evaluate the root mean square error (RMSE) and the single-step time consumption of different methods. Among them, the smaller the RMSE represents the higher the estimation accuracy of the method.

[0213] Figure 5 The box plot of the root mean square error of different methods is shown. Figure 5 PF (Particle Filter) represents the particle filter, SLF (Supervised Learning-based Filter) represents the supervised learning method, DAOFw / o PE represents the data-driven approximate-optimal filtering method without using position encoding, and DAOF represents the data-driven approximate-optimal filtering method proposed in this application. According to Figure 5 It can be seen that the data-driven approximate-optimal filtering method proposed in this application has higher estimation accuracy than the common nonlinear non-Gaussian filtering method, and the use of position encoding operation plays a positive role in improving the estimation accuracy.

[0214] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to execute the methods described in the above method embodiment descriptions, and the specific implementation can refer to the descriptions of the above method embodiments. For brevity, they will not be repeated here.

[0215] The embodiments of the present disclosure also propose a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0216] The embodiments of the present disclosure also propose an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0217] The embodiments of the present disclosure also provide a computer program product, comprising computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device executes the above method.

[0218] Figure 6 A block diagram of a data-driven nonlinear system state estimation apparatus according to an embodiment of the present disclosure is shown. The model includes at least the following modules: a first acquisition module 610, an action estimation module 620, a second acquisition module 630, an estimation evaluation module 640, a first training module 650, a second training module 660, and an online estimation module 670.

[0219] The first acquisition module 610 is configured to acquire historical information of a system state to be estimated of the nonlinear system at a time t, the historical information including state estimation values at N historical times before the time t and historical observation values at a next time of each historical time; the t is a positive integer starting from 0, and the N is a preset positive integer;

[0220] The action estimation module 620 is configured to determine an action at the time t based on a policy function model to be trained using the historical information at the time t, to obtain a current estimation value at the time t;

[0221] The second acquisition module 630 is configured to determine historical information at a time t+1 based on the current estimation value at the time t and a system observation value at the time t+1;

[0222] The estimation evaluation module 640 is configured to determine a first estimation evaluation value corresponding to the historical information at the time t and a second estimation evaluation value corresponding to the historical information at the time t+1 based on a value function model to be trained;

[0223] The first training module 650 is configured to perform iterative training on the value function model to be trained based on the first estimation evaluation value, the second estimation evaluation value, the current estimation value at the time t, and an actual state value at the time t, using a reinforcement learning algorithm, to obtain a trained value function model corresponding to the time t;

[0224] The second training module 660 is configured to perform iterative training on the policy function model to be trained based on a steady-state estimation evaluation value of the trained value function model corresponding to the time t, using the reinforcement learning algorithm, to obtain a trained policy function model corresponding to the time t;

[0225] The online estimation module 670 is configured to determine a state estimation value of the system state based on the trained policy function model using a system observation value of the nonlinear system after the training of the policy function model is completed.

[0226] For related descriptions, refer to the above method embodiments.

[0227] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can be referred to the description of the above method embodiments. For brevity, details are not repeated here.

[0228] Figure 7 is a block diagram of a data-driven nonlinear system state estimation apparatus 1900 according to an exemplary embodiment. For example, the apparatus 1900 can be provided as a server or a terminal device. Referring to Figure 7 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions, such as application programs, executable by the processing component 1922. The application programs stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0229] The apparatus 1900 can also include a power supply component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input output interface 1958 (I / O interface). The apparatus 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0230] In exemplary embodiments, a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions executable by the processing component 1922 of the apparatus 1900 to perform the above method is also provided.

[0231] The above has described the embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical application or technical improvement in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A data-driven method for estimating the state of a nonlinear system, characterized in that, The method includes: The historical information of the system state to be estimated in the nonlinear system at time t is obtained. The historical information includes the state estimates of N historical times before time t and the historical observations of the next time after each historical time. t is an integer starting from 0, and N is a preset positive integer. Based on the policy function model to be trained, the historical information at time t+1 is used to determine the action at time t, and the current estimated value at time t is obtained. Based on the current estimated value of time t and the system observation value of time t+1, determine the historical information of time t+1; Based on the value function model to be trained, determine the first estimated evaluation value corresponding to the historical information at time t, and the second estimated evaluation value corresponding to the historical information at time t+1; Based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t, the value function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained value function model corresponding to time t. Based on the steady-state estimation evaluation value of the trained value function model at time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model at time t. After the policy function model is trained, the system state estimate is determined based on the system observations of the nonlinear system using the trained policy function model.

2. The method according to claim 1, characterized in that, The step of iteratively training the value function model to be trained using a reinforcement learning algorithm based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t, to obtain the trained value function model corresponding to time t, includes: Obtain the first loss function corresponding to the value function model to be trained; The first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t are input into the first loss function to obtain the loss function value for the j-th iteration; where j is a positive integer starting from 1. If the iteration stopping condition is not met in the j-th iteration, the model parameters of the value function model to be trained are updated based on the gradient descent method. After the parameter update, let the value of j equal to j+1, and execute the steps of inputting the first estimated evaluation value, the second estimated evaluation value, the historical information of time t, and the actual state value corresponding to the historical information of time t into the first loss function to obtain the loss function value of the j-th iteration, and the subsequent steps, until the j-th iteration reaches the iteration stopping condition, to obtain the trained value function model corresponding to time t.

3. The method according to claim 2, characterized in that, The first loss function is expressed by the following formula: Among them, J critic This represents the first loss function of the value function model "critic" to be trained; This represents the function for calculating the expected value. This represents the historical information corresponding to the time t. The steady-state distribution that the policy function model to be trained can achieve under the current policy π is obeyed; This represents the historical information corresponding to time t+1. The steady-state distribution that the policy function model to be trained can achieve under the current policy π is obeyed; the l t The loss equation is... This represents the historical information corresponding to time t. Under the given conditions, the current policy π of the policy function model to be trained is based on historical information h. t Choose the current estimate at time t as the action π(h) at time t. t The losses resulting from this; e t The actual state value x at time t is represented by the given value. t The current estimated value at time t The error between; γ represents the preset discount factor; V represents the value function model to be trained; w represents the current model parameters of the value function model to be trained.

4. The method according to claim 3, characterized in that, The model parameters of the value function model to be trained, updated based on gradient descent, are expressed by the following formula: in, The first loss function J represents critic Regarding the gradient of the model parameter w, it is calculated by the first loss function J. critic The partial derivatives with respect to the model parameter w are obtained; α represents the preset learning rate.

5. The method according to claim 1, characterized in that, When the nonlinear system can be modeled based on a state transition model, the step of iteratively training the policy function model to be trained using a reinforcement learning algorithm based on the steady-state estimation evaluation value of the trained value function model at time t to obtain the trained policy function model at time t includes: Obtain the second loss function corresponding to the policy function model to be trained. The second loss function is established based on minimizing the error of the current estimated value at time t and the steady-state estimated evaluation value output by the trained value function model at time t. The historical information corresponding to time t+1 is input into the trained value function model corresponding to time t to obtain the steady-state estimation evaluation value; The current estimated value at time t, the actual state value at time t, and the steady-state estimated evaluation value are input into the second loss function to obtain the loss function value for the kth iteration; where k is a positive integer starting from 1. If the iteration stopping condition is not met in the k-th iteration, the model parameters of the policy function model to be trained are updated based on the gradient descent method. After the parameters are updated, let the value of k equal to k+1, and execute the steps of inputting the current estimated value of time t, the actual state value of time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value of the kth iteration, and the subsequent steps, until the iteration stops when the kth iteration reaches the iteration stopping condition, to obtain the trained policy function model corresponding to time t.

6. The method according to claim 5, characterized in that, The second loss function is expressed by the following formula: Among them, J actor The second loss function represents the policy function model actor to be trained; This represents the function for calculating the expected value. This represents the historical information corresponding to the time t. The steady-state distribution that the policy function model to be trained can achieve under the current policy π is obeyed; This represents the historical information corresponding to time t+1. The steady-state distribution that the policy function model to be trained can achieve under the current policy π is obeyed; the l t The loss equation is... This represents the historical information corresponding to time t. Under the given conditions, the current policy π of the policy function model to be trained is based on historical information h. t Choose the current estimate at time t as the action π(h) at time t. t The losses resulting from this; e t The actual state value x at time t is represented by the given value. t The current estimated value at time t The error between them; γ represents the preset discount factor; This indicates the historical information corresponding to time t+1. The steady-state estimate is obtained by inputting the trained value function model with input model parameters w at time t.

7. The method according to claim 6, characterized in that, The model parameters of the policy function model to be trained, updated based on gradient descent, are expressed by the following formula: in, The second loss function J represents actor Regarding the model parameters of the policy function model to be trained The gradient is obtained by calculating the second loss function J. actor Regarding model parameters The partial derivatives are obtained; β represents the preset learning rate.

8. The method according to claim 1, characterized in that, When the nonlinear system cannot be modeled based on a state transition model, the step of iteratively training the policy function model to be trained using a reinforcement learning algorithm based on the steady-state estimation evaluation value of the trained value function model at time t to obtain the trained policy function model at time t includes: Obtain the third loss function corresponding to the policy function model to be trained. The third loss function is established based on minimizing the steady-state estimation evaluation value output by the trained value function model at time t. The historical information corresponding to time t is input into the trained value function model corresponding to time t to obtain the steady-state estimation evaluation value; The steady-state estimation evaluation value is input into the third loss function to obtain the loss function value of the k-th iteration; where k is a positive integer starting from 1. If the iteration stopping condition is not met in the k-th iteration, the model parameters of the policy function model to be trained are updated based on the gradient descent method. After the parameters are updated, let the value of k equal to k+1, and execute the steps of inputting the steady-state estimation evaluation value into the third loss function to obtain the loss function value of the kth iteration, and the subsequent steps, until the iteration stopping condition is met in the kth iteration, to obtain the trained policy function model corresponding to time t.

9. The method according to claim 8, characterized in that, The third loss function is expressed by the following formula: Among them, J actor This represents the third loss function of the policy function model actor to be trained; This represents the function for calculating the expected value. This represents the historical information corresponding to the time t. The steady-state distribution that the policy function model to be trained can achieve under the current policy π is obeyed; This indicates the historical information corresponding to time t. and the current estimate of time t The steady-state estimate evaluation value is obtained by inputting the trained value function model with input model parameters w at time t. Accordingly, the model parameters of the policy function model to be trained, updated based on gradient descent, are expressed by the following formula: in, The third loss function J represents actor Regarding the model parameters of the policy function model to be trained The gradient is calculated by the third loss function J. actor Regarding model parameters The partial derivatives are obtained; β represents the preset learning rate.

10. A data-driven nonlinear system state estimation device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 9 when executing instructions stored in the memory.

Images