Data-driven nonlinear system state estimation method

Abstract

The invention relates to the technical field of optimal estimation, and discloses a data-driven nonlinear system state estimation method. Determining historical information at a moment t + 1 by acquiring historical information of a system state at a moment t and a to-be-trained strategy function model; based on a to-be-trained value function model, determining a first estimation evaluation value corresponding to the historical information at the moment t and a second estimation evaluation value corresponding to the historical information at the moment t + 1; performing iterative training on the value function model to be trained based on the first estimation evaluation value, the second estimation evaluation value, the current estimation value at the moment t and the actual state value at the moment t; based on the steady state estimation evaluation value of the trained value function model corresponding to the moment t, performing iterative training on a to-be-trained strategy function model to obtain a trained strategy function model corresponding to the moment t; determining a state estimation value of a system state based on the trained strategy function model; the state estimation difficulty can be reduced, and the state estimation accuracy is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of optimal estimation, and in particular to a data-driven nonlinear system state estimation method. Background Art

[0002] State estimation of dynamic systems is used to extract accurate information about the state of the system from observation data that may be contaminated by noise, i.e. filtering. It plays a vital role in various engineering fields such as autonomous driving, robotics and aerospace, and is the basic premise of control work. Common filters are usually based on Bayesian filtering, that is, the posterior distribution of the system state is updated according to the Bayesian formula, and then a point estimate of the system state is obtained by applying specific criteria. The pioneering Kalman filter (KF) is a special case of Bayesian filtering. It is optimal under the minimum mean square error criterion in linear systems with Gaussian noise. For nonlinear systems, the extended Kalman filter (EKF) is one of the most commonly used filtering techniques. This method approximates the nonlinear random state space model through a first-order Taylor series expansion to obtain a suboptimal filter. Since the errors of second-order and higher-order terms are ignored, the algorithm may diverge when facing a strong nonlinear system. Another widely used method is the Unscented Kalman Filter (UKF), which uses a set of sigma points to parameterize the mean and covariance of the posterior distribution, avoiding direct model linearization. However, UKF is prone to divergence when there is a significant difference between the system model and the actual dynamics and the noise is asymmetric. In general, the accuracy of both nonlinear filtering methods will drop significantly when faced with strong nonlinear or non-Gaussian noise. Compared with the variants of the Kalman filter, the particle filter (PF) based on the Monte Carlo sampling principle shows better accuracy in the estimation problem of nonlinear non-Gaussian systems. However, this method requires propagating and resampling a large number of particles at each time step, resulting in high computational complexity, which limits its real-time approximation. Different from using particles to approximate the entire posterior distribution, the Moving Horizon Estimation (MHE) achieves maximum a posteriori estimation by recursively minimizing the estimation cost function defined on a sliding time window. Similar to PF, MHE improves the estimation accuracy, but its computational burden also increases accordingly, and its real-time performance is severely limited in the case of high-dimensional state space.

[0003] In recent years, the rapid development of deep learning technology has led to the emergence of a large number of learning-based filtering methods. A typical method is to combine recurrent neural networks with Kalman filters, which have achieved good performance in numerical experiments. However, such methods usually lack strict theoretical guarantees and it is difficult to establish the equivalence or approximate equivalence between the obtained filter and the optimal filter. The variational Bayesian filter proposes another deep learning-based method, which uses a deep neural network nested in a recursive structure to model the variational distribution of process noise conditioned on the current observation and previous estimates. Although such methods provide a convincing theoretical basis through variational reasoning and show accurate estimation capabilities in actual experiments, their reliance on the Gaussian noise assumption and precise modeling limits their applicability in a wider range of contexts. Another class of methods uses deep neural networks to approximate MHE, but since such methods mainly focus on the mode of the posterior distribution of the estimated state rather than the entire distribution, their accuracy is limited, and solving the arrival loss during training is difficult to achieve in practical applications.

[0004] A common problem with the above estimation methods is their reliance on the accuracy of system models and parameters, or at least on the structural model of the system, which is usually a feature of model-driven estimation. However, the need for clear and accurate models significantly limits the applicability of such estimation methods in complex systems. In the field of industrial control, there are a large number of systems with very complex dynamic characteristics, and it is difficult to accurately model such systems, which leads to the difficulty and low accuracy of system estimation of nonlinear systems. Summary of the invention

[0005] In view of this, the present disclosure proposes a data-driven nonlinear system state estimation method, which can solve the problem of high difficulty and low accuracy in system estimation of traditional nonlinear systems.

[0006] According to one aspect of the present disclosure, a data-driven nonlinear system state estimation method is provided, the method comprising:

[0007] Obtaining historical information of the system state to be estimated of the nonlinear system at time t, the historical information including the state estimation values ​​at N historical moments before the time t and the historical observation value at the next moment of each historical moment; t is a positive integer starting from 0, and N is a preset positive integer;

[0008] Determine the action at time t based on the policy function model to be trained using the historical information at time t to obtain a current estimated value at time t;

[0009] Determine historical information at time t+1 based on the current estimated value at time t and the system observed value at time t+1;

[0010] Determine, based on the value function model to be trained, a first estimated evaluation value corresponding to the historical information at time t and a second estimated evaluation value corresponding to the historical information at time t+1;

[0011] Based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at the time t, and the actual state value at the time t, the value function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain a trained value function model corresponding to the time t;

[0012] Based on the steady-state estimated evaluation value of the trained value function model corresponding to the time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to the time t;

[0013] After the policy function model is trained, a state estimation value of the system state is determined based on the trained policy function model and using the system observation value of the nonlinear system.

[0014] In a possible implementation, based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at the time t, and the actual state value at the time t, the value function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain a trained value function model corresponding to the time t, including:

[0015] Obtaining a first loss function corresponding to the value function model to be trained;

[0016] Input the first estimated evaluation value, the second estimated evaluation value, the current estimated value at the time t, and the actual state value at the time t into the first loss function to obtain the loss function value of the jth iteration; j is a positive integer starting from 1;

[0017] When the iteration stopping condition is not reached in the j-th iteration, the model parameters of the value function model to be trained are updated based on the gradient descent method;

[0018] After the parameters are updated, let the value of j be equal to j+1, and the step of inputting the first estimated evaluation value, the second estimated evaluation value, the historical information at time t, and the actual state value corresponding to the historical information at time t into the first loss function to obtain the loss function value of the j-th iteration and subsequent steps are executed again until the j-th iteration reaches the iteration stopping condition, and the trained value function model corresponding to time t is obtained.

[0019] In a possible implementation, the first loss function is expressed by the following formula:

[0020]

[0021] Among them, J critic A first loss function representing the value function model critic to be trained; represents the expected value calculation function; Represents the historical information corresponding to the time t A steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Indicates the historical information corresponding to the time t+1 The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; t is the loss equation, Represents the historical information corresponding to time t Under the condition of, the current strategy π of the strategy function model to be trained is used according to the historical information h t Select the current estimate at time t as the action π(h t ), the losses caused by e t represents the actual state value x at the time t t The current estimated value at the time t The error between ; γ represents a preset discount factor; V represents the value function model to be trained; w represents the current model parameters of the value function model to be trained.

[0022] In a possible implementation, the model parameters of the value function model to be trained are updated based on the gradient descent method, which is expressed by the following formula:

[0023]

[0024] in, Represents the first loss function J critic With respect to the gradient of the model parameter w, the first loss function J is calculated critic The partial derivative with respect to the model parameter w is obtained; α represents the preset learning rate.

[0025] In a possible implementation, when the nonlinear system can be modeled based on a state transition model, the steady-state estimated evaluation value of the trained value function model corresponding to the time t is used to iteratively train the policy function model to be trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to the time t, including:

[0026] Obtaining a second loss function corresponding to the policy function model to be trained, wherein the second loss function is established based on minimizing the sum of the error of the current estimated value at the time t and the steady-state estimated evaluation value output by the trained value function model corresponding to the time t;

[0027] Inputting the historical information corresponding to the time t+1 into the trained value function model corresponding to the time t to obtain the steady-state estimation evaluation value;

[0028] Input the current estimated value at time t, the actual state value at time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value of the kth iteration; k is a positive integer starting from 1;

[0029] When the iteration stopping condition is not reached in the kth iteration, the model parameters of the policy function model to be trained are updated based on the gradient descent method;

[0030] After the parameters are updated, the value of k is set equal to k+1, and the step of inputting the current estimated value at the time t, the actual state value at the time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value of the kth iteration and subsequent steps are performed again until the kth iteration reaches the iteration stop condition, and the trained policy function model corresponding to the time t is obtained.

[0031] In a possible implementation, the second loss function is expressed by the following formula:

[0032]

[0033] Among them, J actor A second loss function representing the policy function model actor to be trained; represents the expected value calculation function; Represents the historical information corresponding to the time t A steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Indicates the historical information corresponding to the time t+1 The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; t is the loss equation, Represents the historical information corresponding to time t Under the condition of, the current strategy π of the strategy function model to be trained is used according to the historical information h t Select the current estimate at time t as the action π(h t ), the losses caused by e trepresents the actual state value x at the time t t The current estimated value at the time t The error between γ and γ represents the preset discount factor; Indicates the historical information corresponding to time t+1 After inputting the trained value function model corresponding to the time t with model parameter w, the steady-state estimated evaluation value is obtained.

[0034] In a possible implementation, the model parameters of the policy function model to be trained are updated based on the gradient descent method, which is expressed by the following formula:

[0035]

[0036] in, Denotes the second loss function J actor Model parameters of the policy function model to be trained The gradient of the second loss function J is calculated by actor About Model Parameters The partial derivative of is obtained; β represents the preset learning rate.

[0037] In a possible implementation, when the nonlinear system cannot be modeled based on the state transition model, the steady-state estimated evaluation value of the trained value function model corresponding to the time t is used to iteratively train the policy function model to be trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to the time t, including:

[0038] Obtaining a third loss function corresponding to the policy function model to be trained, wherein the third loss function is established based on minimizing a steady-state estimated evaluation value output by the trained value function model corresponding to the time t;

[0039] Inputting the historical information corresponding to the time t into the trained value function model corresponding to the time t to obtain the steady-state estimation evaluation value;

[0040] Inputting the steady-state estimation evaluation value into the third loss function to obtain the loss function value of the kth iteration; wherein k is a positive integer starting from 1;

[0041] When the iteration stopping condition is not reached in the kth iteration, the model parameters of the policy function model to be trained are updated based on the gradient descent method;

[0042] After the parameters are updated, the value of k is set equal to k+1, and the step of inputting the steady-state estimation evaluation value into the third loss function to obtain the loss function value of the kth iteration and the subsequent steps are executed again, until the kth iteration reaches the iteration stop condition, and the trained strategy function model corresponding to time t is obtained.

[0043] In a possible implementation, the third loss function is expressed by the following formula:

[0044]

[0045] Among them, J actor A third loss function representing the policy function model actor to be trained; represents the expected value calculation function; Represents the historical information corresponding to the time t A steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Indicates that the historical information corresponding to time t and the current estimate at time t , after inputting the trained value function model corresponding to the time t with the model parameter w, the steady-state estimated evaluation value is obtained;

[0046] Accordingly, the model parameters of the policy function model to be trained are updated based on the gradient descent method, which is expressed by the following formula:

[0047]

[0048] in, Denotes the third loss function J actor Model parameters of the policy function model to be trained The gradient of the third loss function J is calculated by actor About Model Parameters The partial derivative of is obtained; β represents the preset learning rate.

[0049] According to another aspect of the present disclosure, a data-driven nonlinear system state estimation device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0050] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0051] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0052] By obtaining the historical information of the system state to be estimated of the nonlinear system at time t; determining the action at time t based on the historical information at time t based on the strategy function model to be trained, and obtaining the current estimated value at time t; determining the historical information at time t+1 based on the current estimated value at time t and the system observation value at time t+1; determining the first estimated evaluation value corresponding to the historical information at time t and the second estimated evaluation value corresponding to the historical information at time t+1 based on the value function model to be trained; based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t, iteratively training the value function model to be trained using a reinforcement learning algorithm to obtain the trained state value corresponding to time t value function model; based on the steady-state estimation evaluation value of the trained value function model corresponding to time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to time t; after the policy function model training is completed, the state estimation value of the system state is determined based on the system observation value of the nonlinear system based on the trained policy function model; the policy function model can be trained offline using a reinforcement learning algorithm based on historical information, and the obtained policy function model can be used to perform state estimation on a nonlinear system with non-Gaussian noise online, and the state estimation of the system state can be achieved without relying on accurate system modeling, thereby reducing the difficulty of system estimation of nonlinear systems and improving the accuracy of system estimation.

[0053] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0055] Figure 1 A flow chart showing a data-driven nonlinear system state estimation method according to an embodiment of the present disclosure;

[0056] Figure 2 A flow chart showing a data-driven nonlinear system state estimation method according to another embodiment of the present disclosure;

[0057] Figure 3 A schematic diagram showing the training process of the value function model and the policy function model of an embodiment of the present disclosure;

[0058] Figure 4 A schematic diagram showing a model training code implementation of an embodiment of the present disclosure;

[0059] Figure 5 A schematic diagram showing a comparison of RMSE results of the data-driven nonlinear system state estimation of the present application according to an embodiment of the present disclosure and the existing state estimation method;

[0060] Figure 6 A block diagram showing a data-driven nonlinear system state estimation device according to an embodiment of the present disclosure;

[0061] Figure 7 A block diagram of a data-driven nonlinear system state estimation device according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0062] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0063] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0064] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0065] First, several terms involved in this application are introduced.

[0066] Reinforcement Learning (RL) is a machine learning method that achieves decision-making and behavior optimization through interactive learning between an agent and the environment. In reinforcement learning, the agent explores the environment by trying different actions and adjusts its strategy based on the feedback (rewards) from the environment to maximize the long-term accumulated reward value. Among them, the agent and the environment are the two core components of reinforcement learning.

[0067] Agent: It is the decision maker in the reinforcement learning system, which achieves a certain goal by perceiving the environment and performing actions. The agent can be a robot, a software program, or any entity that can interact with the environment and take actions.

[0068] Environment: It is the external world in which the agent is located, including everything that the agent can perceive and influence. The environment provides state information to the agent and gives corresponding feedback based on the agent's actions. In this application, the agent is a software program, the environment is constructed based on historical information at each moment, and the action indicates the estimated value of the state at each moment. At this time, the agent interacts with the environment through the strategy to determine the estimated value of the state, and the environment provides estimated feedback to the agent based on the estimated value of the state.

[0069] Interaction: The interaction between the agent and the environment is carried out through a cyclic process. The agent perceives the current state (s) at each time step, such as: the current state is the historical information at time t is the estimated value of the state at historical moment k, y k is the observed value of the system at historical time k. Based on the adopted strategy π, an action (a) is selected, such as: the action is the current estimated value of the system state at time t ; Then execute the action, the environment updates the state and gives the agent an evaluation feedback (r), such as updating the current state to the historical information at time t+1, and feeding back the action corresponding to the historical information at time t+1 This process is repeated continuously, allowing the agent to learn the optimal strategy through trial and error to maximize the cumulative reward.

[0070] In reinforcement learning, state, action, and reward are the basic elements to describe the interaction between an agent and its environment.

[0071] The goal of reinforcement learning is to find a strategy that maximizes the cumulative reward of the agent in its interactions with the environment.

[0072] Policy (π): It is the rule or function for the agent to choose an action in each state. It can be deterministic (choose a fixed action in each state) or stochastic (choose different actions with a certain probability in each state). The policy can be expressed as π(s) = a, indicating that action a is chosen in state s, or π(a|s), indicating the probability of choosing action a in state s.

[0073] Schematically, an algorithmic framework for reinforcement learning includes an actor-critic model, which is a reinforcement learning algorithm based on a value function and a policy function. The actor (i.e., the policy function model) is responsible for learning the policy function to select actions based on the current state; the critic (i.e., the value function model) is responsible for learning the value function to evaluate the value of the state estimate of the current state. The actor-critic model continuously optimizes the policy through two steps: policy evaluation (PEV) and policy improvement (PIM).

[0074] In the policy evaluation step, the critic updates the value function based on the current state and the value of the action. In the policy improvement step, the actor updates the policy function based on the evaluation results of the critic. These two steps are performed alternately until the model convergence condition is reached.

[0075] Specifically, in the policy evaluation step, the critic uses a value function to evaluate the value of the current state. The value function can be a state value function (V function) or an action value function (Q function). In the policy improvement step, the actor updates the policy function based on the evaluation results of the critic. A common method is to use a policy gradient algorithm to update the policy parameters by maximizing the expected value of the cumulative reward. The policy gradient algorithm can use the gradient ascent method or the gradient descent method to update the policy parameters so that the agent can choose a better action.

[0076] Since traditional system estimation methods rely on system models and parameter accuracy, or at least on the structural model of the system. Based on this, state estimation needs to shift from model-driven estimation to data-driven estimation, in which system dynamics are represented by data samples rather than explicit system models, thereby reducing dependence on system models. Reinforcement Learning (RL) is becoming a typical method for data-driven control problems. In addition, RL can handle high-dimensional problems by utilizing approximate functions with strong fitting properties (such as deep neural networks). There have been some application attempts of reinforcement learning in the field of optimal estimation, but these methods are limited by the assumption of zero-mean Gaussian noise and explicit modeling of the system.

[0077] Based on this, this application proposes a data-driven nonlinear system state estimation method, called Data-Driven Approximate Optimal Filter (DAOF), which is represented by a deep neural network and trained using a reinforcement learning algorithm. This method is applicable to nonlinear systems with non-Gaussian noise, and can perform state estimation on complex systems where the state space equation cannot be explicitly constructed.

[0078] The data-driven nonlinear system state estimation method proposed in this application is introduced in detail below.

[0079] Figure 1 The flowchart of a data-driven nonlinear system state estimation method according to an embodiment of the present disclosure is shown. This embodiment takes the method used in an electronic device with computing capabilities such as a server or a user terminal as an example for explanation, wherein the user terminal includes but is not limited to: a mobile phone, a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device. Figure 1 As shown, the method includes:

[0080] Step 101, obtaining historical information of a system state to be estimated of a nonlinear system at time t, the historical information including state estimation values ​​at N historical moments before time t and historical observation values ​​at the next moment of each historical moment.

[0081] Wherein, t is a positive integer starting from 0, and N is a preset positive integer.

[0082] In this embodiment, a nonlinear system refers to a physical system to be state estimated. The nonlinear system refers to a dynamic system in reality, such as a vehicle control system, a power system, etc. The system state of a nonlinear system changes with time. This embodiment does not limit the implementation method of the nonlinear system. In this embodiment, a nonlinear system can be a nonlinear industrial object with complex variables and / or variable environmental interference. The system state refers to the system variables in a nonlinear system, such as the system state is the position of a vehicle and / or the speed of a vehicle. Estimating the system state can realize monitoring and early warning of the nonlinear system. This embodiment does not limit the application scenario of the estimation result of the system state.

[0083] In one example, N is all historical moments before time t. In this case, the historical information at time t is: is the estimated value of the state at historical moment k, y k is the observed value of the system at historical time k. The parameters with the same symbols have the same meanings in the following text, and will not be described in detail in this embodiment. tIt includes all the historical moments before time t, that is, all the state estimates from time 0 to time t-1, and the historical observations from time 1 to time t at the next moment of each historical moment. The historical observations refer to the system observations at the next moment of the historical moment. The system observations are data that can be directly obtained from the nonlinear system, and the values ​​usually contain some noise. The system observations are the external manifestation of the system state, but they are not directly equivalent to the system state. The system observations can be used to update the estimate of the system state.

[0084] In another example, N is a fixed value. In this case, the historical information at time t uses a sliding window of fixed length N to approximate all the historical information at time t. Schematically, the historical information at time t is expressed as follows:

[0085]

[0086] The value of N is determined based on real-time requirements and accuracy requirements. By selecting an appropriate sliding window length N, a balance can be achieved between real-time requirements and accuracy.

[0087] Optionally, the historical information in this embodiment is all historical information after position encoding, so that the relative position relationship between the data corresponding to different historical moments can be obtained through subsequent reinforcement learning, thereby improving the accuracy of state estimation. The encoding method of historical information includes but is not limited to: sine-cosine position encoding method, learning encoding method, sequential index encoding method, or Fourier encoding method, etc. This embodiment does not limit the encoding method of historical information.

[0088] Since the sine-cosine position coding method has the characteristics of periodicity, this coding method can be seamlessly extended to sequences of any length; compared with learning-based coding, no additional training is required, reducing coding complexity; compared with sequential index coding, it provides continuous position information rather than discrete indexes, which helps the model better understand the relative position in the sequence; compared with Fourier coding, it is simpler to implement and the frequency control is more intuitive. Based on this, this embodiment uses the sine-cosine position coding method to perform position coding on historical data corresponding to historical moments to obtain historical information as an example for explanation.

[0089] The sine-cosine position encoding method is expressed by the following formula:

[0090] PE(pos,2i)=sin(pos / 10000 2i / d ),

[0091] PE(pos,2i+1)=cos(pos / 10000 2i / d ),

[0092] Among them, PE(pos,2i) represents the position code at the 2i index position at the time step pos (i.e. the moment in this embodiment); PE(pos,2i+1) represents the position code at the 2i+1 index position at the time step pos (i.e. the moment in this embodiment); d represents the data dimension of the position code, which is consistent with the data dimension of the historical data to be encoded, and i represents the index position in the position code, which is an integer starting from 0.

[0093] Step 102, based on the policy function model to be trained, using the historical information at time t to determine the action at time t, and obtain the current estimated value at time t.

[0094] In this embodiment, the policy function model and the value function model described below are determined by a pre-constructed reinforced estimation problem (REP). The reinforced estimation problem refers to an optimal estimation problem constructed with the criterion of minimizing the cumulative error with a discount factor, and the relationship between the reinforced estimation problem and reinforcement learning is established to obtain the policy function model and the value function model.

[0095] The reinforcement estimation problem can be expressed as follows:

[0096]

[0097] in, represents the optimal state estimation sequence from time k to infinity; Represents the historical information h at a given time t t Under the condition of , calculate the expected value of the weighted loss function value of all future time steps; x k is the estimated state value required at time k, or the actual state value of the system state at time k; is the estimated value of the state at time k. The closer the estimated value is to the actual state value, the better. t is the historical information at time t, y k is the system observation value at time k, φ(·,·) is the loss function that measures the gap between the state estimate and the actual state value, and γ∈[0,1) is the discount factor; Represents finding an optimal state estimation sequence from time t to infinity Minimize the expected value.

[0098] Establishing the connection between the reinforcement estimation problem and reinforcement learning includes: transforming the historical information h at time t t As the state in reinforcement learning, the current estimated value at time t As an action in reinforcement learning, square the estimated error Defined as the loss function, the state transition probability is defined as Represents the historical information h at a given time t t and the current estimate at time t In the case of , the next state (i.e. the historical information at time t+1) h t+1 The probability distribution of , the obtained value function can be expressed as follows:

[0099]

[0100] Where V represents the state value function, Represents the historical information h at a given state at time t t In the case of , the weighted sum of the loss function values ​​from time t to each future time is calculated; Q represents the action value function, Represents the historical information h at a given state at time t t , and the action is the current estimate In the case of , the weighted sum of the loss function values ​​from time t to each future time is calculated. Among them, the current estimated value at time t refers to the state estimated value at time t determined using the current policy function.

[0101] In this embodiment, a parameterized neural network is used to approximate the value function to obtain a value function model, namely, critic. The value function model can be expressed as:

[0102]

[0103] in, Represents the historical information at a given state at time t The evaluation value output by the value function in the case of Indicates that the model parameter of the neural network is w, and the state is the historical information h at time t t The evaluation value output by the state value function model is . Here, V includes the above state value function and the above action value function.

[0104] According to the above enhanced estimation problem, we can know that the optimal state estimation sequence is obtained by minimizing the value function, that is, It can be expressed by the following formula:

[0105]

[0106] Among them, V(h t ) represents the historical information h of a given state at time t t The state value function in the case of Represents the historical information h of a given state at time t t , and the action is the current estimate The action-value function in the case of .

[0107] By solving The resulting filter is actually the reinforcement learning strategy π. For reinforcement learning, the optimal strategy can be derived through the Bellman Equation. The basic idea of ​​the Bellman Equation is that the value function of an optimal strategy (i.e., the maximum cumulative reward that can be obtained by following the strategy starting from a certain state) can be expressed by considering the optimal value function of the subsequent state after taking the optimal action in the current state. This equation is a necessary and sufficient condition for optimality. Based on this, in order to obtain the optimal filter (or optimal strategy), the Bellman equation for the reinforcement estimation problem is derived as follows:

[0108]

[0109] Among them, V * (h t ) indicates the state h t The optimal state value function of e t represents the actual state value x at the time t t The current estimated value at the time t The error between Indicates that the state distribution of historical information at time t+1 obeys In the case of , calculate the loss function value of the current state value and the next state h t+1 The expected value of the weighted sum of the optimal state value functions, correspondingly, To find the current state value that minimizes the expected value of the weighted sum , thus obtaining the state h t The optimal state value function of . Indicates the state h t And actions The optimal action value function of Indicates that the state distribution of historical information at time t+1 obeys In the case of t+1 Take the best action The expected value of the optimal action-value function, represents the optimal action that minimizes the expected value of the optimal action-value function Indicates that the state distribution of historical information at time t+1 obeys , calculate the expected value of the loss function value of the current state value.

[0110] According to the above formula, in this embodiment, the Bellman equation is explored one step forward to determine the optimal value function as an example for explanation. In this way, the problem of high computational complexity caused by searching for estimated values ​​at all future moments when directly minimizing the value function is avoided, and computational efficiency can be improved. In other embodiments, the Bellman equation can also be explored at least two steps forward. This embodiment does not limit the implementation method of the Bellman equation for the enhanced estimation problem.

[0111] In this embodiment, different filter structures correspond to state estimation with a model and state estimation without a model, and the state estimation with a model and state estimation without a model are respectively introduced below.

[0112] 1. For state estimation with a model, that is, when the nonlinear system can be modeled based on the state transition model, the state estimation can be implemented based on the state transition model. Accordingly, the structure of the filter is as follows:

[0113]

[0114] Among them, f(·) is the state transfer function in system modeling, and in this embodiment, f(·) is called the prediction part, π(·) is the filtering strategy or strategy function, and in this embodiment, π(·) is called the update part. Under this strategy structure, in order to solve the optimal strategy, the strategy iteration method in reinforcement learning is used to derive the iterative process of strategy evaluation and strategy improvement, which is expressed by the following formula:

[0115]

[0116] Among them, j is the number of iterations (or times) in the strategy evaluation, It means that the state value function uses the current strategy π and the historical information h at time t after the j+1th iteration. t Determine the evaluation value, It means that the state value function uses the current strategy π and the historical information h at time t+1 after the jth iteration. t+1 Determine the evaluation value. p(h t+1 |h t ,π(h t )) means that given the current state h t and by the policy function π in state h t The action π(h t ), the next state h t+1 The probability distribution of . k represents the number of iterations in the strategy improvement, π k+1 (h t ) indicates that after the k+1th iteration, the policy function π is in state h t Next, select the action. For a given strategy πk (i.e., the policy function π of the kth iteration) is the steady-state value function reached after policy evaluation, or the function when the state value function is iterated a certain number of times and reaches convergence.

[0117] 2. For model-free state estimation, that is, when the nonlinear system cannot be modeled based on the state transition model, state estimation cannot be implemented based on the state transition model. Accordingly, the structure of the filter is as follows:

[0118]

[0119] Under this strategy structure, in order to solve the optimal strategy, the strategy iteration method in reinforcement learning is used to derive the iterative process of strategy evaluation and strategy improvement, which can be expressed as follows:

[0120]

[0121] in, It means that the action value function uses the current strategy π to deal with the historical information h at time t after the j+1th iteration. t and current estimates Determine the evaluation value, It means that the action value function uses the current strategy π and the historical information h at time t+1 after the jth iteration. t+1 , and the historical information h at time t+1 t+1 The action π(h t+1 ) determines the evaluation value. For a given strategy π k The steady-state action value function reached after strategy evaluation, or the function when the action value function reaches convergence after a certain number of iterations. k+1 (h t ) indicates that after the k+1th iteration, the policy function π is in state h t Next, select the action. Indicates the search for Strategy to achieve the minimum.

[0122] Optionally, for state estimation with a model, the iterative process in state estimation without a model can also be adopted; for state estimation without a model, the iterative process in state estimation with a model can also be adopted. In this case, since the state transition model of the state estimation without a model cannot be determined, the steady-state distribution of the state can be approximated by multiple samplings to determine the state transition probability.

[0123] In this embodiment, a parameterized neural network is used to approximate the policy function to obtain a policy function model, namely, actor. The policy function model can be expressed as:

[0124]

[0125] in, Represents the historical information at a given state at time t In the case of, the action taken by the policy function, The model parameters of the neural network are represented as , the state is the historical information h at time t t Output action in case of .

[0126] After the policy function model is constructed, the policy function model is the policy function model to be trained. In this embodiment, the policy function model to be trained refers to a policy function model that has not been trained, that is, the policy function model in the training process also belongs to the policy model to be trained.

[0127] At this time, the action at time t is determined based on the historical information at time t based on the policy function model to be trained, and the current estimated value at time t is obtained, including: inputting the historical information at time t into the policy function model to be trained, obtaining the action at time t output by the policy function model to be trained, and the action at time t is the current estimated value at time t

[0128] Step 103, based on the current estimated value at time t and the system observed value at time t+1, determine the historical information at time t+1.

[0129] If the historical information is all the historical data at a certain time, then the historical information h at time t+1 t+1 It can be expressed by the following formula:

[0130]

[0131] in, represents the current estimated value at time t, y t+1 Represents the system observation value at time t+1.

[0132] If the historical information is the historical data of N moments in the past, then the historical information h at time t+1 t+1 It can be expressed by the following formula:

[0133]

[0134] Step 104 : Determine, based on the value function model to be trained, a first estimated evaluation value corresponding to the historical information at time t and a second estimated evaluation value corresponding to the historical information at time t+1.

[0135] The value function model to be trained refers to a value function model that has not been trained. That is, the value function model in the training process also belongs to the strategy model to be trained.

[0136] Based on the value function model to be trained, determine a first estimated evaluation value corresponding to the historical information at time t and a second estimated evaluation value corresponding to the historical information at time t+1, including: inputting the historical information at time t into the value function model to be trained to obtain the first estimated evaluation value output by the value function model; inputting the historical information at time t+1 into the value function model to be trained to obtain the second estimated evaluation value output by the value function model.

[0137] Step 105, based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t, the value function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain a trained value function model corresponding to time t.

[0138] In this embodiment, based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t, the value function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain a trained value function model corresponding to time t, including steps 1051-1054:

[0139] Step 1051, obtaining a first loss function corresponding to the value function model to be trained.

[0140] In this embodiment, the first loss function is determined based on the strategy evaluation process in reinforcement learning. For the above-mentioned state estimation with model and state estimation without model, the first loss function obtained based on the corresponding strategy evaluation process can be expressed by the following formula:

[0141]

[0142] Among them, J critic Represents the first loss function of the value function model critic to be trained; represents the expected value calculation function; Represents the historical information corresponding to time t The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Represents the historical information corresponding to time t+1 The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; l t is the loss equation, Represents the historical information corresponding to time t Under the condition of, the current strategy π of the strategy function model to be trained is used according to the historical information h t Select the current estimate at time t as the action π(h t ), the losses caused by e tRepresents the actual state value x at time t t and the current estimate at time t ; γ represents the preset discount factor; V represents the value function model to be trained, where V includes the value function model corresponding to the action value function and the value function model corresponding to the state value function; w represents the current model parameters of the value function model to be trained.

[0143] The steady-state distribution refers to the distribution when the mean change of the state tends to zero after a certain time step, when the strategy is given and the state transition probability starts to run from the initial state. For state estimation with a model, since the state transition probability p(h t+1 |h t ,π(h t )) can be obtained explicitly through system modeling (such as in the above), so the state can be derived given the current policy π and For model-free state estimation, since both the state transition probability and the strategy are uncertain and in probabilistic form, the state obtained after each time step may be different. In this case, the steady-state distribution of the state can be approximated by sampling, that is, starting from the initial state and running a certain time step according to a given strategy, sampling the state corresponding to the time step, and after multiple runs, using the result of each sampling to approximate the steady-state distribution of the state.

[0144] Step 1052: Input the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t into the first loss function to obtain the loss function value of the jth iteration, where j is a positive integer starting from 1.

[0145] According to the first loss function mentioned above, it is That is the first estimated evaluation value; is the second estimated evaluation value, is the current estimated value, x t is the actual state value at time t, and the loss function value J of the jth iteration is obtained critic .

[0146] Step 1053, when the iteration stopping condition is not reached in the j-th iteration, the model parameters of the value function model to be trained are updated based on the gradient descent method.

[0147] The model parameters of the value function model to be trained are updated based on the gradient descent method, which is expressed by the following formula:

[0148]

[0149]

[0150] in, Represents the first loss function J critic With respect to the gradient of the model parameter w, the first loss function J is calculated critic The partial derivative with respect to the model parameter w is obtained; α represents the preset learning rate.

[0151] Step 1054, after the parameters are updated, let the value of j be equal to j+1, and again execute the step of inputting the first estimated evaluation value, the second estimated evaluation value, the historical information at time t, and the actual state value corresponding to the historical information at time t into the first loss function to obtain the loss function value of the j-th iteration (i.e., step 1052) and subsequent steps, until the j-th iteration reaches the iteration stop condition, and obtains the trained value function model corresponding to time t.

[0152] The iteration stop condition includes but is not limited to: the loss function value of the jth iteration indicates that the value function model converges; or the value of j reaches the first preset number of iterations. At this time, after step 1052, it also includes determining whether the iteration reaches the iteration stop condition. If the iteration stop condition is reached, the model parameters corresponding to the iteration are used as the model parameters of the trained value function model corresponding to time t to obtain the trained value function model corresponding to time t; if the iteration stop condition is not reached, step 1053 is executed.

[0153] Step 106, based on the steady-state estimated evaluation value of the trained value function model corresponding to time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to time t.

[0154] In this embodiment, since the filter structures of the state estimation with model and the state estimation without model are different, the loss function when training the policy function model is also different accordingly.

[0155] 1. When the nonlinear system is modeled based on the state transition model, based on the steady-state estimated evaluation value of the trained value function model corresponding to time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to time t, including steps 1061-1065:

[0156] Step 1061, obtain a second loss function corresponding to the strategy function model to be trained, wherein the second loss function is established based on minimizing the sum of the error of the current estimated value at time t and the steady-state estimated evaluation value output by the trained value function model corresponding to time t.

[0157] The second loss function is expressed as follows:

[0158]

[0159] Among them, J actor Represents the second loss function of the policy function model actor to be trained; represents the expected value calculation function; Represents the historical information corresponding to time t The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Represents the historical information corresponding to time t+1 The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; l t is the loss equation, Represents the historical information corresponding to time t Under the condition of, the current strategy π of the strategy function model to be trained is used according to the historical information h t Select the current estimate at time t as the action π(h t ), the losses caused by e t Represents the actual state value x at time t t and the current estimate at time t The error between γ and γ represents the preset discount factor; Indicates the historical information corresponding to time t+1 After inputting the trained value function model corresponding to the time t with model parameter w, the steady-state estimated evaluation value is obtained.

[0160] Step 1062, input the historical information corresponding to time t+1 into the trained value function model corresponding to time t to obtain a steady-state estimated evaluation value.

[0161] The steady-state estimated evaluation value is the above

[0162] Step 1063, input the current estimated value at time t, the actual state value at time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value of the kth iteration.

[0163] Here, k is a positive integer starting from 1.

[0164] The current estimate at time t is The actual state value at time t is x t .

[0165] Step 1064, when the iteration stopping condition is not reached in the kth iteration, the model parameters of the policy function model to be trained are updated based on the gradient descent method.

[0166] The model parameters of the policy function model to be trained are updated based on the gradient descent method, which is expressed by the following formula:

[0167]

[0168] in, Represents the second loss function J actor Model parameters of the policy function model to be trained The gradient of the second loss function J is calculated actor About Model Parameters The partial derivative of is obtained; β represents the preset learning rate.

[0169] Step 1065, after the parameters are updated, set the value of k equal to k+1, and again execute the step of inputting the current estimated value at time t, the actual state value at time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value of the kth iteration and the subsequent steps, until the kth iteration reaches the iteration stop condition, and obtains the trained policy function model corresponding to time t.

[0170] The iteration stop condition includes but is not limited to: the loss function value of the kth iteration indicates that the policy function model converges; or the value of k reaches the second preset number of iterations. At this time, after step 1063, it also includes determining whether the iteration reaches the iteration stop condition. If the iteration stop condition is reached, the model parameters corresponding to the iteration are used as the model parameters of the trained policy function model corresponding to time t to obtain the trained policy function model corresponding to time t; if the iteration stop condition is not reached, step 1064 is executed.

[0171] 2. When the nonlinear system cannot be modeled based on the state transition model, based on the steady-state estimated evaluation value of the trained value function model corresponding to time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to time t, including the following steps 1066-10691:

[0172] Step 1066, obtaining a third loss function corresponding to the policy function model to be trained, wherein the third loss function is established based on the steady-state estimated evaluation value output by the trained value function model corresponding to the minimization time t.

[0173] The third loss function is expressed as follows:

[0174]

[0175] Among them, J actor Represents the third loss function of the policy function model actor to be trained; represents the expected value calculation function; Represents the historical information corresponding to time t The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Indicates that the historical information corresponding to time t and the current estimate at time t , after inputting the trained value function model corresponding to the time t with the model parameter w, the steady-state estimated evaluation value is obtained.

[0176] Step 1067, input the historical information corresponding to time t into the trained value function model corresponding to time t to obtain a steady-state estimated evaluation value.

[0177] Step 1068: Input the steady-state estimation evaluation value into the third loss function to obtain the loss function value of the kth iteration, where k is a positive integer starting from 1.

[0178] The steady-state estimated evaluation value is the value in the third loss function above.

[0179] Step 1069, when the iteration stopping condition is not reached in the kth iteration, the model parameters of the policy function model to be trained are updated based on the gradient descent method.

[0180] The model parameters of the policy function model to be trained are updated based on the gradient descent method, which is expressed by the following formula:

[0181]

[0182]

[0183] in, Represents the third loss function J actor Model parameters of the policy function model to be trained The gradient of the third loss function J is calculated actor About Model Parameters The partial derivative of is obtained; β represents the preset learning rate.

[0184] Step 10691, after the parameters are updated, set the value of k equal to k+1, and execute again the step of inputting the steady-state estimation evaluation value into the third loss function to obtain the loss function value of the kth iteration and the subsequent steps, until the kth iteration reaches the iteration stop condition, and obtain the trained policy function model corresponding to time t.

[0185] The iteration stop condition in this step is the same as or different from the iteration stop condition in step 1065. The iteration stop condition in this step includes but is not limited to: the loss function value of the kth iteration indicates that the policy function model converges; or, the value of k reaches the third preset number of iterations. This embodiment does not limit the implementation method of the iteration stop condition. At this time, after step 1068, it also includes determining whether the iteration has reached the iteration stop condition. If the iteration stop condition is reached, the model parameters corresponding to the iteration are used as the model parameters of the trained policy function model corresponding to time t to obtain the trained policy function model corresponding to time t; if the iteration stop condition is not reached, step 1069 is executed.

[0186] Step 107, after the policy function model is trained, the state estimation value of the system state is determined based on the trained policy function model using the system observation value of the nonlinear system.

[0187] Optionally, after step 106, determine whether the training stop condition is met; if the training stop condition is met, stop training and obtain the strategy function model; if the training stop condition is not met, set t=t+1 and execute step 101 again until the training stop condition is met.

[0188] Illustratively, the training stop conditions include but are not limited to: the value of t reaches a preset number of steps, such as 500; or the change in model parameters of the strategy function model at two adjacent moments is less than a preset change threshold. This embodiment does not limit the setting method of the training stop conditions.

[0189] After the policy function model training is completed, the policy function model is obtained After that, for the nonlinear system, starting from time 0, the system observation value y1 at time 1 is input into the strategy function model , and obtain the state estimation value at time 1; then, input the system observation value y2 at time 2 into the strategy function, so that the strategy function obtains the state estimation value at time 2 based on the state estimation value at time 1 and the system observation value y2, and repeats this cycle until the state estimation at each time is completed.

[0190] refer to Figure 2 According to the above content, the data-driven nonlinear system state estimation method in this embodiment includes at least the following working processes:

[0191] 1. Strengthen the construction of the estimation problem: In this embodiment, the optimal estimation problem is constructed based on the criterion of minimizing the cumulative error with a discount factor;

[0192] 2. Strengthening estimation strategy structure design: In this embodiment, model-based and model-free strengthening estimation strategies are designed according to the strengthening estimation problem form, that is, the strategy function model mentioned above;

[0193] 3. Offline training of network parameters: In this embodiment, the network model is designed to approximate the reinforcement estimation problem value function and estimation strategy, and the model parameters are solved by the gradient descent method;

[0194] refer to Figure 3 The offline training process shown in the figure obtains the actual state value x of the system at time t. t and the system observation y t ; The historical information at time t Input the policy function model to be trained and the value function model to be trained respectively. represents the estimated value of the state from time tN to time t, y t-N+1:t+1 Represents the system observation value from time t-N+1 to time t+1. The policy function model to be trained outputs the current estimated value at time t , the value function model to be trained outputs the first estimated evaluation value of the current estimated value or ; Based on the current estimate at time t With the actual state value x t The error between t , and the first estimated evaluation value, through the policy evaluation iterative process and the policy improvement iterative process, use the gradient descent algorithm to iteratively update the model parameters of the value function model and the policy function model to obtain the trained policy function model corresponding to time t.

[0195] refer to Figure 4 The iterative update process of the value function model and the policy function model shown in Figure 4 It can be seen that, assuming that the model parameter w of the initialized value function model to be trained is the initialization parameter w0, the model parameter of the initialized strategy function model to be trained is For initialization parameters , the learning rate is initialized to α and β. The historical information h0∈S at the initialization time 0 means that the historical information h0 belongs to the state space S, and the state space S includes the set of all possible states of the system. Repeat the following process: the historical information at time t Input the policy function model π to be trained, obtain and store the historical information at time t+1 ; Calculate the gradient of the value function model to be trained (i.e., critic) through the first loss function, and update the model parameters of the value function model to be trained to After the value function model to be trained meets the iteration stop condition, the gradient of the policy function model to be trained is calculated by the second loss function or the third loss function, and the model parameters of the policy function model to be trained (i.e., actor) are updated to Until the policy function model to be trained meets the iteration stop condition, the trained policy function model corresponding to time t is obtained; until the model training stops when the training stop condition is met, the final policy function model is obtained.

[0196] 4. Online application of estimation network: input the system observation value of nonlinear system into the trained policy function model, and output the state estimation value of system state online.

[0197] In summary, the data-driven nonlinear system state estimation method in this embodiment obtains the historical information of the system state to be estimated of the nonlinear system at time t; determines the action at time t based on the historical information at time t based on the strategy function model to be trained, and obtains the current estimated value at time t; determines the historical information at time t+1 based on the current estimated value at time t and the system observation value at time t+1; determines the first estimated evaluation value corresponding to the historical information at time t and the second estimated evaluation value corresponding to the historical information at time t+1 based on the value function model to be trained; and iteratively trains the value function model to be trained using a reinforcement learning algorithm based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at time t, and the actual state value at time t. Train to obtain the trained value function model corresponding to time t; based on the steady-state estimation evaluation value of the trained value function model corresponding to time t, use the reinforcement learning algorithm to iteratively train the policy function model to be trained to obtain the trained policy function model corresponding to time t; after the policy function model training is completed, use the system observation value of the nonlinear system based on the trained policy function model to determine the state estimation value of the system state; the policy function model can be trained offline using the reinforcement learning algorithm based on historical information, and the obtained policy function model can be used to perform state estimation on nonlinear systems with non-Gaussian noise online, and does not need to rely on accurate system modeling to achieve state estimation of the system state, thereby reducing the difficulty of system estimation of nonlinear systems and improving the accuracy of system estimation.

[0198] The data-driven nonlinear system state estimation method provided by the present application is described below by taking an example. This example takes the nonlinear system as a car center of mass side slip angle model as an example, and establishes the following equation based on tire dynamics:

[0199]

[0200] Where β is the sideslip angle of the center of mass, and accordingly, represents the derivative of the sideslip angle at the center of mass; ω is the yaw rate of the vehicle, and accordingly, represents the derivative of the vehicle's yaw rate; δ fThe steering wheel angle is used as the control input of the model. The tire dynamics model is based on the Pacejka model, or the Magic Formula. The meaning of the model parameters in the system model and the parameter values ​​are shown in Table 1 below.

[0201] Table 1:

[0202]

[0203] The state to be estimated in the above system model is X = [β, ω]. Considering the process noise ξ, the forward Euler method is used to discretize and obtain the state equation:

[0204]

[0205] Among them, X t represents the estimated value of the system state at time t; X t+1 represents the state estimation value of the system state at time t+1; ΔT represents the length of time from time t to time t+1, that is, the time interval; represents the rate of change of the system state, that is, the average rate of change within the time interval ΔT; ξ t represents the process noise at time t.

[0206] The observation of the above system model is Y = [a y ,ω] T , where a y is the longitudinal acceleration of the vehicle, ζ is the observation noise, and the observation equation is as follows:

[0207]

[0208] Where y1 represents the longitudinal acceleration of the vehicle a y y2 represents the system observation value of the vehicle’s yaw angular velocity ω.

[0209] In this example, the process noise ξ t is Gaussian mixture noise, observation noise ζ t Take Laplace noise as an example. t and t satisfy:

[0210]

[0211] in, represents Gaussian distribution (or normal distribution), and Represent the mean vectors of two Gaussian distributions, represents the covariance matrix of two Gaussian distributions, is a 2×2 identity matrix. Laplace(·) represents the Laplace distribution, represents the mean vector of the Laplace distribution, Represents the scale parameter matrix of the Laplace distribution.

[0212] In order to verify the performance of the data-driven approximate optimal filtering algorithm proposed in this application, the algorithm is compared with the particle filter and the supervised learning method. Through 100 Monte Carlo experiments, each experiment includes 500 time steps (or moments), to evaluate the root mean square error (RMSE) and single-step time of different methods. Among them, the smaller the RMSE, the higher the estimation accuracy of the method.

[0213] Figure 5 Box plots of the root mean square error for different methods are shown. Figure 5 PF (Particle Filter) represents particle filtering, SLF (Supervised Learning-based Filter) represents supervised learning method, DAOFw / o PE represents data-driven approximate optimal filtering method without position encoding, and DAOF represents the data-driven approximate optimal filtering method proposed in this application. Figure 5 It can be seen that the data-driven approximate optimal filtering method proposed in this application has higher estimation accuracy than the common nonlinear non-Gaussian filtering, and the use of position encoding operation plays a positive role in improving the estimation accuracy.

[0214] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0215] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0216] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0217] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0218] Figure 6 A block diagram of a data-driven nonlinear system state estimation device according to an embodiment of the present disclosure is shown. The model includes at least the following modules: a first acquisition module 610, an action estimation module 620, a second acquisition module 630, an estimation and evaluation module 640, a first training module 650, a second training module 660, and an online estimation module 670.

[0219] A first acquisition module 610 is used to acquire historical information of the system state to be estimated of the nonlinear system at time t, wherein the historical information includes the state estimation values ​​at N historical moments before the time t and the historical observation value at the next moment of each historical moment; t is a positive integer starting from 0, and N is a preset positive integer;

[0220] An action estimation module 620, configured to determine the action at time t based on the policy function model to be trained using the historical information at time t to obtain a current estimated value at time t;

[0221] A second acquisition module 630, configured to determine historical information at time t+1 based on the current estimated value at time t and the system observed value at time t+1;

[0222] An estimation and evaluation module 640, configured to determine, based on the value function model to be trained, a first estimation and evaluation value corresponding to the historical information at the time t, and a second estimation and evaluation value corresponding to the historical information at the time t+1;

[0223] A first training module 650 is used to iteratively train the value function model to be trained using a reinforcement learning algorithm based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at the time t, and the actual state value at the time t, to obtain a trained value function model corresponding to the time t;

[0224] The second training module 660 is used to iteratively train the policy function model to be trained using a reinforcement learning algorithm based on the steady-state estimated evaluation value of the trained value function model corresponding to the time t, so as to obtain the trained policy function model corresponding to the time t;

[0225] The online estimation module 670 is used to determine the state estimation value of the system state based on the trained policy function model and the system observation value of the nonlinear system after the policy function model is trained.

[0226] For detailed description, please refer to the above method embodiment.

[0227] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0228] Figure 7 1 is a block diagram of a data-driven nonlinear system state estimation device 1900 according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Figure 7 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0229] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0230] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0231] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A data-driven nonlinear system state estimation method, characterized in that: The method comprises: Obtaining historical information of the system state to be estimated of the nonlinear system at time t, the historical information including the state estimation values ​​at N historical moments before the time t and the historical observation value at the next moment of each historical moment; t is an integer starting from 0, and N is a preset positive integer; Determine the action at time t based on the policy function model to be trained using the historical information at time t+1 to obtain a current estimated value at time t; Determine historical information at time t+1 based on the current estimated value at time t and the system observed value at time t+1; Determine, based on the value function model to be trained, a first estimated evaluation value corresponding to the historical information at time t and a second estimated evaluation value corresponding to the historical information at time t+1; Based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at the time t, and the actual state value at the time t, the value function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain a trained value function model corresponding to the time t; Based on the steady-state estimated evaluation value of the trained value function model corresponding to the time t, the policy function model to be trained is iteratively trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to the time t; After the policy function model is trained, a state estimation value of the system state is determined based on the trained policy function model and using the system observation value of the nonlinear system.

2. The method according to claim 1, characterized in that The method of iteratively training the value function model to be trained by using a reinforcement learning algorithm based on the first estimated evaluation value, the second estimated evaluation value, the current estimated value at the time t, and the actual state value at the time t to obtain the trained value function model corresponding to the time t includes: Obtaining a first loss function corresponding to the value function model to be trained; Input the first estimated evaluation value, the second estimated evaluation value, the current estimated value at the time t, and the actual state value at the time t into the first loss function to obtain the loss function value of the jth iteration; j is a positive integer starting from 1; When the iteration stopping condition is not reached in the j-th iteration, the model parameters of the value function model to be trained are updated based on the gradient descent method; After the parameters are updated, let the value of j be equal to j+1, and the step of inputting the first estimated evaluation value, the second estimated evaluation value, the historical information at time t, and the actual state value corresponding to the historical information at time t into the first loss function to obtain the loss function value of the j-th iteration and subsequent steps are executed again until the j-th iteration reaches the iteration stopping condition, and the trained value function model corresponding to time t is obtained.

3. The method according to claim 2, characterized in that The first loss function is expressed by the following formula: Among them, J critic A first loss function representing the value function model critic to be trained; represents the expected value calculation function; Represents the historical information corresponding to the time t A steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Indicates the historical information corresponding to the time t+1 The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; t is the loss equation, Represents the historical information corresponding to time t Under the condition of, the current strategy π of the strategy function model to be trained is used according to the historical information h t Select the current estimate at time t as the action π(h t ), the losses caused by e t represents the actual state value x at the time t t The current estimated value at the time t The error between ; γ represents a preset discount factor; V represents the value function model to be trained; w represents the current model parameters of the value function model to be trained.

4. The method according to claim 3, characterized in that: The model parameters of the value function model to be trained are updated based on the gradient descent method, which is expressed by the following formula: in, Represents the first loss function J critic With respect to the gradient of the model parameter w, the first loss function J is calculated critic The partial derivative with respect to the model parameter w is obtained; α represents the preset learning rate.

5. The method according to claim 1, characterized in that: In the case where the nonlinear system can be modeled based on the state transition model, the steady-state estimated evaluation value of the trained value function model corresponding to the time t is used to iteratively train the policy function model to be trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to the time t, including: Obtaining a second loss function corresponding to the policy function model to be trained, wherein the second loss function is established based on minimizing the sum of the error of the current estimated value at the time t and the steady-state estimated evaluation value output by the trained value function model corresponding to the time t; Inputting the historical information corresponding to the time t+1 into the trained value function model corresponding to the time t to obtain the steady-state estimation evaluation value; Input the current estimated value at time t, the actual state value at time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value of the kth iteration; k is a positive integer starting from 1; When the iteration stopping condition is not reached in the kth iteration, the model parameters of the policy function model to be trained are updated based on the gradient descent method; After the parameters are updated, the value of k is set equal to k+1, and the step of inputting the current estimated value at the time t, the actual state value at the time t, and the steady-state estimated evaluation value into the second loss function to obtain the loss function value of the kth iteration and subsequent steps are performed again until the kth iteration reaches the iteration stop condition, and the trained policy function model corresponding to the time t is obtained.

6. The method according to claim 5, characterized in that The second loss function is expressed by the following formula: Among them, J actor A second loss function representing the policy function model actor to be trained; represents the expected value calculation function; Represents the historical information corresponding to the time t A steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Indicates the historical information corresponding to the time t+1 The steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; t is the loss equation, Represents the historical information corresponding to time t Under the condition of, the current strategy π of the strategy function model to be trained is used according to the historical information h t Select the current estimate at time t as the action π(h t ), the losses caused by e t represents the actual state value x at the time t t The current estimated value at the time t The error between γ and γ represents the preset discount factor; Indicates the historical information corresponding to time t+1 After inputting the trained value function model corresponding to the time t with the model parameter w, the steady-state estimated evaluation value is obtained.

7. The method according to claim 6, characterized in that The model parameters of the strategy function model to be trained are updated based on the gradient descent method, which is expressed by the following formula: in, Denotes the second loss function J actor Model parameters of the policy function model to be trained The gradient of the second loss function J is calculated by actor About Model Parameters The partial derivative of is obtained; β represents the preset learning rate.

8. The method according to claim 1, characterized in that: In the case where the nonlinear system cannot be modeled based on the state transition model, the steady-state estimated evaluation value of the trained value function model corresponding to the time t is used to iteratively train the policy function model to be trained using a reinforcement learning algorithm to obtain the trained policy function model corresponding to the time t, including: Obtaining a third loss function corresponding to the policy function model to be trained, wherein the third loss function is established based on minimizing a steady-state estimated evaluation value output by the trained value function model corresponding to the time t; Inputting the historical information corresponding to the time t into the trained value function model corresponding to the time t to obtain the steady-state estimation evaluation value; Inputting the steady-state estimation evaluation value into the third loss function to obtain the loss function value of the kth iteration; wherein k is a positive integer starting from 1; When the iteration stopping condition is not reached in the kth iteration, the model parameters of the policy function model to be trained are updated based on the gradient descent method; After the parameters are updated, the value of k is set equal to k+1, and the step of inputting the steady-state estimation evaluation value into the third loss function to obtain the loss function value of the kth iteration and the subsequent steps are executed again, until the kth iteration reaches the iteration stop condition, and the trained strategy function model corresponding to time t is obtained.

9. The method according to claim 8, characterized in that The third loss function is expressed by the following formula: Among them, J actor A third loss function representing the policy function model actor to be trained; represents the expected value calculation function; Represents the historical information corresponding to the time t A steady-state distribution that can be achieved under the current strategy π of the strategy function model to be trained; Indicates that the historical information corresponding to time t and the current estimate at time t After inputting the trained value function model corresponding to the time t with model parameter w, the steady-state estimated evaluation value is obtained; Accordingly, the model parameters of the policy function model to be trained are updated based on the gradient descent method, which is expressed by the following formula: in, Denotes the third loss function J actor Model parameters of the policy function model to be trained The gradient of the third loss function J is calculated by actor About Model Parameters The partial derivative of is obtained; β represents the preset learning rate.

10. A data-driven nonlinear system state estimation device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

Images