A target tracking state estimator based on deep reinforcement learning

Abstract

The present invention discloses a target tracking state estimator based on deep reinforcement learning. The present invention belongs to the field of state estimator design in the field of signal processing. It comprises the following steps: first, the state estimation problem of target tracking is modeled as a partially observable Markov decision process, wherein the observation is designed as the measurement of the sensor node, the action is designed as the estimated value of the target state, and the reward of the intelligent agent is designed based on the modal trajectory estimation criterion. Subsequently, a cyclic determination policy gradient for solving the POMDP problem is derived, and finally a target tracking state estimator based on deep reinforcement learning is constructed based on RDPG. The target tracking state estimator based on deep reinforcement learning proposed in this invention can effectively improve the state estimation accuracy of nonlinear target tracking systems, overcome the defects of existing nonlinear filtering technology, and obtain good state estimation performance in nonlinear non-Gaussian target tracking systems.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking state estimator design, and in particular to a target tracking state estimator based on deep reinforcement learning. Background Art

[0002] Target tracking involves the problem of estimating the target state, also known as filtering. Filtering involves inferring the state value using noise-contaminated measurement data and a basic model of the dynamic system. Filtering has important applications in control engineering, robotics, target tracking, and navigation. For linear stochastic systems, the Kalman filter has been shown to obtain optimal solutions under Gaussian noise. However, for nonlinear stochastic systems, state estimation often fails to achieve optimal solutions because the probability distribution of the state no longer has a Gaussian property. Therefore, researchers have proposed many improved filters to address the state estimation problem for nonlinear systems. The most commonly used are the extended Kalman filter (EKF), the unscented Kalman filter (UKF), and the particle filter (PF). Due to initial estimation errors and linear errors, the EKF can only guarantee local asymptotic stability. The UKF also suffers from similar issues. Although particle filters can achieve good state estimation performance with a large number of particles, increasing the number of particles significantly increases the cost of online computation.

[0003] Therefore, how to use deep reinforcement learning technology to overcome the above shortcomings, use the interaction data between the deep reinforcement learning agent and the environment to learn, and design a target tracking state estimator based on deep reinforcement learning to obtain the optimal solution for target state estimation is an urgent problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies and propose a target tracking state estimator based on deep reinforcement learning. The specific steps are as follows:

[0005] S1: The state estimation problem of target tracking is modeled as a partially observable Markov decision process, where observations are designed as measurements of sensor nodes, actions are constructed as estimates of the target state, and the agent's reward is designed based on the modal trajectory estimation criterion;

[0006] S2: Determine the policy gradient for solving partially observable Markov decision processes through the value function of historical trajectories, the action-value function of historical trajectory action pairs, and the expected reward;

[0007] S3: Build a target tracking state estimator based on deep reinforcement learning based on the determined policy gradient; and train it to obtain the optimal network parameters;

[0008] S4: Based on the trained target tracking state estimator, the state of the target tracking system is estimated.

[0009] Furthermore, the parameters of the state estimation problem of target tracking in S1 are specifically:

[0010] Considering the target motion in three-dimensional space in a rectangular coordinate system, the target state at time k is:

[0011]

[0012] Its elements represent the position, velocity, and acceleration of the target on the x-axis, y-axis, and z-axis respectively;

[0013] The target motion model is expressed as:

[0014] x k =F k-1 x k-1 +w k-1

[0015] Among them F k-1 is the state transition matrix; w k-1 is the process noise that obeys the zero-mean Gaussian distribution, and its covariance is Q k-1 ;

[0016] The target moves at a constant speed, then the state transfer matrix is

[0017]

[0018] Where T is the sampling interval; the process noise covariance matrix is:

[0019]

[0020] Where q is the process noise intensity;

[0021] The measurement model of the sensor at time k is:

[0022]

[0023] in is the measurement function, is zero-mean white Gaussian noise, and its covariance matrix is ​​R k ;

[0024] are the measurement vectors of azimuth and elevation, whose true values ​​are:

[0025]

[0026]

[0027] where [x s ,y s ,z s ] T are the coordinates of the sensor node.

[0028] Furthermore, the state estimation problem of target tracking is modeled as a partially observable Markov decision process as follows:

[0029] Partially Observable Markov Decision Process Using Arrays Description, where is the state space, is the action space, is the transition probability of the environment, is the reward function, Ω is the observation probability, is the observation space, γ is the discount factor;

[0030] The design rules for the agent's observation o, action a, and reward r are as follows:

[0031] Observation o:k moment, the agent's observation o k is the observation data of the sensor, that is

[0032]

[0033] Action a: the action a of the agent at time k k is the estimated value of the target state, that is

[0034]

[0035] Reward r: The goal of the MTE criterion is to maximize the joint posterior probability density function p(x 0:k |y 0:k ), where x 0:k and y 0:k Represent all state vectors and observation vectors before the current moment. Using the Bayesian rule, it is expressed as:

[0036]

[0037] The above formula only has the numerator and x 0:k The target tracking duration is N. Based on the independence of measurements, we get:

[0038]

[0039] where p vis the probability density function of the measurement error; due to the Markov property of state transition, we have:

[0040]

[0041] Where f(·) is the system function of the target motion, p(x0) is the probability density function of the initial state, and p w is the probability density function of the system state error;

[0042] Based on the above results, we can get:

[0043]

[0044] Where c is a constant and x 0:k irrelevant;

[0045] Because of w k and v k are all Gaussian noise, so we have

[0046]

[0047] Where p0 is the error covariance of the initial state estimate, and the covariance matrix is ​​R k ;

[0048] The state estimation problem of target tracking is constructed as the following optimization problem;

[0049]

[0050] The reward of the agent at time k is defined as:

[0051]

[0052] Furthermore, the specific process of S2 is as follows:

[0053] Since the observation sequence of the partially observable Markov decision process does not satisfy the Markov property:

[0054] p(o k+1 |a k ,o k ,a k-1 ,o k-1 ,…,o0)≠p(o k+1 |o k ,a k )

[0055] Therefore, the agent needs to be based on the existing historical trajectory h k =(o k ,a k-1 ,o k-1 ,…,o0) to infer the hidden state s k, and make decisions based on the historical trajectory; Under the partially observable setting, the goal of reinforcement learning is to learn an optimal deterministic policy π=μ(h k ) to maximize the cumulative reward:

[0056]

[0057] Defined when observing h k Afterwards, the hidden state s k The value function is:

[0058]

[0059] Defined when observing h k Afterwards, the hidden state action pair (s k ,a k )’s action-value function is:

[0060]

[0061] Defined when observing h k Afterwards, in the hidden state s k Take action a k The expected reward is:

[0062]

[0063] respectively and About the hidden state distribution s k …p(s k |h k ) Find the expectation and get the historical trajectory h k The value function is:

[0064]

[0065] The action-value function of the historical trajectory and action pair is:

[0066]

[0067] The expected reward for a historical trajectory and action pair is:

[0068]

[0069] Based on the above definition, the objective function of reinforcement learning is reconstructed as:

[0070]

[0071] Where the expectation of the initial trajectory h0 is: h0~p(h0), and h0 is equal to o0, which is the initial observation;

[0072] In order to derive RDPG, the derivative of the Dirac function is introduced:

[0073]

[0074] Where g(·) is an arbitrary function and g′ is the derivative of the function. Combining the derivative of the Dirac function, the derivative of the objective function η(π) is derived as follows:

[0075]

[0076] where l(h,a)=μ(h)-a,d π (h) is the distribution of historical trajectories.

[0077] Furthermore, the specific process of constructing the target tracking state estimator based on deep reinforcement learning in S3 is as follows:

[0078] Determining Policy Gradients Based on POMDP Use recurrent neural network to analyze Q π (h,a) and μ(h) are approximated to construct a target tracking state estimator with an actor-critic structure, which includes an actor network μ θ (h) and critic network Q φ (h, a), where θ and φ are network parameters, and both the actor network and the critic network are composed of RNN and fully connected layers in series.

[0079] Furthermore, the specific process of estimator training in S3 is as follows:

[0080] S3.1 Exploring the Environment

[0081] At time t, the agent is based on the historical trajectory h t ={o 1:t ,a 1:t-1}Execute action a t ; In action a t After acting on the environment, the reward r of environmental feedback is obtained t , at the same time, the environment moves to the next state s t+1 , the agent observes the next state o t+1 ; Data that interacts with the environment (o1, a1, r1, ..., o T ,a T ,r T ) is stored in the memory pool

[0082] S3.2 Training network parameters

[0083] In the learning phase, first from the memory pool Randomly sample N rounds of data:

[0084]

[0085] And use the randomly sampled round data to construct the historical trajectory:

[0086]

[0087] Secondly, for each round of sampling Using a recurrent target neural network Q φ′ Calculate the target value:

[0088]

[0089] Where θ′ is the parameter of the recurrent target actor network, and φ′ is the parameter of the recurrent target critic network; the update amount of the critic value network is calculated using the backpropagation algorithm over time:

[0090]

[0091] in is the number of trajectories sampled during training, is the length of the trajectory; BPTT is also used to calculate the parameter update of the actor network:

[0092]

[0093] Therefore, the parameters of the recurrent actor and critic networks are updated as follows:

[0094] θ←θ+λ θ Δθ

[0095] φ←φ-λ φ Δφ,

[0096] where λ θ and λ φ are the learning rates of the recurrent actor network and the recurrent critic network respectively;

[0097] Finally, the parameters θ′ and φ′ of the recurrent target actor network and the recurrent target critic network are updated using Polyak averaging:

[0098] θ′←τθ+(1-τ)θ′

[0099] φ′←τφ+(1-τ)φ′

[0100] Where τ is the polyak update coefficient;

[0101] S3.3 State Estimation

[0102] After the training reaches the preset maximum number of training rounds M, the training process ends and the optimal network parameters θ are obtained. * and φ * ; Therefore, the target tracking state estimator based on deep reinforcement learning is approximately:

[0103]

[0104] The target state is tracked based on the state estimator.

[0105] Compared with the prior art, the advantages and positive effects of the present invention are:

[0106] This paper uses deep reinforcement learning to design a novel offline target tracking state estimator. This problem is modeled as a POMDP problem with continuous action outputs. This deep reinforcement learning-based target tracking state estimator combines the advantages of continuous control and reinforcement learning. By using continuous action outputs, the agent directly outputs an estimate of the target state during an iterative process, while simultaneously learning through trial and error in the environment to find the optimal solution for target state estimation. This method overcomes the shortcomings of existing nonlinear filtering techniques and achieves good state estimation performance in nonlinear, non-Gaussian target tracking systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 This is an overall flow chart of a target tracking state estimator based on deep reinforcement learning disclosed in the present invention.

[0108] Figure 2 It is a recurrent actor network structure of the target tracking state estimator based on deep reinforcement learning.

[0109] Figure 3 It is a recurrent critic network structure in the target tracking state estimator based on deep reinforcement learning.

[0110] Figure 4 Schematic diagram of the offline training process of the target tracking state estimator based on deep reinforcement learning. DETAILED DESCRIPTION

[0111] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0112] The overall flow chart of a filter based on deep reinforcement learning disclosed in the present invention is as follows: Figure 1 As shown, the specific implementation process is as follows:

[0113] Partially Observable Markov Decision Processes (POMDP) ​​modeling of target tracking state estimation problem.

[0114] The specific steps are:

[0115] Considering the target motion in three-dimensional space in a rectangular coordinate system, the target state at time k is:

[0116]

[0117] Its elements represent the position, velocity, and acceleration of the target on the x-axis, y-axis, and z-axis, respectively.

[0118] The target motion model can be expressed as:

[0119] x k =F k-1 x k-1 +w k-1

[0120] Among them F k-1 is the state transition matrix; w k-1 is the process noise that obeys the zero-mean Gaussian distribution, and its covariance is Q k-1 .

[0121] Assuming the target moves at a constant speed (CV), the state transfer matrix is

[0122]

[0123] Where T is the sampling interval. The process noise covariance matrix is:

[0124]

[0125] Where q is the process noise intensity.

[0126] Assume that the measurement model of the sensor at time k is:

[0127]

[0128] in is zero-mean white Gaussian noise, and its covariance matrix is ​​R k . are the measurement vectors of azimuth and elevation, whose true values ​​are:

[0129]

[0130]

[0131] where [x s ,ys ,z s ] T are the coordinates of the sensor node.

[0132] Considering that the state estimation problem of target tracking is the process of deriving unobservable hidden state variables from observable measurements, we model it as a POMDP, which uses the array Description, where is the state space, is the action space, is the transition probability of the environment, is the reward function, Ω is the observation probability, is the observation space, and γ is the discount factor.

[0133] The design rules for the agent's observation o, action a, and reward r are as follows:

[0134] 1) Observation o:k moment, the agent's observation o k is the observation data of the sensor, that is

[0135]

[0136] 2) Action a: Define the action a of the agent at time k k is the estimated value of the target state, that is

[0137]

[0138] 3) Reward r: The goal of the Modal Trajectory Estimation (MTE) criterion is to maximize the conditional probability density function p(x 0:k |y 0:k ). Using Bayes’ rule, it can be expressed as:

[0139]

[0140] The above formula only has the numerator and x 0:k Assuming that the target tracking duration is N, based on the independence of measurement, we can get:

[0141]

[0142] where p v is the probability density function of the measurement error. Due to the Markov property of state transition, we have:

[0143]

[0144] Where p(x0) is the probability density function of the initial state, p wis the probability density function of the system state error.

[0145] Based on the above results, we can get:

[0146]

[0147] Where c is a constant and x 0:k Not relevant.

[0148] Because of w k and v k are all Gaussian noise, so we have

[0149]

[0150] Where p0 is the error covariance of the initial state estimate.

[0151] Assume that we have no prior information about the initial state estimate, so This term disappears, and the state estimation problem of target tracking can be formulated as the following optimization problem.

[0152]

[0153] Therefore, in order to complete the state estimation of the target tracking process, the reward of the agent at time k can be designed as:

[0154]

[0155] (2) Deriving the cyclic deterministic policy gradient for solving POMDP

[0156] Since the observation sequence of the partially observable Markov decision process does not satisfy the Markov property:

[0157] p(o k+1 |a k ,o k ,a k-1 ,o k-1 ,…,o0)≠p(o k+1 |o k ,a k )

[0158] Therefore, the agent needs to be based on the existing historical trajectory h k =(o k ,a k-1 ,o k-1 ,…,o0) to infer the hidden state s k , and make decisions based on this historical trajectory. In a partially observable setting, the goal of reinforcement learning is to learn an optimal deterministic policy π=μ(h k ) to maximize the cumulative reward:

[0159]

[0160] Defined when observing h k Afterwards, the hidden state s k The value function is:

[0161]

[0162] Defined when observing h k Afterwards, the hidden state action pair (s k ,a k )’s action-value function is:

[0163]

[0164] Defined when observing h k Afterwards, in the hidden state s k Take action a k The expected reward is:

[0165]

[0166] respectively and About the hidden state distribution s k …p(s k |h k ) to find the expectation, we can get the historical trajectory h k The value function is:

[0167]

[0168] The action-value function of the historical trajectory and action pair is:

[0169]

[0170] The expected reward for a historical trajectory and action pair is:

[0171]

[0172] Based on the above definition, the objective function of reinforcement learning can be reconstructed as:

[0173]

[0174] In the formula, the expectation of the initial trajectory h0 is: h0~p(h0), and h0 is equal to o0, which is the initial observation.

[0175] In order to derive the Recurrent Deep Deterministic Policy Gradient (RDPG), the derivative of the Dirac function is introduced:

[0176]

[0177] Combined with the derivative of the Dirac function, the derivative of the objective function η(π) can be derived as:

[0178]

[0179] where l(h,a)=μ(h)-a,d π (h) is the distribution of historical trajectories.

[0180] (3) Building a target tracking state estimator based on deep reinforcement learning based on RDPG

[0181] Determining Policy Gradients Based on POMDP Use Recurrent Neural Network (RNN) to analyze Q π (h,a) and μ(h) are approximated, and the RDPG deep reinforcement learning algorithm with an actor-critic structure can be constructed, which includes the actor network μ θ (h) and critic network Q φ (h, a), where θ and φ are network parameters, and the cyclic actor network structure and cyclic critic network structure diagrams are as follows: Figure 2 and Figure 3 As shown, both the actor network and the critic network are composed of RNN and fully connected layers in series, a0,…,a t is the estimated value of the target state at time 0-t, o1,…,o t is the sensor observation value at time 1-t, h1,…,h t is the hidden vector output by the memory neural network at time 1-t. The training process of the target tracking state estimator based on deep reinforcement learning is as follows: Figure 4 As shown: The estimator alternately performs environment exploration and updates network parameters. Its main process is as follows:

[0182] 1) Explore the environment

[0183] At time t, the agent is based on the historical trajectory h t ={o 1:t ,a 1:t-1}Execute action a t In action a t After acting on the environment, you can get the reward r of environmental feedback t , at the same time, the environment moves to the next state s t+1 , the agent observes the next state o t+1 Data that interacts with the environment (o1, a1, r1, ..., o T ,aT ,r T ) is stored in the memory pool

[0184] 2) Training network parameters

[0185] In the learning phase, first from the memory pool Randomly sample N rounds of data:

[0186]

[0187] And use the randomly sampled round data to construct the historical trajectory:

[0188]

[0189] Secondly, for each round of sampling Using a recurrent target neural network Q θ′ Calculate the target value:

[0190]

[0191] Where θ′ is the parameter of the recurrent target actor network, and φ′ is the parameter of the recurrent target critic network. The update amount of the critic value network is calculated using the Back-Propagation Through Time (BPTT) algorithm:

[0192]

[0193] in is the number of trajectories sampled during training, is the length of the trajectory. BPTT is also used to calculate the parameter update amount of the actor network:

[0194]

[0195] Therefore, the parameters of the recurrent actor and critic networks can be updated as follows:

[0196] θ←θ+λ θ Δθ

[0197] φ←φ-λ φ Δφ,

[0198] where λ θ and λ φ are the learning rates of the recurrent actor network and the recurrent critic network, respectively.

[0199] Finally, the parameters θ′ and φ′ of the recurrent target actor network and the recurrent target critic network are updated using Polyak averaging:

[0200] θ′←τθ+(1-τ)θ′

[0201] φ′←τφ+(1-τ)φ′

[0202] Where τ is the polyak update coefficient.

[0203] 3) State Estimation

[0204] After the training reaches the preset maximum number of training rounds M, the training process ends and the optimal network parameters θ are obtained. * and φ * Therefore, the target tracking state estimator based on deep reinforcement learning can be approximated as:

[0205]

[0206] The target state can be tracked based on the state estimator.

[0207] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A target tracking state estimator based on deep reinforcement learning, characterized in that The following steps are involved: S1: The state estimation problem of target tracking is modeled as a partially observable Markov decision process, where observations are designed as measurements of sensor nodes, actions are constructed as estimates of the target state, and the agent's reward is designed based on the modal trajectory estimation criterion; S2: Determine the policy gradient for solving partially observable Markov decision processes through the value function of historical trajectories, the action-value function of historical trajectory action pairs, and the expected reward; S3: Build a target tracking state estimator based on deep reinforcement learning based on the determined policy gradient; and train it to obtain the optimal network parameters; The specific process of estimator training in S3 is as follows: S3.1 Exploring the Environment At time t, the agent is based on the historical trajectory h t ={o 1:t ,a 1:t-1 }Execute action a t ; In action a t After acting on the environment, the reward r of environmental feedback is obtained t , at the same time, the environment moves to the next state s t+1 , the agent observes the next state o t+1 ; Data that interacts with the environment (o1, a1, r1, ..., o T ,a T ,r T ) is stored in the memory pool S3.2 Training network parameters In the learning phase, first from the memory pool Randomly sample N rounds of data: And use the randomly sampled round data to construct the historical trajectory: Secondly, for each round of sampling Using a recurrent target neural network Q φ′ Calculate the target value: Where θ′ is the parameter of the recurrent target actor network, and φ′ is the parameter of the recurrent target critic network; the update amount of the critic value network is calculated using the backpropagation algorithm over time: in is the number of trajectories sampled during training, is the length of the trajectory; BPTT is also used to calculate the parameter update amount of the actor network: Therefore, the parameters of the recurrent actor and critic networks are updated as follows: θ←θ+λ θ Dth φ←φ-λ φ Df, where λ θ and λ φ are the learning rates of the recurrent actor network and the recurrent critic network respectively; Finally, the parameters θ′ and φ′ of the recurrent target actor network and the recurrent target critic network are updated using Polyak averaging: θ′←τθ+(1-τ)θ′ φ′←τφ+(1-τ)φ′ Where τ is the polyak update coefficient; S3.3 State Estimation After the training reaches the preset maximum number of training rounds M, the training process ends and the optimal network parameters θ are obtained. * and φ * ; Therefore, the target tracking state estimator based on deep reinforcement learning is approximately: Tracking the target state based on the state estimator; S4: Based on the trained target tracking state estimator, the state of the target tracking system is estimated.

2. The target tracking state estimator based on deep reinforcement learning according to claim 1, characterized in that The parameters of the state estimation problem of target tracking in S1 are specifically: Considering the target motion in three-dimensional space in a rectangular coordinate system, the target state at time k is: Its elements represent the position, velocity, and acceleration of the target on the x-axis, y-axis, and z-axis respectively; The target motion model is expressed as: x k =F k-1 x k-1 +w k-1 Among them F k-1 is the state transition matrix; w k-1 is the process noise that obeys the zero-mean Gaussian distribution, and its covariance is Q k-1 ; The target moves at a constant speed, then the state transfer matrix is Where T is the sampling interval; the process noise covariance matrix is: Where q is the process noise intensity; The measurement model of the sensor at time k is: in is the measurement function, is zero-mean white Gaussian noise, and its covariance matrix is ​​R k ; are the measurement vectors of azimuth and elevation, whose true values ​​are: where [x s ,y s ,z s ] T are the coordinates of the sensor node.

3. The target tracking state estimator based on deep reinforcement learning according to claim 2, characterized in that The state estimation problem of target tracking is modeled as a partially observable Markov decision process as follows: Partially Observable Markov Decision Process Using Arrays Description, where is the state space, is the action space, is the transition probability of the environment, is the reward function, Ω is the observation probability, is the observation space, γ is the discount factor; The design rules for the agent's observation o, action a, and reward r are as follows: Observation o:k moment, the agent's observation o k is the observation data of the sensor, that is Action a: the action a of the agent at time k k is the estimated value of the target state, that is Reward r: The goal of the MTE criterion is to maximize the joint posterior probability density function p(x 0:k |y 0:k ), where x 0:k and y 0:k Represent all state vectors and observation vectors before the current moment respectively; using the Bayesian rule, it is expressed as: The above formula only has the numerator and x 0:k The target tracking duration is N. Based on the independence of measurements, we get: where p v is the probability density function of the measurement error; due to the Markov property of state transition, we have: Where f(·) is the system function of the target motion, p(x0) is the probability density function of the initial state, and p w is the probability density function of the system state error; Based on the above results, we can get: Where c is a constant and x 0:k irrelevant; Because of w k and v k are all Gaussian noise, so we have Where p0 is the error covariance of the initial state estimate, and the covariance matrix is ​​R k ; The state estimation problem of target tracking is constructed as the following optimization problem; The reward of the agent at time k is defined as:

4. The target tracking state estimator based on deep reinforcement learning according to claim 1, characterized in that The specific process of S2 is as follows: Since the observation sequence of the partially observable Markov decision process does not satisfy the Markov property: dust k+1 |a k ,the k ,the k-1 ,the k-1 ,…,o0)≠p(o k+1 |the k ,the k ) Therefore, the agent needs to be based on the existing historical trajectory h k =(o k ,a k-1 ,o k-1 ,…,o0) to infer the hidden state s k , and make decisions based on the historical trajectory; Under the partially observable setting, the goal of reinforcement learning is to learn an optimal deterministic policy π=μ(h k ) to maximize the cumulative reward: Defined when observing h k Afterwards, the hidden state s k The value function is: Defined when observing h k Afterwards, the hidden state action pair (s k ,a k )’s action-value function is: Defined when observing h k Afterwards, in the hidden state s k Take action a k The expected reward is: respectively and About the hidden state distribution s k ~p(s k |h k ) Find the expectation and get the historical trajectory h k The value function is: The action-value function of the historical trajectory and action pair is: The expected reward for a historical trajectory and action pair is: Based on the above definition, the objective function of reinforcement learning is reconstructed as: Where the expectation of the initial trajectory h0 is: h0~p(h0), and h0 is equal to o0, which is the initial observation; In order to derive RDPG, the derivative of the Dirac function is introduced: Where g(·) is an arbitrary function and g′ is the derivative of the function. Combining the derivative of the Dirac function, the derivative of the objective function η(π) is derived as follows: where l(h,a)=μ(h)-a,d π (h) is the distribution of historical trajectories.

5. The target tracking state estimator based on deep reinforcement learning according to claim 4, characterized in that The specific process of constructing the target tracking state estimator based on deep reinforcement learning in S3 is as follows: Determining Policy Gradients Based on POMDP Use recurrent neural network to calculate Q π (h,a) and μ(h) are approximated to construct a target tracking state estimator with an actor-critic structure, which includes an actor network μ θ (h) and critic network Q φ (h, a), where θ and φ are network parameters, and both the actor network and the critic network are composed of RNN and fully connected layers in series.

Images