A vehicle tracking method based on deep Q network learning, a storage medium and an equipment

Abstract

The application discloses a vehicle tracking method based on deep Q network learning, a storage medium and equipment, and belongs to the technical field of vehicle tracking.The application solves the problem that high-precision positioning and tracking of a vehicle cannot be realized by using the prior art in a situation where GPS signals cannot be covered and a scene is complex.The application is aimed at a complex situation where a state transition probability matrix of an IMM Kalman filter is unknown, obtains the state transition probability matrix of the IMM Kalman filter based on DQN learning, and utilizes the perception and decision-making capability of deep reinforcement learning to effectively perceive an external environment, obtain optimal judgment, and further has the implementation process of the method not dependent on GPS signals, so that the method can still improve vehicle tracking performance and realize high-precision positioning and tracking of a vehicle in a situation where GPS signals cannot be covered and a scene is complex.The method can be applied to the technical field of vehicle tracking.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of vehicle tracking, and particularly relates to a vehicle tracking method based on deep Q network learning, a storage medium and equipment. BACKGROUND

[0002] Vehicle tracking is an important research direction in the fields of wireless networks and intelligent transportation. At present, the road-side unit (RSU) tracking technology is a commonly used technology in the intelligent transportation system. Unlike the global positioning system (GPS) and the global system for mobile communications (GSM), the RSU tracking technology allows vehicles to realize vehicle trajectory tracking through RSU signals without installing special devices such as GPS. More importantly, the RSU can effectively cover positions in the city that are difficult for GPS signals to cover (such as tunnels and building obstructions), and the positioning accuracy can meet the actual tracking requirements. Therefore, the vehicle tracking method using the RSU is a good choice for vehicle target tracking in the urban traffic environment.

[0003] In the vehicle tracking problem, Kalman filtering is a practical and common tracking algorithm that can be organically combined with the RSU signal. However, the traditional Kalman filtering is used to solve linear space problems by using the state space method and the recursive algorithm. However, for complex scenes with multiple vehicle motion conditions, it is not possible to use a fixed state space for solving, so in the case where GPS signals cannot be covered and the scene is complex, the existing technology still cannot realize high-precision positioning and tracking of vehicles. SUMMARY

[0004] The purpose of the present application is to solve the problem that the existing technology cannot realize high-precision positioning and tracking of vehicles in the case where GPS signals cannot be covered and the scene is complex, and to propose a vehicle tracking method based on deep Q network learning, a storage medium and equipment.

[0005] The technical solution adopted by the present application to solve the above technical problems is as follows:

[0006] A vehicle tracking method based on deep Q network learning, the method specifically comprising the following steps:

[0007] Step one, constructing a vehicle tracking system model;

[0008] The step one specifically comprises:

[0009] Step one, installing RSUs on both sides of the road;

[0010] Steps 1 and 2: Construct a vehicle motion state prediction model;

[0011] Step 13: Based on the Time of Arrival (TOA) information received from the RSU during the vehicle's movement, construct a vehicle motion state observation model;

[0012] Step 2: Based on the state prediction and state observation results obtained in Step 1, track the vehicle using a deep Q-network and an IMM (Interacting Multiple Model) Kalman filter.

[0013] Step two specifically includes:

[0014] Step 21: Use IMM Kalman filtering to track vehicles;

[0015] Step 22: Obtain the state array (i, a) based on the tracking results from Step 21. k-1 ,R k ,j), and obtain the state array (i,a k-1 ,R k ,j) Store in the experience replay unit, and update the parameters of the value function of the deep Q network according to the state array stored in the experience replay unit;

[0016] Steps 2 and 3: Update the Markov state transition probability matrix according to the updated value function parameters;

[0017] Step 24: Determine whether the vehicle has reached the end of its motion. If it has, end the tracking; otherwise, if it has not reached the end of its motion, return to Step 21 using the updated Markov state transition probability matrix.

[0018] For the first tracking moment, the Markov state transition probability matrix used is randomly set.

[0019] Furthermore, the vehicle motion state prediction model is as follows:

[0020] x k =Fx k-1 +Gω k-1

[0021] Where, x k Indicates t k The motion state vector of the vehicle at time x k =[x,y,v x ,v y ,a x ,a y ] Tx represents the vehicle's position along the X-axis of the Cartesian coordinate system, y represents the vehicle's position along the Y-axis of the Cartesian coordinate system, and v x The velocity v of the vehicle along the X-axis of a Cartesian coordinate system represents its velocity. y a represents the vehicle's velocity along the Y-axis of a Cartesian coordinate system. x a represents the acceleration of the vehicle along the X-axis of a Cartesian coordinate system. y This represents the acceleration of the vehicle along the Y-axis in a Cartesian coordinate system. The superscript T indicates transpose. k-1 Indicates t k-1 The vehicle's motion state vector at time ω, F is the state transition matrix, and ω k-1 Indicates t k-1 The acceleration perturbation noise is defined at any given time, and G is the noise matrix of the prediction model.

[0022] Furthermore, the vehicle motion state observation model is as follows:

[0023] z k =Hx k +v k

[0024] Among them, z k For t k The observed values ​​of the vehicle's motion state at time v, where H is the mapping matrix of arrival times received from the RSU, and v k It is observation noise.

[0025] Furthermore, the specific process of step two-one is as follows:

[0026] Step 2.11: Calculate t k The predicted probability of motion state j at time j:

[0027]

[0028] Where, μ i,k-1 For t k-1 The probability of the vehicle's motion state at time i. For t k The predicted probability of motion state j at time j, p ij For t k-1 Markov state transition probability matrix updated at each time step The elements in the array are i = 1, 2, ..., r, j = 1, 2, ..., r;

[0029] According to μ i,k-1 The mixed probability of the vehicle changing from motion state i to motion state j is calculated as follows:

[0030]

[0031] Where, μij,k-1k-1 Let be the mixed probability of the vehicle changing from motion state i to motion state j;

[0032] Using μ ij,k-1k-1 Calculate the mixed state estimate and the mixed covariance estimate of motion state j:

[0033]

[0034]

[0035] in, For the mixed state estimation of motion state j, It is t k-1 State estimation of vehicle motion state i at time i, P 0j,k-1|k-1 For the mixture covariance estimation of motion state j, P i,k-1|k-1 It is t k-1 Covariance estimation of the vehicle's motion state at time i;

[0036] Step 2.12, according to and P 0j,k-1k-1 One-step prediction of state and one-step prediction of covariance:

[0037]

[0038] P j,kk-1 =F j P 0j,k-1|k-1 F j T +G j Q j G j T

[0039] K j,k =P j,kk-1 H T HP j,kk-1 H T +R] -1

[0040] in, For t k Predicting the state of motion j at time step P j,k|k-1 For t k The covariance of the motion state j at time j is predicted in one step, F j Let G be the state transition matrix for motion state j. j Let Q be the model noise matrix for motion state j. j Let K be the covariance matrix of the disturbance noise in motion state j. j,k For t kThe Kalman gain of motion state j at time j, where R is the observation covariance noise matrix of RSU, and the superscript -1 represents the inverse of the matrix;

[0041] according to P j,k|k-1 and K j,k Calculate t k State estimation and covariance estimation of vehicle motion state j at time j:

[0042]

[0043] P j,kk =[IK j,k H]P j,k|k-1

[0044] in, For t k State estimation of vehicle motion state j at time j, P j,k|k For t k The covariance estimate of the vehicle's motion state j at time j, where I is the identity matrix;

[0045] Steps 2-13: The likelihood function of motion state j is:

[0046]

[0047] Among them, Λ j,k It is t k The likelihood function value of motion state j at time n, where z is the value of the state j. k The dimension of S j,k S is an intermediate variable matrix. j,k =HP j,kk-1 H T +R, |·| represents taking the modulus, and the superscript -1 represents taking the inverse. The probability of motion state j is updated as follows: In the formula, μ j,k For t k The probability of motion state j at time t, where c is the normalization coefficient.

[0048] Step 214, based on t k The probability μ of motion state j at time j j,k Calculate t k Final motion state estimation at time:

[0049]

[0050] in, It is t k The final motion state estimation result at time t k The results of continuous vehicle tracking.

[0051] Further, the specific process of step two is as follows:

[0052] Step two one, according to the following formula, calculate the reward function R at time t k : k :

[0053]

[0054] Wherein, E[·] represents the expectation;

[0055] Step two two, according to R k , construct the state array (i, a k-1 , R k , j), and store the obtained state array (i, a k-1 , R k , j) to the experience replay unit, a k-1 is the action selected at time t k-1 ;

[0056] Step two three, select a small batch of state arrays from the experience replay unit to update the value function parameter of the deep Q network, and the value function parameter is updated by using the gradient descent method.

[0057] Further, the specific process of step two three is as follows:

[0058] Calculate the cumulative Q value y k-1 at time t k-1 :

[0059]

[0060] Wherein, γ is a weight factor, Q(j', a k ; θ k ) is the value function value at time t k , j' is the target state at time t k , a k is the action at time t k ;

[0061] The value function parameter θ k at time t k is:

[0062]

[0063] In the formula, θ k-1 is the value function parameter at time t k-1 , α is the learning rate, Q(i, a k-1 ; θ k-1 ) is the value function value at time tk-1 the value function value of the moment, Q(i, a k-1 ; θ k-1 ) is the partial derivative of Q(i, a k-1 ; θ j ) with respect to θ k .

[0064] Further, the Markov state transition probability matrix is updated according to the updated value function parameter, and specifically:

[0065]

[0066]

[0067] wherein i = 1, 2, …, r, j = 1, 2, …, r, q j is the score of the vehicle motion state j under the action a k learned by the deep Q network.

[0068] Further, the action a k-1 is randomly selected by an epsilon-greedy method.

[0069] A computer storage medium, characterized in that the storage medium stores at least one instruction, the at least one instruction is loaded and executed by a processor to realize the vehicle tracking method based on deep Q network learning.

[0070] A vehicle tracking device based on deep Q network learning, the device comprising a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to realize the vehicle tracking method based on deep Q network learning.

[0071] The beneficial effects of the present application are:

[0072] The present application is aimed at the complex situation that the state transition probability matrix of the IMM Kalman filter is unknown, and the state transition probability matrix of the IMM Kalman filter is learned based on DQN, and the perception and decision-making ability of deep reinforcement learning can effectively perceive the external environment and obtain the optimal judgment. Moreover, the implementation process of the method of the present application does not need to rely on GPS signals, so that in the case where GPS signals cannot be covered and the scene is complex, the vehicle tracking performance can still be improved by the method of the present application, and high-precision positioning and tracking of the vehicle can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 is a flowchart of the vehicle tracking method based on deep Q network learning according to the present application;

[0074] Figure 2 is a vehicle tracking simulation diagram of a traditional Kalman filtering method;

[0075] Figure 3 is a vehicle tracking method simulation diagram based on deep Q network learning. DETAILED DESCRIPTION

[0076] The application will be described in further detail below with specific reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the application.

[0077] DETAILED DESCRIPTION Figure 1 The application is described. The vehicle tracking method based on deep Q network learning described in the embodiment specifically includes the following steps:

[0078] Step one, constructing a vehicle tracking system model;

[0079] The step one specifically includes:

[0080] Step one, installing RSU on both sides of the road;

[0081] Step two, constructing a vehicle motion state prediction model;

[0082] Step three, constructing a vehicle motion state observation model according to the time of arrival (TOA) information received by the vehicle from the RSU during the motion process;

[0083] Step two, tracking the vehicle based on the state prediction and state observation results obtained in step one, and based on deep Q network and IMM (Interacting Multiple Model, IMM) Kalman filtering;

[0084] The step two specifically includes:

[0085] Step two one, tracking the vehicle by IMM Kalman filtering;

[0086] Step two two, obtaining a state array (i, a k-1 ,R k ,j) according to the tracking result of step two one, and storing the obtained state array (i, a k-1 ,R k ,j) to an experience replay unit, and updating the parameters of the value function of the deep Q network according to the state array stored in the experience replay unit;

[0087] Step two three, update the Markov state transition probability matrix according to the updated value function parameters;

[0088] Step two four, judge whether the vehicle reaches the termination time of motion, if the termination time of motion is reached, end the tracking, otherwise the termination time of motion is not reached, return to step two one using the updated Markov state transition probability matrix.

[0089] For the first tracking time, the Markov state transition probability matrix used is randomly set.

[0090] Specific implementation two: the difference between this embodiment and specific implementation one is that the vehicle motion state prediction model is:

[0091] x k =Fx k-1 +Gω k-1

[0092] Wherein, x k represents the motion state vector of the vehicle at time t k , x k =[x,y,v x ,v y ,a x ,a y ] T , x represents the position of the vehicle along the X axis direction of the space rectangular coordinate system, y represents the position of the vehicle along the Y axis direction of the space rectangular coordinate system, v x represents the speed of the vehicle along the X axis direction of the space rectangular coordinate system, v y represents the speed of the vehicle along the Y axis direction of the space rectangular coordinate system, a x represents the acceleration of the vehicle along the X axis direction of the space rectangular coordinate system, a y represents the acceleration of the vehicle along the Y axis direction of the space rectangular coordinate system, the upper index T represents transposition, x k-1 represents the motion state vector of the vehicle at time t k-1 , F is the state transition matrix, ω k-1 represents the acceleration disturbance noise at time t k-1 , and G is the prediction model noise matrix.

[0093] The other steps and parameters are the same as those in specific implementation one.

[0094] Specific implementation three: the difference between this embodiment and specific implementation one or two is that the vehicle motion state observation model is:

[0095] z k =Hx k +v k

[0096] Among them, z k For t k The observed values ​​of the vehicle's motion state at time v, where H is the mapping matrix of arrival times received from the RSU, and v k It is observation noise.

[0097] Other steps and parameters are the same as in specific implementation method one or two.

[0098] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that the specific process of step two is as follows:

[0099] Step 2.11: Calculate t k The predicted probability of motion state j at time j:

[0100]

[0101] Where, μ i,k-1 For t k-1 The probability of the vehicle's motion state at time i. For t k The predicted probability of motion state j at time j, p ij For t k-1 Markov state transition probability matrix updated at each time step The elements in the array are i = 1, 2, ..., r, j = 1, 2, ..., r;

[0102] According to μ i,k-1 The mixed probability of the vehicle changing from motion state i to motion state j is calculated as follows:

[0103]

[0104] Where, μ ij,k-1|k-1 Let be the mixed probability of the vehicle changing from motion state i to motion state j;

[0105] Using μ ij,k-1|k-1 Calculate the mixed state estimate and the mixed covariance estimate of motion state j:

[0106]

[0107]

[0108] in, For the mixed state estimation of motion state j, It is t k-1 State estimation of vehicle motion state i at time i, P 0j,k-1|k-1 For the mixture covariance estimation of motion state j, P i,k-1|k-1 It is t k-1 Covariance estimation of the vehicle's motion state at time i;

[0109] Step two, according to and P 0j,k-1|k-1 One-step prediction of state and one-step prediction of covariance:

[0110]

[0111] P j,k|k-1 = F j P 0j,k-1|k-1 F j T + G j Q j G j T

[0112] K j,k = P j,k|k-1 H T [H P j,k|k-1 H T + R] -1

[0113] wherein, is the one-step prediction of state of motion state j at time t k P j,k|k-1 is the one-step prediction of covariance of motion state j at time t k F j is the state transition matrix of motion state j, G j is the model noise matrix of motion state j, Q j is the covariance matrix of disturbance noise of motion state j, K j,k is the Kalman gain of motion state j at time t k R is the observation covariance noise matrix of RSU, and the superscript -1 represents the inverse of the matrix;

[0114] According to P j,k|k-1 and K j,k , the state estimation and covariance estimation of vehicle motion state j at time t k are calculated:

[0115]

[0116] P j,k|k = [I - K j,k H] P j,k|k-1

[0117] wherein, is the state estimation of vehicle motion state j at time t k P j,k|k is the covariance estimation of vehicle motion state j at time t kThe covariance estimate of the vehicle's motion state j at time j, where I is the identity matrix;

[0118] Steps 2-13: The likelihood function of motion state j is:

[0119]

[0120] Among them, Λ j,k It is t k The likelihood function value of motion state j at time n, where z is the value of the state j. k The dimension of S j,k S is an intermediate variable matrix. j,k =HP j,k|k-1 H T +R, |·| represents taking the modulus, and the superscript -1 represents taking the inverse. The probability of motion state j is updated as follows: In the formula, μ j,k For t k The probability of motion state j at time t, where c is the normalization coefficient.

[0121] Step 214, based on t k The probability μ of motion state j at time j j,k Calculate t k Final motion state estimation at time:

[0122]

[0123] in, It is t k The final motion state estimation result at time t k The results of continuous vehicle tracking.

[0124] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0125] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the specific process of step two is as follows:

[0126] Step Two Two One, according to Calculate t k Reward function R at time step k :

[0127]

[0128] Where E[·] represents the expectation;

[0129] Step 222, according to R k Construct the state array (i, a) k-1 ,R kj), and the obtained state array (i, a k-1 , k j) is stored to the experience replay unit, a k-1 is the action selected at the time t k-1 ;

[0130] Step two three, randomly select a small batch of state arrays from the experience replay unit to update the parameters of the value function of the deep Q network, and the gradient descent method is used for updating the parameters of the value function.

[0131] The other steps and parameters are the same as one of the first to fourth embodiments.

[0132] The sixth embodiment is different from the first to fifth embodiments in that the specific process of step two three is as follows:

[0133] Calculate the cumulative Q value y k-1 at the time t k-1 :

[0134]

[0135] Wherein, γ is a weight factor, Q(j', a k ; θ k ) is the value function value at the time t k , j' is the target state at the time t k , a k is the action at the time t k ;

[0136] Then the value function parameter θ k at the time t k is:

[0137]

[0138] In the formula, θ k-1 is the value function parameter at the time t k-1 , α is the learning rate, Q(i, a k-1 ; θ k-1 ) is the value function value at the time t k-1 , represents the partial derivative of Q(i, a k-1 ; θ k-1 ) with respect to θ k-1 .

[0139] The other steps and parameters are the same as one of the first to fifth embodiments.

[0140] Specific implementation seven: different from one of the specific implementations one to six, the Markov state transition probability matrix is updated according to the updated value function parameter, which is specifically:

[0141]

[0142]

[0143] Wherein, i = 1, 2…, r, j = 1, 2…, r, q j is the value function Q (i, a k ; θ k ) learned by the deep Q network to score the vehicle motion state j under the action a k .

[0144] The updated Markov state transition probability matrix of the embodiment is used for tracking the vehicle motion state at the next time. Moreover, it needs to be particularly pointed out that the motion state i and the motion state j in the application are not used to limit a specific motion state. For any current tracking time, the motion state of the previous time of the current time is represented as the motion state i, and the motion state from the motion state i to the motion state of the current tracking time after selecting the action of the previous time is represented as the motion state j.

[0145] The other steps and parameters are the same as one of the specific implementations one to six.

[0146] Specific implementation eight: different from one of the specific implementations one to seven, the action a k-1 is randomly selected by an ε-greedy method.

[0147] The other steps and parameters are the same as one of the specific implementations one to seven.

[0148] Specific implementation nine: the embodiment is a computer storage medium, and the storage medium stores at least one instruction. The at least one instruction is loaded and executed by a processor to realize the vehicle tracking method based on deep Q network learning.

[0149] It should be understood that the instructions include a computer program product, software or computerized method corresponding to any method described in the application; the instructions can be used to program a computer system or other electronic device. The computer storage medium can include a readable medium having instructions stored thereon, and can include but is not limited to a magnetic storage medium, an optical storage medium, a magneto-optical storage medium, a read-only memory (ROM), a random access memory (RAM), an erasable programmable memory (such as an EPROM and an EEPROM), and a flash memory layer, or other types of media suitable for storing electronic instructions.

[0150] Specific implementation ten: the present embodiment is a vehicle tracking device based on deep Q network learning, the device comprises a processor and a memory, it should be understood that any device comprising the processor and the memory described in the present application includes the device can also include other units, modules, etc. through signal or instruction display, interaction, processing, control and other functions and other functions;

[0151] The memory stores at least one instruction, which is loaded and executed by the processor to implement the vehicle tracking method based on deep Q network learning.

[0152] Embodiments

[0153] The present application assumes that RSU has been installed on both sides of the road at a known location, and the vehicle moves in a Z-shaped trajectory in a two-dimensional plane, corresponding to three motion states: "uniform speed", "slow turn" and "fast turn"; as Figure 1 The MATLAB simulation experiment steps of the vehicle tracking method based on deep Q network learning are as follows:

[0154] 1. Establish a vehicle tracking system model based on RSU received signal

[0155] 1) Construct vehicle motion scene

[0156] It is assumed that RSU has been installed on both sides of the road at a known location, and all vehicles are equipped with radio equipment and can realize vehicle ad hoc network (VANET) communication, and all VANET nodes will periodically broadcast signals; the vehicle-mounted inertial navigation system (INS) can obtain the rough position estimation of the vehicle itself, and the RSU can also obtain the time of arrival (TOA) estimation between the vehicle and the RSU; by obtaining the two estimates, the RSU can analyze the position of the vehicle and perform real-time tracking;

[0157] The number of RSUs in the simulation experiment is 6;

[0158] 2) Construct vehicle motion state prediction model

[0159] Let x k =[x,y,v x ,v y ,a x ,a y ] T represent the motion state vector of the vehicle at time t k , wherein x, y, v x , v y and a x,a y Let X represent the vehicle's position, velocity, and acceleration along the X and Y axes, respectively. Then, the prediction equation for the motion state vector is:

[0160] x k =Fx k-1 +Gω k-1

[0161] Where F is the state transition matrix, representing the specific mode of vehicle motion, x k Indicates t k The vehicle's motion state vector at time ω k-1 Indicates t k-1 The acceleration disturbance noise at each time step is a zero-mean independent Gaussian random vector with a covariance matrix of Q; G is the model noise matrix, representing the degree of influence of the disturbance noise on the system state;

[0162] In the simulation experiment, the initial state of the vehicle is x. k =[2000,10000,0,-15,0,0] T First, it moves at a constant velocity in the current direction. After 400 seconds, it accelerates at a speed of 0.075 m / s² in both the X and Y axes. 2 The vehicle makes a slow turn. After 200 seconds, it moves at a constant speed of 15 m / s along the X-axis. After 10 seconds, it accelerates at 0.3 m / s² along both the X and Y axes. 2 The fast turning motion, after 50s, moves at a constant linear speed of 15m / s along the negative y-axis for 240s. The scan period of the RSU is T = 2s.

[0163]

[0164] These three models correspond to the motion states of "uniform speed," "slow turn," and "fast turn," respectively. The first model is a non-maneuvering model, while the second and third models are maneuvering models. The model noise matrices are respectively...

[0165]

[0166] ω k-1 Given a random 2×1 zero-mean independent Gaussian random vector, the covariance matrices for its corresponding three motion states—"uniform speed," "slow turn," and "fast turn"—are respectively...

[0167] 3) Construct a vehicle motion state observation model

[0168] The vehicle receives TOA information from multiple RSUs simultaneously while in motion. k The observation equation for the vehicle's motion state at time t is:

[0169] z k = Hx k + v k

[0170] where H is the mapping matrix of TOA information, v k is the observation noise, is a zero-mean independent Gaussian random vector with covariance matrix R, and v k is independent of ω k ;

[0171] In the simulation experiment the standard deviation of the observation noise is δ = 100,

[0172] 2. Establishing an IMM vehicle tracking system based on deep Q network

[0173] 1) Constructing an IMM Kalman filter model of vehicle motion state

[0174] (1) Assuming that the vehicle has r motion states, such as turning, accelerating, decelerating, etc., the transition probability between all motion states is determined by the Markov state transition probability matrix, denoted as:

[0175]

[0176] where p ij represents the transition probability between motion state i and motion state j, which is learned by DQN;

[0177] In the simulation experiment, r = 3, the initial value of is randomly set, and the iteration number in the DQN training process is 100 times;

[0178] (2) Assuming that the probability of vehicle motion state i at time t k-1 is μ i,k-1 , the predicted probability of motion state j at time t k is The mixing probability from motion state i to motion state j is The estimation of vehicle motion state i at time t k-1 is The covariance estimation is P i,k-1|k-1 , and the mixed state estimation of motion state j is The mixed covariance estimation of motion state j is and the mixed state estimation and the mixed covariance estimation P 0j,k-1|k-1 are taken as the initial state of the current cycle;

[0179] In the simulation experiment, μ i,k-1The initial value of is set randomly;

[0180] (3) Taking P 0j,k-1|k-1 and z k as the inputs of the Kalman filter, the Kalman filtering is performed on the motion state j to obtain the predicted motion state The predicted covariance is P j,k|k-1 = F j P 0j,k-1|k-1 F j T + G j Q j G j T , the Kalman gain is K j,k = P j,k|k-1 H T [H P j,k|k-1 H T + R] -1 , wherein G j , F j and Q j are respectively the model noise matrix, the state transition matrix and the covariance matrix of the disturbance noise of the motion state j, the updated predicted motion state is The updated predicted covariance is P j,k|k = [I - K j,k H] P j,k|k-1 ;

[0181] (4) The likelihood function of the motion state j is:

[0182]

[0183] wherein, S j,k = H P j,k|k-1 H T + R, n is the dimension of the vector z k ; the probability of the motion state j is updated as In the formula, is the normalized coefficient;

[0184] In the simulation experiment, n = 6;

[0185] (5) According to the probability μ k of the motion state j at the time t j,k , the weighted sum of the predicted motion state of each filter is performed to obtain the final motion state estimation as

[0186] 2) Constructing a Markov decision process

[0187] Markov Decision Process (MDP) is the basis of reinforcement learning, an MDP is described by the array (I, A, p, R, γ), where I is the set of all possible states, also known as state space, each state i∈I represents a unique state information in the environment information, each state information is a six-dimensional vector, i.e. k , represents the position, velocity and acceleration information of the vehicle along the X and Y axes at time t k ; the action space A is the set of all actions that the vehicle can use to control or change the motion state, including turning, accelerating, decelerating, etc. The dynamics model that may occur during vehicle motion; the future state of the Markov chain only depends on the current state and the current action, and the state transition probability is re-expressed as p ij = p(i, a k-1 , j) = p(j | i, a k-1 ), which represents the probability of transitioning from motion state i at time t k-1 to motion state j at time t k-1 when performing action a k ; the reward function is R(i, a k-1 , j), which represents the average reward accumulated when the vehicle transitions from motion state i to motion state j when performing action a k-1 at time t k-1 , and the reward function is specifically represented as where E is the expectation; γ∈[0, 1] is the weight factor, which simulates the preference of current reward for future reward, when γ is close to 0, the tracking tends to obtain the reward immediately after performing an action, and the future reward is seriously ignored, when γ is close to 1, it tends to remote and long-term reward;

[0188] In the simulation experiment, γ = 0.9;

[0189] 3) Using DQN to learn the approximate solution of the MDP process

[0190] The goal of DQN is to find the optimal policy for a given MDP process, which is a mapping between states and actions, usually denoted by the symbol π, DQN method learns the approximate solution of the MDP process by directly learning the state-action value function, the state-action value function Q π (i, a k-1 ) at time t k-1 is represented as:

[0191]

[0192] DQN uses a neural network Q(i, a; θ) to approximate the state-action value function Q π (i, ak-1 ), DQN uses gradient descent method to update network parameters θ, t k Network parameters θ are updated at time t as:

[0193]

[0194] In the formula, α is the learning rate, represents the partial derivative of Q(i, a k-1 ; θ) with respect to θ, y k-1 is the cumulative Q value, which is represented as:

[0195]

[0196] Using the updated network parameters θ, the updated neural network Q(i, a; θ) is obtained;

[0197] In the simulation experiment, α = 0.9, and Q(i, a; θ) is initialized using random network parameters θ;

[0198] 4) Update the Markov state transition probability

[0199] Given the vehicle t k-1 , the motion state is i, and the action a k-1 is performed at t k , the motion state is changed to j at t k-1 , and the action a k-1 under Q(i, a j ; θ) is learned using DQN to score q k-1 , 1≤j≤r, then the Markov state transition probability is

[0200] 3) Realize vehicle tracking

[0201] 1) Initialization

[0202] Q(i, a; θ) is initialized using random network parameters θ, the initial motion state is set to i, and the initial value of the state transition probability matrix is randomly set;

[0203] In the simulation experiment, the initial motion state is set to uniform speed, and the initial value of the state transition probability matrix is randomly set;

[0204] 2) Update the Markov state transition probability

[0205] An action a k-1 is randomly selected using the ε-greedy method, and the reward R k and the new motion state j are obtained by executing this action, and p ij is updated using step 2) 4);

[0206] In the simulation experiment, an action a is randomly selected by using an epsilon-greedy method k-1 ;

[0207] 3) Vehicle tracking is realized by using an IMM Kalman filter

[0208] The IMM Kalman filter is realized by using 1) in step 2, and a final motion state estimation is obtained That is, vehicle tracking is realized

[0209] 4) The neural network is updated

[0210] (i, a k-1 , R k , j) is stored in an experience replay unit, a stored state array is taken out from the experience replay unit in small batches, y k-1 is obtained by using 3) in step 2, network parameters θ k are updated, and an updated network Q(i, a; θ) is obtained

[0211] In the simulation experiment, an action a is randomly selected by using an epsilon-greedy method The network Q(i, a; θ) is updated once, and all motion state estimations are completed.

[0212] The present application adopts an IMM Kalman filter, uses two or more models to describe possible states in the vehicle motion process, and applies effective weighted fusion to obtain system state estimation. The present application can overcome the problem that a traditional Kalman filter cannot accurately describe possible motion states in the vehicle motion process under various vehicle motion states, resulting in inaccurate tracking. The present application uses a deep Q network (DQN) to learn the state transition probability of the IMM Kalman filter, which is beneficial to overcoming the problem that the state transition probability matrix between different models of the IMM Kalman filter is unknown in a complex scene, and improves the robustness of the vehicle tracking method. In the case where GPS signals cannot be covered and the scene is complex, the present application can still realize high-precision positioning and tracking of vehicles.

[0213] Vehicle motion trajectory tracking is realized by using a traditional Kalman filtering method and the present application, and simulation results are as shown in Figure 2 and Figure 3The actual trajectory in the simulation diagram refers to the real motion trajectory of the tracked vehicle, and the observation trajectory is measured by the RSU. It can be seen from the simulation diagram that the traditional Kalman filtering method has good performance in the early stage of tracking and is almost completely consistent with the motion trajectory. However, when the vehicle appears a maneuver change, its tracking ability decreases significantly, which shows that a single motion model is insufficient to support the vehicle tracking performance requirement in a complex motion state. In contrast, the method of the application uses IMM, and uses a deep Q network to learn the Markov state transition probability, thereby improving the vehicle tracking performance in a complex motion state.

[0214] The above calculation examples of the application are only to illustrate the calculation model and calculation process of the application, and are not a limitation on the embodiments of the application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art, and it is impossible to enumerate all the embodiments here, and any obvious changes or variations derived from the technical solutions of the application still fall within the protection scope of the application.

Claims

1. A vehicle tracking method based on deep Q network learning, characterized in that, The method specifically comprises the following steps: Step one, constructing a vehicle tracking system model; The step one specifically comprises: Step one one, installing RSUs on both sides of the road; Step one two, constructing a vehicle motion state prediction model; The vehicle motion state prediction model is: x k = Fx k-1 + Gω k-1 wherein x k represents the motion state vector of the vehicle at time t k , x k = [x, y, v x , v y , a x , a y ] T , x represents the position of the vehicle along the X-axis direction of the space rectangular coordinate system, y represents the position of the vehicle along the Y-axis direction of the space rectangular coordinate system, v x represents the speed of the vehicle along the X-axis direction of the space rectangular coordinate system, v y represents the speed of the vehicle along the Y-axis direction of the space rectangular coordinate system, a x represents the acceleration of the vehicle along the X-axis direction of the space rectangular coordinate system, a y represents the acceleration of the vehicle along the Y-axis direction of the space rectangular coordinate system, the upper index T represents transposition, x k-1 represents the motion state vector of the vehicle at time t k-1 , F is a state transition matrix, ω k-1 represents the acceleration disturbance noise at time t k-1 , and G is a prediction model noise matrix; Step one three, constructing a vehicle motion state observation model according to the time of arrival information received by the vehicle from the RSU during the motion process; The vehicle motion state observation model is: z k = Hx k + v k where z k is the observation of the vehicle motion state at time t k , H is the mapping matrix of the received time of arrival from the RSU, v k is the observation noise; Step two, tracking the vehicle according to the state prediction and state observation results obtained in step one, and based on deep Q network and IMM Kalman filtering; The step two specifically comprises: Step two one, tracking the vehicle by IMM Kalman filtering; The specific process of the step two one is: Step two-1, calculate t k Predicted probability of motion state j at time instant where μ i,k-1 is the probability of the vehicle motion state i at time t k-1 is the probability of the vehicle motion state i at time t is the predicted probability of the motion state j at time t k is the predicted probability of the motion state j at time t ij is the predicted probability of the motion state j at time t k-1 is the element in the updated Markov state transition probability matrix at time t , i = 1, 2, …, r, j = 1, 2, …, r. According to μ i,k-1 The mixing probability of a vehicle from motion state i to motion state j is calculated as: where μ ij,k-1|k-1 is the mixing probability of the vehicle from motion state i to motion state j; Utilizing μ ij,k-1|k-1 compute a mixed state estimate of motion state j and a mixed covariance estimate of motion state j: wherein is a hybrid state estimate of motion state j, is a hybrid state estimate of motion state j, k-1 is a state estimate of vehicle motion state i at time instant t 0j,k-1|k-1 is a hybrid covariance estimate of motion state j, i,k-1|k-1 is a hybrid covariance estimate of motion state j, k-1 is a covariance estimate of vehicle motion state i at time instant t Step two, according to and P 0j,k-1|k-1 One-step prediction of the state and one-step prediction of the covariance: P j,k|k-1 = F j P 0j,k-1|k-1 F j T + G j Q j G j T K j,k = P j,k|k-1 H T [HP j,k|k-1 H T +R] -1 in, For t k Predicting the state of motion j at time step P j,k|k-1 For t k One-step prediction of the covariance of motion state j at time j, F j Let G be the state transition matrix for motion state j. j Let Q be the model noise matrix for motion state j. j Let K be the covariance matrix of the disturbance noise in motion state j. j,k For t k The Kalman gain of motion state j at time j, where R is the observation covariance noise matrix of RSU, and the superscript -1 represents the inverse of the matrix; according to P j,k|k-1 and K j,k Calculate t k State estimation and covariance estimation of vehicle motion state j at time j: P j,k|k = [I - K j,k H]P j,k|k-1 wherein is the state estimate of the vehicle motion state j at time t k is the state estimate of the vehicle motion state j at time t j,k|k is the covariance estimate of the vehicle motion state j at time t k is the covariance estimate of the vehicle motion state j at time t Step two one three, the likelihood function of the motion state j is: where Λ j,k is the likelihood function value of the motion state j at time t k , n is the dimension of z k , S j,k is an intermediate variable matrix, S j,k = H P j,k|k-1 H T + R, |·| represents the modulus length, The probability of the motion state j is updated as: where μ j,k is the probability of the motion state j at time t k , and c is a normalization coefficient, Step two, four, according to t k The probability μ of the motion state j at time instant t j,k The calculation of t k The final motion state estimate at time instant t wherein, is t k the final motion state estimation result at time t k the result of the vehicle tracking at time t Step two, obtaining a state array (i, a k-1 ,R k ,j) according to the tracking result of step two one, and storing the obtained state array (i, a k-1 ,R k ,j) to an experience replay unit, and updating the parameters of the value function of the deep Q network according to the state array stored by the experience replay unit; Step two three, updating the Markov state transition probability matrix according to the updated value function parameters; Step two four, judging whether the vehicle reaches the termination time of the motion, if yes, ending the tracking, otherwise, if not, returning to step two one by using the updated Markov state transition probability matrix. 2.The vehicle tracking method based on deep Q network learning according to claim 1, wherein, The specific process of the step two two is: Step two one, according to Computing t k Reward function R k : Wherein, E[·] represents the expectation; Step two two two, according to R k Construct a state array (i, a k-1 , R k , j) and store the obtained state array (i, a k-1 , R k , j) to the experience replay unit, a k-1 is the action selected at time t k-1 . Step two two three, randomly selecting a state array from the experience replay unit to update the value function parameters of the deep Q network, and the gradient descent method is used for updating the value function parameters. 3.The vehicle tracking method based on deep Q network learning according to claim 2, wherein, The specific process of the step two two three is: Compute t k-1 Q value y accumulated at time t k-1 : where γ is a weighting factor, Q(j',a k ) is the value function value at time t k , j' is the target state at time t k , and a k is the action at time t k . k ​ t k the value function parameter θ at time t k is: In the formula, θ k-1 is a value function parameter at time t k-1 , α is a learning rate, Q(i,a k-1 ; θ k-1 ) is a value function value at time t k-1 , and ∇Q(i,a k-1 ; θ k-1 ) represents a partial derivative of Q(i,a k-1 ; θ k-1 ) with respect to θ k-1 . 4.The vehicle tracking method based on deep Q network learning according to claim 3, wherein, The Markov state transition probability matrix is updated according to the updated value function parameters, which is specifically: where i = 1, 2,..., r, j = 1, 2,..., r, q j is learned by a deep Q network to score the vehicle motion state j under action a k . 5.The vehicle tracking method based on deep Q network learning according to claim 4, wherein, The action a k-1 Randomly selected with the ε-greedy method.

6. A computer storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to realize the vehicle tracking method based on deep Q network learning according to any one of claims 1 to 5. 7.A vehicle tracking device based on deep Q network learning, characterized in that, The device comprises a processor and a memory, and the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to realize the vehicle tracking method based on deep Q network learning according to any one of claims 1 to 5.

Images