TD3-based extended target tracking optimization method in non-stationary noise environment

Abstract

The invention relates to a TD3-based extended target tracking optimization method in a non-stationary noise environment, which belongs to the technical field of target tracking and noise modeling, and realizes dynamic switching between Gaussian noise and thick-tail noise by modeling a noise switching probability into an action variable of a reinforcement learning agent and utilizing a Gaussian-student t mixed distribution model. Therefore, the adaptability of the target tracking system to a complex noise scene is enhanced; in the aspect of geometric modeling, a star convex random hypersurface model is adopted to accurately describe the motion state and the geometric shape of an extended target, and key filtering parameters are dynamically adjusted in combination with an unscented Kalman filter, so that the tracking precision and the robustness are remarkably improved. The method can greatly reduce the target state estimation error in a non-stationary noise environment, shows excellent real-time performance and stability, and is suitable for the fields of unmanned driving, intelligent monitoring, military and national defense and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking and noise modeling, and in particular to an extended target tracking optimization method in a non-stationary noise environment based on TD3. Background Art

[0002] Extended target tracking is a core task in modern sensor data processing, widely used in fields such as autonomous driving, intelligent surveillance, and military defense. Unlike traditional point target tracking methods, extended target tracking requires not only estimating the target's motion state but also modeling and tracking its geometry. However, traditional methods face significant challenges in handling complex dynamic environments, particularly in nonstationary, heavy-tailed noise environments, where noise characteristics vary dramatically and the distribution has a heavy-tailed nature. This places higher demands on the robustness and real-time performance of tracking systems.

[0003] In existing technologies, the random hypersurface model, a classic geometric modeling method, can parameterize the target boundary. However, this method typically assumes a Gaussian noise distribution and is not adaptable to non-stationary noise environments. In recent years, researchers have proposed the Gaussian-Student t mixed distribution model to combine the accuracy of the Gaussian distribution with the robustness of the Student t distribution. However, its optimization methods often rely on variational Bayesian derivation, which is computationally complex and difficult to meet real-time requirements.

[0004] At the same time, deep reinforcement learning, as an efficient optimization method, has been widely used in dynamic decision-making problems. Among them, the TD3 algorithm significantly improves the performance of reinforcement learning in continuous action spaces by introducing a dual value network, delayed policy updates, and action smoothing mechanisms. However, current research has mostly focused on point target tracking tasks and has not yet fully incorporated the geometric characteristics of extended target tracking and the requirements of dynamic noise modeling. Therefore, the present invention proposes a TD3-based extended target tracking optimization method in a non-stationary noise environment. Summary of the Invention

[0005] The purpose of the present invention is to provide an extended target tracking optimization method in a non-stationary noise environment based on TD3, which dynamically optimizes the noise switching probability through reinforcement learning, combines the star-convex random hypersurface model and the unscented Kalman filter, and realizes the accurate estimation and tracking of the extended target state and shape.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A TD3-based extended target tracking optimization method in a non-stationary noise environment, comprising:

[0008] S1. Based on the Gaussian-Student t mixture distribution model, the noise switching probability optimization problem is constructed as a Markov decision process, and the measurement information of the extended target is obtained;

[0009] S2. Based on the measurement information, the extended target is modeled using a star-convex random hypersurface, and an unscented Kalman filter is used to estimate the state of the extended target, obtain an estimated position and a covariance matrix, and calculate the trace of the covariance matrix, the center of mass position error, and the shape matching error;

[0010] S3. Inputting the trace of the covariance matrix, the center of mass position error, and the shape matching error into a pre-built reward function to obtain a reward value;

[0011] S4. Update the policy network and value network in the TD3 algorithm based on the reward value, and dynamically optimize the noise switching probability in combination with the policy action generated in the current state;

[0012] S5. Feedback the noise switching probability into the extended target state estimate of S2, iterate S2-S4 in a loop until a preset iteration target is reached, obtain an optimal noise switching probability learning strategy, and track the extended target in a non-stationary noise environment by real-time optimizing the state estimate of the extended target based on the optimal noise switching probability learning strategy.

[0013] Optionally, in S1, according to the Gaussian-Student t mixture distribution model, the noise switching probability optimization problem is constructed as a Markov decision process <S, A, P, R>, where S is the state space, consisting of the trace of the noise covariance and the noise switching probability; A is the action space, consisting of the decision set; P is the state transition probability, consisting of the dynamic update of the noise switching probability and the state evolution of the filter; R is the reward function, consisting of the trace of the noise covariance matrix, the shape matching error, and the center of mass position error.

[0014] Optionally, the reward function is:

[0015] r t =α(Tr(C k )-Tr(C k+1 ))+β(1-IoU Loss)+γ(1-RMSE);

[0016] Among them, Tr(Σ k ) is the trace of the covariance matrix at time k; IoU Loss represents the shape matching error; RMSE represents the center of mass position error; α, β, γ are the weights of the reward item, r t is the reward value.

[0017] Optionally, in S2, performing star-convex random hypersurface modeling on the extended target based on the measurement information includes:

[0018] Establishing a measurement source model based on the spatial distribution hypothesis, and generating a plurality of measurement sources of the extended target using the measurement source model;

[0019] generating a plurality of measurements of an extended target based on the measurement source mixed with sensor noise;

[0020] Based on the measurement source and the measurement, a star-convex random hypersurface is used for modeling to obtain an observation model of a star-convex extended target.

[0021] Optionally, the observation model of the star convex extended target is:

[0022]

[0023] Among them, h(·) is the pseudo-measurement equation, x k is the state of the extended target at time k, υ k,l is the measurement noise of the lth sampling point at the extended target k moment, s k,l is the scale factor of the lth sampling point at the kth moment of the extended target, z k,l is the measurement of the lth sampling point at the extended target k moment, are the Fourier coefficients, is the shape parameter vector of the extended target at time k, is the direction vector, m k is the center of mass of the extended target at time k, is the angle between the vector between the center of mass and the measurement source at time k and the x-axis.

[0024] Optionally, in S4, updating the policy network and the value network in the TD3 algorithm based on the reward value, and dynamically optimizing the noise switching probability in combination with the policy action generated in the current state includes:

[0025] Initialize the target state and network parameters of the TD3 algorithm;

[0026] According to the current strategy, the switching probability at time k is input into the strategy network to obtain the action at time k+1;

[0027] Based on the switching probability at time k and the action at time k+1, the switching probability at time k+1 is obtained;

[0028] Substitute the action and switching probability at time k+1 into the filter, update the state vector and covariance matrix, calculate the reward value and execute the action to obtain the noise switching probability at time k+1.

[0029] Optionally, updating the state vector and the covariance matrix includes:

[0030]

[0031] Among them, a k+1 is the action at time k+1, It is k+1 time according to action a k+1 The predicted state vector obtained, ω k+1,j It is the weight coefficient of the j-th predicted state at time k+1, that is, the weight of each sigma point in UKF. is the state prediction of the jth sigma point at time k+1, is the error between the predicted state of the jth sigma point and the weighted average predicted state, is the covariance matrix of the state and measurement at time k+1, Represents the error between the predicted state vector and the actual target state, x k+1|k represents the prior state estimate at time k+1, P k+1|k represents the prior covariance matrix at time k+1, x k+1|k+1 (a k+1 ) represents the posterior state estimate at time k+1, K is the Kalman gain, P k+1|k+1 (a k+1 ) is the k+1 time posterior covariance matrix, is the prediction covariance matrix at time k+1.

[0032] Optionally, in the loop iteration, updating the value network and the policy network includes:

[0033] Calculate the target value y i :

[0034] y i =R j +γmin(Q1(s k+1,j ,a' k+1,j |θ'1),Q2(s k+1,j ,a' k+1,j |θ'2));

[0035] Update the value network:

[0036]

[0037] Update the policy network:

[0038]

[0039] Soft Update:

[0040]

[0041] Among them, y i is the value function of the target action-value function, R jis the instantaneous reward at the current moment, γ is the discount factor, Q1 and Q2 are the estimated values obtained by the dual value network, and s k+1,j and a' k+1,j is the state and action at the next moment, θ1 and θ2 are the current value network parameters, φ is the current strategy network parameter, and are the loss functions of the two value networks, s k,j and a k,j is the state and action at the current moment, is the gradient of the policy network, represents the gradient of the Q1 network, a=μ(s k,j |φ) represents the state s k,j When the action output by the policy network is represents the gradient of the policy network, θ'1 and θ'2 are the target value network parameters, φ' is the target policy network parameter, and τ is the update coefficient.

[0042] The beneficial effects of the present invention are:

[0043] The present invention proposes a TD3-based extended target tracking optimization method in a non-stationary noise environment, which can significantly improve the tracking accuracy and real-time performance of extended targets in non-stationary noise environments. The present invention dynamically optimizes the noise switching probability, adaptively switches between Gaussian distribution and fat-tailed distribution, combines a star-convex random hypersurface model to capture the global and local characteristics of the target, and uses a reward function to optimize the trace value of the covariance matrix, thereby achieving joint optimization of the state and shape of the extended target. The present invention gives full play to the advantages of the TD3 algorithm and the random hypersurface model, and provides an efficient and reliable solution for complex dynamic target tracking in the fields of unmanned driving, intelligent monitoring, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a flow chart of an extended target tracking optimization method in a non-stationary noise environment based on TD3 according to an embodiment of the present invention;

[0046] Figure 2 The structure and flow chart of the star convex extended target tracking algorithm based on TD3 according to an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of the TD3 neural network structure according to an embodiment of the present invention;

[0048] Figure 4 Target trajectory tracking diagrams of an embodiment of the present invention, where (a) is a single extended target tracking diagram, (b) is a complete tracking result diagram of the target trajectory by the present method and the comparative method, and (c) is an enlarged diagram of the target trajectory tracking details by the present method and the comparative method;

[0049] Figure 5 Magnified diagrams of target trajectory tracking in three stages of an embodiment of the present invention, where (a) is the starting stage, (b) is the middle stage, and (c) is the ending stage;

[0050] Figure 6 IoU curves of the present method and the comparative method according to the embodiment of the present invention;

[0051] Figure 7 This is a comparison chart of the RMSE of the estimated centroid of the tracking target of the present method and the comparative method according to an embodiment of the present invention;

[0052] Figure 8 This is a reward curve diagram of the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] This embodiment provides an extended target tracking optimization method in a non-stationary noise environment based on TD3. Figure 1 Shown, including:

[0056] S1. According to the Gaussian-Student t mixture distribution model, the noise switching probability optimization problem is constructed as a Markov decision process, and the measurement information of the extended target is obtained. The constructed Markov decision process is specifically:<S,A,P,R> , where S is the state space, which consists of the trace of the noise covariance and the noise switching probability; A is the action space, which consists of the decision set; P is the state transition probability, which consists of the dynamic update of the noise switching probability and the state evolution of the filter; R is the reward function, which consists of the trace of the noise covariance matrix, the shape matching error, and the center of mass position error.

[0057] S2. Based on the measurement information, the extended target is modeled using a star-convex random hypersurface, and an unscented Kalman filter is used to estimate the state of the extended target, obtain an estimated position and a covariance matrix, and calculate the trace of the covariance matrix, the center of mass position error, and the shape matching error;

[0058] The step of performing star-convex random hypersurface modeling on the extended target based on the measurement information specifically includes:

[0059] Establishing a measurement source model based on the spatial distribution hypothesis, and generating a plurality of measurement sources of the extended target using the measurement source model;

[0060] generating a plurality of measurements of an extended target based on the measurement source mixed with sensor noise;

[0061] Based on the measurement source and the measurement, a star-convex random hypersurface is used for modeling to obtain an observation model of a star-convex extended target.

[0062] S3. Inputting the trace of the covariance matrix, the center of mass position error, and the shape matching error into a pre-built reward function to obtain a reward value;

[0063] S4. Update the policy network and value network in the TD3 algorithm based on the reward value, and dynamically optimize the noise switching probability in combination with the policy action generated by the current state, specifically including:

[0064] Initialize the target state and network parameters of the TD3 algorithm;

[0065] According to the current strategy, the switching probability at time k is input into the strategy network to obtain the action at time k+1;

[0066] Based on the switching probability at time k and the action at time k+1, the switching probability at time k+1 is obtained;

[0067] Substitute the action and switching probability at time k+1 into the filter, update the state vector and covariance matrix, calculate the reward value and execute the action to obtain the noise switching probability at time k+1.

[0068] S5. Feedback the noise switching probability into the extended target state estimate of S2, iterate S2-S4 in a loop until a preset iteration target is reached, obtain an optimal noise switching probability learning strategy, and track the extended target in a non-stationary noise environment by real-time optimizing the state estimate of the extended target based on the optimal noise switching probability learning strategy.

[0069] Specifically, this embodiment dynamically optimizes the noise switching probability, adaptively switching between Gaussian and fat-tailed distributions, combining a star-convex random hypersurface model to capture the global and local characteristics of the target, and using a reward function to optimize the trace value of the covariance matrix, thereby achieving joint optimization of the extended target state and shape. In summary, this invention fully leverages the advantages of the TD3 algorithm and random hypersurface model, providing an efficient and reliable solution for complex dynamic target tracking in fields such as unmanned driving and intelligent monitoring.

[0070] The measurement information used for extended target tracking optimization must be obtained from sensors. First, a sensor extended target tracking database is established, storing the measurement data corresponding to extended targets detected by the sensors at different time points. Based on this measurement information, the extended target's geometry is modeled using a star-convex random hypersurface model, and the noise characteristics are described using a Gaussian-Student t mixture distribution model. The TD3 algorithm is then used to dynamically optimize the noise switching probability, obtaining the optimal noise distribution parameters at each time point and an estimate of the extended target's state and shape.

[0071] During the optimization process, the TD3 algorithm's agent determines whether the current strategy outperforms the previous strategy based on the current reward value and dynamically updates its strategy and value function based on the results. After repeated iterations of this process, the reinforcement learning agent learns the optimal noise switching strategy, enabling highly accurate tracking of extended targets in complex dynamic environments. Ultimately, using optimized noise parameters and geometric modeling parameters, the agent achieves optimal contour estimation accuracy, center of mass estimation accuracy, and tracking stability for extended target tracking.

[0072] The multi-feature state of the extended target at time k contains motion parameters and shape parameters, which can be expressed as Among them, m k is the center of mass of the expanded target at time k, M k represents the motion parameter, p k is the shape contour parameter of the target, then the motion equation of the extended target is:

[0073] x k+1 =f k (x k )+w k (1);

[0074] Among them, f k (·) is the system state evolution map, w k is the process noise.

[0075] As an implementation, the measurement modeling process for extended targets can be broken down into two steps. First, a measurement source model is established based on reasonable spatial distribution assumptions, and this model is used to generate multiple measurement sources for the extended target. Second, the measurement sources are mixed with sensor noise to produce multiple measurements of the target. The sensor measurement model can be described as follows:

[0076] z k,l =y k,l +υ k,l (2);

[0077] Among them, y k,l is the measurement source position of the lth sampling point at the extended target k moment, z k,l To expand the target measurement, υ k,l is the measurement noise (l=1,...,N).

[0078] This embodiment uses the star convex extended target model algorithm as the target tracking algorithm. The star convex is defined as: if a set All points on the line segment from any point in to the centroid still belong to the set, then the shape formed by the set S is a star convex shape. Assume that the shape is parameterized by the set express:

[0079]

[0080] Among them, the scale factor s k,l ∈[0,1] can be regarded as multiplicative noise. This embodiment assumes a Gaussian distribution, which can reflect the relative distance from the measurement source to the center of mass; φ k ∈[0,2π] represents the measurement source and the center of mass m k The angle between the vector and the x-axis; e(φ k )=[cos(φ k ),sin(φ k )] T is the direction vector. N can be expanded by the Fourier series of the radial function f The low-frequency harmonic components of the Fourier series describe the basic outline of the star convex shape, while the high-frequency harmonic components show the local outline characteristics of the star convex shape.

[0081]

[0082] Among them, R(φ k ) is the Fourier coefficient, a k and b k are the target shape parameters The weight.

[0083] If mk represents the center of mass of the extended target at time k, then the measurement source y at time k k,l It can be expressed as:

[0084]

[0085] The angle between the target mass center, the measurement source and the x-axis is φ k,l =∠(y k,l -m k )express; For shape The boundaries of , satisfy:

[0086]

[0087] From the implicit description of the star-convex curve, we can get:

[0088] g(y k,l ,x k )=‖‖y k,l -m k ‖‖ 2 -r 2 (9);

[0089] in, So we get the scaled bounds:

[0090]

[0091] Combining the measurement equation, we can get the measurement equation of the star convex extension target:

[0092]

[0093] Arranged where φ k,l It is unknown, and the point estimate of the target center of mass at the current moment is generally used and measure z k,l The angle between the vector and the axis is replaced by Combining these we get:

[0094]

[0095] in, Then the pseudo-measurement equation is:

[0096]

[0097] The observation model of the star-shaped extended target can be described by using formula (13), that is, the pseudo-measurement equation h(·) establishes the extended target state x k , measurement noise υ k,l , scale factor s k,lAnd measure z k,l The relationship between them.

[0098] As an implementation method, the TD3 algorithm uses a delayed dual deterministic strategy, which is applied to optimize the noise switching probability. Through a continuously cyclic feedback process, an optimal noise switching strategy is gradually learned, thereby achieving accurate tracking estimation of the extended target state and shape.

[0099] This embodiment is based on the Markov decision process framework, which can formulate the noise modeling and parameter optimization problem in extended target tracking into a Markov decision process.<S,A,P,R> First, define the relevant elements:

[0100] 1) The state space S consists of two main variables: the trace of the noise covariance, which reflects the non-stationary nature of the current noise; and the switching probability, which is the switching weight between the Gaussian distribution and the Student t distribution. Therefore, the state is defined as:

[0101]

[0102] 2) The action space A represents the set of decisions that the reinforcement learning agent can take. The agent's action represents the adjustment of the switching probability to adapt to the current noise characteristics and is defined as:

[0103] a k ∈[-Δπ max ,Δπ max ](15);

[0104] The action range is clipped to ensure that the switching probability is always between [0,1].

[0105] 3) P represents the state transition probability. The system state transition at a moment describes the impact of the current state and action on the state at the next moment. State transition includes the dynamic update of the switching probability and the evolution of the filter state:

[0106] π k+1 =π k +a k (16);

[0107]

[0108] Among them, F k is the state transfer matrix, Q k is the process noise covariance matrix.

[0109] 4) R represents the reward function that maps parameter states and agent action choices to rewards. It is used to measure the quality of actions and aims to guide the agent to learn action strategies that can improve target tracking accuracy. This embodiment designs a composite reward function that considers the following three factors:

[0110] ① The trace of the noise covariance matrix: reflects the uncertainty of the target state. A decrease in the covariance indicates a decrease in the uncertainty of the estimate, thus giving a positive reward;

[0111] ②IOU loss: the matching error between the true target shape and the estimated shape. The higher the IoU, the greater the reward.

[0112] ③ RMSE of the center of mass: The error between the target center of mass estimate and the true center of mass, which is used to measure the accuracy of the target state estimation. The smaller the error, the greater the reward.

[0113] After each reward is normalized, the mathematical form of the reward function is as follows:

[0114] r t =α(Tr(C k )-Tr(C k+1 ))+β(1-IoU Loss)+γ(1-RMSE) (18);

[0115] Among them, Tr(Σ k ) is the covariance matrix trace at time k; IoU Loss represents the loss of shape matching; RMSE represents the centroid matching error; α, β, and γ are the weights of the reward items.

[0116] To address optimization problems in continuous action spaces, the TD3 algorithm uses a deep reinforcement learning network based on an actor-critic architecture, consisting of a policy network and a value network. To improve training stability, TD3 incorporates a dual-value network mechanism, soft updates to the target network, and an action noise smoothing strategy. This example implements the following optimizations for non-stationary noise scenarios.

[0117] The policy network is responsible for k Generate action a k , used to interact with the environment, namely:

[0118] a k =μ(s k |θ μ ) (19);

[0119] Among them, μ(·) represents the function mapping of the policy network, θ μ is the network parameter. The optimization goal of the policy network is to learn the optimal action by maximizing the Q value of the Critic network:

[0120] L a = -Ε[Q1(s,a)] (20);

[0121] The action-value network evaluates the value Q(s,a) of a state-action pair (s,a).

[0122] TD3 uses dual value networks Q1 and Q2 to alleviate the problem of overestimation of value:

[0123] Q(s k ,a k |θ Q )=f(s k ,a k |θ Q ) (twenty one);

[0124] TD network adopts a three-layer fully connected architecture, designed as follows Figure 3 shown.

[0125] As an implementation method, this embodiment uses a dynamic optimization filter based on the TD3 algorithm to jointly estimate the state and shape of the star-shaped extended target, such as Figure 2 As shown, the control process of the extended target tracking sensor based on the TD3 algorithm is described below.

[0126] This embodiment combines a reinforcement learning agent with a star-convex stochastic hypersurface model and utilizes the TD3 algorithm to dynamically adjust the noise switching probability, resulting in more accurate state estimation of the extended target. The sensor control process includes sensor acquisition of extended target measurement information, recursive target state estimation based on a pseudo-measurement equation, and dynamic optimization of a Gaussian-Student t mixture distribution through reinforcement learning. Based on reward function feedback, the agent dynamically adjusts the system noise modeling and filter parameters, thereby improving the robustness and real-time performance of target tracking.

[0127] In complex dynamic environments, non-stationary, thick-tailed noise poses a significant challenge to extended target tracking accuracy. This embodiment combines a random hypersurface model with an unscented Kalman filter and uses the TD3 algorithm to dynamically optimize the switching probability of the mixed noise distribution, achieving accurate estimation of the extended target's motion state and geometry. The following describes the TD3-based optimization of extended target tracking in the presence of non-stationary noise.

[0128] Initialize the target state (x k ,P k|k ) and TD3 network parameters (θ1, θ2, θ'1, θ'2), use formula (1) to perform one-step update prediction and obtain (x k+1|k ,P k+1|k ).

[0129] The process noise and measurement noise of the extended target may exhibit nonstationary characteristics. To describe these characteristics, noise modeling is performed. The switching probability π is dynamically optimized through reinforcement learning to adapt to the variations in nonstationary noise. During filtering, π approaches 1 when the noise distribution is close to a Gaussian distribution and approaches 0 when the noise has a thick tail.

[0130] According to the current strategy μ, the switching probability s at time k is k Input into the policy network and get the action at time k+1:

[0131] a k+1 =μθ(s k )+N (22);

[0132] Where N is the introduced Gaussian noise. The output action set in this embodiment is an increment, which represents the adjustment of the switching probability, and the switching probability s at time k+1 is obtained. k+1 (a k+1 ):

[0133] s k+1 (a k+1 )=s k +a k+1 (twenty three);

[0134] Among them, s k+1 (a k+1 )=[G k ,π k (a k+1 )] T The first term is the trace of the noise covariance during the filtering process, and the second term is the trace of the noise covariance after the action is performed at time k+1. k+1 The obtained switching probability.

[0135] The switching probability s at the current moment k and the action a selected by the policy network k+1 Get the switching probability s at time k+1 k+1 (a k+1 ), substitute it into the filter to calculate. Suppose there is a set of [x k+1|k ,a k ,s k ] to obtain the sampling points Corresponding weight ω k+1,j , then the sampling point after propagation is defined as:

[0136]

[0137] in, For the sampling point in action step a k+1 Then update the state vector and covariance matrix:

[0138]

[0139] Get the reward value R according to the reward function k After that, the output action is executed. The switching probability s at the next moment k+1 is obtained k+1 , so far one cycle ends, the state, action and other data are collected into data group {s k+1 (a k+1 ),a k+1 ,R k ,s k+1}, store it in the experience replay pool H, and loop this scene. Set the number of loops to T. After the loop ends, randomly extract n groups of data from the experience replay pool H to calculate the target value.

[0140] Target value calculation Calculate the target value y using the minimum value in the dual value network Q1 and Q2 i :

[0141] y i =R j +γmin(Q1(s k+1,j ,a' k+1,j |θ'1),Q2(s k+1,j ,a' k+1,j |θ'2)) (31);

[0142] The target action is:

[0143] a' k+1,j =μ'(s k+1 |φ')+ε,ε~clip(N(0,σ 2 ),-c,c) (32);

[0144] The value network update updates the parameters of the two value networks through the loss functions Q1 and Q2:

[0145]

[0146] Policy Network Update The policy network is updated every d cycles, and the parameter φ is updated using the following gradient formula:

[0147]

[0148] Finally, the target network parameters θ'1, θ'2 and the policy network target parameter φ' are updated through the soft update strategy:

[0149]

[0150] Here, τ is the update coefficient. In order to reduce the recurrence data correlation, the update coefficient is generally set to a smaller value.

[0151] In the star-convex random hypersurface model, the noise switching probability and geometric modeling parameters have a direct impact on the accuracy of the measurement equation and state estimation. If the noise model parameters or the geometric model are improperly set, it may lead to large estimation errors and instability in the tracking process. Therefore, it is necessary to find an optimal dynamic optimization scheme to reduce the adverse effects of noise and modeling parameters on target state estimation and enhance its adaptability. The iterative optimization algorithm in reinforcement learning provides an effective solution to the above problems, especially for the optimization problems in non-stationary noise scenarios. There are few related studies. Therefore, this embodiment proposes a star-convex extended target tracking optimization method based on the TD3 algorithm, which dynamically optimizes the state and shape estimation process of the extended target, thereby significantly improving the accuracy and robustness of extended target tracking.

[0152] This embodiment uses a star-convex random hypersurface model to model the geometric contour of the extended target. The model can flexibly adapt to the complex geometric shape of the target and can still obtain a good estimation effect when there is a significant difference between the actual shape of the target and the modeled shape. Therefore, this method has important practical application value in solving target recognition, detection and tracking problems. On this basis, combined with the parameter optimization method based on TD3 reinforcement learning, by dynamically adjusting the noise switching probability and key model parameters, the estimation effect of the target center of mass motion state and geometric shape is significantly improved, thereby providing higher tracking accuracy and system stability in a non-stationary noise environment.

[0153] Simulation analysis:

[0154] Set the system sampling period T s = 1s, tracking time is N = 200 sampling cycles. The number of measurements collected by the extended target n mea The compliance strength is β D = 40 Poisson distribution. The initial state of the expansion target is defined as a circle with a radius of 8 cm, and the Fourier expansion order in the shape parameter is N f =16, the initial value of the switching probability is set to 0.7. The number of Monte Carlo experiment cycles T = 200. The extended target shape is set to a cross. The initial target state parameters include the position, velocity, and shape parameters in the x and y axis directions. The initial target parameters and state covariance are:

[0155] x0=[10,10,10,10,8,0,...,0] 1×20 (36);

[0156] C0=diag([0.3,0.3,0.3,0.3,0.02,…,0.02]) 20×20 (37);

[0157] The covariance matrix of the system measurement noise is R, the system noise is zero-mean Gaussian noise, the covariance matrix is Q, and the corresponding state transfer matrix is F:

[0158]

[0159]

[0160] Where q1 = 0.01 2 ,q2=0.03 2 ,ν=5.

[0161] The hyperparameters of the TD3 deep reinforcement learning algorithm are set as shown in Table 1.

[0162] Table 1

[0163]

[0164] In the above simulation scenario, the star convex extended target tracking optimization method of the TD3 algorithm is used to track the extended target. Figure 4 As shown, Figure 4 Figure (a) shows the tracking of a single extended target, showing the estimated center of mass, tracking curve, and measurement data points. The target shape and position estimates are relatively accurate, with most measurement points falling within the tracking boundary. This method effectively tracks the motion and extended shape of a single extended target, and the model is highly robust to noise. The extended target tracking results show that the estimated shape fits the target's uncertainty region well. The measurement points are largely distributed within the target estimation region, further validating the accuracy of the method. Figure 4 (b) shows the complete tracking results of the target trajectory by the two comparison methods. It can be clearly seen that the overall trajectory is not a straight line due to the non-stationarity of measurement noise and process noise, but both methods can track it well. Figure 4 (c) shows a zoomed-in view of target trajectory tracking details from the comparison method, demonstrating the difference in shape tracking details between the reinforcement learning method and the traditional method. The reinforcement learning method fits the target boundary more accurately, while the traditional method exhibits deviations at the target boundary. In terms of shape tracking, the reinforcement learning method is more capable of capturing subtle changes in the target, demonstrating its superiority in complex noisy environments.

[0165] Figure 5Zoomed-in images of the target's motion at any three stages are shown, with (a) representing the start, (b) the middle, and (c) the end. As can be seen from the figure, the target trajectory generated by the reinforcement learning algorithm TD3 is more accurate, as evidenced by a better fit between the trajectory and the target center than the traditional variational Bayesian method. Under complex non-Gaussian noise conditions, the TD3 algorithm maintains low deviation, demonstrating its ability to accurately estimate the target's motion state. The TD3 algorithm's estimates of the target's extended and motion states are more accurate, closer to the true target state. In the complex noisy environment of the experimental setup, the TD3 algorithm effectively avoids the overfitting and divergence that can occur with traditional methods. In contrast, traditional methods exhibit greater deviation in target trajectories at different stages, particularly when the target position changes significantly, resulting in a weakened trajectory tracking effect. This demonstrates the algorithm's greater adaptability and robustness, making it suitable for complex, dynamically changing scenarios.

[0166] Figure 6 The IoU curves for the two methods are shown. The reinforcement learning algorithm TD3 exhibits higher IoU values throughout the entire time step, with its curve stabilizing around 0.8, indicating more accurate predictions of the target region. While the traditional variational Bayesian method VB gradually improves early on, it exhibits significant fluctuations later on, with the IoU dropping below 0.6 for most time steps. This indicates that the VB method is less stable when handling complex scenarios. In time steps 50-100, which simulates the impact of a sudden, unknown, thick-tailed noise event, the TD3 method responds quickly, and the IoU curve quickly returns to around 0.8 while maintaining tracking. In contrast, the VB method exhibits significant overall fluctuations, and its tracking performance is far inferior to that of the TD3 method. The TD3 method's IoU curve converges quickly and reaches a stable state around time steps 20-30, demonstrating that the TD3 algorithm can quickly adapt to the target's characteristics and generate stable predictions. The VB method improves more slowly initially and does not achieve complete stability throughout the entire time period, reflecting its weaker adaptability to complex scenarios. The TD3 algorithm exhibits smaller fluctuations, indicating greater robustness for target tracking in dynamic environments. In contrast, the VB method exhibits more significant fluctuations, indicating its greater sensitivity to noise, occlusion, and complex target changes. The TD3 algorithm exhibits a higher and more stable overall IoU value, demonstrating superior prediction accuracy and reliability in target tracking tasks. While the VB method performs reasonably well at some time steps, its overall performance is limited, particularly when dealing with non-Gaussian noise and complex scenes.

[0167] Figure 7Comparison of the RMSE for estimating the center of mass of a tracked target. The RMSE of the TD3 algorithm remained low throughout the entire timeframe, primarily between 0.2 and 0.4, with minimal fluctuations, demonstrating its superior tracking accuracy and stability. The RMSE of the VB algorithm fluctuated significantly, reaching 0.5 or even 1.0 on several occasions, indicating large prediction errors in some time steps. The TD3 algorithm exhibited minimal error fluctuations throughout the entire timeframe, demonstrating its robustness and ability to accurately estimate the target's motion state. During time steps 50-100, the simulation encountered a sudden burst of unknown, thick-tailed noise. The figure shows that the TD3 method exhibited relatively stable fluctuations and returned to normal more quickly than the VB method. The VB algorithm's RMSE curve exhibited significant fluctuations, particularly during time steps 20-60, with multiple instances of large errors, indicating instability in complex scenarios. The trend over the first 20 time steps shows that the TD3 algorithm converged quickly to a stable error level, demonstrating its high learning efficiency. In contrast, the VB algorithm exhibited large error fluctuations in the early stages and showed no subsequent stabilization trend, indicating poor adaptability to complex dynamic environments. The TD3 algorithm exhibited smaller errors and limited fluctuations across time steps, demonstrating strong robustness. However, the VB algorithm was highly sensitive to noise or changes in target motion, prone to large errors and lacking robustness.

[0168] Figure 8 This is the reward curve obtained by the TD3 reinforcement learning algorithm during training. The reward value shows a gradual increase with each time step, indicating that the algorithm gradually improves during the learning process. The reward value gradually stabilizes around -50 from an initial low level, demonstrating that the algorithm has achieved its learning objective and achieved a stable policy. The reward value remains relatively stable in the second half of the time step, indicating that the algorithm has completed most of the learning and is gradually converging. The post-convergence reward value exhibits minimal fluctuation, indicating that the algorithm's policy is relatively stable. Throughout the learning process, the reward value exhibits some fluctuation, especially with occasional large drops after convergence. Therefore, the algorithm's robustness in handling complex or unusual conditions requires further improvement. The stability and trend of the reward value indicate that the algorithm is well adapted to dynamic environments and gradually learns an effective policy, but additional optimization may be required in some extreme cases.

[0169] This embodiment proposes an extended target tracking optimization method in a non-stationary noise environment based on the TD3 algorithm. By modeling the noise switching probability as the action of the reinforcement learning agent and combining it with the Gaussian-Student t mixed distribution model, this embodiment realizes the dynamic switching of Gaussian and thick-tailed noise. The extended target is geometrically modeled based on the star-convex random hypersurface model, and the key parameters of the filter are dynamically adjusted under the TD3 framework, thereby effectively improving the accuracy and robustness of state estimation. Through simulation experiments, it is verified that compared with the traditional variational Bayesian method, the method of this embodiment significantly reduces the estimation error in complex noise scenarios and exhibits stronger robustness and real-time performance. The experimental results show that the adaptive optimization capability of the reinforcement learning algorithm TD3 in a dynamic noise environment can achieve accurate tracking of the motion state and geometric shape of the extended target, and provides a new idea for solving the extended target tracking problem under non-Gaussian noise.

[0170] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A TD3-based extended target tracking optimization method in a non-stationary noise environment, characterized in that: Including: S1. According to the Gaussian - Student t mixture distribution model, construct the noise switching probability optimization problem as a Markov decision process, and obtain the measurement information of the extended target; S2. Based on the measurement information, perform star - convex stochastic hypersurface modeling on the extended target, and combine the unscented Kalman filter to estimate the state of the extended target, obtain the estimated position and covariance matrix, and calculate the trace of the covariance matrix, centroid position error, and shape matching error; S3. Input the trace of the covariance matrix, centroid position error, and shape matching error into a pre - constructed reward function to obtain a reward value; S4. Based on the reward value, update the policy network and value network in the TD3 algorithm, and dynamically optimize the noise switching probability by combining the policy actions generated by the current state; S5. Feed back the noise switching probability to the extended target state estimation in S2, loop and iterate S2 - S4 until a preset iteration target is reached, obtain the optimal noise switching probability learning strategy, and based on the optimal noise switching probability learning strategy, track the extended target in a non - stationary noise environment by real - time optimizing the state estimation of the extended target.

2. The extended target tracking optimization method in a non-stationary noise environment based on TD3 according to claim 1 is characterized in that: In S1, according to the Gaussian - Student t mixture distribution model, construct the noise switching probability optimization problem as a Markov decision process <S, A, P, R>, where S is the state space, composed of the trace of the noise covariance and the noise switching probability; A is the action space, composed of a decision set; P is the state transition probability, composed of the dynamic update of the noise switching probability and the state evolution of the filter; R is the reward function, composed of the trace of the noise covariance matrix, shape matching error, and centroid position error.

3. The extended target tracking optimization method in a non-stationary noise environment based on TD3 according to claim 2 is characterized in that: The reward function is: r t =α(Tr(C k )-Tr(C k+1 ))+β(1-IoU Loss)+γ(1-RMSE); Among them, Tr(Σ k ) is the trace of the covariance matrix at time k; IoU Loss represents the shape matching error; RMSE represents the center of mass position error; α, β, γ are the weights of the reward item, r t is the reward value.

4. The extended target tracking optimization method in a non-stationary noise environment based on TD3 according to claim 1 is characterized in that: In S2, performing star - convex stochastic hypersurface modeling on the extended target based on the measurement information includes: Establish a measurement source model according to the spatial distribution hypothesis, and use the measurement source model to generate several measurement sources of the extended target; Generate several measurements of the extended target based on the measurement sources and mixed sensor noise; Based on the measurement sources and measurements, use a star - convex stochastic hypersurface for modeling to obtain the observation model of the star - convex extended target.

5. The extended target tracking optimization method in a non-stationary noise environment based on TD3 according to claim 4 is characterized in that: The observation model of the star - convex extended target is: Among them, h(·) is the pseudo-measurement equation, x k is the state of the extended target at time k, υ k,l is the measurement noise of the lth sampling point at the extended target k moment, s k,l is the scale factor of the lth sampling point at the kth moment of the extended target, z k,l is the measurement of the lth sampling point at the extended target k moment, are the Fourier coefficients, is the shape parameter vector of the extended target at time k, is the direction vector, m k is the center of mass of the extended target at time k, is the angle between the vector between the center of mass and the measurement source at time k and the x-axis.

6. The extended target tracking optimization method in a non-stationary noise environment based on TD3 according to claim 1 is characterized in that: In S4, updating the policy network and value network in the TD3 algorithm based on the reward value and dynamically optimizing the noise switching probability by combining the policy actions generated by the current state includes: Initialize the target state and the network parameters of the TD3 algorithm; According to the current policy, input the switching probability at time k into the policy network to obtain the action at time k + 1; Based on the switching probability at time k and the action at time k + 1, obtain the switching probability at time k + 1; Substitute the action and switching probability at time k + 1 into the filter, update the state vector and covariance matrix, calculate the reward value and execute the action, and obtain the noise switching probability at time k + 1.

7. The extended target tracking optimization method in a non-stationary noise environment based on TD3 according to claim 6 is characterized in that: Updating the state vector and covariance matrix includes: Among them, a k+1 is the action at time k+1, It is k+1 time according to action a k+1 The predicted state vector obtained, ω k+1,j It is the weight coefficient of the j-th predicted state at time k+1, that is, the weight of each sigma point in UKF. is the state prediction of the jth sigma point at time k+1, is the error between the predicted state of the jth sigma point and the weighted average predicted state, is the covariance matrix of the state and measurement at time k+1, Represents the error between the predicted state vector and the actual target state, x k+1|k represents the prior state estimate at time k+1, P k+1|k represents the prior covariance matrix at time k+1, x k+1|k+1 (a k+1 ) represents the posterior state estimate at time k+1, K is the Kalman gain, P k+1|k+1 (a k+1 ) is the k+1 time posterior covariance matrix, is the prediction covariance matrix at time k+1.

8. The extended target tracking optimization method in a non-stationary noise environment based on TD3 according to claim 1 is characterized in that: In the loop iteration, the update of the value network and policy network includes: Calculate the target value y i : y i =R j +γmin(Q1(s k+1,j ,a' k+1,j |θ'1),Q2(s k+1,j ,a' k+1,j |θ'2)); Update the value network: Update the policy network: Soft update: Among them, y i is the value function of the target action-value function, R j is the instantaneous reward at the current moment, γ is the discount factor, Q1 and Q2 are the estimated values obtained by the dual value network, and s k+1,j and a' k+1,j is the state and action at the next moment, θ1 and θ2 are the current value network parameters, φ is the current strategy network parameter, and are the loss functions of the two value networks, s k,j and a k,j is the state and action at the current moment, is the gradient of the policy network, represents the gradient of the Q1 network, a=μ(s k,j |φ) represents the state s k,j When the action output by the policy network is represents the gradient of the policy network, θ'1 and θ'2 are the target value network parameters, φ' is the target policy network parameter, and τ is the update coefficient.

Images