Reinforcement learning driven interactive multi-model aircraft filtering algorithm

Abstract

The invention discloses an interactive multi-model aircraft filtering algorithm and system driven by reinforcement learning, and the algorithm comprises the steps: firstly constructing a multi-model state prediction set composed of a Singer model, a current statistical model and a Jerk model, and presetting a state transition probability matrix according to the state evolution correlation between the models, so as to define the interaction relation between the models; in the filtering stage, state estimation preliminary fusion of the three maneuvering models is achieved according to a traditional interactive multi-model algorithm, the fusion proportion of each model is adjusted in real time in combination with a weight correction vector output by a reinforcement learning strategy function, and strategy iteration optimization is conducted through a reward function containing an error improvement item, a matching item and a balance item. The method can significantly improve the filtering precision and fusion stability of the aircraft maneuvering target.

Description

Technical Field

[0001] This invention relates to the field of target tracking and intelligent perception technology, specifically to a reinforcement learning-driven interactive multi-model aircraft filtering algorithm. Background Technology

[0002] In modern complex warfare and aerospace defense systems, the flight states of highly maneuverable targets (such as hypersonic vehicles and tactical missiles) exhibit highly nonlinear and time-varying characteristics. Their motion patterns typically involve high-speed maneuvering, non-constant acceleration, random abrupt changes, and complex external interference. Traditional target tracking methods, such as Kalman filters (KF) or extended Kalman filters (EKF) based on a single dynamic model, often suffer from significant estimation biases when facing targets with abrupt changes in maneuverability due to the fixed model, sometimes even leading to target loss. Figure 1 In the positioning and tracking scenario shown, the red aircraft represents our probe aircraft, and the blue aircraft represents the target to be tracked. The measurement model includes three dimensions: heading angle, pitch angle, and distance.

[0003] To improve the stability and accuracy of maneuvering target tracking, researchers proposed an Interacting Multiple Model (IMM) filtering algorithm. This method jointly estimates the target motion using multiple parallel state models and fuses them based on model probabilities, thus achieving adaptive switching between models to some extent. However, the model transition behavior of traditional IMM algorithms relies on a pre-set Markov state transition matrix, which is usually empirically set and kept fixed. When the target undergoes sudden maneuvers, the fixed transition probabilities cannot reflect the changing relationships between models in a timely manner, leading to a lag in model switching and resulting in significant instantaneous errors and tracking delays in the early stages of maneuvering.

[0004] Furthermore, traditional multi-model filtering methods still suffer from efficiency bottlenecks under environments with limited computational and communication resources. As the number of maneuvering models increases, the computational cost of state estimation and covariance updates grows exponentially, making it difficult to meet the demands of real-time, high-precision tracking. Although some studies have attempted to improve filtering performance by adaptively adjusting model weights or employing parameter estimation methods, a unified framework that can dynamically optimize model weight distribution and maintain a stable switching mechanism is still lacking.

[0005] In view of this, the present invention proposes a reinforcement learning-driven interactive multi-model aircraft filtering algorithm and system. Summary of the Invention

[0006] The purpose of this invention is to provide a reinforcement learning-driven interactive multi-model aircraft filtering algorithm and system, which aims to solve the problems of model switching lag and untimely weight updates in the traditional interactive multi-model filtering (IMM) algorithm for tracking highly maneuverable targets.

[0007] In a first aspect, the present invention provides a reinforcement learning-driven interactive multi-model aircraft filtering algorithm, comprising the following steps:

[0008] S1. Construct a multi-model state prediction set consisting of three maneuvering models, including the Singer model, the current statistical model, or the Jerk model; each model independently generates state predictions, and a model probability transition matrix is ​​preset based on the state evolution correlation between models to define the interaction relationship between models; at the same time, a measurement model consisting of azimuth, pitch, and relative distance is constructed.

[0009] S2. In the filtering stage, each maneuver model in the multi-model state prediction set achieves preliminary fusion of state estimation based on the traditional interactive multi-model algorithm; at the same time, the likelihood and residual covariance of each maneuver model are calculated to construct the reinforcement learning state space.

[0010] S3. In the weight update process, a reinforcement learning agent is introduced, which takes the model likelihood, residual change rate and historical weight distribution as policy input, outputs a weight correction vector through the policy function and adjusts the fusion ratio of each model in real time; the reinforcement learning process performs policy iterative optimization through a reward function containing error improvement term, model matching term and weight smoothing term, so as to realize the self-learning and convergence performance improvement of the filtering process.

[0011] As a preferred technical solution of the first aspect of the present invention, the step of defining the interaction relationship between models based on the preset Markov model transition logic includes the following steps:

[0012] Define the model transition probability matrix: Construct a fixed Markov probability transition matrix P, where the matrix element pᵢⱼ represents the prior probability that model i will transition to model j in the next time step;

[0013] Calculate the mixture probability: In the interactive input step of each filtering cycle, based on the transition matrix P and the posterior probabilities of each model at the previous time step, calculate the mixture probability of transitioning from model i to model j.

[0014] Generate mixed state estimates: Using the calculated mixed probabilities, the state estimates and covariances of all models at the previous time step are weighted and summed to generate a new, mixed initial state estimate and covariance for each model, which serves as the input for filtering at the current time step.

[0015] As a preferred technical solution of the first aspect of the present invention, adaptive adjustment and stability are achieved through a reinforcement learning weight adaptive unit, including the following steps:

[0016] Dynamic weight adjustment: The algorithm's adaptability is not achieved by adjusting the transition probabilities between models, but by the reinforcement learning agent adjusting the weights of the Singer, CS, and Jerk models in real time during the final state fusion at each filtering cycle.

[0017] Reward function-driven feedback: The error improvement term and the model matching term in the reward function form a feedback loop. When the weight adjustment can effectively reduce the overall filtering error, or when a higher weight is assigned to a model with a smaller error, the agent receives a positive reward, thereby learning a better weight allocation strategy.

[0018] As a preferred technical solution of the first aspect of the present invention, the stability constraint of the filtering process is achieved through the weight smoothing term in the reward function, including the following steps:

[0019] Define a weight smoothing term: Set a weight smoothing term in the reward function, which is defined as the negative value of the magnitude of the weight adjustment action vector;

[0020] Punishing drastic adjustments: When the agent chooses an excessively large weight adjustment, this smoothing term generates a negative reward, thereby suppressing drastic oscillations in the weight distribution;

[0021] Encouraging smooth transitions: Through this penalty mechanism, the agent tends to make smooth, small weight adjustments. When the system response is stable, the weight distribution naturally remains stable, thereby ensuring the overall convergence of the filtering process.

[0022] As a preferred technical solution of the first aspect of the present invention, the weight correction is a decision-making process that the reinforcement learning agent continuously performs in each filtering cycle, including the following steps:

[0023] Constructing a continuous state space: In each filtering cycle, the current fused target state residual, the weight information of each model, and the likelihood function of each model are combined to form the real-time state space of the reinforcement learning agent.

[0024] Continuous decision-making and updates: The reinforcement learning agent receives the state input at each time step and continuously outputs a weight adjustment action through its policy network. This action is immediately used to calculate the model fusion weights for the current cycle. The entire "perception-decision-action" cycle is completed uninterruptedly within each filtering cycle.

[0025] As a preferred technical solution of the first aspect of the present invention, the algorithm does not include explicit weight freezing control logic. The stability of the weights is implicitly guaranteed through the design of the reward function, including the following steps:

[0026] Implicit stabilization mechanism: By including a penalty term for the magnitude of weight adjustment in the reward function, the agent is encouraged to learn autonomously and output a small weight adjustment amount when the filtering error does not change significantly.

[0027] Maintaining dynamic adjustment capability: This mechanism avoids hard "freeze" commands, enabling the system to quickly release the implicit stable state and restore the adaptive adjustment capability of weights when faced with sudden changes in the target's maneuvering state, thus ensuring the robustness of tracking.

[0028] As a preferred technical solution of the first aspect of the present invention, the process of outputting the weight correction vector through the policy function and adjusting the fusion ratio of each model includes the following steps:

[0029] Constructing the policy function input: The input set is precisely defined as including the weights of each current model, the model likelihood function, and the residuals of the fused target state;

[0030] Policy function derivation: The policy function is implemented by a pre-trained deep neural network (such as DQN or D3QN), which maps the input state to the adjustment of each model weight;

[0031] Weight update and normalization: The weight correction vector output by the policy function is applied to the weights of the previous time step; to ensure that the sum of all model weights is always 1, the updated weight vector is normalized to generate the final fused weights for the current period.

[0032] As a preferred technical solution of the first aspect of the present invention, the iterative update of the policy function is completed during the offline training phase, including the following steps:

[0033] An offline training-online inference architecture is adopted: to ensure the real-time performance and low computational overhead of online filtering, the parameter updates of the policy network are completed using historical or simulation data in the offline stage; during online inference, only a fixed pre-trained model is used for fast decision-making.

[0034] Training based on experience replay: In offline training, the agent interacts with the simulation environment and stores the generated experience tuples such as states, actions, and rewards into the experience replay pool. During training, data is sampled from this pool.

[0035] Parameter update based on loss function: Using DQN or D3QN algorithms, the weight parameters of the policy network are iteratively updated by minimizing the loss function composed of the temporal difference error (TD-error) calculated based on the main network and the target network.

[0036] As a preferred technical solution of the first aspect of the present invention, the process of iteratively optimizing the reward function is characterized in that the reward function is composed of three weighting factors:

[0037] Filtering error improvement factor: defined as the reduction in total filtering error before and after the agent takes an action. This directly motivates the agent to learn weight adjustment strategies that can reduce the final fusion error.

[0038] Model matching factor: When the error of a specific model is less than the total error benchmark and its weight increases, or when its error is greater than the total error benchmark and its weight decreases, a positive reward is given. This incentive agent will allocate high weights to models with a higher matching degree to the current target motion state.

[0039] Weight smoothing factor: defined as the negative value of the magnitude of the weight adjustment vector, which ensures the stability and convergence of the filtering process by penalizing drastic weight changes.

[0040] In a second aspect, the present invention provides a reinforcement learning-driven interactive multi-model aircraft filtering system for performing the first aspect, comprising:

[0041] The three-dimensional aerodynamic modeling unit is used to establish the target dynamic equations in the inertial coordinate system, where lift, drag and side force are jointly determined by aerodynamic coefficients, air density, flight speed and reference area. The spatial attitude evolution of the target under the action of aerodynamic force and gravity is described by the coupling relationship between velocity, pitch angle and tilt angle. It is used to construct the real motion trajectory of the target and provide a simulation environment for highly maneuverable targets for the algorithm.

[0042] The multi-model state prediction unit constructs a prediction set in parallel, consisting of the Singer model, the current statistical model, and the Jerk model. Each model independently generates state predictions and forms a state transition probability matrix based on the correlation between state evolutions between models, so as to achieve preliminary updates of the model probabilities of each model.

[0043] The multi-model interaction fusion unit is bidirectionally connected to the prediction unit. The three maneuvering models achieve preliminary fusion of state estimation based on the traditional interactive multi-model algorithm. At the same time, the likelihood and residual covariance of each model are calculated to provide data support for the subsequent construction of the reinforcement learning state space.

[0044] The reinforcement learning weight adaptive unit is bidirectionally connected to the fusion unit. Based on the model likelihood, residual change rate and historical weight distribution input, it outputs a weight correction vector through the policy function and performs self-optimization through a reward function containing error improvement, model matching and smoothing constraints, thereby realizing a self-learning closed loop of dynamics layer, fusion layer and policy layer.

[0045] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0046] This invention models the model weight switching process of the interactive multi-model filtering algorithm as a Markov decision process and introduces a reinforcement learning mechanism to dynamically optimize and adjust the weights of each model. This enables the filtering system to adaptively allocate model weights based on the real-time observation residuals and the degree of model matching, thereby overcoming the model switching lag and error accumulation problems caused by the fixed transition probability matrix in traditional interactive multi-model algorithms. This achieves rapid and stable switching between multiple models and high-precision state estimation. Therefore, this invention significantly reduces instantaneous tracking errors during the sudden maneuvering phase of highly maneuverable targets, improves the real-time performance, stability, and environmental adaptability of the filtering system, and ensures continuous and reliable target tracking performance even under complex interference conditions. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0048] Figure 1 This is a schematic diagram of the measurement information of a maneuvering target in the background art of this application;

[0049] Figure 2 This is a flowchart illustrating the process of updating model weights using reinforcement learning-assisted interactive multi-model filtering in this application.

[0050] Figure 3 This is a diagram showing the actual trajectory of the moving target in this application;

[0051] Figure 4 This is a graph showing the relationship between the loss function and the number of iterations during reinforcement learning training in this application.

[0052] Figure 5 A comparison chart of the errors of three filtering models and traditional interactive multi-model filtering;

[0053] Figure 6 This is a comparison chart of traditional interactive multi-model filtering and four types of reinforcement learning-driven interactive multi-model adaptive filtering. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0055] Throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The described embodiments are only a part of the embodiments of this application, not all of them. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0056] Example 1

[0057] Please see Figure 1 As shown, this embodiment provides a reinforcement learning-driven interactive multi-model aircraft filtering algorithm for state estimation and trajectory tracking of highly maneuverable targets (such as hypersonic vehicles). Based on the traditional interactive multi-model (IMM) filtering framework, this algorithm introduces a reinforcement learning decision-making mechanism to achieve dynamic adaptive adjustment of model weights, including the following steps:

[0058] S1: Establish a three-dimensional measurement model and state model set for the maneuvering target, which are used in the observation update stage and state prediction stage of the filtering algorithm, respectively;

[0059] Specifically, the three-dimensional measurement model is used in the observation update phase to map the azimuth, elevation, and relative distance measured by radar into the target's position and state in the inertial coordinate system, thereby correcting the predicted state. By introducing a state model set containing multiple maneuvering models and corresponding noise features, joint estimation and dynamic fusion of multiple motion modes are achieved, effectively addressing the problems of uncertainty in the maneuvering mode and frequent state switching of maneuvering targets. These two elements work together to form the prediction and update structure of the filtering algorithm, providing a unified mathematical modeling foundation for subsequent interactive multi-model filtering and reinforcement learning adaptive optimization. Among them:

[0060] Observation Model Construction: When radar or photoelectric sensor observations are present, a three-dimensional measurement model is established to achieve a nonlinear correspondence between the observed data and the target's true state. This three-dimensional model maps radar-measured azimuth, elevation, and relative distance measurements to the target's three-dimensional position in an inertial coordinate system, thus enabling the observation update of the filtering algorithm. Based on... Figure 1 Based on the geometric relationships shown, the observation equation formula is as follows:

[0061] ;

[0062] In the formula, The azimuth angle of the maneuvering target relative to the observation point. The pitch angle of the maneuvering target relative to the horizontal plane. This is the relative distance between the moving target and the observation point; , and Let be the spatial coordinates of the maneuvering target in the inertial coordinate system. For the first Time-observation vector, For the first The time-series measurement noise follows a mean of 0 and a covariance matrix of... , ~ , This is the noise covariance matrix, used to characterize the statistical correlation between angle measurement and distance measurement noise.

[0063] State model set construction: Specifically, the state model set consists of multiple sub-models with different dynamic assumptions running in parallel within the same filtering cycle. State prediction and observation update operations are performed on each sub-model, and the model weights are dynamically adjusted according to the transition probabilities between models. Each sub-model can correspond to different maneuvering modes to ensure high robustness and continuous tracking capability under complex maneuvering conditions.

[0064] To address highly maneuverable targets, this embodiment plans to combine three highly maneuverable models—the Singer model, the Current Statistical (CS) model, and the Jerk model—to construct a set of state models for this problem. The state equation for the maneuvering target motion model can be expressed as:

[0065] ;

[0066] In the formula: The motion state of the maneuvering target at the previous moment; The current motion state of the maneuvering target; Here is the state transition matrix. The noise is zero-mean Gaussian white noise. Under different maneuvering assumptions, the state transition matrix is ​​adjusted. By defining the structure and parameters of the process noise covariance matrix Q, we can model different motion characteristics of maneuvering targets. However, a single model often struggles to simultaneously account for the dynamic characteristics of maneuvering targets under various states, such as stable flight, accelerated turns, and sudden maneuvers.

[0067] Therefore, within the framework of this general state equation, this embodiment selects three representative types of maneuvering targets to form a state model set, wherein:

[0068] The Singer model is used to characterize sustained maneuvering behavior with time-dependent acceleration;

[0069] The current statistical model (CS model) is used to describe the weak maneuvering state of a maneuvering target where the acceleration tends to be stable within a short-term prediction interval;

[0070] The Jerk model is used to characterize strong maneuvering processes with significant rate of change of acceleration and continuously differentiable jerk.

[0071] The three models correspond to different dynamic assumptions. Their state transition matrices and noise feature matrices are formally independent, but they are parallelized and dynamically fused through model probabilities within the filtering framework, thus forming the basic unit of the interactive multi-model filter. Specifically, the structure and numerical description of the Singer model, the current statistical model, and the Jerk model will be explained in detail below.

[0072] The Singer model, equivalent to a correlated noise model for maneuvering targets rather than the commonly assumed white noise model, assumes that the acceleration a(t) of the maneuvering target is not a white noise model, but rather a zero-mean stochastic process with exponential autocorrelation. Its state transition matrix F and state noise covariance matrix Q in the one-dimensional case are as follows:

[0073] , ;

[0074] ;

[0075] In the formula: T is the sampling interval, which determines the state update frequency; The maneuver frequency reflects the acceleration-related time and is usually selected based on experience. Generally, values ​​of 1, 1 / 20, and 1 / 60 are taken according to the maneuverability of the target, corresponding to strong, medium, and weak maneuver values, respectively. The standard deviation of acceleration is used to describe the randomness of maneuver intensity. This represents the time decay factor of the acceleration autocorrelation term. Wherein, , , These correspond to the noise variance components of position, velocity, and acceleration, respectively. , , This describes the covariance relationship between state variables, used to reflect the stochastic coupling effect between position, velocity, and acceleration.

[0076] The Current Statistical Model (CS model) is essentially an algorithmic model of acceleration with a non-zero mean. This algorithm assumes that when a maneuvering target is moving at a certain acceleration, the acceleration value at the next moment is finite and can only be within the neighborhood of the "current" acceleration. The "current" statistical probability density of the maneuvering acceleration is described using a modified Rayleigh distribution, with the mean being the predicted value of the "current" acceleration. That is, the target's acceleration is approximately constant in the short term, but exhibits smooth changes over longer time scales, making it suitable for describing the target state during weak maneuvers or quasi-uniform maneuvers.

[0077] The state transition matrix and state noise covariance matrix of this algorithm are the same as those of the Singer model. The CS model, compared to the Singer model, has a control input matrix, and its discrete state equations are expressed as follows: Where G is the input control matrix. Let be the mean acceleration. The state transition equation can be expressed as:

[0078] ;

[0079] in: For the first The state vector at any given time; For the previous moment The state vector at any given time; This is the state transition matrix (structure consistent with the Singer model). To control the input matrix;

[0080] ;

[0081] Unlike the Singer model, the CS model explicitly introduces a "predicted acceleration term" during state propagation. , The predicted acceleration from the current state estimate can be used as a substitute. ; through matrix This prediction is applied to the state update process to reflect the inertial trend of the target during short-term steady acceleration. The first row of elements represents the integral effect of the current acceleration on the corresponding position; the second row of elements represents the linear change of velocity under the influence of acceleration; the third row of elements... This reflects the time smoothing of acceleration changes. Among them: the exponential decay term... Controlling the "memory depth" of the model The larger the value, the faster the model responds to sudden changes in acceleration.

[0082] Taking the x-axis as an example, the variance of the current acceleration can be obtained from the mean of the current acceleration:

[0083]

[0084] In the formula and The maximum and minimum accelerations are set for each axis, typically based on prior information. Used to constrain the range of acceleration changes and prevent filtering from causing problems.

[0085] The Jerk model assumes that the rate of change of acceleration of a maneuvering target follows a zero-mean, stationary, first-order time-dependent process. It introduces zero-mean colored noise to represent acceleration, building upon the Singer model. The state vector under this model is: The discretized state transition matrix and state noise covariance matrix of the model are as follows:

[0086] , ;

[0087] , ;

[0088] in, The sampling time interval, The variance of the jerk; For maneuver frequency; each item Describes the stochastic correlation between position, velocity, acceleration, and jerk. , , and These correspond to the noise variance components of different order states; the remaining terms reflect the interrelationships between states to ensure the smoothness and continuity of the model.

[0089] S2: The three maneuvering models achieve preliminary fusion of state estimation based on the traditional interactive multi-model algorithm; at the same time, the likelihood and residual covariance of each model are calculated to provide data support for the subsequent construction of the reinforcement learning state space.

[0090] To achieve adaptive tracking of various maneuver states, this embodiment employs an Interacting Multiple Model (IMM) filtering algorithm structure to perform parallel estimation and dynamic fusion of the target state based on the state model set composed of the aforementioned Singer, CS, and Jerk models. By introducing probability transfer and likelihood evaluation mechanisms between different models, model interaction, filtering prediction, and weighted fusion are completed at each time step, thereby achieving robust tracking of the target's complex maneuver states.

[0091] Since the maneuver characteristics described by each state model are different, the target state at any given time may switch between different models. The interaction relationship between the models is defined according to the preset Markov model transition logic, including the following steps:

[0092] Define the model transition probability matrix: Construct a fixed Markov probability transition matrix P, where the matrix elements are... The prior probability that model i will transition to model j in the next time step:

[0093] ;

[0094] Where N represents the number of models (3 in this embodiment). This is represented as the target being in the model at the previous time step. When in a state, transition to the model at the current time. The probability satisfies: ;

[0095] This transition matrix reflects the prior relationships between models and is used to initialize the model interaction weights in each filtering cycle. Before performing filtering prediction, the state estimates of each model at the previous time step need to be weighted and fused to generate the time step prediction. The initial conditions for filtering,

[0096] In the interactive input section, the output values ​​of each model from the previous time step are used to obtain new filtered state values ​​and covariance values ​​based on the interaction of Markov matrices. The specific calculation is as follows:

[0097] ;

[0098] In the formula For the model Transfer to model The posterior probability of the transition; For mobile targets in The probability of model i at time i. Assume that at... At any given time, each model has an optimal estimate. and estimate covariance Based on the correctness probability of the IMM filter, the estimated states and covariance matrices of each model can be weighted and fused to obtain the fused maneuvering target. optimal state at time 1 and covariance matrix :

[0099] ;

[0100] ;

[0101] ;

[0102] At the same time, it can be obtained Model probability at time: In the model probability update part, the previously obtained initial state values ​​are filtered and calculated to obtain the residuals. and residual covariance :

[0103] ;

[0104] ;

[0105] Calculated Time model Maximum likelihood function Assuming it follows a normal distribution, the posterior probability of model j can be obtained as:

[0106] ;

[0107] in Let be the normalization constant, expressed as ;

[0108] In the preliminary fusion section of state estimation, based on the model probabilistic fusion of the filtered predictions from each model, the maneuvering target state and covariance matrix output at time k are obtained as follows:

[0109] ;

[0110] ;

[0111] Finally, we construct the state space at this moment by combining the data obtained in the previous steps. The state space includes the model's weights at the current moment, the model's likelihood function at the current moment, and the residual after fusing the multiple model states at this moment, i.e. .

[0112] S3: Reinforcement learning-driven adaptive filtering adjustment mechanism. In online reinforcement learning algorithms, whether on-policy or off-policy, they share a common feature: the agent can continuously interact with the environment during training, obtaining new feedback data. The main difference lies in that online policy algorithms directly use this feedback data, while offline policy algorithms first store the data in an experience replay pool and then sample it when needed. The mobile objective of offline reinforcement learning is to obtain a better policy from a pre-collected, defined dataset without the agent interacting with the environment.

[0113] In the offline phase, the neural network's weight update strategy is fully trained using historical data, avoiding the high computational overhead and potential instability of online learning, and ensuring that the model learns the optimal weight allocation pattern in complex environments. In the online inference phase, the pre-trained model is directly utilized to quickly respond to dynamic changes and adjust the weights of the multi-model filtering in real time. This not only meets the requirements for low-latency interaction but also adapts to unknown scenarios through the generalization ability of the pre-trained model. This architecture separates the computational burden of training and deployment while ensuring the system's robustness and efficiency in dynamic environments.

[0114] We currently employ two modeling methods to construct reinforcement learning datasets for training.

[0115] The first modeling approach involves dynamically optimizing the weights by adjusting each model individually, i.e., multi-agent modeling. We define the state space as containing the weight information of the model at the current time step and the likelihood function of the model. And the residual of the current fused maneuvering target state, i.e. ,in It represents the weights of model i at that moment; the model likelihood function. It contains the residuals and residual covariance matrix of the predicted state and the true state at that moment. Therefore, it can well reflect the degree of matching between model i and the current motion state. At the same time, it is also an inherent intermediate calculation result of IMM, which can be reused as the input of reinforcement learning without additional computing resources, realizing lightweight integration, avoiding the introduction of redundant modules, and maintaining the efficiency of the system. The residual of the fused maneuvering target state reflects the accuracy of the fused state.

[0116] In this scenario, the action in reinforcement learning is to adjust the model's weights. Under this modeling approach, the action... This can be defined as the update amount of each model weight. We set the maximum magnitude of the action chosen by each agent, i.e., the weight adjustment of a single model, to 0.3. Therefore, the action space can be set as follows: The purpose of this action is to adjust the model weights to obtain a more accurate state estimate.

[0117] An effective reward function needs to comprehensively consider the reduction of total error, the matching of model weights and error, and the stability of actions. We divide the reward term into three parts. The first part is to encourage the agent to reduce the total error of interactive multi-model filtering. That is:

[0118] ;

[0119] in, It is the total error before the agent takes an action, i.e., the error after the initial fusion of state estimates. This is the total error after the agent takes an action, i.e., the error after the final fusion of state estimates. This reward is given when the total error decreases. The second part is to incentivize the agent to adjust weights based on model error. Models with smaller errors should have their weights increased, while those with larger errors should have their weights decreased. That is:

[0120] ;

[0121] in, This is the filtered error of model i. The total error as the reward serves as a baseline. When the error of model i is smaller than the total error and the action is positive (weight increases), the reward increases; when the error of model i is larger than the total error and the action is negative (weight decreases), the reward increases. The third part is to prevent excessive weight adjustments and maintain stability. That is:

[0122] ;

[0123] Finally, by integrating the above, we obtain the final reward function:

[0124] ;

[0125] in, , , These are the weighting parameters for each reward.

[0126] Another modeling approach involves dynamically adjusting the weights of all models as a whole, treating them as a single agent, i.e., a single-agent model. In this model, the state space consists of the weights of each model at the current time step, the model likelihood function of each model at the current time step, and the residuals after fusing the states of all models at that time step. .

[0127] In this model, we define the agent's chosen action as adjusting the weights of all models. Similarly, the maximum adjustment amount for a single model's weight is 0.3. Discretizing this value yields the action space as follows:

[0128] The reward function of this model is constructed with reference to the reward function of the multi-agent model. We can obtain the reward function of this model as follows:

[0129] ;

[0130] After constructing the dataset, we need to select a reinforcement learning algorithm for training. We selected the DQN and D3QN algorithms for training, as follows: Figure 4The diagram illustrates the training processes of four reinforcement learning frameworks in this invention: Multi-Agent Model (DQN), Multi-Agent Model (D3QN), Single-Agent Model (DQN), and Single-Agent Model (D3QN). The vertical axis represents "MSE Loss" (Mean Squared Error Loss). In machine learning, Loss can be understood as the "degree of error" or the "gap between the model and the correct answer." A higher Loss indicates that the model is "more wrong." The horizontal axis represents "Epoch" (number of iterations). This represents the "number of learning rounds." As the number of iterations increases, the loss function gradually approaches 0, indicating that these four network frameworks gradually stabilize. Internally, this verifies that the reinforcement learning frameworks proposed in this invention (whether DQN or D3QN) are effective and trainable, and they have successfully learned how to optimize the weight allocation strategy. Figure 4 Successful convergence is a prerequisite for the algorithm of this invention to achieve excellent filtering results.

[0131] Deep Q-Network (DQN) is a reinforcement learning algorithm that combines deep learning and Q-learning. Compared to traditional Q-learning, DQN improves the framework by introducing an experience replay mechanism and a double-delay deep deterministic policy gradient (TD3), effectively solving the discrete action selection problem in high-dimensional state spaces. During training, an ε-greedy exploration policy is used to balance experience accumulation and utilization, priority experience replay (PER) is used to improve sample efficiency, and gradient clipping techniques are used to control Q-value overestimation.

[0132] D3QN is an efficient combination of Double DQN and Dueling DQN, simultaneously solving the problems of Q-value overestimation and state-action value decomposition. Double DQN improves upon the Q-value overestimation problem of traditional DQN. In DQN, the calculation of the maneuvering target's Q-value involves the maneuvering target network selecting actions and evaluating Q-values. However, this process can lead to an overestimation of the optimal Q-value. Double DQN decouples action selection and Q-value evaluation: the main network selects the optimal action (…). The maneuvering target network only evaluates the Q-value of the action. By separating the two, the bias caused by the same network being responsible for both selection and evaluation is reduced, thus improving the accuracy of Q-value estimation. The formula is expressed as:

[0133] ;

[0134] Dueling DQN decomposes the Q-value into a state value function by reconstructing the neural network structure. ) and action advantage function This allows for a better distinction between the inherent value of a state and the relative advantages derived from an action. Its core formula is:

[0135] ;

[0136] in, Representing state long-term value Measuring actions The mean term, subtracted from the average advantage, is used to enhance the discriminativeness of the advantage function, preventing the network from relying solely on advantage function updates. This decomposition makes the algorithm more stable in scenarios dominated by state value, while retaining sensitivity to differences in key actions, significantly improving policy evaluation efficiency.

[0137] The final fusion of state estimates is achieved by calling the training framework:

[0138] For each time k, the reinforcement learning agent is based on the current state. Choose an action (That is, adjusting the model weights) and obtaining rewards through environmental feedback. This is used to optimize the strategy. The reinforcement learning strategy is represented as... ,in It is the policy function, which takes the current state as input. Output action This refers to the adjustment amount of the model weights. The weights of each model... The update method is .

[0139] After the weights are updated, in order to ensure that the sum of the weights of each model is still 1, they need to be normalized, i.e. Obtain the final weights of each model after adjustment at time k.

[0140] Example 2

[0141] Based on Example 1, to verify the effectiveness of the method of the present invention, a tracking experiment targeting a maneuvering target was conducted. In the experiment, the initial altitude of the hypersonic vehicle was set to 70 km, the initial speed to 5500 m / s, and the initial velocity tilt angle and velocity deflection angle to 0°. A highly realistic simulation environment was constructed, such as... Figure 3 The image shows the actual trajectory of the maneuvering target in this simulation experiment. This trajectory is a standard trajectory of a hypersonic vehicle generated by solving complex dynamic equations using the three-dimensional aerodynamic modeling unit described in this application.

[0142] Specifically, the realistic modeling of the maneuvering target adopts a three-degree-of-freedom hypersonic vehicle model, and its equations of motion are as follows:

[0143] ;

[0144] in: The yaw angle is the speed of the aircraft. The pitch angle is the speed of the aircraft. The speed of the aircraft; , and These are the position coordinates of the aircraft in the inertial coordinate system; The tilt angle is the angle of inclination for flight speed; For lift; As resistance; It is a lateral force; For the mass of the aircraft; This is the acceleration due to gravity.

[0145] Expressions for lift, drag, and lateral force:

[0146] ;

[0147] In the formula: It represents the dynamic pressure of an aircraft, and the kinetic pressure of a fluid per unit area; , , These are the aerodynamic coefficients (drag coefficient, lift coefficient, and side force coefficient), which are determined by aerodynamic characteristics; denoted by , where represents atmospheric density, and S represents the aerodynamic reference area.

[0148] The above dynamic equations can describe the evolution of the velocity, attitude and spatial position of a maneuvering target over time under the combined action of aerodynamic force and gravity, and realize continuous state propagation modeling. The dynamic equations can be used to generate three-dimensional simulation trajectory samples, which represent the real values ​​used for comparison in this experiment.

[0149] During training, we used the DQN and D3QN algorithms to train the multi-agent model and the single-agent model, respectively. The algorithm had 500 epochs. In the multi-agent model, the reward function values ​​for α, β, and γ were 1, 0.5, and 0.1, respectively; in the single-agent model, the reward function values ​​for α, β, and γ were 3, 0.5, and 0.1, respectively. Experimental results are as follows: Figure 5-6 As shown, where: Figure 5This chart compares the errors of three model filtering methods with those of the traditional Interactive Multi-Model (IMM) filtering method. It compares the filtering performance of the widely recognized "traditional Interactive Multi-Model (IMM) algorithm" (red line) with its three constituent single models (Singer, CS, and Jerk). The tracking error of the traditional IMM algorithm (red line) is significantly lower than that of any single model in most cases. More importantly, when a single model (such as the blue and yellow lines in rounds 9 and 13) experiences performance degradation and a sharp increase in error due to target maneuvering, the IMM algorithm (red line) still maintains excellent stability and low error. Therefore, the IMM algorithm, based on the traditional IMM algorithm, possesses a very powerful and stable technical benchmark. The experimental results show that the IMM algorithm, through model switching during tracking, effectively reduces tracking error and achieves more stable tracking performance compared to the other three types of single-model filtering.

[0150] Based on this, such as Figure 6 As shown, this invention employs a reinforcement learning-assisted interactive multi-model weight update algorithm, reducing the latency impact of the error transition matrix and further reducing tracking error based on the interactive multi-model algorithm. In terms of reinforcement learning algorithms, D3QN performs better than the traditional DQN; in terms of reinforcement learning modeling, single-agent modeling provides more state space information and can better describe the current state, thus its performance is better than multi-agent modeling.

[0151] Example 2

[0152] The parts not detailed in this embodiment are as described in Embodiment 1. This embodiment provides a reinforcement learning-driven interactive multi-model aircraft filtering system, including:

[0153] The three-dimensional aerodynamic modeling unit is used to establish the target dynamic equations in the inertial coordinate system, where lift, drag and side force are jointly determined by aerodynamic coefficients, air density, flight speed and reference area. The spatial attitude evolution of the target under the action of aerodynamic force and gravity is described by the coupling relationship between velocity, pitch angle and tilt angle. It is used to construct the real motion trajectory of the target and provide a simulation environment for highly maneuverable targets for the algorithm.

[0154] The multi-model state prediction unit constructs a prediction set in parallel, consisting of the Singer model, the current statistical model, and the Jerk model. Each model independently generates state predictions and a state transition probability matrix is ​​preset based on the correlation between state evolutions of the models to define the interaction relationships between the models.

[0155] The multi-model interaction fusion unit is bidirectionally connected to the prediction unit. The three maneuvering models achieve preliminary fusion of state estimation based on the traditional interactive multi-model algorithm. At the same time, the likelihood and residual covariance of each model are calculated to provide data support for the subsequent construction of the reinforcement learning state space.

[0156] The reinforcement learning weight adaptive unit is bidirectionally connected to the fusion unit. Based on the model likelihood, residual change rate and historical weight distribution input, it outputs a weight correction vector through the policy function and performs self-optimization through a reward function containing error improvement, model matching and smoothing constraints, thereby realizing a self-learning closed loop of dynamics layer, fusion layer and policy layer.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An interactive multi-model aircraft filtering algorithm driven by reinforcement learning, characterized in that, The method comprises the following steps: S1, constructing a multi-model state prediction set composed of three maneuver models, the maneuver models including a Singer model, a current statistical model or a Jerk model; each model independently generates a state prediction, and a model probability transition matrix is preset according to the state evolution correlation between the models to define the interaction between the models; and a measurement model composed of a bearing angle, a pitch angle and a relative distance is constructed; S2, in the filtering stage, each maneuver model in the multi-model state prediction set realizes preliminary fusion of state estimation according to a traditional interacting multiple model algorithm; meanwhile, the likelihood of each maneuver model and the residual covariance are calculated to construct a reinforcement learning state space; S3, in the weight updating process, a reinforcement learning agent is introduced, the model likelihood, the residual change rate and the historical weight distribution are taken as policy inputs, a weight correction vector is output through a policy function, and the fusion proportion of each model is adjusted in real time; the reinforcement learning process is optimized through policy iteration of a reward function containing an error improvement term, a model matching term and a weight smoothing term, so that the self-learning and convergence performance of the filtering process are improved.

2. The interactive multi-model aircraft filtering algorithm driven by reinforcement learning according to claim 1, wherein, The interaction between the corresponding models in the multi-model state prediction set is defined by a preset state transition probability matrix as follows: The state transition probability matrix is preset based on the state evolution correlation between the models; A mixed probability is calculated according to the state transition probability matrix and the model posterior probability at the last time; The model state estimation and covariance at the last time are weighted according to the mixed probability to obtain the mixed initial state and covariance at the current filtering period.

3. The reinforcement learning driven interactive multiple model aircraft filtering algorithm of claim 1, wherein, The adaptive adjustment process comprises: The reinforcement learning agent adjusts the fusion weights of the Singer model, the current statistical model and the Jerk model in real time at each filtering period; When the weight adjustment reduces the overall filtering error or improves the model matching degree, the error improvement term and the model matching term in the reward function produce positive rewards.

4. The reinforcement learning driven interactive multiple model aircraft filtering algorithm of claim 1, wherein, The weight change amplitude is constrained by the weight smoothing term in the reward function, including: The weight smoothing term is set as a negative reward for the amplitude of the weight adjustment; When the weight adjustment amplitude is too large, a penalty is generated; through this constraint, the agent performs smooth weight updating to ensure the convergence and stability of the filtering.

5. The reinforcement learning driven interactive multiple model aircraft filtering algorithm of claim 1, wherein, The weight correction is a continuous decision-making process of the reinforcement learning agent at each filtering period, comprising the following steps: A continuous state space containing the current residual, the model weight and the model likelihood is constructed; The agent outputs a weight adjustment action through a policy network according to the state space to generate the fusion weight at the current period.

6. The reinforcement learning driven interactive multiple model aircraft filtering algorithm of claim 1, wherein, Instead of using explicit weight freezing control logic, stability is achieved implicitly through the reward function, including: A penalty term for large amplitude adjustment of the weight is set in the reward function; When the error change is small, the agent naturally produces a small adjustment amount; When the target maneuver changes increase, the constraint is automatically released to restore the dynamic adjustment ability of the weight.

7. The reinforcement learning driven interactive multiple model aircraft filtering algorithm of claim 1, wherein, The process of outputting the weight correction vector by the policy function comprises: The model weight, the model likelihood and the fusion residual are taken as the policy input set; A deep neural network is used to map the input to the weight adjustment amount; The weight is updated according to the adjustment amount, and the normalized processing is performed on the updated weight vector, so that the weight sum is kept as 1.

8. The reinforcement learning driven interactive multiple model aircraft filtering algorithm of claim 1, wherein, The iterative update of the policy function is completed in an offline training phase, comprising: An offline training-online inference architecture is adopted; An experience replay pool is used to store state, action and reward data; The policy network parameters are updated according to the time difference error.

9. The reinforcement learning driven interactive multiple model aircraft filtering algorithm of claim 1, wherein, The reward function is used for policy iteration optimization, and the reward function is composed of three weighted factors: Filter error improvement factor: defined as the total filter error reduction before and after the agent takes an action, this directly encourages the agent to learn a weight adjustment strategy that can reduce the final fusion error; Model matching factor: when the error of a specific model is less than the total error benchmark and its weight increases, or its error is greater than the total error benchmark and its weight decreases, a positive reward is given, which encourages the agent to assign higher weights to models that have higher matching degrees for the current target motion state; Weight smoothing factor: defined as the negative value of the weight adjustment vector amplitude, which punishes drastic weight changes to ensure the stability and convergence of the filtering process.

10. A reinforcement learning driven interactive multiple model aircraft filtering system for implementing the reinforcement learning driven interactive multiple model aircraft filtering algorithm of any one of claims 1-9, characterized in that, Comprise: A three-dimensional aerodynamic modeling unit is used to establish a target dynamics equation in an inertial coordinate system, wherein: the lift, drag and side force are determined by the aerodynamic coefficients, air density, flight speed and reference area, the spatial attitude evolution of the target under the action of aerodynamic force and gravity is described through the coupling relationship of speed, pitch angle and tilt angle, and the target real motion trajectory is constructed to provide a high-maneuvering target simulation environment for the algorithm. A multi-model state prediction unit is used to construct a prediction set composed of a Singer model, a current statistical model and a Jerk model in parallel, each model independently generates state prediction, and a state transition probability matrix is formed according to the correlation between the models to realize preliminary update of the model probability of each model; A multi-model interactive fusion unit is bidirectionally connected with the prediction unit, and the three maneuvering models realize preliminary fusion of state estimation according to the traditional interactive multiple model algorithm; meanwhile, the likelihood and residual covariance of each model are calculated to provide data support for subsequent construction of a reinforcement learning state space; A reinforcement learning weight adaptive unit is bidirectionally connected with the fusion unit, based on the model likelihood, residual change rate and historical weight distribution input, a weight correction vector is output by the policy function and self-optimized through a reward function containing error improvement, model matching and smoothing constraints, realizing the self-learning closed loop of the dynamics layer, the fusion layer and the policy layer.

Images