Reinforcement learning based radar anti-jamming decision method based on recursive game training

Abstract

The application discloses a radar anti-jamming decision method based on recursive game training reinforcement learning and belongs to the technical field of radar anti-jamming. The application aims at the problems of insufficient self-learning ability and poor dynamic adaptability of the existing radar anti-jamming decision method. The radar decision system is established as a radar agent, the interference decision system is established as an interference agent, and a stahlkberg game model is formed. The radar agent and the interference agent of the stahlkberg game model respectively adopt a on-policy optimization algorithm to perform recursive game training reinforcement learning, and finally reach a bayesian nash equilibrium state, so that the radar agent has the anti-jamming strategy generation ability for coping with dynamic change interference, and radar anti-jamming is realized. The application is used for radar anti-jamming decision.

Description

Technical Field

[0001] This invention relates to a radar anti-jamming decision-making method based on recursive game training reinforcement learning, belonging to the field of radar anti-jamming technology. Background Technology

[0002] With the continuous development of jamming technology, jamming methods with cognitive capabilities are gradually being applied. In the confrontation between radar and jamming, the jammer no longer generates jamming strategies based on random or fixed patterns. Instead, through intelligent decision-making systems, it adjusts its jamming methods in real time according to the radar's transmitted waveform, producing complex and unpredictable jamming effects. At this point, the confrontation between radar and jamming has evolved into a non-cooperative, time-series game process based on incomplete information.

[0003] Traditional radar anti-jamming decision-making methods largely rely on manual adjustments or preset rules, which exhibit drawbacks in intelligent jamming scenarios, including slow response speed, poor dynamic adaptability, insufficient self-learning ability, and limited anti-jamming benefits. Therefore, there is an urgent need to develop intelligent radar anti-jamming decision-making methods to address issues such as information asymmetry and the "dimensional explosion" of strategies. Summary of the Invention

[0004] To address the shortcomings of existing radar anti-jamming decision-making methods in terms of insufficient self-learning ability and poor dynamic adaptability, this invention provides a radar anti-jamming decision-making method based on recursive game training reinforcement learning.

[0005] This invention provides a radar anti-jamming decision-making method based on recursive game-theoretic training reinforcement learning, comprising:

[0006] The radar decision-making system is established as the radar agent, and the jamming decision-making system is established as the jamming agent. They are then combined into a Steinkelberg game model.

[0007] The radar agent and the jamming agent in the Stankelberg game model are trained and reinforced by recursive game training using the proximal policy optimization algorithm, and finally reach a Bayesian Nash equilibrium. This enables the radar agent to generate anti-jamming strategies to cope with dynamically changing interference, thus achieving radar anti-jamming.

[0008] According to the radar anti-jamming decision-making method based on recursive game training reinforcement learning of the present invention, during the recursive game training reinforcement learning process:

[0009] The radar agent senses the current interference state of the interfering agent, combines historical information of the interference state, optimizes the anti-interference strategy using the near-end policy optimization algorithm, generates an anti-interference decision, and finally generates a radar transmission signal.

[0010] Meanwhile, the jamming agent senses the current radar state of the radar agent, combines the historical information of the radar state, uses the near-end policy optimization algorithm to optimize the jamming strategy, generates jamming decisions, and finally generates jamming signals.

[0011] The radar agent and the jamming agent continuously adjust their anti-jamming decisions and jamming strategies, and through multiple rounds of game and strategy optimization, they eventually reach a Bayesian Nash equilibrium.

[0012] According to the radar anti-jamming decision-making method based on recursive game training reinforcement learning of the present invention, the recursive game training reinforcement learning process includes the following three stages:

[0013] Initial phase: The radar agent and the jamming agent generate the initial anti-jamming strategy and the initial jamming strategy, respectively;

[0014] Recursive Phase: In each round of the game, the radar agent and the jamming agent learn each other's historical strategies, predict each other's future decisions, and optimize their own strategies.

[0015] Game convergence phase: In the recursive phase, the radar agent and the jamming agent continuously optimize their own strategies and neural network parameters until the strategies of both sides gradually converge and reach a Bayesian Nash equilibrium.

[0016] According to the radar anti-jamming decision-making method based on recursive game training reinforcement learning of the present invention, the recursive stage specifically includes:

[0017] In the first round of the game, based on the initial interference strategy of the interference agent, the radar agent uses the near-end policy optimization algorithm to optimize the anti-interference strategy. After multiple cycles of interactive learning and training, the radar agent continuously optimizes and adjusts the anti-interference strategy based on the collected historical information of interference status and environmental feedback information, and generates anti-interference decisions. Then, the radar agent's current anti-interference strategy and neural network parameters are fixed to play against the interference agent.

[0018] Then, based on the radar agent's current anti-jamming strategy, the jamming agent optimizes the initial jamming strategy using a near-end strategy optimization algorithm. After multiple cycles of interactive learning and training, the jamming agent continuously optimizes and adjusts the jamming strategy based on the collected historical radar state information and environmental feedback information, generating jamming decisions. Then, the jamming agent's current jamming strategy and neural network parameters are fixed to play against the radar agent in the next round of the game.

[0019] According to the radar anti-jamming decision-making method based on recursive game training reinforcement learning of the present invention, the near-end policy optimization algorithm is improved based on the policy gradient algorithm, adopts the trust region policy optimization algorithm, introduces KL divergence constraints, and restricts the difference between the old and new policies. The policy is updated by calculating the objective function.

[0020] According to the radar anti-jamming decision-making method based on recursive game training reinforcement learning of the present invention, the decision-making process of each radar agent perceiving the jamming agent is described by Markov decision process.

[0021] According to the radar anti-jamming decision-making method based on recursive game training reinforcement learning of the present invention, the radar agent selects the optimal anti-jamming decision from the anti-jamming knowledge base according to the anti-jamming strategy.

[0022] According to the radar anti-jamming decision-making method based on recursive game training reinforcement learning of the present invention, both the initial anti-jamming strategy and the initial jamming strategy are randomly generated based on a normal distribution.

[0023] According to the radar anti-jamming decision-making method based on recursive game training reinforcement learning of the present invention, the environmental feedback information corresponding to the radar agent is the anti-jamming benefit value, and the environmental feedback information corresponding to the jamming agent is the jamming benefit value.

[0024] The interference gain value is the negative of the anti-interference gain value; the anti-interference gain value is equal to the signal-to-interference-to-noise ratio.

[0025] The beneficial effects of the present invention are as follows: The method of the present invention solves the problems of information asymmetry and strategy "dimensional explosion" in the existing methods by using intelligent radar anti-interference decision-making means, and can realize the radar to generate anti-interference strategy targets efficiently and accurately in the Nash equilibrium state of the game.

[0026] The method of this invention is based on recursive game training reinforcement learning to realize radar anti-interference decision-making. When faced with intelligent interference scenarios, the trained radar agent can quickly and accurately generate anti-interference strategies according to decision criteria, select the optimal interference suppression means from the anti-interference knowledge base, and achieve a game Nash equilibrium between the interference and the radar, thus realizing the radar's successful anti-interference. Attached Figure Description

[0027] Figure 1 This is a Steinkelberg game model diagram of the radar anti-jamming decision-making method based on recursive game training reinforcement learning described in this invention.

[0028] Figure 2 This is a schematic diagram of the game tree based on the two-agent Stankelberg game model;

[0029] Figure 3 This is a convergence graph of the average reward function value of the PPO algorithm, A3C algorithm, and PG algorithm of this invention in the reinforcement learning stage of recursive game training, which verifies Experiment 1.

[0030] Figure 4This is a convergence graph of the average reward function values ​​of the PPO algorithm, A3C algorithm, and PG algorithm of this invention during the testing phase in Experiment 1;

[0031] Figure 5 This is a graph showing the performance test results of the PPO algorithm, A3C algorithm, and PG algorithm of this invention in the third round of the game against the radar agent in Experiment 2.

[0032] Figure 6 This is a graph showing the performance test results of the PPO algorithm, A3C algorithm, and PG algorithm of this invention in the third round of the game, which verifies the interference agent.

[0033] Figure 7 This is a comparison chart of the performance test results of the PPO algorithm, A3C algorithm, and PG algorithm of this invention in the third round of the game in Experiment 2.

[0034] Figure 8 This is a schematic diagram illustrating the opposite of the maximum radar reward and the maximum interference reward in the four-round game in Experiment 2.

[0035] Figure 9 This is a schematic diagram illustrating the difference between the opposite values ​​of the maximum radar reward value and the maximum interference reward value in Experiment 2. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0038] The present invention will be further described below with reference to the accompanying drawings, but this should not be construed as limiting the invention.

[0039] Combination Figure 1 and Figure 2 As shown, this invention provides a radar anti-jamming decision-making method based on recursive game training reinforcement learning, including:

[0040] The radar decision-making system is established as the radar agent, and the jamming decision-making system is established as the jamming agent. They are then combined into a Steinkelberg game model.

[0041] The radar agent and the jamming agent in the Stankelberg game model are trained and reinforced by recursive game training using the proximal policy optimization algorithm, and finally reach a Bayesian Nash equilibrium. This enables the radar agent to generate anti-jamming strategies to cope with dynamically changing interference, thus achieving radar anti-jamming.

[0042] This implementation method, based on game theory and reinforcement learning theory, establishes a dual-agent Steinkelberg game model of radar and jamming, designs an anti-jamming strategy generation algorithm based on recursive game training reinforcement learning, solves the Bayesian Nash equilibrium strategy of both parties, thereby enabling the radar to autonomously learn and optimize its strategy in intelligent jamming scenarios, achieving the goal of fast and accurate anti-jamming.

[0043] The confrontation between radar and jammers is characterized by dynamism, incomplete information, and non-cooperation. The radar typically acts as the leader, while the jammer adjusts its jamming behavior according to the radar's strategy. To effectively cope with intelligent jamming environments, this implementation method constructs a two-agent Stackelberg game model based on deep reinforcement learning and Stackelberg game theory, such as... Figure 1 As shown.

[0044] In this implementation method, during the game process of training reinforcement learning through recursive game theory:

[0045] The radar agent senses the current interference state of the interfering agent, combines historical information of the interference state, optimizes the anti-interference strategy using the near-end policy optimization algorithm, generates an anti-interference decision, and finally generates a radar transmission signal.

[0046] Meanwhile, the jamming agent senses the current radar state of the radar agent, combines the historical information of the radar state, uses the near-end policy optimization algorithm to optimize the jamming strategy, generates jamming decisions, and finally generates jamming signals.

[0047] The radar agent and the jamming agent continuously adjust their anti-jamming decisions and jamming strategies, and through multiple rounds of game and strategy optimization, they eventually reach a Bayesian Nash equilibrium.

[0048] Furthermore, the recursive game training reinforcement learning process includes the following three stages:

[0049] Initial Phase: In the initial phase of the game, the radar agent and the jamming agent generate initial anti-jamming strategies and initial jamming strategies, respectively.

[0050] Recursive Phase: In each round of the game, the radar agent and the jamming agent learn each other's historical strategies, predict each other's future decisions, and optimize their own strategies.

[0051] Game convergence phase: In the recursive phase, the radar agent and the jamming agent continuously optimize their own strategies and neural network parameters until the strategies of both sides gradually converge and reach a Bayesian Nash equilibrium.

[0052] The recursive phase specifically includes:

[0053] First, in the first round of the game, based on the initial interference strategy of the interference agent, the radar agent uses a near-end policy optimization algorithm to optimize the anti-interference strategy. After multiple cycles of interactive learning and training, the radar agent continuously optimizes and adjusts the anti-interference strategy based on the collected historical information of interference status and environmental feedback information, and generates the optimal anti-interference decision; it attempts to avoid being interfered with in the next round of the game; then, the radar agent's current anti-interference strategy and neural network parameters are fixed and used to play against the interference agent.

[0054] Then, based on the radar agent's current anti-jamming strategy, the jamming agent optimizes the initial jamming strategy using a near-end policy optimization algorithm. After multiple cycles of interactive learning and training, the jamming agent continuously optimizes and adjusts its jamming strategy based on collected historical radar state information and environmental feedback information, generating the optimal jamming decision. It then attempts to jam the radar in the next round of the game to prevent it from detecting the real target. Finally, the jamming agent's current jamming strategy and neural network parameters are fixed to engage in a game with the radar agent in the next round.

[0055] In this embodiment, the environmental feedback information corresponding to the radar agent is the anti-interference gain value, and the environmental feedback information corresponding to the interference agent is the interference gain value.

[0056] The interference gain value is the negative of the anti-interference gain value; the anti-interference gain value is equal to the signal-to-interference-to-noise ratio.

[0057] Game convergence phase: The above process is recursive, with the agents continuously optimizing their own strategies and neural network parameters until both sides' strategies gradually converge, reaching a Bayesian Nash equilibrium. At this point, both the interfering agent and the radar agent have optimized to the optimal strategy in the current adversarial environment. That is, without changing the opponent's strategy, neither side can gain an additional advantage by unilaterally adjusting its own strategy.

[0058] Furthermore, the near-end policy optimization algorithm is an improvement on the policy gradient algorithm. It adopts a trust region policy optimization algorithm, introduces KL divergence constraints, and limits the differences between the old and new policies. The policy is updated by calculating the objective function.

[0059] The principle of the Proximal Policy Optimization (PPO) algorithm.

[0060] Methods for solving reinforcement learning problems are mainly divided into two categories: value-based methods and policy gradient-based methods. The Deep Q-Network (DQN) algorithm is a typical value-based method that estimates the state-action value function from samples to find the optimal policy. However, its use of neural networks to parameterize the state-action value function can lead to instability in the learning process. Value-based methods, on the other hand, effectively avoid this problem and can easily handle high-dimensional and continuous action spaces. The PPO algorithm, such as Trust Region Policy Optimization (TRPO), is a typical value-based method designed to address the high complexity and difficulty in debugging traditional policy gradient methods during training. The following section will derive the principles of the PPO algorithm in detail.

[0061] The PPO algorithm is an improvement on the Policy Gradient (PG) algorithm. The goal of the PG algorithm is to maximize a policy π(a) t |s t ) expected cumulative reward. This indicates that the agent starts from state s0 and adopts policy π. θ (a t |s t After that, the weighted expectation of the probability of a certain state s occurring is given, where θ is a parameter of the policy π. Therefore, the objective function of the PG algorithm is:

[0062]

[0063] By differentiating the objective function J(θ), the gradient direction is obtained, and the policy parameter θ is updated accordingly.

[0064]

[0065] However, the update strategy based on the above formula has two problems: first, low data utilization, with trajectories collected in a round being discarded after only one use; second, large policy update magnitudes can lead to policy instability. To address these issues, the Trust Region Policy Optimization (TRPRO) algorithm was proposed. This algorithm introduces a KL (Kullback-Leibler) divergence constraint to limit the difference between the old and new policies: maximizing the objective function J(θ) while requiring the KL divergence between the old and new policies to not exceed a certain threshold δ. The objective function of TPRO is:

[0066]

[0067] in Given a policy The dominant function, D KL (·||·) calculates the KL distance, and δ is the maximum KL distance.

[0068] Based on the state-action value function Q(s,a) and the state value function V(s), the dominance function A π (s,a) is defined as:

[0069]

[0070] The TRPO algorithm, through the new strategy π θ (a|s) and the old strategy Imposing constraints ensures policy stability, but the solution process requires the use of natural gradients and conjugate gradients, necessitating quadratic programming for each update, significantly increasing algorithm complexity and computational cost. To avoid this problem while simultaneously improving performance, the PPO algorithm improves upon TRPO by modifying the objective function and introducing a shearing operation to limit the update magnitude of the old and new policies. The objective function of the PPO algorithm is:

[0071]

[0072] in It is the importance sampling ratio. It is the estimate of the dominance function, and cl is the shearing operation.

[0073]

[0074] Where ε is a scalar between 0 and 1.

[0075] The PPO algorithm utilizes the advantage function To update the strategy, introduce a cut operation cl: if Within the range [1-ε, 1+ε], the policy update magnitude is moderate, utilizing... Update the strategy; if If the range is outside [1-ε, 1+ε], the cut ratio cl is used to limit the policy update.

[0076] The PPO algorithm is based on the old strategy. The generated trajectory is used to estimate the objective function and calculate the gradient, thereby improving data utilization. Meanwhile, for The truncation operation can help avoid significant differences between the old and new strategies, thus improving the stability of the algorithm. Furthermore, due to the presence of the "min" operation in PPO, L... CLIP (θ) is an untruncation function The lower bound, L CLIP (θ) boosting will also make This improvement ensures the stability of strategy updates.

[0077] As an example, the decision-making processes of radar agents in perceiving interference agents are described using Markov decision processes.

[0078] In this embodiment, the radar agent selects the optimal anti-jamming decision from the anti-jamming knowledge base according to the anti-jamming strategy.

[0079] In this embodiment, both the initial anti-interference strategy and the initial interference strategy are generated randomly based on a normal distribution.

[0080] In this embodiment, a Bayesian Nash equalization strategy is used for radar and jamming. Details are as follows:

[0081] Through the interaction and policy evolution of the two agents, the game eventually converges to a Bayesian Nash equilibrium. In this equilibrium, both the radar and jammer's strategies are optimal given the opponent's strategy. This means that the radar and jammer have found a mutually adaptive and stable strategy combination during the game; that is, under conditions of incomplete information, neither party can gain a greater benefit by unilaterally changing their strategy.

[0082] In the two-agent Stankelberg game model, the game process is viewed as a tree structure, such as... Figure 2 As shown. Assume the radar has three possible decisions. There are four interference options. The initial node information set of the entire game tree is represented as follows: Solid rectangles and circles represent radar nodes and jamming nodes, respectively, while dashed rectangles represent an information set. Radar and jamming agents take turns acting, with each edge representing a transition from one decision point to another. In this model, each agent chooses an action at a decision node, and each action by each agent leads to a new node. After a series of actions, the agent reaches a leaf node in the game tree, signifying the termination state. The state indicated by the symbol 'stop' signifies the end of the game and the attainment of equilibrium. At this point, the final payoffs are distributed among the agents based on the outcome of their game.

[0083] Verification Experiment 1:

[0084] To verify whether the radar can effectively cope with dynamically changing interference strategies and improve anti-interference performance in intelligent jamming scenarios, simulation experiments were conducted. The stability and convergence of the proposed method were observed in the first round of the game. The PPO algorithm of this invention was compared with the A3C and PG algorithms. The three algorithms were trained and tested in the same environment, and the convergence of the average reward function value was observed. Figure 3 and Figure 4 As shown in the figure. The red curve represents the PPO algorithm; the green curve represents the A3C algorithm; and the blue curve represents the PG algorithm.

[0085] Figure 3 The curves showing the change in average reward during the training process for the three algorithms are presented. Figure 3 The curves in the figure represent the average reward of each algorithm under different training rounds, with the radar agent's reward value ranging from [0,1]. The reward curve of the PPO algorithm rises rapidly from 0.32, reaching 0.925 after about 30,000 rounds, and then gradually stabilizes, with the average reward remaining between 0.9 and 1, demonstrating good convergence and stability. The average reward of the A3C algorithm increases rapidly from 0.33, reaching 0.885 within a few hundred rounds, and then gradually stabilizes, with the average reward remaining between 0.88 and 0.9. The average reward of the PG algorithm reaches about 0.624 from the beginning of training and maintains a large fluctuation, oscillating between 0.75 and 0.86 throughout the training process, showing large fluctuations and poor policy stability.

[0086] Figure 4 The average reward changes of the three algorithms during the testing process are shown. The PG algorithm exhibits the largest fluctuation in its reward curve, averaging between 0.75 and 0.86. The A3C algorithm performs better than the PG algorithm in terms of reward, and its reward increases faster, especially in the first 10 test periods, where the reward rises rapidly and eventually stabilizes between 0.85 and 0.9. The PPO algorithm shows the most outstanding reward performance, reaching 0.92 in the 18th test period. Its reward curve fluctuates less, and the reward value remains consistently between 0.9 and 0.96, demonstrating its optimal learning effect and more stable strategy.

[0087] In summary: the A3C algorithm has the fastest convergence speed, but its ability to find the optimal policy is insufficient, and it may stabilize at a local optimum for a short period of time. The PG algorithm performs poorly throughout the training and testing process, with a slow rate of reward increase and greater volatility. The PPO algorithm has a relatively slow convergence speed, but it can stabilize in the high reward region, demonstrating its superior policy learning ability, indicating that this method may be the most advantageous in specific tasks. Considering convergence, stability, and average optimal reward, the PPO algorithm of this invention exhibits the best performance.

[0088] Verification Experiment 2:

[0089] To verify the performance of the method of this invention in solving Bayesian Nash equilibrium strategies, the average reward curves and optimal reward values ​​of the radar agent and the interference agent in the third round of the game were observed, such as... Figures 5 to 7As shown. For either the radar or the jammer, the goal is to find the maximum reward value. The higher the reward value, the higher their score in the game. If the radar agent's reward value is less than 0.5, it means the radar's anti-jamming has failed and the jammer's jamming has succeeded; if the jammer's reward value is less than -0.5, it means the jammer's jamming has failed and the radar's anti-jamming has succeeded. Figure 5 The test results of the radar agent in the third round of the game. Figure 6 The test results for the interfering agent in the third round of the game. Figure 7 The values ​​represent the inverses of the maximum reward values ​​of the radar agent and the jamming agent during the testing of the three algorithms in the third round of the game. Overall, the PG and A3C algorithms reached Nash equilibrium, while the PPO algorithm reached an approximate Nash equilibrium. In the radar-jamming intelligent game implemented using the PG algorithm, the radar failed to resist jamming; in the radar-jamming intelligent game implemented using the A3C algorithm, the radar and the jammer were tied; only in the radar-jamming intelligent game implemented using the PPO algorithm did the radar successfully resist jamming.

[0090] To more intuitively analyze the radar gain value during the game-theoretic process, the following is given: Figure 8 The PPO algorithm is shown in the diagram, representing the performance of the PPO algorithm over four rounds of the game. The horizontal axis represents the number of rounds (from 1 to 4), and the vertical axis represents the maximum reward value of the radar.

[0091] Figure 8 This represents the maximum radar reward value obtained by the PPO algorithm over four game rounds. Both sides reach Nash equilibrium in the third round, and the radar successfully resists interference. The difference between the maximum radar reward value and the inverse of the maximum interference reward value was observed, such as... Figure 9 As shown, all three algorithms can reach Nash equilibrium. The PPO and A3C algorithms both reach Nash equilibrium after the second round of the game, with a decision efficiency improvement of approximately 33.33% compared to the PG algorithm.

[0092] The method of the present invention can also be used for various other data and scenarios. Without departing from the spirit and essence of the present invention, those skilled in the art can process different data in different scenarios according to the present invention, but these should all fall within the protection scope of the appended claims.

[0093] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A radar anti-jamming decision-making method based on recursive game training reinforcement learning, characterized in that... include, The radar decision-making system is established as the radar agent, and the jamming decision-making system is established as the jamming agent. They are then combined into a Steinkelberg game model. The radar agent and the jamming agent in the Stankelberg game model are trained and reinforced by recursive game training using the proximal policy optimization algorithm, and finally reach a Bayesian Nash equilibrium state. This enables the radar agent to generate anti-jamming strategies to cope with dynamic interference, thus achieving radar anti-jamming. In the process of training reinforcement learning through recursive game theory: The radar agent senses the current interference state of the interfering agent, combines historical information of the interference state, optimizes the anti-interference strategy using the near-end policy optimization algorithm, generates an anti-interference decision, and finally generates a radar transmission signal. Meanwhile, the jamming agent senses the current radar state of the radar agent, combines the historical information of the radar state, uses the near-end policy optimization algorithm to optimize the jamming strategy, generates jamming decisions, and finally generates jamming signals. The radar agent and the jamming agent continuously adjust their anti-jamming decisions and jamming strategies, and through multiple rounds of game and strategy optimization, they eventually reach a Bayesian Nash equilibrium. The recursive game training reinforcement learning process includes the following three stages: Initial phase: The radar agent and the jamming agent generate the initial anti-jamming strategy and the initial jamming strategy, respectively; Recursive Phase: In each round of the game, the radar agent and the jamming agent learn each other's historical strategies, predict each other's future decisions, and optimize their own strategies. Game convergence phase: The recursive phase continues to recursively, with the radar agent and the jamming agent continuously optimizing their own strategies and neural network parameters until the strategies of both sides gradually converge and reach a Bayesian Nash equilibrium. The recursive phase specifically includes: In the first round of the game, based on the initial interference strategy of the interference agent, the radar agent uses the near-end policy optimization algorithm to optimize the anti-interference strategy. After multiple cycles of interactive learning and training, the radar agent continuously optimizes and adjusts the anti-interference strategy based on the collected historical information of interference status and environmental feedback information, and generates anti-interference decisions. Then, the radar agent's current anti-interference strategy and neural network parameters are fixed to play against the interference agent. Then, based on the radar agent's current anti-jamming strategy, the jamming agent optimizes the initial jamming strategy using a near-end strategy optimization algorithm. After multiple cycles of interactive learning and training, the jamming agent continuously optimizes and adjusts the jamming strategy based on the collected radar state history information and environmental feedback information, generating jamming decisions. Then, the jamming agent's current jamming strategy and neural network parameters are fixed to play against the radar agent in the next round of the game. The near-end policy optimization algorithm is an improvement on the policy gradient algorithm. It adopts a trust region policy optimization algorithm, introduces KL divergence constraints, and limits the difference between the old and new policies. The policy is updated by calculating the objective function.

2. The radar anti-jamming decision-making method based on recursive game training reinforcement learning according to claim 1, characterized in that, The decision-making processes of the radar agent and the jamming agent are described using Markov decision processes.

3. The radar anti-jamming decision-making method based on recursive game training reinforcement learning according to claim 1, characterized in that, The radar agent selects the optimal anti-jamming decision from the anti-jamming knowledge base based on the anti-jamming strategy.

4. The radar anti-jamming decision-making method based on recursive game training reinforcement learning according to claim 1, characterized in that, Both the initial anti-interference strategy and the initial interference strategy are generated randomly based on a normal distribution.

5. The radar anti-jamming decision-making method based on recursive game training reinforcement learning according to claim 1, characterized in that, The environmental feedback information corresponding to the radar agent is the anti-jamming gain value, and the environmental feedback information corresponding to the jamming agent is the jamming gain value. The interference gain value is the negative of the anti-interference gain value; the anti-interference gain value is equal to the signal-to-interference-to-noise ratio.

Images