A minmax-q learning based jamming and radar frequency dynamic game method

Abstract

The application discloses a kind of Minmax-Q learning-based interference and radar frequency dynamic game method, first establish the signal model of jammer and radar, design the game reward function based on frequency aiming probability, then construct the dynamic frequency stochastic game model between jammer and radar under the game theory framework, and the Nash equilibrium of this game is analyzed, finally propose the Nash equilibrium solution algorithm based on Minmax-Q learning, for generating the optimal countermeasure strategy of jammer in frequency dimension.The method of the application provides new theory and method support for interference countermeasure intelligent decision, the optimal decision interference frequency resource scheduling strategy in the dynamic game confrontation of jammer and intelligent radar, improves the success probability of interference countermeasure under the condition of intelligent radar, especially suitable for the dynamic frequency confrontation between jammer and radar under the condition of limited spectrum resource, with good practical value and application prospect.

Description

Technical Field

[0001] This invention belongs to the field of radar jamming technology, specifically relating to a dynamic game method for jamming and radar frequency based on Minmax-Q learning. Background Technology

[0002] Radar jamming, a typical electronic countermeasure, uses electromagnetic interference signals to make it difficult for the adversary's radar to obtain information about friendly targets. Based on the jamming principle, jamming can be divided into two categories: cover-up jamming and deception jamming. Cover-up jamming raises the noise floor of the radar's received signal by releasing high-power noise interference signals, causing the target echo signal to be submerged in noise, thus preventing the radar from detecting the target. Deception jamming, based on Digital Radio Frequency Memory (DRFM) technology, intercepts, stores, modulates, and forwards radar signals to deceive the radar in terms of range, speed, and angle. Regardless of whether it is cover-up jamming or deception jamming, the core countermeasure technology lies in the control and utilization of electromagnetic spectrum resources.

[0003] Regarding the generation of intelligent control strategies for frequency domain resources in covert jamming, existing covert jamming methods mainly employ broadband suppression, narrowband frequency targeting, or frequency sweeping. With the expansion of radar spectrum resources (e.g., the availability of the 2GHz-18GHz band), broadband suppression jamming struggles to cover such a wide band and is detrimental to the efficient utilization of spectrum resources. With the development of cognitive intelligent radar technology, radar systems can sense changes in jamming frequencies and select frequency agility strategies. They may even possess intelligent learning and decision-making capabilities, dynamically learning the frequency change patterns of jammers to optimize frequency hopping strategies and avoid jamming bands. These advanced cognitive intelligent radar anti-jamming technologies pose a significant challenge to jamming countermeasures.

[0004] The confrontation between radar and jamming is a dynamic game. From a developmental perspective, the advancement of cognitive intelligent radar technology will inevitably drive the development of jammers towards cognitive intelligence. Frequency domain intelligent game-theoretic confrontation offers greater freedom, and the struggle for and utilization of spectrum resources is the core of future electromagnetic spectrum warfare. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a dynamic frequency game method for jamming and radar based on Minmax-Q learning, which addresses the dynamic frequency confrontation problem between jammers and radars under conditions of limited spectrum resources, effectively improving the jamming success rate.

[0006] The technical solution adopted in this invention is: a dynamic game method for interference and radar frequency based on Minmax-Q learning, the specific steps of which are as follows:

[0007] S1. Establish signal models for the jammer and radar;

[0008] First, let's assume the baseband linear frequency modulated signal transmitted by the radar. The expression is as follows:

[0009] (1);

[0010] in, Indicates time, Indicates the pulse width. Indicates the frequency modulation slope. Indicates bandwidth. Represents a rectangle function.

[0011] Then, assuming the radar employs a frequency agility strategy, the transmitted... radar pulse signal The expression is as follows:

[0012] (2);

[0013] in, Indicates radar transmit power. Indicates the first The carrier frequency of each radar pulse signal, corresponding to the frequency band is The expression is as follows:

[0014] (3);

[0015] Then set the interference waveform as Then the first radar pulse signal The transmitted interference signal is The expression is as follows:

[0016] (4);

[0017] in, Indicates the jamming transmission power. Indicates the first The carrier frequency of each interference signal, and the corresponding frequency band is The expression is as follows:

[0018] (5);

[0019] in, This indicates the bandwidth of the interference signal.

[0020] S2. Construct a frequency game model between the jammer and the radar;

[0021] First, let's set the assumptions: the instantaneous bandwidth of the jammer. Smaller than the entire selectable frequency band of the radar, i.e. .

[0022] in, This indicates the number of selectable frequency points for the radar.

[0023] Redefining the first The frequency targeting probability of a single pulse interference signal on a radar is the proportion of the radar frequency band covered by the interference signal within the radar signal frequency band. The expression is as follows:

[0024] (6);

[0025] The mathematical model of the frequency game between the jammer and the radar is represented by a seven-tuple. The specific definitions of each element are as follows:

[0026] 1) Set of game participants This includes jammers and radar; 1 represents a jammer, and 2 represents a radar.

[0027] 2) Jammer Action Set The jammer's action set is its selectable set of frequency points, i.e. .

[0028] in, This indicates the number of selectable frequency points for the jammer; the jammer is at the [number]th frequency. The action of each pulse is .

[0029] 3) Radar Action Set The radar action set is its selectable set of frequency points, i.e. .

[0030] in, This indicates the number of selectable frequency points for the radar. The action of each pulse is .

[0031] 4) State set The set of frequency points where jammers and radars may appear, i.e. .

[0032] in, Represents the Cartesian product; radar and jammers in the first... The state of the second game is as follows: .

[0033] 5) State transition function In state The jammer took action. The radar took action. The state transitions to The state transition is represented as .

[0034] 6) Jammer report function The jammer was at the The reward obtained in this game confrontation is That is, the first The probability of a pulse targeting the radar frequency.

[0035] in, Represents the real number field.

[0036] 7) Radar report function Radar at the The reward obtained in this game confrontation is That is, the first The probability of a single pulse interference causing the radar's frequency targeting is negative.

[0037] The reward functions for the jammer and the radar are expressed as follows: That is, in the state Next, both sides took action. and The state transitions to Time game participants The rewards obtained. And the game participants. The return its opponents The return The sum is zero, meaning the frequency game between the jammer and the radar is a zero-sum game.

[0038] S3. Establish a frequency-based stochastic game model between the jammer and the radar;

[0039] S31. Constructing the Nash equilibrium of frequency matrix games;

[0040] S32. Construct a frequency-based random game model;

[0041] S4. Based on the game model established in step S3, a Nash equilibrium solution algorithm based on Minmax-Q learning is proposed to generate frequency dynamic game strategies.

[0042] For each state There exists a game between a jammer and a radar frequency matrix, and the goal is to find a Nash equilibrium strategy within this matrix game. The Nash equilibrium strategy for the random game between the jammer and the radar frequency is the set of Nash equilibrium strategies for each specific state matrix game. If the action-value function corresponding to each state is known... Then, the game participants are obtained by solving the linear programming problem using the Minmax-Q JRFG algorithm, which is based on Minmax-Q learning to generate interference-radar frequency game strategies. Nash equilibrium strategy The linear programming problem is expressed as follows:

[0043] (7);

[0044] in, Represents the state-value function under the Nash equilibrium strategy. In random games, the game participants strategy Let the probability distribution be defined in the action space, i.e. , Indicates the participants in the game Select an action in the current state. The probability of and satisfying .

[0045] Furthermore, step S3 is specifically as follows:

[0046] S31. Constructing the Nash equilibrium of frequency matrix games;

[0047] In each pulse confrontation, the jammer and the radar engage in a static matrix game. In this game, the jammer's objective is to maximize the frequency targeting probability, while the radar's objective is to minimize it. In a stochastic game, the game participants... strategy Let the probability distribution be defined in the action space, i.e. .

[0048] The Nash equilibrium of the frequency matrix game between the jammer and the radar is obtained by solving a linear programming model. The expression is as follows:

[0049] (8);

[0050] in, This indicates the strategy employed by the jammer. Countering radar actions Expected returns obtained .

[0051] S32. Construct a frequency-based random game model;

[0052] The frequency confrontation between jammers and radar is modeled as a generalized Markov matrix process. The expected payoffs of the game participants are expressed as... Its meaning is: from the state Initially, the jammer took action. And the radar took action The expected returns obtained.

[0053] A random game of frequency between jammers and radar regarding states The state value of the interference machine is defined as The expression is as follows:

[0054] (9);

[0055] in, Represent each state Instant payouts for unrelated matrix game sequences, and single game solutions based on the optimization method of equation (8).

[0056] jammer in status Use action Countering radar actions The value of , i.e., the action value function, is defined as follows:

[0057] (10);

[0058] in, This represents the instantaneous reward function obtained by the jammer, i.e., in the state... The jammer and radar then took separate actions. and At that time, the first The probability of a radar frequency targeting by a single pulse interference; This represents the discount factor.

[0059] The objective of a random frequency game between a jammer and a radar is to find a Nash equilibrium, which is represented by a strategy combination. For all states And all game participants The following condition expression should be satisfied:

[0060] (11);

[0061] in, This represents the optimal mixing strategy for the jammer. This represents the optimal hybrid strategy for radar. Indicates the participants in the game opponent, This represents the game participants under the given current state and equilibrium strategies of all participants. Expected return on discount; This indicates that, given the current state, when the game participants... Adopting strategies Its opponent employs a Nash equilibrium strategy. At that time, participants The expected return from the discount received.

[0062] Reuse Represents the state-value function under the Nash equilibrium strategy. Combining the Q-learning algorithm and the definition of Nash equilibrium, let the action-value function... This represents the game participants given the current state and the current actions of the jammer and radar. The sum of the expected returns after discounts is obtained, and then a Nash equilibrium strategy is applied to get... and The specific expression is as follows:

[0063] (12);

[0064] (13);

[0065] in, Indicates the participants in the game In state Actions under Nash equilibrium strategy The probability distribution.

[0066] The Nash equilibrium of the random frequency game between the jammer and the radar can then be rewritten as follows:

[0067] (14);

[0068] in, Indicates the participants in the game In state Use action The probability distribution. And when state dependencies are not particularly emphasized, let's denote... for The abbreviated form of .

[0069] Finally, if all states Given the values, the participants are obtained by solving the game between the jammer and the radar frequency matrix in a specific state. The Nash equilibrium strategy.

[0070] Furthermore, in step S4, the Minmax-Q JRFG method for solving the linear programming problem is as follows:

[0071] S41, Parameter initialization;

[0072] Input jammer action set Radar Action Set State set Jammer and radar reward function Total number of antagonistic pulse groups and the number of pulses in each pulse group Initialize action value function State value function ,Strategy Discount Factor and learning rate ;

[0073] in, That is, the number of iterations. That is, the number of iterations per scene.

[0074] S42, Game Participants Based on the exploration-development strategy, from the current state Take action ;

[0075] In the initial learning phase of the exploration-exploitation strategy, the game participants try each type of action at a high frequency, collect reward information under different states, and gradually increase the probability of choosing the action with the best estimated reward during the learning process, guiding the strategy to converge toward Nash equilibrium.

[0076] S43, State transition Game participants Calculate the return obtained And observe the opponent In the previous state Actions taken ;

[0077] S44, Game Participants Update its action value function;

[0078] During the Nash equilibrium solution process of the random frequency game between the jammer and the radar, the action value function is updated according to the following expression:

[0079] (15);

[0080] in, This represents the learning rate.

[0081] S45, Game Participants By using linear programming to solve equation (7), the Nash equilibrium strategy under the corresponding state can be obtained;

[0082] The linear programming aims to maximize or minimize the expected return of the game participants in a given state. Under the condition of satisfying the policy probability constraints, the policy probability corresponding to each action is obtained, and the Nash equilibrium policy defined by equation (7) is obtained.

[0083] S46. Repeat steps S42 to S45 to obtain the updated version. and Iterate through each scene sequentially until the preset total number of scenes is reached, and output the Nash equilibrium strategy. .

[0084] Within each scene, decisions and learning are made step by step in a pulse sequence.

[0085] The beneficial effects of this invention are as follows: First, the method of this invention establishes signal models for both the jammer and the radar, designs a game-theoretic reward function based on frequency aiming probability, then constructs a dynamic frequency stochastic game model between the jammer and the radar within the framework of game theory, analyzes the Nash equilibrium of this game, and finally proposes a Nash equilibrium solving algorithm based on Minmax-Q learning to generate the optimal countermeasure strategy for the jammer in the frequency dimension. This invention provides new theoretical and methodological support for intelligent decision-making in jamming countermeasures. In the dynamic game-theoretic confrontation between the jammer and the intelligent radar, the optimal decision-making jamming frequency resource scheduling strategy improves the success probability of jamming countermeasures under intelligent radar conditions. It is particularly suitable for dynamic frequency confrontation between jammers and radar under conditions of limited spectrum resources, and has good practical value and application prospects. Attached Figure Description

[0086] Figure 1 This is a flowchart of a dynamic game method for interference and radar frequency based on Minmax-Q learning according to the present invention.

[0087] Figure 2 The jammer and radar in the embodiments of the present invention A schematic diagram of the payoff matrix for a frequency game.

[0088] Figure 3 This is a diagram showing the results of frequency countermeasures between radar and interference based on different strategies in an embodiment of the present invention.

[0089] Figure 4 This is a diagram showing the aiming probability results based on different interference strategies in an embodiment of the present invention.

[0090] Figure 5 This is a graph showing the radar pulse count results of successful interference based on different interference strategies in embodiments of the present invention. Detailed Implementation

[0091] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0092] like Figure 1 The flowchart of a dynamic game theory method for interference and radar frequency based on Minmax-Q learning according to the present invention is shown below. The specific steps are as follows:

[0093] S1. Establish signal models for the jammer and radar;

[0094] First, let's assume the baseband linear frequency modulated signal transmitted by the radar. The expression is as follows:

[0095] (1);

[0096] in, Indicates time, Indicates the pulse width. Indicates the frequency modulation slope. Indicates bandwidth. Represents a rectangle function.

[0097] Then, assuming the radar employs a frequency agility strategy, the transmitted... radar pulse signal The expression is as follows:

[0098] (2);

[0099] in, Indicates radar transmit power. Indicates the first The carrier frequency of each radar pulse signal, corresponding to the frequency band is The expression is as follows:

[0100] (3);

[0101] Then set the interference waveform as It can be a noise-modulated signal (such as noise frequency modulation / amplitude modulation / phase modulation and radio frequency noise signals), or a modulated and relayed radar signal. Then the... radar pulse signal The transmitted interference signal is The expression is as follows:

[0102] (4);

[0103] in, Indicates the jamming transmission power. Indicates the first The carrier frequency of each interference signal, and the corresponding frequency band is The expression is as follows:

[0104] (5);

[0105] in, This indicates the bandwidth of the interference signal.

[0106] S2. Construct a frequency game model between the jammer and the radar;

[0107] This embodiment constructs a frequency game model between a jammer and a radar. Both sides possess the capabilities of perception, decision-making, action, and evaluation, and are rational agents. Taking full account of the opponent, each side makes optimal decisions based on perceived information, thereby achieving its own optimal strategy under the condition that the opponent's optimal strategy is in effect. First, the following assumptions are set: the instantaneous bandwidth of the jammer... Smaller than the entire selectable frequency band of the radar, i.e. (That is, this embodiment considers the problem of intelligent decision-making to combat interference under the condition of limited spectrum resources).

[0108] in, This indicates the number of selectable frequency points for the radar.

[0109] Redefining the first The frequency targeting probability of a single pulse interference signal on a radar is the proportion of the radar frequency band covered by the interference signal within the radar signal frequency band. The expression is as follows:

[0110] (6);

[0111] The mathematical model of the frequency game between the jammer and the radar is represented by a seven-tuple. The specific definitions of each element are as follows:

[0112] 1) Set of game participants This includes jammers and radar; 1 represents a jammer, and 2 represents a radar.

[0113] 2) Jammer Action Set The jammer's action set is its selectable set of frequency points, i.e. .

[0114] in, This indicates the number of selectable frequency points for the jammer; the jammer is at the [number]th frequency. The action of each pulse is .

[0115] 3) Radar Action Set The radar action set is its selectable set of frequency points, i.e. .

[0116] in, This indicates the number of selectable frequency points for the radar. The action of each pulse is .

[0117] 4) State set The set of frequency points where jammers and radars may appear, i.e. .

[0118] in, Represents the Cartesian product; radar and jammers in the first... The state of the second game is as follows: .

[0119] 5) State transition function In state The jammer took action. The radar took action. The state transitions to The state transition is represented as .

[0120] 6) Jammer report function The jammer was at the The reward obtained in this game confrontation is That is, the first The frequency aiming probability of a single pulse to the radar, frequency aiming probability The larger the size, the more likely the jammer is to be in the first position. The greater the reward gained in the next game confrontation.

[0121] in, Represents the real number field.

[0122] 7) Radar report function Radar at the The reward obtained in this game confrontation is That is, the first The frequency aiming probability of a single pulse interference pulse to the radar is negative; the frequency aiming probability... The larger the radar is, the more effective it is in the first place. The smaller the reward value obtained in the next game confrontation.

[0123] The reward functions for the jammer and the radar are expressed as follows: That is, in the state Next, both sides took action. and The state transitions to Time game participants The rewards obtained. And the game participants. The return its opponents The return The sum is zero, meaning the frequency game between the jammer and the radar is a zero-sum game.

[0124] S3. Establish a frequency-based stochastic game model between the jammer and the radar;

[0125] S31. Constructing the Nash equilibrium of frequency matrix games;

[0126] S32. Construct a frequency-based random game model;

[0127] S4. Based on the game model established in step S3, a Nash equilibrium solution algorithm based on Minmax-Q learning is proposed to generate frequency dynamic game strategies.

[0128] For each state There exists a game between a jammer and a radar frequency matrix, and the goal is to find a Nash equilibrium strategy within this matrix game. The Nash equilibrium strategy for the random game between the jammer and the radar frequency is the set of Nash equilibrium strategies for each specific state matrix game. If the action-value function corresponding to each state is known... Then, the game participants are obtained by solving the linear programming problem using the Minmax-Q JRFG algorithm, which is based on Minmax-Q learning to generate interference-radar frequency game strategies. Nash equilibrium strategy The linear programming problem is expressed as follows:

[0129] (7);

[0130] in, Represents the state-value function under the Nash equilibrium strategy. In random games, the game participants strategy Let the probability distribution be defined in the action space, i.e. , Indicates the participants in the game Select an action in the current state. The probability of and satisfying .

[0131] The solution to the linear programming problem uses both minmax optimization and Q-learning. Therefore, the proposed algorithm is named the Minmax-Q based Jammer-RadarFrequency Game (Minmax-Q JRFG) strategy generation algorithm.

[0132] In this embodiment, step S3 is specifically as follows:

[0133] S31. Constructing the Nash equilibrium of frequency matrix games;

[0134] In each pulse countermeasure, the jammer and the radar engage in a static matrix game, with the payoff matrix as follows: Figure 2 As shown. Furthermore, in this matrix game, the objective of the jammer is to maximize the frequency targeting probability, while the objective of the radar is to minimize the frequency targeting probability. In a stochastic game, the game participants... strategy Let the probability distribution be defined in the action space, i.e. .

[0135] The Nash equilibrium of the frequency matrix game between the jammer and the radar is obtained by solving a linear programming model. The expression is as follows:

[0136] (8);

[0137] in, This indicates the strategy employed by the jammer. Countering radar actions Expected returns obtained (Subscripts omitted here for simplicity) ).

[0138] S32. Construct a frequency-based random game model;

[0139] The confrontation between jammers and radar is a multi-stage process. The frequency confrontation between jammers and radar can be modeled as a generalized Markov matrix process (also known as a Markov game or stochastic game). The expected payoffs of the game participants are expressed as... Its meaning is: from the state Initially, the jammer took action. And the radar took action The expected returns obtained.

[0140] A random game of frequency between jammers and radar regarding states The state value of the interference machine is defined as The expression is as follows:

[0141] (9);

[0142] in, Represent each state Instant payouts for unrelated matrix game sequences, where a single game can be solved based on the optimization method of equation (8).

[0143] jammer in status Use action Countering radar actions The value of , i.e., the action value function, is defined as follows:

[0144] (10);

[0145] in, The function representing the immediate reward obtained by the jammer is denoted by , which represents the state . The jammer and radar then took separate actions. and At that time, the first The probability of frequency targeting by a single pulse interference pulse on the radar. This represents the discount factor, which characterizes the degree to which an agent values ​​future rewards during multi-stage learning.

[0146] The objective of a random frequency game between a jammer and a radar is to find a Nash equilibrium, which is represented by a strategy combination. For all states And all game participants The following condition expression should be satisfied:

[0147] (11);

[0148] in, This represents the optimal mixing strategy for the jammer. This represents the optimal hybrid strategy for radar. Indicates the participants in the game opponent, This represents the game participants under the given current state and equilibrium strategies of all participants. Expected return on discount; This indicates that, given the current state, when the game participants... Adopting strategies Its opponent employs a Nash equilibrium strategy. At that time, participants The expected return from the discount received.

[0149] Reuse Represents the state-value function under the Nash equilibrium strategy. Combining the Q-learning algorithm and the definition of Nash equilibrium, let the action-value function... This represents the game participants given the current state and the current actions of the jammer and radar. The sum of the expected returns after discounts is obtained, and then a Nash equilibrium strategy is applied to get... and The specific expression is as follows:

[0150] (12);

[0151] (13);

[0152] in, Indicates the participants in the game In state Actions under Nash equilibrium strategy The probability distribution.

[0153] The Nash equilibrium of the random frequency game between the jammer and the radar can then be rewritten as follows:

[0154] (14);

[0155] in, Indicates the participants in the game In state Use action The probability distribution. And when state dependencies are not particularly emphasized, let's denote... for The abbreviated form of .

[0156] Finally, the Nash equilibrium of the random game between the jammer and the radar frequency can be described as a set of Nash equilibrium strategies in a matrix game between the jammer and the radar frequency associated with specific states. If all states... Given the values, the participants are obtained by solving the game between the jammer and the radar frequency matrix in a specific state. The Nash equilibrium strategy.

[0157] In this embodiment, step S4, specifically the Minmax-Q JRFG solution of the linear programming problem, is as follows:

[0158] S41, Parameter initialization;

[0159] Input jammer action set Radar Action Set State set Jammer and radar reward function Total number of antagonistic pulse groups and the number of pulses in each pulse group Initialize action value function State value function ,Strategy Discount Factor and learning rate ;

[0160] in, That is, the number of iterations. That is, the number of iterations per scene.

[0161] S42, Game Participants Based on the exploration-development strategy, from the current state Take action ;

[0162] The core idea of ​​the exploratory-explosive strategy is to balance the exploration of unknown actions with the utilization of known optimal actions. Specifically, in the initial learning phase, game participants try various actions at a high frequency to collect information on rewards under different states. As the learning process progresses, the system gradually increases the probability of selecting the action with the currently estimated best reward, thereby guiding the strategy towards Nash equilibrium. Through this exploratory-explosive strategy, the learning efficiency of game participants in a dynamic frequency game environment can be improved while ensuring strategy convergence.

[0163] S43, State transition Game participants Calculate the return obtained And observe the opponent In the previous state Actions taken ;

[0164] S44, Game Participants Update its action value function;

[0165] During the Nash equilibrium solution process of the random frequency game between the jammer and the radar, the action value function is updated according to the following expression:

[0166] (15);

[0167] in, This represents the learning rate.

[0168] S45, Game Participants By using linear programming to solve equation (7), the Nash equilibrium strategy under the corresponding state can be obtained;

[0169] The linear programming aims to maximize or minimize the expected return of the game participants in a given state. Under the condition of satisfying the policy probability constraints, the policy probability corresponding to each action is obtained, and the Nash equilibrium policy defined by equation (7) is obtained.

[0170] S46. Repeat steps S42 to S45 to obtain the updated version. and Iterate through each scene sequentially until the preset total number of scenes is reached, and output the Nash equilibrium strategy. .

[0171] Within each scene, decisions and learning are made step by step in a pulse sequence.

[0172] This embodiment further verifies and analyzes the method of the present invention through simulation. The simulation scenario involves a jammer countering an adversary's cognitive frequency-agile radar. The radar has a transmit power of 50kW, with one pulse group containing 128 pulses, a selectable frequency band range of [2GHz, 4GHz], an instantaneous bandwidth of 20MHz, and 200 selectable frequency points, covering the entire selectable frequency band without overlap. The jammer has a transmit power of 500kW, and its selectable frequency band range is also [2GHz, 4GHz]. For the Minmax-Q JRFG algorithm, the discount factor... .

[0173] To verify the effectiveness of the Minmax-Q JRFG algorithm proposed in this invention, the following three methods were selected for performance comparison: (1) Frequency targeting strategy: The jammer measures the frequency of a pulse on the radar and then uses the measured center frequency to carry out jamming. The instantaneous jamming bandwidth is 80MHz (the frequency targeting jamming bandwidth is 3 to 5 times the radar bandwidth, and here we take 4 times); (2) Frequency sweeping strategy (instantaneous jamming bandwidth of 200MHz): The jammer periodically scans the entire frequency band, changing the frequency once for each pulse; (3) Frequency sweeping strategy (instantaneous jamming bandwidth of 1GHz): The jammer periodically scans the entire frequency band, changing the frequency once for each pulse. This strategy is a typical broadband frequency sweeping jamming.

[0174] The results of radar and jamming frequency band variations with pulses based on different strategies are as follows: Figure 3 As shown, this result represents the outcome of the 200th pulse group. Figure 3 (a) shows the outcome of the countermeasures using the targeting strategy. Figure 3 (b) is a frequency sweeping strategy. The outcome of the confrontation Figure 3 (c) is a frequency sweeping strategy. The outcome of the confrontation; Figure 3(d) shows the adversarial results of the Minmax-Q JRFQ algorithm. The probability of a jammer's frequency band covering the radar's frequency band is very small, because the radar has learning capabilities and can learn the frequency change patterns of the jammer, thus avoiding the jamming frequency band. Due to the time lag in the frequency jamming decision-making process, the radar can avoid the jamming frequency band through continuous frequency agility. For frequency sweeping jamming, the wider the instantaneous bandwidth, the better the jamming effect. However, even with wideband frequency sweeping jamming (instantaneous bandwidth of 1 GHz), the radar can learn the frequency sweeping pattern of the jamming, thus formulating a frequency agility strategy to avoid the jamming frequency band. The jamming effect of the Minmax-Q JRFQ algorithm proposed in this invention is significantly better than other methods based on fixed rules, even if its instantaneous bandwidth is smaller (or much smaller) than that of frequency sweeping jamming, its advantage is still quite significant. This is because the proposed Minmax-Q JRFQ algorithm fully considers the situation where the opponent is also an intelligent agent. In the game between the jammer and the radar's rational intelligent agent, it obtains the optimal adversarial strategy through interactive learning.

[0175] The targeting probability and the number of radar pulses for successful jamming based on different jamming strategies are as follows: Figure 4-5 As shown, the frequency aiming probability of jammers based on the frequency targeting strategy and the frequency sweeping strategy is relatively low (below 0.1), resulting in a lower number of successfully jammed radar pulses (less than 10). The jammer based on the proposed Minmax-Q JRFG algorithm, because it fully considers the intelligent behavior of the adversary radar, has a higher frequency aiming probability and a significantly higher number of successfully jammed radar pulses than the jammer based on the wideband frequency sweeping strategy, and significantly higher than the other two strategies. This verifies the superiority of the proposed algorithm.

[0176] In summary, the method of this invention verifies the effectiveness of the Nash equilibrium solution algorithm based on Minmax-Q learning through digital simulation. A rational and intelligent jammer, fully sensing and understanding the frequency agility behavior of the adversary radar, optimizes its own frequency selection strategy to obtain the best jamming frequency selection strategy under the radar's optimal frequency agility strategy. Simulation results show that the method of this invention achieves a 40% success rate in jamming intelligent radars under a game-theoretic equilibrium strategy (the success rate can be significantly improved against non-cognitive intelligent radars). The method of this invention can be extended to jamming countermeasures decisions in other dimensions, such as waveform, time, beam, and power resource decisions.

[0177] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A dynamic game theory method for interference and radar frequency based on Minmax-Q learning, the specific steps of which are as follows: S1. Establish signal models for the jammer and radar; First, let's assume the baseband linear frequency modulated signal transmitted by the radar. The expression is as follows: （1）； in, Indicates time, Indicates the pulse width. Indicates the frequency modulation slope. Indicates bandwidth. Represents a rectangular function; Then, assuming the radar employs a frequency-agile strategy, the transmitted... radar pulse signal The expression is as follows: （2）； in, Indicates radar transmit power. Indicates the first The carrier frequency of each radar pulse signal, corresponding to the frequency band is The expression is as follows: （3）； Then set the interference waveform as Then the first radar pulse signal The transmitted interference signal is The expression is as follows: （4）； in, Indicates the jamming transmission power. Indicates the first The carrier frequency of each interference signal, corresponding to the frequency band is The expression is as follows: （5）； in, Indicates the bandwidth of the interference signal; S2. Construct a frequency game model between the jammer and the radar; First, let's set the assumptions: the instantaneous bandwidth of the jammer. Smaller than the entire selectable frequency band of the radar, i.e. ; in, Indicates the number of selectable frequency points for the radar; Redefining the first The frequency targeting probability of a single pulse interference signal on a radar is the proportion of the radar frequency band covered by the interference signal within the radar signal frequency band. The expression is as follows: （6）； The mathematical model of the frequency game between the jammer and the radar is represented by a seven-tuple. The specific definitions of each element are as follows: 1) Set of game participants This includes jammers and radar; 1 represents jammer, and 2 represents radar. 2) Jammer Action Set The jammer's action set is its selectable set of frequency points, i.e. ; in, This indicates the number of selectable frequency points for the jammer; the jammer is at the [number]th frequency. The action of each pulse is ; 3) Radar Action Set The radar action set is its selectable set of frequency points, i.e. ; in, This indicates the number of selectable frequency points for the radar. The action of each pulse is ; 4) State set The set of frequency points where jammers and radars may appear, i.e. ; in, Represents the Cartesian product; radar and jammers in the first... The state of the second game is as follows: ; 5) State transition function In state The jammer took action. The radar took action. The state transitions to The state transition is represented as ; 6) Jammer report function The jammer was at the... The reward obtained in this game confrontation is That is, the first The probability of a pulse targeting the radar frequency; in, Represents the real number field; 7) Radar report function Radar at the The reward obtained in this game confrontation is That is, the first The frequency targeting probability of a single pulse interference pulse is negative for the radar. The reward functions for the jammer and the radar are expressed as follows: That is, in the state Next, both sides took action. and The state transitions to Time game participants The rewards obtained; and the participants in the game The return its opponents The return The sum is zero, meaning the frequency game between the jammer and the radar is a zero-sum game. S3. Establish a frequency-based stochastic game model between the jammer and the radar; S31. Constructing the Nash equilibrium of frequency matrix games; S32. Construct a frequency-based random game model; S4. Based on the game model established in step S3, a Nash equilibrium solution algorithm based on Minmax-Q learning is proposed to generate frequency dynamic game strategies. For each state There exists a game between a jammer and a radar frequency matrix, and a Nash equilibrium strategy is sought in the corresponding matrix game. The Nash equilibrium strategy of the random game between the jammer and the radar frequency is the set of Nash equilibrium strategies in each specific state matrix game. If the action-value function corresponding to the state is known... Then, the game participants are obtained by solving the linear programming problem using the Minmax-Q JRFG algorithm, which is based on Minmax-Q learning to generate interference-radar frequency game strategies. Nash equilibrium strategy The linear programming problem is expressed as follows: （7）； in, Represents the state-value function under the Nash equilibrium strategy. In random games, the game participants strategy Let the probability distribution be defined in the action space, i.e. , Indicates the participants in the game Select an action in the current state The probability of and satisfying .

2. The method for dynamic game theory of interference and radar frequencies based on Minmax-Q learning according to claim 1, characterized in that, Step S3 is as follows: S31. Constructing the Nash equilibrium of frequency matrix games; In each pulse confrontation, the jammer and the radar engage in a static matrix game. In this game, the jammer's objective is to maximize the frequency targeting probability, while the radar's objective is to minimize it. In the stochastic game, the game participants... strategy Let the probability distribution be defined in the action space, i.e. ; The Nash equilibrium of the frequency matrix game between the jammer and the radar is obtained by solving a linear programming model. The expression is as follows: （8）； in, This indicates the strategy employed by the jammer. Countering radar actions Expected returns obtained ; S32. Construct a frequency-based random game model; The frequency confrontation between jammers and radar is modeled as a generalized Markov matrix process; the expected payoffs of the game participants are expressed as... Its meaning is: from the state Initially, the jammer took action. The radar took action. Expected returns; A random game of frequency between jammers and radar regarding states The state value of the interference mechanism is defined as follows: The expression is as follows: （9）； in, Represent each state Instant payouts for unrelated matrix game sequences, with each game solved using the optimization method based on equation (8); jammer in status Use action Countering radar actions The value of , i.e., the action value function, is defined as follows: （10）； in, This represents the instantaneous reward function obtained by the jammer, i.e., in the state... The jammer and radar then took separate actions. and At that time, the first The probability of a radar frequency targeting by a single pulse interference; Indicates the discount factor; The objective of a random frequency game between a jammer and a radar is to find a Nash equilibrium, which is represented by a strategy combination. For all states And all game participants The following condition expression should be satisfied: （11）； in, This represents the optimal mixing strategy for the jammer. This represents the optimal hybrid strategy for radar; Indicates the participants in the game opponent, This represents the game participants under the given current state and equilibrium strategies of all participants. Expected return on discount; This indicates that, given the current state, when the game participants... Adopting strategies Its opponent employs a Nash equilibrium strategy. At that time, participants The expected return from the discount received; Reuse Represents the state-value function under the Nash equilibrium strategy. Combining the Q-learning algorithm and the definition of Nash equilibrium, let the action-value function... This represents the game participants given the current state and the current actions of the jammer and radar. The sum of the expected returns after discounts is obtained, and then a Nash equilibrium strategy is applied to get... and The specific expression is as follows: （12）； （13）； in, Indicates the participants in the game In state Actions under Nash equilibrium strategy The probability distribution; The Nash equilibrium of the random frequency game between the jammer and the radar can then be rewritten as follows: （14）； in, Indicates the participants in the game In state Use action The probability distribution; and when state dependencies are not particularly emphasized, let for The abbreviated form; Finally, if all states Given the values, the participants are obtained by solving the game between the jammer and the radar frequency matrix in a specific state. The Nash equilibrium strategy.

3. The method for dynamic game theory of interference and radar frequencies based on Minmax-Q learning according to claim 2, characterized in that, In step S4, the Minmax-Q JRFG method for solving the linear programming problem is as follows: S41, Parameter initialization; Input jammer action set Radar Action Set State set Jammer and radar reward function Total number of antagonistic pulse groups and the number of pulses in each pulse group Initialize action value function State value function ,Strategy Discount Factor and learning rate ; in, That is, the number of iterations. That is, the number of iterations per scene; S42, Game Participants Based on the exploration-development strategy, from the current state Take action ; In the early stages of learning, the exploration-exploitation strategy involves game participants trying each type of action at a high frequency to collect reward information under different states. During the learning process, the probability of choosing the action with the best estimated reward will be gradually increased, guiding the strategy to converge toward Nash equilibrium. S43, State transition Game participants Calculate the return obtained And observe the opponent In the previous state Actions taken ; S44, Game Participants Update its action value function; During the Nash equilibrium solution process of the random frequency game between the jammer and the radar, the action value function is updated according to the following expression: （15）； in, Indicates the learning rate; S45, Game Participants By using linear programming to solve equation (7), the Nash equilibrium strategy under the corresponding state can be obtained; The linear programming aims to maximize or minimize the expected return of the game participants in a given state. Under the condition of satisfying the policy probability constraints, the policy probability corresponding to each action is obtained, and the Nash equilibrium policy defined by equation (7) is obtained. S46. Repeat steps S42 to S45 to obtain the updated version. and Iterate through each scene sequentially until the preset total number of scenes is reached, and output the Nash equilibrium strategy. ; Within each scene, decisions and learning are made step by step in a pulse sequence.

Images