An interference strategy perception method based on generative adversarial imitation learning

Abstract

The application discloses an interference strategy perception method based on a generative adversarial imitation learning, and comprises the following steps: based on a turn-based Markov decision process, obtaining an online sample set according to the interaction between a radar and a jammer; inputting an online sample in the online sample set into a jamming strategy network, updating the parameters of a discriminator network first, updating the parameters of a generator network based on the reward output by the discriminator network, obtaining the generator network and the discriminator network meeting preset requirements, and obtaining a trained jamming strategy network; obtaining an interference strategy for training an anti-interference strategy network according to a strategy evaluation error; training the anti-interference strategy network based on the interference strategy obtained in step 3, obtaining a trained anti-interference strategy network; and using the anti-interference strategy output by the trained anti-interference strategy network to online counteract a real interference strategy. Compared with a supervised learning method, the method can relieve the composite error problem in the perception process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of radars, and particularly relates to a jamming strategy perception method based on generative adversarial imitation learning. BACKGROUND

[0002] With the rapid development of modern electronic technology, the survival of radars is facing greater challenges. Among them, the main lobe jamming poses a serious threat to radars because it is on the main lobe of the radar antenna, and the jamming signal is completely matched with the target signal in the spatial domain, making it difficult for the radar system to extract the real target. In addition, a more intelligent jammer can adjust its jamming strategy according to the intercepted radar signal. Therefore, it is of great significance to study a more intelligent radar system.

[0003] The traditional main lobe jamming suppression method is to suppress the jamming signal in the received echo data through signal processing. For example, researchers use eigenmatrix projection (EMP) preprocessing to suppress the main lobe jamming, or use blind source separation technology to separate the jamming signal from the target signal. However, when the jamming signal is close to the target echo in the direction of arrival, these methods will fail, for example, when the target is equipped with a self-defense jammer, the above methods not only suppress the jamming signal, but also suppress the target echo signal, resulting in extremely low output signal-to-noise ratio. For main lobe deception jamming, Lan Lan et al. of Xi'an University of Electronic Science and Technology adopted a new phase modulation method based on an array element pulse coding MIMO (EPC-MIMO) radar, which can separate the true and false targets corresponding to different pulses. However, when the jamming pattern is suppression jamming, this method cannot distinguish the target signal from the jamming signal. Polarization mode as an important degree of freedom for radar anti-jamming has attracted widespread research interest. Radar can filter out jamming by using polarization filtering in received signal processing by adopting variable polarization. However, if the jammer adopts the same polarization mode, the polarization filtering will fail. Unlike signal processing, active countermeasures have received widespread attention in recent years. Researchers avoid being jammed by designing appropriate signal transmission strategies. However, advanced jammers can take more intelligent methods to analyze radar anti-jamming strategies by intercepting previous radar signals, and design corresponding jamming strategies. Therefore, when facing more intelligent jammers, how to quickly develop anti-jamming strategies is the key to winning in the radar and jammer confrontation process.

[0004] Current intelligent anti-jamming strategy design mainly focuses on game theory and reinforcement learning. Using game theory, Li Kang et al. from Xi'an University of Electronic Science and Technology modeled the competition between frequency-agile radar and jammers as an extended game process and used a neural virtual self-game (NFSP) algorithm to find an approximate Nash equilibrium (ANE) strategy. This ANE strategy relies on two assumptions: both the radar and the jammer are rational, and the jammer's utility function is the negative of the radar's utility function. However, in real-time adversarial scenarios, the jammer's utility function is difficult for the radar to determine, and the ANE strategy is not the optimal adversarial strategy against different jamming strategies. As an effective tool for learning decision-making strategies, many studies in recent years have attempted to use reinforcement learning to solve for optimal anti-jamming strategies. In this approach, the jammer is modeled as the environment, and the frequency-agile radar is the agent making decisions through multiple interactions with the environment. Researchers have used reinforcement learning algorithms such as DQN and PPO to design optimal anti-jamming strategies. However, due to the high interaction costs and large interaction volume of current model-free reinforcement learning algorithms, directly interacting with the real jamming environment to train anti-jamming strategies online is impractical. To improve sampling efficiency, Fang Yuyuan et al. from Sun Yat-sen University proposed an online anti-jamming strategy learning method based on a multi-armed radar. However, this method only considers the anti-jamming strategy design for single-pulse detection and does not take into account the long-term rewards of multiple pulses. Furthermore, current anti-jamming strategy design methods lack an understanding of jamming strategies, making it unsafe to solve for the optimal anti-jamming strategy solely through online exploration. For example, the jammer might develop a deception strategy to disable the radar's detection strategy.

[0005] Unlike reinforcement learning, which learns the optimal policy from delayed feedback, imitation learning learns a good policy more efficiently from expert examples based on timely feedback. However, in imitation learning, directly learning the state-action mapping of the interfering machine is prone to the problem of "compound error" with small sample sizes. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a method for perceiving interference policies based on generative adversarial imitation learning. The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] An interference policy perception method based on generative adversarial imitation learning, the interference policy perception method comprising:

[0008] Step 1: Based on the turn-based Markov decision process, obtain an online sample set according to the interaction between the radar and the jammer. The online sample set is the collected interaction data between the radar and the jammer.

[0009] Step 2: Input the online samples from the online sample set into the interference policy network, and first update the parameters of the discriminator network. Then, update the parameters of the generator network based on the reward output by the discriminator network to obtain the generator network and the discriminator network that meet the preset requirements, so as to obtain the trained interference policy network, which includes the generator network and the discriminator network.

[0010] Step 3: Based on the expected value of the reward corresponding to the interference policy output by the trained interference policy network, obtain the policy evaluation error between the interference policy and the real interference policy, and obtain the interference policy for training the anti-interference policy network based on the policy evaluation error.

[0011] Step 4: Train the anti-interference strategy network based on the interference strategy obtained in Step 3 to obtain the trained anti-interference strategy network.

[0012] Step 5: Use the anti-interference strategy output by the trained anti-interference strategy network to counter real interference strategies online.

[0013] In one embodiment of the present invention, step 1 includes:

[0014] Step 1.1: Initialize the radar's guidance strategy;

[0015] Step 1.2: Based on the round-based Markov decision process, the radar is modeled as an intelligent agent in the round-based Markov decision process, and the jammer is the environment. The radar observes a preset number of online samples by interacting with the jammer, and the online sample set is composed of all observed online samples.

[0016] In one embodiment of the present invention, step 2 includes:

[0017] Step 2.1: Sample the online sample set and input the sampled online samples into the interference strategy network;

[0018] Step 2.2: Update the parameters of the discriminator network using gradient descent.

[0019] Step 2.3: Treat the reward output by the discriminator network as a cost function, and use the TRPO algorithm to perform gradient descent on the generator network to update the parameters of the generator network;

[0020] Step 2.4: Repeat steps 2.1 to 2.3 until the generator network and the discriminator network reach Nash equilibrium, thus obtaining the trained interference policy network.

[0021] In one embodiment of the present invention, the gradient corresponding to the gradient descent method is:

[0022]

[0023] Among them, c ω (s,b) represents the reward output by the discriminator network, where s is the radar's state information and b is the jammer's action. Let the gradient of the reward output by the discriminator network be the gradient of the discriminator network parameters ω. This is a mean estimate of the gradient of the reward value of a state-action pair (s,b) in a real sample with respect to the discriminator network parameters ω. This is the mean estimate of the gradient of the discriminator network parameter ω to the reward value of the state-action pair (s,b) output by the generator after passing through the discriminator.

[0024] In one embodiment of the present invention, the optimal discriminator network in the trained interference policy network is:

[0025]

[0026] Among them, c * (s h ,b h ) represents the reward output by the optimal discriminator network, s h b represents the radar's state information at time h. h For the action of the jammer at time h, Radar guidance strategy at time h In interference strategy M θ Distribution of state actions of the jammer in the interaction between the two. Radar guidance strategy at time h The state-action distribution of the jammers interacting in jamming strategy M, M θ M represents the interference policy output by the interference policy network, where M is the actual interference policy. Let h be the radar's guidance strategy at time h.

[0027] In one embodiment of the present invention, step 3 includes:

[0028] Step 3.1: Obtain the unknown real interference policy M, and derive the interference policy M based on the trained interference policy network. θ ;

[0029] Step 3.2: Obtain the expected value of the reward based on the unknown real interference strategy M. According to interference strategy M θ Expected value of receiving a reward

[0030] Step 3.3: Expected Value Based on Rewards And the expected value of rewards The policy evaluation error is obtained, and the policy evaluation error is:

[0031] Step 3.4: Determine whether the policy evaluation error remains constant. If not, proceed to step 2 until the policy evaluation error becomes constant. If so, use the output of the currently trained interference policy network as the interference policy for training the anti-interference policy network.

[0032] In one embodiment of the present invention, the expected value of the reward corresponding to the unknown real interference strategy is expressed as:

[0033]

[0034] The expected value of the reward corresponding to the interference policy output by the trained interference policy network is expressed as:

[0035]

[0036] Among them, a h For the radar action at time h, s h Let r be the radar state information at time h, 1≤h≤H. h (s h ,a h Let ρ be the reward of the radar pulse signal at time h, ρ0 be the initial state distribution of the radar, and π be the initial state distribution of the radar. h (·|s h M represents the radar's guidance strategy at time h. h (·|s h ,a h The jamming strategy of the jammer at time h; the reward of the radar pulse signal at time h is equal to the signal-to-interference-plus-noise ratio of the radar pulse signal at time h.

[0037] In one embodiment of the present invention, step 4 includes:

[0038] Based on the PPO algorithm, the interference strategy obtained in step 3 is used to train the anti-interference strategy network to obtain the trained anti-interference strategy network.

[0039] In one embodiment of the present invention, the objective function corresponding to the anti-interference strategy network is:

[0040]

[0041] in, The expected value of the reward corresponding to the interference policy output by the trained interference policy network.

[0042] The beneficial effects of this invention are:

[0043] 1. This invention addresses the problem of rapid perception of jamming strategies during the confrontation between FA radar and targeting jammers. It proposes a jamming strategy perception method based on generative adversarial imitation learning. Compared with supervised learning methods, this method can alleviate the problem of compound errors in the perception process. After the radar perceives the jamming strategy, it learns anti-jamming strategies through virtual interaction, thereby countering real jamming strategies online.

[0044] 2. This invention models the jamming strategy as an environment in a turn-based Markov decision process, with the FA radar as the agent and the jammer adopting deterministic or random jamming strategies that are time-varying in units of pulses.

[0045] 3. This invention collects online interactive samples through a radar-induced jammer. The online samples and samples sampled by the generator are fed into the discriminator network for training. When Nash equilibrium is reached, the jamming strategy is perceived. The method proposed in this invention can effectively perceive both deterministic and random jamming strategies. After perceiving the jamming strategy, the PPO algorithm is then used to virtually interact and train the anti-jamming strategy. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an interference strategy perception method based on generative adversarial imitation learning provided in an embodiment of the present invention.

[0047] Figure 2 This is a flowchart illustrating the process of online collection of interactive samples for training anti-interference strategies in virtual interaction, as provided in an embodiment of the present invention.

[0048] Figure 3 This is a flowchart of an algorithm for generative adversarial imitation learning provided in an embodiment of the present invention;

[0049] Figure 4 This is a discriminator network structure diagram provided in an embodiment of the present invention;

[0050] Figure 5 This is an Actor network structure diagram in GAIL provided by an embodiment of the present invention;

[0051] Figure 6 This is a Critic network structure diagram in GAIL provided by an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of a random and deterministic interference strategy provided by an embodiment of the present invention;

[0053] Figure 8 This is a reward evaluation curve diagram of a deterministic interference strategy provided in an embodiment of the present invention;

[0054] Figure 9This is a reward evaluation curve diagram of a random interference strategy provided in an embodiment of the present invention;

[0055] Figure 10 This is a graph showing the change of strategy evaluation error as a function of the complexity of the interference strategy, provided by an embodiment of the present invention.

[0056] Figure 11 This is a graph showing how the strategy evaluation error changes with the number of online samples, as provided in an embodiment of the present invention. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0058] Example 1

[0059] Existing radar anti-jamming strategy design methods suffer from low sample efficiency when using reinforcement learning algorithms for online training, while supervised learning methods for jamming strategy perception exhibit compound errors in small sample sizes. This invention proposes a jamming strategy perception method based on generative adversarial imitation learning. Please refer to... Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an interference policy perception method based on generative adversarial imitation learning provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the process of online collection of interactive samples for training anti-interference strategies in virtual interaction, as provided in an embodiment of the present invention. The interference strategy perception method based on generative adversarial imitation learning provided by the present invention includes:

[0060] Step 1: Based on the turn-based Markov decision process, obtain the online sample set according to the interaction between the radar and the jammer. The online sample set is the collected interaction data between the radar and the jammer, which reflects the jammer's decision-making process.

[0061] Specifically, the radar is used as an agent in a turn-based Markov decision process to select actions based on the current strategy and interact with the jammer to collect interaction sample information.

[0062] In one specific embodiment, step 1 includes:

[0063] Step 1.1: Initialize the radar's guidance strategy.

[0064] Here, the radar's guidance strategy is that, when the radar and the jammer are in confrontation, the radar needs to formulate a guidance strategy to interact with the jammer because the jamming strategy is unknown.

[0065] When faced with unknown jamming strategies, the radar needs to transmit induced signals to obtain samples for the jammer to make decisions. In a time-varying jamming environment, assuming the radar transmits H pulses of induced signals, where the transmission strategy for the h-th pulse is... It is defined as a given state s at time h. h Below, radar selects action a h The probability of [the jamming strategy]. Generally speaking, when faced with an unknown jamming strategy, radar can choose a uniform random sampling strategy to collect samples that better reflect the complete jamming strategy. Wherein, s h For the environmental information sensed by radar, including historical radar and jamming action information, a h This is real radar action information.

[0066] Step 1.2: Based on the round-based Markov decision process, the radar is modeled as an intelligent agent in the round-based Markov decision process, and the jammer is the environment. The radar observes a preset number of online samples by interacting with the jammer, and the online sample set is composed of all observed online samples.

[0067] Here, based on a turn-based Markov decision process, the radar is modeled as an agent in the turn-based Markov decision process, and the jammer is the environment. When the radar induces the jammer to make a decision, the strategy is the induction strategy formulated in step 1.1.

[0068] This invention assumes that the jammer operates in a time-division multiplexing mode. Furthermore, the jammer is capable of synchronously intercepting or jamming each sub-pulse of the radar, and one radar pulse consists of K sub-pulses. The action of the k-th sub-pulse of the h-th radar pulse is represented as follows: So, the radar's action a in a pulse h Represented as 1≤k≤K. Correspondingly, the action of the jammer in the k-th sub-pulse of the h-th radar pulse is expressed as: When the jammer is operating in reconnaissance mode, When the value is α, and the machine is in interference mode, The value is taken as the center frequency of the interference signal. Therefore, the action of the jammer at time h is expressed as:

[0069] This invention views radar as an observer collecting information on jammer actions, and the unknown jamming environment as an expert strategy to be perceived. Radar can observe a certain number of online samples by interacting with the jammer. (Online sample set) The dataset contains collected interaction data between the radar and the jammer, representing a sample of the jammer's decision-making processes.

[0070] Step 2: Input the online samples from the online sample set into the interference policy network, first update the parameters of the discriminator network, and then update the parameters of the generator network based on the reward output by the discriminator network. Obtain the generator network and discriminator network that meet the preset requirements to obtain the trained interference policy network, which includes the generator network and the discriminator network. The preset requirement is to achieve Nash equilibrium.

[0071] In one specific embodiment, step 2 includes:

[0072] Step 2.1: Sample the online sample set and input the sampled online samples into the interference policy network.

[0073] Specifically, firstly, the learning rate, generator network, and discriminator network are set, and the step size is updated; then, the radar and the learned jamming strategy interact to generate an online sample set for sampling, and the sampled state-action pairs (s, b) (i.e., online samples) are mapped to the sample set. middle, Online samples used to train the interference strategy network, where radar is sampled using a uniform random sampling method.

[0074] Step 2.2: Update the parameters ω of the discriminator network using gradient descent. i →ω i+1 .

[0075] Here, to alleviate the compound error problem caused by insufficient samples when perceiving interference policies, this invention uses generative adversarial learning (GAP) in imitation learning to perceive interference policies. Compared with the behavior cloning algorithm in imitation learning, it can alleviate the compound error problem caused by insufficient samples. Furthermore, for random interference policies, GAP can more accurately imitate the probability distribution of the interference policy. For the interference policy network... Its objective function can be expressed as:

[0076]

[0077] Represented as the radar guidance strategy at time h. Distribution of jammer state actions in the jamming strategy M. Let c(s) be the radar's guidance strategy at time h. h ,b h Let be the given state sh at time h and the action b of the jammer. h The reward output by the discriminator network under certain conditions. In fact, the above optimization objective applies to the online sample set. The optimization objective follows the optimization problem as follows:

[0078]

[0079] The principle of solving the above optimization problem is as follows: Figure 3 As shown, the discriminator network is a binary classifier whose output represents the probability that the input data belongs to an expert example. Its goal is to maximize the probability from the expert example and minimize the output probability of the data sampled by the generator. The generator network is a reinforcement learning process in which the output of the discriminator network is used as the reward for the generator network, and its goal is to maximize the probability of its sampled data.

[0080] Therefore, in this embodiment, the gradient corresponding to the gradient descent method is:

[0081]

[0082] Among them, c ω (s,b) represents the reward output by the discriminator network, where s is the radar's state information and b is the jammer's action. Let the gradient of the reward output by the discriminator network be the gradient of the discriminator network parameters ω. This is a mean estimate of the gradient of the reward value of a state-action pair (s,b) in a real sample with respect to the discriminator network parameters ω. This is the mean estimate of the gradient of the discriminator network parameter ω to the reward value of the state-action pair (s,b) output by the generator after passing through the discriminator.

[0083] In this embodiment, the discriminator network is referred to as follows: Figure 4 The state and actions of the jammer are input into the network. The network structure is a fully connected network with 256 hidden units in each layer. The activation function is the leaky ReLU function. The output unit is processed by the sigmoid function, and the output is the probability that the input state-action pair belongs to the real jammer's state-action pair.

[0084] Step 2.3: Calculate the reward c output by the discriminator network. ω (s,b) is considered as the cost function. The generator network is subjected to gradient descent using the TRPO (TrustRegion Policy Optimization) algorithm to update the parameters of the generator network.

[0085] Step 2.4: Repeat steps 2.1 to 2.3 until the generator network and the discriminator network reach Nash equilibrium, and obtain the trained interference policy network.

[0086] Here, when the discriminator network achieves its optimal goal, that is, when the generator network and the discriminator network reach Nash equilibrium, the resulting interference policy network composed of the generator network and the discriminator network is the trained interference policy network. The optimal discriminator is represented as follows:

[0087]

[0088] Among them, c * (s h ,b h ) represents the reward output by the optimal discriminator network, s h b represents the radar's state information at time h. h For the action of the jammer at time h, Radar guidance strategy at time h In interference strategy M θ Distribution of state actions of the jammer in the interaction between the two. Radar guidance strategy at time h The state-action distribution of the jammers interacting in jamming strategy M, M θ M represents the interference policy output by the interference policy network, where M is the actual interference policy. Let h be the radar's guidance strategy at time h.

[0089] It can be deduced that the objective of generative adversarial imitation learning is to minimize the distribution error of the state-action pair (s,b) between the real perturbing policy and the imitation learning policy. Imitation learning represented by JS divergence is as follows:

[0090]

[0091] Therefore, the essence of adopting generative adversarial imitation learning to perceive jamming strategies is to minimize the state-action distribution of online samples collected by radar and the state-action distribution output by the perceived jamming strategy network.

[0092] This invention collects online interaction samples using a radar-induced jammer. The online samples and samples sampled by the generator are fed into a discriminator network for training. When a Nash equilibrium is reached, the jamming strategy is perceived. The method proposed in this invention can effectively perceive both deterministic and random jamming strategies. After perceiving the jamming strategy, the PPO algorithm is then used to virtually interact and train an anti-jamming strategy.

[0093] Step 3: Based on the expected value of the reward corresponding to the interference policy output by the trained interference policy network, obtain the policy evaluation error between the interference policy and the real interference policy, and obtain the interference policy for training the anti-interference policy network based on the policy evaluation error.

[0094] Specifically, the policy evaluation error is obtained based on the expected value of the reward corresponding to the unknown real interference policy and the expected value of the reward corresponding to the interference policy output by the trained interference policy network. The interference policy used to train the anti-interference policy network is obtained based on the policy evaluation error.

[0095] In one specific embodiment, step 3 may include:

[0096] Step 3.1: Obtain the unknown real interference policy M, and derive the interference policy M based on the trained interference policy network. θ .

[0097] Step 3.2: Obtain the expected value of the reward based on the unknown real interference strategy M. According to interference strategy M θ Expected value of receiving a reward

[0098] Here, the expected value of the reward corresponding to the unknown real interference strategy is expressed as:

[0099]

[0100] Here, the expected value of the reward corresponding to the interference policy output by the trained interference policy network is expressed as:

[0101]

[0102] Among them, a h For the radar action at time h, s h Let r be the radar state information at time h, 1≤h≤H. h (s h ,a h Let ρh be the reward of the radar pulse signal at time h, ρ0 be the initial state distribution, and π be the initial state distribution. h (·|s h M represents the radar's guidance strategy at time h. h (·|s h ,a h The jamming strategy of the jammer at time h; the reward of the radar pulse signal at time h is equal to the signal-to-interference-plus-noise ratio (SIR) of the radar pulse signal at time h, which is expressed as:

[0103]

[0104] Among them, P T For radar transmission power, G T For radar antenna gain, Let be the wavelength and carrier frequency of the signal transmitted by the radar at the k-th sub-pulse of the h-th pulse, respectively. σ is the center frequency of the corresponding interference signal. RCS R is the radar cross-section of the target. d This refers to the distance between the radar and the target. For receiver noise variance, This represents the noise variance of the interference signal.

[0105]

[0106] in, This is an indication function that characterizes whether the radar signal has been successfully jammed.

[0107] Step 3.3: Expected Value Based on Rewards And the expected value of rewards The policy evaluation error is obtained, and the policy evaluation error is:

[0108] Generally, the accuracy of a jamming environment model can be measured using mean squared error (MSE). Given an unknown real jamming environment M and a jamming environment perceived by a behavioral clone, the accuracy of the learned environment can be evaluated using MSE. This method of measuring the jamming environment using MSE can reflect the accuracy of the learned jamming environment, but its accuracy still needs further improvement. Therefore, to further evaluate the performance differences of radar strategies in perceived jamming environments, the accuracy of the jamming environment model in this embodiment is measured using the policy evaluation error between environment models. Given an unknown real jamming strategy M and a jamming strategy perceived by a generative adversarial learning algorithm, the policy evaluation error is... in Let Π be the expected reward of radar strategy Π in the anti-jamming environment M, which reflects the radar's anti-jamming performance.

[0109] The strategy evaluates the error to reflect the inherent differences in the interference environment that are subject to the signal-to-interference-plus-noise ratio reward of the radar received echo signal, and can more accurately evaluate the learned interference environment compared to mean square error.

[0110] Step 3.4: Determine whether the policy evaluation error remains constant. If not, proceed to step 2 until the policy evaluation error becomes constant. If so, use the current interference policy network as the trained interference policy network.

[0111] Step 4: Train the anti-interference strategy network based on the interference strategy obtained in Step 3 to obtain the trained anti-interference strategy network.

[0112] Specifically, based on the PPO algorithm, the interference strategy obtained in step 3 is used to train the anti-interference strategy network to obtain the trained anti-interference strategy network.

[0113] For reinforcement learning tasks, it is impractical and unsafe to learn the optimal anti-jamming strategy online from scratch due to a lack of understanding of the jamming strategy. After sensing the jamming strategy, the radar can quickly solve for the anti-jamming strategy through offline interaction. Based on the jamming strategy learned using the generative adversarial imitation learning algorithm in step 3, the radar and the sensed jamming strategy perform virtual trial-and-error interaction to solve for the optimal anti-jamming strategy. This invention uses the PPO algorithm in reinforcement learning to solve for the optimal anti-jamming strategy. The optimization objective is:

[0114]

[0115] Furthermore, the parameters of the radar anti-jamming strategy network are optimized using the PPO algorithm in reinforcement learning, mainly adopting the Actor-Critic framework, which includes the Actor network and the Critic network.

[0116] See the Actor network structure. Figure 5 The state data is input into a fully connected network with 256 hidden units, where it is activated using the tanh function. This data is then passed to the next fully connected layer, which has the same structure as the first layer and uses the tanh function as its activation function. Finally, the data is passed to the third fully connected layer, where the output unit represents the total number of radar actions. The softmax function is used as the activation function to output the probability of each jamming action.

[0117] See the Critic network structure. Figure 6 The state data is input into the network, which is a fully connected network with 16 hidden units in each layer. The activation function is the ReLU function, and the output unit is 1. The output is a state value function that outputs state data and action data.

[0118] Step 5: Use the anti-jamming strategy output by the trained anti-jamming strategy network to counter real-world jamming strategies online. This is the anti-jamming strategy Π obtained in Step 4. θ Used to counter real-world interference strategies online.

[0119] The following simulation experiments further verify and illustrate the effectiveness of the invention.

[0120] (1) Simulation conditions:

[0121] The radar has a transmit power of 30KW, an antenna gain of 32dB, a radar subpulse bandwidth of 2MHz, a distance of 100km between the radar and the jammer, and an antenna gain of 3dB for the jammer.

[0122] The FA radar transmits pulse signals, each pulse containing four sub-pulses. Three carrier frequencies are selectable: f1 = 10 GHz, f2 = 10.1 GHz, and f3 = 10.2 GHz. The jammer operates in two possible modes: reconnaissance mode and targeted jamming signal transmission mode. In reconnaissance mode, the jammer acquires the radar's carrier frequencies; in jamming mode, it transmits narrowband noise jamming signals according to its strategy. This invention assumes that the jammer synchronously intercepts or jams the radar signal. Therefore, for each sub-pulse of the radar, the jammer can choose to intercept or jam. For ease of representation, this invention specifies the jammer's actions for each sub-pulse as representable jamming rules, including deterministic jamming rules and random jamming rules:

[0123] J0: The jammer is operating in reconnaissance mode. And radar carrier frequency Intercepted by jamming equipment;

[0124] J1: The jammer interferes with the previously intercepted radar carrier frequency;

[0125] J2: The jammer interferes with the radar carrier frequency that appears the least frequently among all intercepted historical frequencies;

[0126] J3: The jammer interferes with the radar carrier frequency that appears most frequently among all intercepted historical frequencies;

[0127] J4: The jammer interferes with the radar carrier frequency that appears second to last in all intercepted historical frequencies;

[0128] J5: The jammer executes jamming rules J0 and J1 with a probability of 0.5 respectively;

[0129] J0, J1, J2, J3, and J4 are deterministic rules, and J5 is a random rule. Figure 7 In this invention, different rules are used to construct the jamming strategy for each pulse. To verify the jamming strategy perception method based on generative adversarial learning proposed in this invention, several realistic jamming strategies are assumed based on the above-defined jamming rules, see [link to relevant documentation]. Figure 7 As shown, the randomness strategy refers to the interference rules comprising the interference strategy including J5. The interference strategy considered in this invention is time-varying in each pulse, with a time-varying length of H. For example, when H = 2, the jammer's strategy is that every two sub-pulses constitute one round. As H increases, the time within one round of the interference strategy becomes longer, and the interference strategy becomes more complex.

[0130] Figure 8 Given 1000 sub-pulses in an online interactive sample under a deterministic interference strategy, a generative adversarial imitation learning algorithm is used to perceive the reward curves of the interference environment and the real interference environment.Figure 8 In the diagram, (a) and (b) are the reward evaluation curves for H=1 and H=2, respectively. Where M and Π represents the real interference environment and the interference environment perceived using a generative adversarial imitation learning algorithm. * and Π GAIL The curves R(Π) represent the optimal anti-interference strategies trained in real-world interference environments and interference environments perceived by the GAIL algorithm, respectively. * Let M represent the reward evaluation curve of the anti-interference strategy Π over 32 pulses in a real interference environment M. For the anti-interference strategy learning task, the purpose of this invention is to provide an anti-interference strategy Π trained in a perceived interference environment. GAIL To counteract the real interference environment M, which is labeled R(Π) in the diagram. GAIL The curve of M). From Figure 8 It can be seen that when H=1 and H=2, the three curves eventually converge to the same signal-to-interference-plus-noise ratio (SIR / NNR) value, which means that the perceived interference environment can completely replace the real interference environment.

[0131] Figure 9 Given 10,000 online interaction samples of sub-pulses under a random interference strategy, the reward evaluation curve of the interference environment perceived by the generative adversarial imitation learning algorithm is obtained. Figure 9 In (a) of the diagram, when H = 1, the three curves eventually converge to the same reward value. Figure 9 In (b) of the diagram, when H=2, as the randomness of the jamming strategy increases, the trained radar strategy will inevitably have a probability of being successfully jammed. Therefore, the variance of the curve is not 0, but the mean and variance of the three curves are similar. Thus, when the jammer adopts a random strategy, the GAIL algorithm can accurately perceive the real jamming strategy.

[0132] Figure 10 To illustrate the policy evaluation error curve for deterministic interference strategies, given online interaction data of 1000 sub-pulses, as the complexity H of the interference strategy increases, the curve represents the average result of 20 random seeds. Experimental results show that the policy evaluation error increases with the complexity H of the interference strategy. After perceiving an unknown interference environment, the purpose of this invention is to train an anti-interference strategy through virtual interaction to counter real interference strategies online. The table below shows the reward values ​​for training in a virtual environment and evaluating in a real environment under different interference strategy complexities. The table shows that as the complexity H of the interference strategy increases, the error increases. GAIL In M GAIL The reward is very high, but when facing a real interference environment M, the reward loss becomes larger, for example, when H=8 and H=12, R(Π) GAILThe reward loss (M) is very significant, and the loss increases as H increases. This loss can be reduced by increasing the number of online samples.

[0133]

[0134] Figure 11 The figure shows the change in the policy evaluation error curve as the number of collected online samples increases under a deterministic interference policy with a policy complexity H=8. The horizontal axis represents the number of collected sub-pulse interaction samples, and the vertical axis represents the error value. It can be seen that the policy evaluation error gradually decreases as the number of samples increases.

[0135] 1. This invention addresses the problem of rapid perception of jamming strategies during the confrontation between FA radar and targeting jammers. It proposes a jamming strategy perception method based on generative adversarial imitation learning. Compared with supervised learning methods, this method can alleviate the problem of compound errors in the perception process. After the radar perceives the jamming strategy, it learns anti-jamming strategies through virtual interaction, thereby countering real jamming strategies online.

[0136] 2. This invention models the jamming strategy as an environment in a turn-based Markov decision process, with the FA radar as the agent and the jammer adopting deterministic or random jamming strategies that are time-varying in units of pulses.

[0137] 3. This invention collects online interactive samples through a radar-induced jammer. The online samples and samples sampled by the generator are fed into the discriminator network for training. When Nash equilibrium is reached, the jamming strategy is perceived. The method proposed in this invention can effectively perceive both deterministic and random jamming strategies. After perceiving the jamming strategy, the PPO algorithm is then used to virtually interact and train the anti-jamming strategy.

[0138] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0139] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0140] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for perceiving interference policies based on generative adversarial imitation learning, characterized in that, The interference strategy perception method includes: Step 1: Based on the turn-based Markov decision process, obtain an online sample set according to the interaction between the radar and the jammer. The online sample set is the collected interaction data between the radar and the jammer. Step 2: Input the online samples from the online sample set into the interference policy network, and first update the parameters of the discriminator network. Then, update the parameters of the generator network based on the reward output by the discriminator network to obtain the generator network and the discriminator network that meet the preset requirements, so as to obtain the trained interference policy network, which includes the generator network and the discriminator network. Step 3: Based on the expected value of the reward corresponding to the interference policy output by the trained interference policy network, obtain the policy evaluation error between the interference policy and the real interference policy, and obtain the interference policy for training the anti-interference policy network based on the policy evaluation error. Step 3 includes: Step 3.1: Obtain the unknown real interference strategy And based on the trained interference policy network, the interference policy is obtained. ; Step 3.2: Based on the unknown real interference strategy Expected value of receiving a reward According to the interference strategy Expected value of receiving a reward ; Step 3.3: Expected Value Based on Rewards And the expected value of rewards The policy evaluation error is obtained, and the policy evaluation error is: ; Step 3.4: Determine whether the policy evaluation error is constant. If not, proceed to step 2 until the policy evaluation error is constant. If so, use the output of the currently trained interference policy network as the interference policy for training the anti-interference policy network. Step 4: Train the anti-interference strategy network based on the interference strategy obtained in Step 3 to obtain the trained anti-interference strategy network. Step 5: Use the anti-interference strategy output by the trained anti-interference strategy network to counter real interference strategies online.

2. The interference policy perception method based on generative adversarial imitation learning according to claim 1, characterized in that, Step 1 includes: Step 1.1: Initialize the radar's guidance strategy; Step 1.2: Based on the round-based Markov decision process, the radar is modeled as an intelligent agent in the round-based Markov decision process, and the jammer is the environment. The radar observes a preset number of online samples by interacting with the jammer, and the online sample set is composed of all observed online samples.

3. The interference policy perception method based on generative adversarial imitation learning according to claim 1, characterized in that, Step 2 includes: Step 2.1: Sample the online sample set and input the sampled online samples into the interference strategy network; Step 2.2: Update the parameters of the discriminator network using gradient descent. Step 2.3: Treat the reward output by the discriminator network as a cost function, and use the TRPO algorithm to perform gradient descent on the generator network to update the parameters of the generator network; Step 2.4: Repeat steps 2.1 to 2.3 until the generator network and the discriminator network reach Nash equilibrium, thus obtaining the trained interference policy network.

4. The interference policy perception method based on generative adversarial imitation learning according to claim 3, characterized in that, The gradient corresponding to gradient descent is: in, The reward output by the discriminator network. This is the radar's status information. For the operation of the jammer, The reward output by the discriminator network is relative to the discriminator network parameters. gradient, State-action pairs in real samples The reward value output by the discriminator affects the discriminator network parameters. The mean estimate of the gradient, The state-action pair output by the generator The reward value output by the discriminator affects the discriminator network parameters. Mean estimate of the gradient.

5. The interference policy perception method based on generative adversarial imitation learning according to claim 3, characterized in that, The optimal discriminator network in the trained interference policy network is: in, The reward output by the optimal discriminator network. For the first h Real-time radar status information, For the first h Constantly interfering with the machine's operation For the first h Time-based radar guidance strategy In interference strategy Distribution of state actions of the jammer in the interaction between the two. For the first h Time-based radar guidance strategy In interference strategy Distribution of state actions of the jammer in the interaction between the two. The interference strategy output by the interference strategy network. For real interference strategies, For the first h The guidance strategy of real-time radar.

6. The interference policy perception method based on generative adversarial imitation learning according to claim 1, characterized in that, The expected value of the reward corresponding to the unknown real interference strategy is expressed as: The expected value of the reward corresponding to the interference policy output by the trained interference policy network is expressed as: in, For the first h The radar's movements are constantly monitored. For the first h The status information of the radar at any time, 1≤ h ≤ H , For the first h The reward of the radar pulse signal at any time. This represents the initial state distribution of the radar. For the first h The guidance strategy of real-time radar For the first h The jamming strategy of the time jammer; the first h The reward for the radar pulse signal at a given moment is equal to the first... h The signal-to-interference-plus-noise ratio of the radar pulse signal at any given time.

7. The interference policy perception method based on generative adversarial imitation learning according to claim 1, characterized in that, Step 4 includes: Based on the PPO algorithm, the interference strategy obtained in step 3 is used to train the anti-interference strategy network to obtain the trained anti-interference strategy network.

8. The interference policy perception method based on generative adversarial imitation learning according to claim 7, characterized in that, The objective function corresponding to the anti-interference strategy network is: in, The expected value of the reward corresponding to the interference policy output by the trained interference policy network.

Images