An interference strategy perception method based on behavior cloning

Abstract

The application discloses an interference strategy perception method based on behavior cloning, which comprises the following steps: obtaining the interference strategy of a jammer from the perspective of a radar; obtaining an online sample set according to the state information and action of the radar at different time and the interference strategy of the jammer; inputting the online sample in the online sample set into an interference strategy network to train the interference strategy network and obtain a trained interference strategy network; obtaining an interference strategy for training an anti-interference strategy network based on the expected value of a reward corresponding to an unknown real interference strategy and the expected value of a reward corresponding to the interference strategy output by the trained interference strategy network; training the anti-interference strategy network by using the interference strategy based on a PPO algorithm to obtain an anti-interference strategy output by the trained anti-interference strategy network; and using the anti-interference strategy to online counteract the real interference strategy. The application avoids trial-and-error learning in the online counteraction process.

Description

Technical Field

[0001] This invention belongs to the field of radar technology, specifically relating to a method for sensing jamming strategies based on behavioral cloning. Background Technology

[0002] In modern radar and jamming countermeasures, the anti-jamming capability of radar is increasingly emphasized because jammers can employ advanced jamming measures to interfere with radar's use of the electromagnetic spectrum. Furthermore, more intelligent jammers can employ different jamming strategies and utilize historical radar information to analyze radar anti-jamming strategies. Therefore, research on intelligent radar anti-jamming technology is of great significance.

[0003] Frequency-agile (FA) radar is widely used due to its strong anti-jamming capability and low probability of intercept. However, frequency agility also causes random phase fluctuations, disrupting the consistency between pulse radar echoes and making coherent accumulation difficult. Although some coherent accumulation methods exist to compensate for the phase fluctuations caused by frequency agility, traditional random frequency hopping or stepped frequency hopping is not the optimal operating mode for radar. This is because jammers can formulate simple strategies to successfully jam radar systems. Faced with complex and variable electromagnetic environments, researchers utilize the adaptive and learning capabilities of reinforcement learning to enable radar to adaptively learn an optimal anti-jamming strategy through interaction with the environment. These methods construct the frequency-agile radar and jammer into a Markov decision process, treating the radar as an agent and the jammer as the environment through which the agent interacts by observing its state and collecting rewards. The reward can be the signal-to-interference-plus-noise ratio (SINR) or detection probability, and the number of carrier frequencies can be regarded as the action. However, existing methods mainly employ model-free reinforcement learning algorithms to learn the optimal anti-jamming strategy through trial and error interaction with the environment, which typically requires a large number of samples. For example, the typical model-free reinforcement learning algorithm Double DQN requires millions of interaction samples in Atari games. In complex electromagnetic environments with large state and action spaces, the number of interaction samples required for online radar-jamming confrontation increases exponentially. Therefore, using model-free reinforcement learning to learn the optimal anti-jamming strategy through trial and error with real jamming strategies is impractical. This method is only applicable when the jamming strategy is known, allowing for offline virtual interaction training of the anti-jamming strategy. When facing unknown jamming strategies and the radar lacks prior information about them, a crucial step to maintain radar advantage is to perceive the jamming strategy. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a method for perceiving interference strategies based on behavioral cloning. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] A method for perceiving interference policies based on behavioral cloning, the method comprising:

[0006] Step 1: Initialize the radar's guidance strategy;

[0007] Step 2: Obtain the radar's actions at each moment and, from the radar's perspective, derive the jamming strategy of the jammer;

[0008] Step 3: Obtain an online sample set based on the radar's status information and actions at different times, and the jamming strategy of the jammer;

[0009] Step 4: Input the online samples from the online sample set into the interference policy network to train the interference policy network and obtain the trained interference policy network.

[0010] Step 5: Based on the expected value of the reward corresponding to the unknown real interference policy and the expected value of the reward corresponding to the interference policy output by the trained interference policy network, obtain the interference policy used to train the anti-interference policy network.

[0011] Step 6: Based on the PPO algorithm, train the anti-interference strategy network using the interference strategy obtained in Step 5 to obtain the anti-interference strategy output by the trained anti-interference strategy network.

[0012] Step 7: Use the anti-interference strategy obtained in Step 6 to counter real interference strategies online.

[0013] In one embodiment of the present invention, the jamming strategy of the jammer is obtained from the radar perspective, including:

[0014] Treating the radar as an agent in a turn-based Markov decision process, it selects actions based on the current induction strategy and interacts with the jammer. According to the definition of state transition in a turn-based Markov decision process, the state transition represents the jammer's jamming strategy from the radar's perspective. Therefore, the jammer's jamming strategy at time h is expressed as:

[0015]

[0016] in, From the radar's perspective, the jamming strategy of the jammer at time h, b h For the action of the jammer at time h, a h For the radar action at time h, s h For the radar state information at time h, s h+1 This represents the radar's state information at time h+1.

[0017] In one embodiment of the present invention, the online sample set is represented as:

[0018]

[0019] in, For online sample sets, tr i For the i-th trajectory information collected by the radar, a h For the radar action at time h, s h Here, ρh represents the radar's state information at time h, where 1 ≤ h ≤ H, and ρ0 represents the initial state distribution. For the initialization radar guidance strategy at time h, M h (·|s h ,a h ) represents the jamming strategy of the jammer at time h, and m represents the total number of trajectories.

[0020] In one embodiment of the present invention, step 4 includes:

[0021] Step 4.1: Collect the online samples (s) from the online sample set. h ,a h ,s h+1 The input is fed into the interference policy network, and the output is the probability of each interference action, where (s) h ,a h ) represents network input, s h+1 Labels for interference strategy networks;

[0022] Step 4.2: Update the parameters of the interference policy network using stochastic gradient descent.

[0023] Step 4.3: Repeat steps 4.1 and 4.2 until convergence, and obtain the trained interference policy network.

[0024] In one embodiment of the present invention, step 4.1 includes:

[0025] The online samples (s) in the online sample set h ,a h ,s h+1 The input is fed into a fully connected network and activated using the tanh function. Then it is passed to the next fully connected network layer and activated again using the tanh function. Finally, it is passed to the third fully connected network layer. The output unit is the total number of actions of the jammer. The softmax function is used as the activation function to output the probability of each jamming action.

[0026] In one embodiment of the present invention, the gradient corresponding to the stochastic gradient descent method is:

[0027]

[0028] in, This is a mean estimate of the gradient of the likelihood function with respect to the parameters θ of the neural network under finite sample conditions. M is the gradient of the objective function with respect to the network parameters θ of the interference policy. θ (s h+1|s h ,a h ) is the mathematical expression for the interference policy network, where (s h ,a h ) represents the network input, s h+1 This is the output of the network.

[0029] In one embodiment of the present invention, step 5 includes:

[0030] Step 5.1: Obtain the unknown real interference policy M, and derive the interference policy M based on the trained interference policy network. θ ;

[0031] Step 5.2: Obtain the expected value of the reward based on the unknown real interference strategy M. According to interference strategy M θ Expected value of receiving a reward

[0032] Step 5.3: Expected Value Based on Rewards and the expected value of rewards The evaluation error is obtained, and the evaluation error is...

[0033] Step 5.4: Determine whether the evaluation error remains constant. If not, proceed to step 4 until the evaluation error becomes constant. If so, use the output of the currently trained interference policy network as the interference policy for training the anti-interference policy network.

[0034] In one embodiment of the present invention, the expected value of the reward corresponding to the unknown real interference strategy is expressed as:

[0035]

[0036] The expected value of the reward corresponding to the interference policy output by the trained interference policy network is expressed as:

[0037]

[0038] Among them, a h For the radar action at time h, s h Let r be the radar state information at time h, 1≤h≤H. h (s h ,a h Let ρh be the reward of the radar pulse signal at time h, ρ0 be the initial state distribution, and π be the initial state distribution. h (·|s h M represents the radar's guidance strategy at time h. h (·|s h ,a hThe jamming strategy of the jammer at time h; the reward of the radar pulse signal at time h is equal to the signal-to-interference-plus-noise ratio (SINR) of the radar pulse signal at time h, which is expressed as:

[0039]

[0040] Among them, P T For radar transmission power, G T For radar antenna gain, Let be the wavelength and carrier frequency of the signal transmitted by the radar at the k-th sub-pulse of the h-th pulse, respectively. σ is the center frequency of the corresponding interference signal. RCS R is the radar cross-section of the target. d This refers to the distance between the radar and the target. For receiver noise variance, This represents the noise variance of the interference signal. The indication function characterizing whether the radar signal has been successfully jammed is expressed as:

[0041]

[0042] In one embodiment of the present invention, the anti-interference strategy network includes an Actor network and a Critic network.

[0043] In one embodiment of the present invention, the objective function corresponding to the anti-interference strategy network is:

[0044]

[0045] in, The expected value of the reward corresponding to the interference policy output by the trained interference policy network.

[0046] The beneficial effects of this invention are:

[0047] 1. This invention addresses the problem of excessively large online interactive samples required for training anti-jamming strategies in FA radar and narrowband targeting jamming countermeasures, leading to exploration security risks. It proposes a jamming strategy perception scheme. After sensing a jamming strategy, the radar learns its anti-jamming strategy through virtual interaction, thus avoiding trial-and-error learning during online countermeasures. Specifically, different jamming strategies create different jamming environments; this invention models the jamming strategy as a jamming environment. Therefore, the jamming strategy perception problem is transformed into a jamming environment learning problem.

[0048] 2. Faced with unknown time-varying jamming strategies, existing methods based on discounted Markov decision processes cannot accurately model the time-varying nature of jamming strategies. This invention models the confrontation process between the FA radar and the jammer as a round-based Markov decision process, allowing the jammer to formulate time-varying jamming strategies according to the changes in pulses.

[0049] 3. This invention employs an online sample collection method, an offline perception method for interference strategies, and a method for training anti-interference strategies. Based on an imitation learning architecture, this invention proposes a behavior cloning-based interference strategy perception method. This method collects online observation data from the interference machine's decision-making process and perceives interference strategies offline. Then, the PPO algorithm is used to train an anti-interference strategy. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating an interference strategy perception method based on behavior cloning provided in an embodiment of the present invention.

[0051] Figure 2 This is a Markov decision process diagram for radar and jamming countermeasures provided in an embodiment of the present invention;

[0052] Figure 3 This is a structural diagram of an interference strategy network provided in an embodiment of the present invention;

[0053] Figure 4 This is a structural diagram of an Actor network provided in an embodiment of the present invention;

[0054] Figure 5 This is a structural diagram of a Critic network provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of an interference strategy provided by an embodiment of the present invention;

[0056] Figure 7 This is a graph showing the MSE curves of the real interference environment and the perceived interference environment under different interference strategy lengths H during the training process, provided by an embodiment of the present invention.

[0057] Figure 8 This is the signal-to-interference-plus-noise ratio (SIR) evaluation curve of radar echo provided in the embodiments of the present invention;

[0058] Figure 9 This is a graph showing the change in policy evaluation error as the interference policy length H changes, provided by an embodiment of the present invention.

[0059] Figure 10 This is a strategy evaluation error curve diagram for different sample numbers when H=8, provided by an embodiment of the present invention. Detailed Implementation

[0060] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0061] Example 1

[0062] Existing radar anti-jamming strategy design methods, which rely on model-free reinforcement learning algorithms for online training, suffer from low efficiency with interactive samples and insecurity due to a lack of understanding of the jamming strategy. The purpose of this invention is to use the behavior cloning algorithm from imitation learning as a framework, treating the radar as a proxy in deep reinforcement learning and the jamming machine as the jamming environment. This aims to solve for the optimal anti-jamming strategy for application in the anti-jamming field.

[0063] For the reasons mentioned above, please refer to Figure 1 , Figure 1 This is a flowchart illustrating a behavior cloning-based interference strategy perception method provided in an embodiment of the present invention. The present invention provides a behavior cloning-based interference strategy perception method, which includes:

[0064] Step 1: Initialize the radar's guidance strategy π D .

[0065] Here, the radar's guidance strategy is employed when engaging a jammer. Since the jamming strategy is unknown, the radar needs to develop a guidance strategy to interact with the jammer. The radar can generally choose a uniform sampling strategy or a deceptive guidance strategy to interact with the jammer.

[0066] Step 2: Obtain the radar's actions at each moment and, from the radar's perspective, obtain the jamming strategy of the jammer.

[0067] Specifically, this invention models the radar-jamming confrontation process as a turn-based Markov decision process, with the radar acting as the agent in the turn-based Markov decision process and the jammer as the environment. This turn-based Markov decision process allows the jammer to adopt time-varying jamming strategies. Different jamming strategies lead to different environment transition probabilities.

[0068] The radar acts as a deep reinforcement learning agent, interacting with the jammer according to the induction strategy in step 1.

[0069] To avoid self-excitation response from the jamming system, jammers typically operate in a time-division multiplexing (TDM) mode. Therefore, jammers operate in two modes: reconnaissance mode and jamming mode. Assume the jammer can synchronously intercept and jam every sub-pulse of the radar, and that one radar pulse consists of K sub-pulses. The radar's action in each sub-pulse is represented as... So, the radar's action a in a pulse h Represented as:

[0070]

[0071] Meanwhile, the action of the jammer at time h can be expressed as:

[0072]

[0073] in, Let α represent the action of the jammer on the k-th sub-pulse of the h-th pulse of the radar. When the jammer is operating in reconnaissance mode, α represents the action of the jammer, i.e., b. h Let α be the value of the jammer. When the jammer is operating in targeted jamming mode, the jammer's action is the center frequency of the targeted jamming signal. Assume the jammer can select a set of center frequencies as follows: at this time

[0074] Based on the above description of the jammer and radar operations, please refer to [link / reference needed]. Figure 2 In the h-th pulse, the radar's guidance strategy is mathematically represented as π. h (a h |s h ), which is represented as in a given state s h At that time, the radar took action a h The induced probability. The jamming strategy adopted by the jammer itself can be represented mathematically as: in, The jammer senses the radar's state information. Let be the action information perceived by the jammer regarding the h-th radar pulse. Since different jamming strategies lead to different environmental transition probabilities, the radar cannot accurately know the specific prior information upon which the jammer relies to take action. Therefore, the safest way for the radar to perceive the jamming strategy is to infer the jammer's strategy using as much complete state information as possible. Mathematically, this can be expressed as: Among them, s h For the environmental information sensed by radar, including historical radar and jamming action information, a h This represents the actual radar's action information. According to the definition of state transition in a turn-based Markov decision process, the state transition is the jammer's strategy as represented from the radar's perspective, and can be expressed as:

[0075]

[0076] That is, the interference policy is the state transition function in a turn-based Markov decision process. Therefore, the interference policy perception problem described in this invention can be transformed into an interference environment learning problem under a turn-based Markov decision process.

[0077] Step 3: Obtain an online sample set based on the radar's status information and actions at different times, and the jamming strategy of the jammer.

[0078] Specifically, the radar's induction strategy described in the above steps is used to collect interaction samples in a real jamming environment, and the jamming samples are stored in an online sample set. The online sample set, which includes radar status and action information, is represented as follows:

[0079]

[0080] in, For online sample sets, tr i For the i-th trajectory information collected by the radar, a h For the radar action at time h, s h Here, ρh represents the radar's state information at time h, where 1 ≤ h ≤ H, and ρ0 represents the initial state distribution. Let m be the initialization radar guidance strategy at time h, where m is the total number of trajectories.

[0081] Step 4: Input the online samples from the online sample set into the interference policy network to train the interference policy network and obtain the trained interference policy network.

[0082] Here, through the above modeling, the present invention transforms the interference policy perception problem into the interference environment recovery problem under a given online sample set, and models the interference environment recovery problem as a behavior cloning problem in imitation learning, which has the advantages of fast convergence speed and easy algorithm implementation.

[0083] In one specific embodiment, step 4 may specifically include:

[0084] Step 4.1: Collect the online samples (s) from the online sample set. h ,a h ,s h+1 The input is fed into the interference policy network, and the output is the probability of each interference action, where (s) h ,a h ) represents network input, s h+1 Labels for interference strategy networks.

[0085] Specifically, first, the learning rate and the number of network training iterations are set. Then, the online samples (s) in the online sample set are... h ,a h ,s h+1 The input is given to the interference policy network, specifically: please refer to [link to relevant documentation]. Figure 3 The online samples (s) in the online sample set h ,a h ,s h+1The input is fed into a fully connected network and activated using the tanh function. Then it is passed to the next fully connected network layer and activated again using the tanh function. Finally, it is passed to the third fully connected network layer. The output unit is the total number of actions of the jammer. The softmax function is used as the activation function to output the probability of each jamming action.

[0086] Step 4.2: Update the parameters of the interference policy network using stochastic gradient descent.

[0087] Here, the problem of perturbed policy awareness in turn-based Markov decision-making can also be considered as a problem of perturbed environment learning. Through the above modeling, this invention transforms the perturbed environment learning problem into a problem of perturbed environment recovery under a given online sample set. This problem can generally be viewed as an imitation learning task. This invention models the perturbed environment learning problem as a behavior cloning problem in imitation learning, which has the advantages of fast convergence speed and easy algorithm implementation. For a perturbed environment M... θ Its objective function can be expressed as:

[0088]

[0089] in, This indicates that at time h, strategy π D State-action distribution in the interfering environment M This represents the expected value, s. h+1 This represents the state at time h+1 after the jammer takes jamming action. In fact, the above optimization objective is based on the online sample set. The optimization objective follows the optimization problem as follows:

[0090]

[0091] Therefore, the gradient corresponding to stochastic gradient descent is:

[0092]

[0093] in, This is a mean estimate of the gradient of the likelihood function with respect to the parameters θ of the neural network under finite sample conditions. M is the gradient of the objective function with respect to the network parameters θ of the interference policy. θ (s h+1 |s h ,a h ) is the mathematical expression for the interference policy network, where (s h ,a h ) represents the network input, s h+1 This is the output of the network.

[0094] Step 4.3: Repeat steps 4.1 and 4.2 until convergence, and obtain the trained interference policy network.

[0095] Step 5: Based on the expected value of the reward corresponding to the unknown real interference policy and the expected value of the reward corresponding to the interference policy output by the trained interference policy network, obtain the interference policy used to train the anti-interference policy network.

[0096] In one specific embodiment, step 5 may include:

[0097] Step 5.1: Obtain the unknown real interference policy M, and derive the interference policy M based on the trained interference policy network. θ .

[0098] Step 5.2: Obtain the expected value of the reward based on the unknown real interference strategy M. According to interference strategy M θ Expected value of receiving a reward

[0099] Here, the expected value of the reward corresponding to the unknown real interference strategy is expressed as:

[0100]

[0101] Here, the expected value of the reward corresponding to the interference policy output by the trained interference policy network is expressed as:

[0102]

[0103] Among them, a h For the radar action at time h, s h Let r be the radar state information at time h, 1≤h≤H. h (s h ,a h Let ρh be the reward of the radar pulse signal at time h, ρ0 be the initial state distribution, and π be the initial state distribution. h (·|s h M represents the radar's guidance strategy at time h. h (·|s h ,a h The jamming strategy of the jammer at time h; the reward of the radar pulse signal at time h is equal to the signal-to-interference-plus-noise ratio (SINR) of the radar pulse signal at time h, which is expressed as:

[0104]

[0105] Among them, P T For radar transmission power, G T For radar antenna gain, Let be the wavelength and carrier frequency of the signal transmitted by the radar at the k-th sub-pulse of the h-th pulse, respectively. σ is the center frequency of the corresponding interference signal. RCS R is the radar cross-section of the target. d This refers to the distance between the radar and the target. For receiver noise variance, This represents the noise variance of the interference signal. The indication function characterizing whether the radar signal has been successfully jammed is expressed as:

[0106]

[0107] Step 5.3: Expected Value Based on Rewards and the expected value of rewards The evaluation error is obtained, and the evaluation error is...

[0108] Generally, the accuracy of a disturbance environment model can be measured using mean squared error distance; given an unknown real disturbance environment M and a disturbance environment perceived by a behavioral clone, the accuracy of the learned environment can be evaluated using mean squared error.

[0109]

[0110] The aforementioned mean square error method for measuring the interference environment can reflect the accuracy of the learned interference environment. To further evaluate the performance differences of radar strategies in perceived interference environments, this invention proposes a method that uses the strategy evaluation error between different environments to measure the learned interference environment. in π represents the expected reward of radar strategy π in an anti-jamming environment M, reflecting the radar's anti-jamming performance.

[0111] Therefore, the strategy evaluation error reflects the inherent differences in the interference environment that are subject to the signal-to-interference-plus-noise ratio reward of the radar received echo signal, and can more accurately evaluate the learned interference environment compared to the mean square error.

[0112] Step 5.4: Determine whether the evaluation error remains constant. If not, proceed to step 4 until the evaluation error becomes constant. If so, use the output of the currently trained interference policy network as the interference policy for training the anti-interference policy network.

[0113] Step 6: Based on the PPO algorithm, train the anti-interference strategy network using the interference strategy obtained in Step 5 to obtain the anti-interference strategy output by the trained anti-interference strategy network.

[0114] Specifically, for reinforcement learning tasks, it is impractical and unsafe to learn the optimal anti-jamming strategy online from scratch due to a lack of understanding of the jamming strategy. After perceiving the jamming strategy, the radar can quickly solve for the anti-jamming strategy through offline interaction. Based on the jamming environment learned using the behavior cloning algorithm in step 5, the radar learns anti-jamming strategies through virtual interaction, thereby coping with real jamming strategies in online scenarios. The optimization objective is:

[0115]

[0116] Furthermore, the parameters of the radar anti-jamming strategy network are optimized using the PPO algorithm in reinforcement learning, mainly adopting the Actor-Critic framework, which includes the Actor network and the Critic network.

[0117] See the Actor network structure. Figure 4 The state data is input into a fully connected network with 256 hidden units, where it is activated using the tanh function. This data is then passed to the next fully connected layer, which has the same structure as the first layer and uses the tanh function as its activation function. Finally, the data is passed to the third fully connected layer, where the output unit represents the total number of radar actions. The softmax function is used as the activation function to output the probability of each jamming action.

[0118] See Critic network structure Figure 5 The state data is input into a fully connected network with 256 hidden units, where it is activated using the tanh function. This data is then passed to the next fully connected layer, which has the same structure as the first layer and uses the tanh function as its activation function. Finally, the data is passed to the third fully connected layer, which outputs a 1-unit function representing the state value of the state and action data.

[0119] Step 7: Use the anti-interference strategy obtained in Step 6 to counter real interference strategies online.

[0120] That is, the anti-interference strategy π obtained in step 6 θ Used to counter real-world interference strategies online.

[0121] The following simulation experiments further verify and illustrate the effectiveness of the invention.

[0122] (1) Simulation conditions:

[0123] The radar has a transmit power of 30KW, an antenna gain of 32dB, a radar subpulse bandwidth of 2MHz, a distance of 100km between the radar and the jammer, and an antenna gain of 3dB for the jammer.

[0124] The FA radar transmits pulse signals, each pulse containing four sub-pulses. Three carrier frequencies are selectable: f1 = 10 GHz, f2 = 10.1 GHz, and f3 = 10.2 GHz. The jammer operates in two possible modes: reconnaissance mode and targeted jamming signal transmission mode. In reconnaissance mode, the jammer acquires the radar's carrier frequencies; in jamming mode, it transmits narrowband noise jamming signals according to its strategy. This invention assumes that the jammer synchronously intercepts or jams the radar signal. Therefore, for each sub-pulse of the radar, the jammer can choose to intercept or jam. For ease of representation, this invention specifies the jammer's actions for each sub-pulse as representable jamming rules:

[0125] J0: The jammer is operating in reconnaissance mode. And radar carrier frequency Intercepted by jamming equipment;

[0126] J1: The jammer interferes with the previously intercepted radar carrier frequency;

[0127] J2: The jammer interferes with the radar carrier frequency that appears the least frequently among all intercepted historical frequencies;

[0128] J3: The jammer interferes with the radar carrier frequency that appears most frequently among all intercepted historical frequencies;

[0129] J4: The jammer interferes with the radar carrier frequency that appears second to last in all intercepted historical frequencies;

[0130] To verify the interference strategy perception method based on behavior cloning proposed in this invention, several real interference strategies are assumed, see [link to relevant documentation]. Figure 6 As shown: Each of the four jamming rules is combined into one pulse jamming strategy. H represents the length of the jamming strategy, which can also be considered as the complexity of the jamming strategy. When H=1, the jamming strategy for each pulse is the same. When H=2, the jamming strategy for every two pulses is one round. Correspondingly, the radar's anti-jamming strategy is also one round for every two pulses. The larger H is, the more complex the jamming strategy.

[0131] (2) Simulation Experiment Content

[0132] Collect 1000 sub-pulse interaction data points between radar and jammer, and use the row cloning algorithm to detect jamming strategies. Figure 7The MSE curves for the real and perceived interference environments under different interference policy lengths H during training are shown in (a) and (b). As shown in (a) and (b), when H is small, the variance and mean of the converged curve are approximately 0, which means that the real interference policy is accurately perceived. However, as H increases, as shown in (c), (d), (e), and (f), the variance and mean of the MSE curves increase, indicating that the perceived interference policy is not accurate. More online sample data is needed to more accurately perceive the interference policy.

[0133] Figure 8 The signal-to-interference-plus-noise ratio (SIR) of the radar echo is given under the interaction data of 32 pulses between the radar and the jammer, where R(π) * ,) indicates training an anti-jamming strategy π in a real-world interference environment. * Reward evaluation curve at that time; This indicates training an anti-interference strategy π in a BC algorithm-perceived interference environment. BC Reward evaluation curve at time; R(π) BC ,) indicates that during training π BC During the process, the anti-interference strategy was implemented in the real interference environment M. BC The reward curves in the figure represent the training of anti-interference strategies in the perceived environment to counteract real interference environments as described in step 7 of Embodiment 1 of this invention. (a) and (b) represent the reward evaluation results when the interference strategy lengths H=1 and H=2, respectively. The three reward evaluation curves in the figure converge to the same signal-to-interference-plus-noise ratio (SNR) value, and the curve variance is 0, thus concluding that the perceived interference environment can completely replace the real interference environment.

[0134] Figure 9 The figure shows the variation of the policy evaluation error as the interference policy length H changes, given 1000 sub-pulse interaction data. It can be seen from the figure that the policy evaluation error gradually increases with the complexity of the interference policy. The table below shows the reward values ​​for different levels of interference policy complexity:

[0135]

[0136] As the complexity of the jamming strategy increases, π BC In the perceived interference environment The sample size has a high reward value; however, its reward loss against real-world interference environments increases with the value of H. This error can be mitigated by increasing the number of online samples to obtain the environmental impact assessment.

[0137] Figure 10 The graph shows the policy evaluation error curves for different sample numbers when H=8. The horizontal axis represents the number of collected sub-pulse interaction samples. As the number of samples increases, the policy evaluation error decreases and eventually converges to a smaller value.

[0138] In contrast to the low sample efficiency of reinforcement learning, imitation learning learns policies that match expert data directly from expert examples, resulting in higher sample efficiency. Among these, the behavior cloning algorithm, by directly learning the agent's actions and states, features fast convergence and high learning efficiency. Therefore, FA radar can accurately detect interference strategies using a small number of online samples through the behavior cloning algorithm.

[0139] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0140] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0141] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for perceiving interference strategies based on behavioral cloning, characterized in that, The interference strategy perception method includes: Step 1: Initialize the radar's guidance strategy; Step 2: Obtain the radar's actions at each moment and, from the radar's perspective, derive the jamming strategy of the jammer; Step 3: Obtain an online sample set based on the radar's status information and actions at different times, and the jamming strategy of the jammer; Step 4: Input the online samples from the online sample set into the interference policy network to train the interference policy network and obtain the trained interference policy network. Step 5: Based on the expected value of the reward corresponding to the unknown real interference policy and the expected value of the reward corresponding to the interference policy output by the trained interference policy network, obtain the interference policy used to train the anti-interference policy network. Step 6: Based on the PPO algorithm, train the anti-interference strategy network using the interference strategy obtained in Step 5 to obtain the anti-interference strategy output by the trained anti-interference strategy network. Step 7: Use the anti-interference strategy obtained in Step 6 to counter real interference strategies online; From the radar's perspective, the jamming strategy of the jammer is obtained, including: Treating the radar as an agent in a turn-based Markov decision process, it selects actions based on the current induction strategy and interacts with the jammer. According to the definition of state transition in a turn-based Markov decision process, the state transition represents the jammer's jamming strategy from the radar's perspective. Then, the... h The jamming strategy of the time jammer is expressed as: in, From the radar perspective, the first h The jamming strategy of the time-based jammer. For the first h Constantly interfering with the machine's operation For the first h The radar's movements are constantly monitored. For the first h Real-time radar status information, For the first h Radar status information at time +1.

2. The interference strategy perception method based on behavior cloning according to claim 1, characterized in that, The online sample set is represented as: in, For online sample sets, The first collected by radar Track information, For the first h The radar's movements are constantly monitored. For the first h The status information of the radar at any time, 1≤ h ≤ H , The initial state distribution, For the first h The radar's guidance strategy is initialized at specific times. For the first h The jamming strategy of the time-based jammer. The total number of trajectories.

3. The interference strategy perception method based on behavior cloning according to claim 2, characterized in that, Step 4 includes: Step 4.1: Add online samples from the online sample set. Input to the interference policy network, output is the next state The probability of , where, For network input, Labels for interference strategy networks; Step 4.2: Update the parameters of the interference policy network using stochastic gradient descent. Step 4.3: Repeat steps 4.1 and 4.2 until convergence, and obtain the trained interference policy network.

4. The interference strategy perception method based on behavior cloning according to claim 3, characterized in that, Step 4.1 includes: Online samples in the online sample set The input is fed into a fully connected network and activated using the tanh function. It is then passed to the next fully connected network layer and activated again using the tanh function. Finally, it is passed to the third fully connected network layer. The output unit is the total number of actions of the jammer. The softmax function is used as the activation function to output the probability of each jamming action.

5. The interference strategy perception method based on behavior cloning according to claim 4, characterized in that, The gradient corresponding to stochastic gradient descent is: in, For the likelihood function of neural network parameters under finite samples The mean estimate of the gradient, The objective function is the network parameters of the interference policy. gradient, Here is the mathematical expression for the interference policy network, where For network input, This is the output of the network.

6. The interference strategy perception method based on behavior cloning according to claim 5, characterized in that, Step 5 includes: Step 5.1: Obtain the unknown real interference strategy M And based on the trained interference policy network, the interference policy is obtained. ; Step 5.2: Based on the unknown real interference strategy Expected value of receiving a reward According to the interference strategy Expected value of receiving a reward ; Step 5.3: Expected Value Based on Rewards and the expected value of rewards The evaluation error is obtained, and the evaluation error is... ; Step 5.4: Determine whether the evaluation error remains constant. If not, proceed to step 4 until the evaluation error becomes constant. If so, use the output of the currently trained interference policy network as the interference policy for training the anti-interference policy network.

7. The interference strategy perception method based on behavior cloning according to claim 6, characterized in that, The expected value of the reward corresponding to the unknown real interference strategy is expressed as: The expected value of the reward corresponding to the interference policy output by the trained interference policy network is expressed as: in, For the first h The radar's movements are constantly monitored. For the first h The status information of the radar at any time, 1≤ h ≤ H , For the first h The reward of the radar pulse signal at any time. This represents the initial state distribution of the radar. For the first h The guidance strategy of the radar at any time; h The reward for the radar pulse signal at a given moment is equal to the first... h The signal-to-interference-plus-noise ratio of the radar pulse signal at any given time, the first h The signal-to-interference-plus-noise ratio (SIR) of the radar pulse signal at any given time is expressed as: in, For radar transmission power, For radar antenna gain, , The radar at the 1st The first pulse The wavelength and carrier frequency of the signal transmitted at each sub-pulse moment. The center frequency of the corresponding interference signal. The radar cross section of the target. This refers to the distance between the radar and the target. For receiver noise variance, The noise variance of the interference signal. The indication function characterizing whether the radar signal has been successfully jammed is expressed as: 。 8. The interference strategy perception method based on behavior cloning according to claim 1, characterized in that, Anti-jamming strategy networks include Actor networks and Critic networks.

9. The interference strategy perception method based on behavior cloning according to claim 1, characterized in that, The objective function corresponding to the anti-interference strategy network is: in, The expected value of the reward corresponding to the interference policy output by the trained interference policy network.

Images