Radar anti-jamming strategy generation method based on model-dependent reinforcement learning

Abstract

The application provides a radar anti-jamming strategy generation method based on model-dependent reinforcement learning, comprising: acquiring current real interactive trajectory data; performing model-free reinforcement learning (MFRL) training by using radar side data in the current real interactive trajectory data to obtain a current real anti-jamming strategy; judging whether to start a virtual jammer model training strategy based on the current real anti-jamming strategy; when the virtual jammer model training strategy is started, training a preset supervised model by using jammer side data in the current real interactive trajectory data to obtain a virtual jammer model; generating a virtual anti-jamming strategy through the virtual jammer model and the radar; performing confrontation between the virtual anti-jamming strategy and a real jammer to generate an interactive trajectory; and determining a final anti-jamming strategy according to the interactive trajectory and the nearest current real interactive trajectory data, thereby solving the problem of low sample efficiency in the prior art.

Description

Technical Field

[0001] This invention relates to the field of radar communication technology, and specifically to a method for generating radar anti-jamming strategies based on model-dependent reinforcement learning. Background Technology

[0002] In modern electronic warfare environments, with the rapid development of advanced jamming technologies such as Digital Radio Frequency Memory (DRFM), radars face increasingly complex and variable electromagnetic interference environments. These jamming technologies can generate highly realistic deceptive or targeting signals, severely impacting radar detection performance and survivability. Traditional radar anti-jamming strategies often rely on manual pre-setting, i.e., designing corresponding anti-jamming measures in advance based on known jamming types and parameters. However, when facing complex and varied unknown jamming, this manual pre-setting approach proves inadequate, making it difficult to quickly and accurately identify jamming types and adjust anti-jamming strategies. Therefore, improving radar's autonomous perception and intelligent decision-making capabilities in complex electromagnetic environments has become a critical issue that urgently needs to be addressed in the current radar technology field.

[0003] To address these challenges, deep reinforcement learning has been introduced into radar anti-jamming strategy research in recent years, achieving significant progress. As an important branch of machine learning, deep reinforcement learning constructs an interaction model between the agent and its environment, enabling the agent to continuously optimize its behavioral strategies based on environmental feedback to maximize long-term gains. In the field of radar anti-jamming, deep reinforcement learning is used to train agents to autonomously perceive jamming strategies from radar transmitted signals and echo signals containing jamming components, and to generate corresponding anti-jamming strategies. Particularly for frequency-agile radars, deep reinforcement learning has successfully induced jammers to interfere in the wrong frequency bands by optimizing carrier frequency selection strategies, thereby improving the separability of radar signals and jamming signals in the time and frequency domains and enhancing the radar's anti-jamming capability.

[0004] While deep reinforcement learning-based methods for acquiring radar anti-jamming strategies have shown great potential, existing technologies still face several pressing issues. Specifically, low sample efficiency is one of the main bottlenecks hindering the application of deep reinforcement learning in radar anti-jamming. Deep reinforcement learning algorithms require a large amount of interactive data to train the agent's policy, but in practical applications, radar often struggles to acquire a sufficient number of effective echo signals from moving targets to support policy learning. This results in a time-consuming and costly training process, making it difficult to meet the demands of real-time anti-jamming. Therefore, improving the sample efficiency of deep reinforcement learning in acquiring radar anti-jamming strategies has become a crucial problem that urgently needs to be solved. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a radar anti-jamming strategy generation method based on model-dependent reinforcement learning.

[0006] The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] In a first aspect, the present invention provides a radar anti-jamming strategy generation method based on model-dependent reinforcement learning, comprising:

[0008] Acquire current real interaction trajectory data; the current real interaction trajectory data is the interaction data between the radar and the real jammer during the online confrontation process;

[0009] Model-free reinforcement learning (MFRL) training is performed using radar-side data from current real-world interaction trajectory data to obtain the current real anti-jamming strategy.

[0010] Determine whether to activate the virtual jamming machine model training strategy based on the current real anti-jamming strategy;

[0011] When the virtual jammer model training strategy is started, the jammer-side data in the current real interaction trajectory data is used to train the pre-set supervised model to obtain the virtual jammer model.

[0012] Virtual anti-jamming strategies are generated using a virtual jammer model and radar.

[0013] The virtual anti-jamming strategy is used to counter real jammers, generating an adversarial interaction trajectory.

[0014] Based on the adversarial interaction trajectory and the most recent current real interaction trajectory data, the final anti-jamming strategy is determined; the final anti-jamming strategy is used to distinguish target signals and jamming signals in radar echo signals.

[0015] Optionally, the current real-time interaction trajectory data includes: radar status, radar actions, jammer status, jammer actions, and radar anti-jamming capability rewards;

[0016] The radar status, radar actions, and radar anti-jamming capability rewards are radar-side data from the current real-time interactive trajectory data.

[0017] The jammer's status and jammer maneuvers are considered as jammer-side data in the current real-world interactive trajectory data.

[0018] Optionally, determining whether to activate the virtual jamming machine model training strategy based on the current real anti-jamming strategy includes:

[0019] When the preset activation conditions are met, the virtual jamming machine model training strategy is activated.

[0020] If the preset start conditions are not met, the virtual jamming machine model training strategy will not be started, and the current real anti-jamming strategy will be directly adopted as the final anti-jamming strategy.

[0021] The preset start-up condition is based on the loss value of the current real anti-interference strategy.

[0022] Optionally, the preset start conditions include: the number of MFRL training iterations meets the preset number of iterations and the persistence of the loss of the current real anti-interference strategy is greater than the preset first loss threshold.

[0023] Optionally, before generating a virtual anti-jamming strategy using a virtual jammer model and radar, the following steps are also included:

[0024] When the loss value of the virtual jamming machine model continues to exceed the preset second loss threshold, the current real anti-jamming strategy is directly adopted as the final anti-jamming strategy.

[0025] Optionally, a virtual anti-jamming strategy is generated using a virtual jammer model and radar, including:

[0026] When the loss value of the virtual jammer model is continuously less than the preset second loss threshold, the virtual jammer model interacts with the radar to generate virtual interaction trajectory data.

[0027] A virtual anti-jamming strategy is obtained by using radar-side data from virtual interactive trajectory data for model-free reinforcement learning (MFRL) training.

[0028] Optionally, based on the adversarial interaction trajectory and the most recent current real interaction trajectory data, a final anti-interference strategy is determined, including:

[0029] Compare the radar anti-jamming capability reward in the adversarial interaction trajectory with the radar anti-jamming capability reward in the most recent current real interaction trajectory data.

[0030] When the radar anti-jamming capability reward in the adversarial interaction trajectory is greater than the radar anti-jamming capability reward in the most recent current real interaction trajectory data, the virtual anti-jamming strategy is used as the final anti-jamming strategy.

[0031] When the radar anti-jamming capability reward in the adversarial interaction trajectory is less than the radar anti-jamming capability reward in the most recent current real interaction trajectory data, the current real anti-jamming strategy is used as the final anti-jamming strategy.

[0032] In a second aspect, the present invention provides a radar anti-jamming strategy generation device based on model-dependent reinforcement learning. The radar anti-jamming strategy generation device based on model-dependent reinforcement learning includes: an acquisition unit, a training unit, a judgment unit, a generation unit, an adversarial unit, and a determination unit.

[0033] The acquisition unit is used to: acquire the current real interaction trajectory data; the current real interaction trajectory data is the interaction data between the radar and the real jammer during the online confrontation process;

[0034] The training unit is used to: perform model-free reinforcement learning (MFRL) training using radar-side data from the current real interactive trajectory data to obtain the current real anti-jamming strategy;

[0035] The decision unit is used to: determine whether to start the virtual jamming machine model training strategy based on the current real anti-jamming strategy;

[0036] The training unit is also used to: when the virtual jammer model training strategy is started, use the jammer-side data in the current real interaction trajectory data to train the preset supervised model to obtain the virtual jammer model;

[0037] The generation unit is used to: generate virtual anti-jamming strategies using a virtual jammer model and radar;

[0038] The countermeasure unit is used to: counter virtual anti-jamming strategies against real jammers and generate countermeasure interaction trajectories;

[0039] The determining unit is used to: determine the final anti-jamming strategy based on the adversarial interaction trajectory and the most recent current real interaction trajectory data; the final anti-jamming strategy is used to distinguish the target signal and the jamming signal in the radar echo signal.

[0040] Thirdly, the present invention provides a radar anti-jamming strategy generation device based on model-dependent reinforcement learning, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the radar anti-jamming strategy generation device based on model-dependent reinforcement learning is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the radar anti-jamming strategy generation method based on model-dependent reinforcement learning as described in the first aspect above.

[0041] This invention provides a radar anti-jamming strategy generation method based on model-dependent reinforcement learning, comprising: acquiring current real interaction trajectory data; the current real interaction trajectory data is the interaction data between the radar and a real jammer during online confrontation; performing model-free reinforcement learning (MFRL) training using radar-side data in the current real interaction trajectory data to obtain the current real anti-jamming strategy; determining whether to activate a virtual jammer model training strategy based on the current real anti-jamming strategy; when the virtual jammer model training strategy is activated, training a pre-set supervised model using jammer-side data in the current real interaction trajectory data to obtain a virtual jammer model; generating a virtual anti-jamming strategy through the virtual jammer model and the radar; engaging the virtual anti-jamming strategy against a real jammer to generate an adversarial interaction trajectory; determining the final anti-jamming strategy based on the adversarial interaction trajectory and the most recent current real interaction trajectory data; the final anti-jamming strategy is used to distinguish target signals and jamming signals in radar echo signals. In this invention, a virtual anti-jamming strategy is generated through the virtual jammer model and the radar, and an adversarial interaction trajectory is generated by engaging the virtual anti-jamming strategy against a real jammer. This process essentially expands the training dataset by generating a large amount of virtual interaction data based on real interaction data, thereby reducing the dependence of existing reinforcement learning on real interaction data. Secondly, by pitting the virtual anti-jamming strategy against the real jammer, the effectiveness of the virtual anti-jamming strategy can be verified, and the strategy can be further optimized based on the adversarial results, improving the robustness and adaptability of the final anti-jamming strategy. Furthermore, the final anti-jamming strategy is dynamically adjusted based on the judgment results of the adversarial interaction trajectory and the real interaction trajectory, enabling the final anti-jamming strategy to adapt to changes in the jammer in the real environment and maintaining its effectiveness. These improvements collectively enhance the sample efficiency in radar anti-jamming strategy acquisition, solving the problem of low sample efficiency in existing solutions.

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0043] Figure 1 A flowchart illustrating a radar anti-jamming strategy generation method based on model-dependent reinforcement learning provided in an embodiment of the present invention;

[0044] Figure 2 An exemplary schematic diagram illustrates the interaction between frequency-agile radar and narrowband targeted jamming;

[0045] Figure 3 A schematic diagram of the network model structure corresponding to the anti-interference strategy is shown as an example;

[0046] Figure 4 An exemplary schematic diagram of a radar state slice structure is shown;

[0047] Figure 5 An exemplary schematic diagram illustrating the differences between a virtual jammer model and a real jammer;

[0048] Figure 6 An exemplary diagram of an anti-jamming strategy generated based on a virtual jammer model is shown.

[0049] Figure 7 An exemplary comparison of the learning curves between the method of the present invention and model-free reinforcement learning is shown in the figure.

[0050] Figure 8 A schematic diagram of a radar anti-jamming strategy generation device based on model-dependent reinforcement learning provided in an embodiment of the present invention;

[0051] Figure 9 This is a schematic diagram of a radar anti-jamming strategy generation device based on model-dependent reinforcement learning, provided as an embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0053] To improve the sample efficiency in acquiring radar anti-jamming strategies and enhance the robustness and adaptability of the final anti-jamming strategy, this invention provides a radar anti-jamming strategy generation method based on model-dependent reinforcement learning. Figure 1 This is a flowchart illustrating a radar anti-jamming strategy generation method based on model-dependent reinforcement learning, provided as an embodiment of the present invention. Figure 1 As shown, it includes:

[0054] S101. Obtain the current real interaction trajectory data.

[0055] The current real interaction trajectory data comprises all interaction data between the radar and the real jammer during the online confrontation process up to the current time point. Optionally, the current real interaction trajectory data includes: radar status, radar actions, jammer status, jammer actions, and radar anti-jamming capability rewards.

[0056] The radar status, radar actions, and radar anti-jamming capability rewards are radar-side data from the current real-time interactive trajectory data.

[0057] The jammer's status and jammer maneuvers are considered as jammer-side data in the current real-world interactive trajectory data.

[0058] It should be noted that the radar in this embodiment is an intra-pulse frequency-agile radar with multiple operating frequencies, and each sub-pulse selects one frequency as the carrier frequency. The real jammer has digital radio frequency storage capabilities and adopts a time-division multiplexing system, transmitting narrowband targeted noise suppression jamming signals. The operating time (interception time) of the real jammer's receiver and the operating time (jamming time) of its transmitter are both integer multiples of the radar sub-pulse width, allowing it to jam multiple frequencies simultaneously. The differences between different jamming strategies mainly lie in the jammer's interception time, jamming time, and selection of jamming frequencies.

[0059] In this embodiment, see Figure 2 , Figure 2 This is a schematic diagram illustrating the interaction between frequency-agile radar and narrowband targeted jamming: For the t-th radar pulse signal, the intra-pulse frequency-agile radar, based on the echo signal (state s) t ) and carrier frequency selection strategy π selects the sub-pulse carrier frequency and transmits the pulse signal (action a) t The jammer first intercepts the radar's transmitted signal, then transmits a corresponding targeted jamming signal; the radar receives the new echo signal (state s). t+1 And from this, one can learn about one's own ability to resist interference (reward r) t The above interaction process continues until the end of a coherent processing interval (CPI), at which point the radar updates its carrier frequency selection strategy based on the reward obtained from each interaction. It is important to note that radar echo signals are not the only option for representing the agent's state; any tensor containing both "radar transmitted signals" and "jamming signals" can be used as state input to the radar. Therefore, the term "echo signal" mentioned later refers to this type of tensor.

[0060] In this embodiment, a radar transmitted pulse is divided into M sub-pulses, and the carrier frequency of each sub-pulse can be arbitrarily selected from N given frequency points. The radar transmitted pulse signal is considered as an agent's action, and the radar action expression is:

[0061]

[0062] in, Let d represent the t-th pulse signal transmitted by the radar. M Let N represent the frequency point corresponding to the Mth sub-pulse, and let N represent the total number of frequency points corresponding to the Mth sub-pulse.

[0063] Since a radar pulse contains multiple sub-pulses, and all sub-pulses share a single set of selectable frequency points. Therefore, radar action set It can be represented as:

[0064]

[0065] Where × represents the Cartesian product.

[0066] In this embodiment, the radar echo signal contains interference components. The time-division multiplexing jammer (real jammer) operates with alternating receiver and transmitter modes when transmitting narrowband targeted jamming: When the receiver is active, the jammer intercepts a segment of the radar transmitted signal, analyzes the waveform to obtain its frequency information, and stores its carrier frequency (frequency point) in the jammer's memory; when the transmitter is active, the jammer selects a certain number of frequency points from its memory according to a preset jamming strategy and transmits narrowband targeted jamming signals to the selected frequencies. The jammer's operating mode repeatedly switches between signal interception and jamming until the receiver can no longer receive the radar transmitted signal, at which point the jammer stops operating. Therefore, the jammer's operation... It can be represented as:

[0067]

[0068] in, Let represent the t-th jamming pulse signal emitted by the jammer. It is an M-row N-column matrix, where each row represents a jamming sub-pulse. The m-th jamming sub-pulse is used to jam the m-th radar sub-pulse. This indicates whether the m-th interference sub-pulse covers the radar operating frequency band corresponding to the n-th frequency point. The corresponding unmasked action. Analogous to the action set of an intelligent agent, the action set of the jammer is defined as follows: if All elements in the m-th row are 0, that is:

[0069]

[0070] This indicates that the jammer has intercepted the m-th sub-pulse emitted by the radar, at which point the jammer's transmitter stops operating.

[0071] In this embodiment, since the power of the interference component received by the radar is generally much greater than the power of the echo component of the real target in a real electromagnetic game scenario, the radar can easily obtain the time-frequency diagram of the interference signal using methods such as spectrum analysis. Therefore, by combining its own transmitted signal and the received interference signal, the radar can construct an "echo signal" representing the state tensor, that is, the state s that the radar is in when it transmits the t-th pulse. t The radar state s can be represented using the past k "echo signals". t Represented as:

[0072]

[0073] in, This represents the (t-1)th pulse signal transmitted by the radar. This represents the (t-1)th jamming pulse signal emitted by the jammer.

[0074] In this embodiment, the jammer uses the last T actions performed by the radar as the jammer state. Right now:

[0075]

[0076] In this embodiment, the radar uses the non-obstruction rate as an indicator to measure its own anti-jamming capability (i.e., radar anti-jamming capability bonus), that is:

[0077]

[0078] Where UCR represents the unobstructed rate of the radar pulse signal, l m ∈{0,1} indicates whether the m-th radar sub-pulse is jammed, l m =0 indicates that the m-th jamming sub-pulse did not allocate jamming power on the operating frequency band corresponding to the radar carrier frequency (i.e. The m-th radar subpulse is completely unaffected by interference. Radars can enhance their anti-jamming capabilities by improving their UCR (Unified Radar Response).

[0079] In addition, the current real interaction trajectory data in S101 is also obtained by acquiring and analyzing radar echo data.

[0080] S102. Use radar-side data from the current real interactive trajectory data to perform model-free reinforcement learning (MFRL) training to obtain the current real anti-jamming strategy.

[0081] In an embodiment of the present invention, the model-free reinforcement learning (MFRL) training based on the proximal policy optimization (PPO) algorithm can be used to obtain the current real anti-interference policy.

[0082] The loss function l(φ) during model-free reinforcement learning (MFRL) training guided by the proximal policy optimization algorithm can be expressed as:

[0083]

[0084] in, This represents the mathematical expectation, where the radar's state and action combinations follow the strategy. The distribution of ; φ represents the policy network parameters of the agent (radar) at the current moment; φ old Represents the initial parameters of the policy network; Let represent the advantage function; ε is the cutoff parameter taking the value (0,1); clip(x,l,r) = max(min(x,r),l), meaning to restrict x to the interval [l,r]. By reducing the loss function, the radar can optimize its anti-jamming strategy and obtain the current true anti-jamming strategy. Here, x represents the input parameter, l represents the assumed minimum value of the input parameter x, and r represents the assumed maximum value of the input parameter x.

[0085] It is important to emphasize that the processing steps S101-S102 are continuously executed. Therefore, the current actual anti-interference strategy is also continuously updated.

[0086] S103. Determine whether to start the virtual jamming machine model training strategy based on the current real anti-interference strategy.

[0087] Optionally, S103 may specifically include:

[0088] When the preset activation conditions are met, the virtual jamming machine model training strategy is activated.

[0089] If the preset start conditions are not met, the virtual jamming machine model training strategy will not be started, and the current real anti-jamming strategy will be directly adopted as the final anti-jamming strategy.

[0090] The preset start-up condition is based on the loss value of the current real anti-interference strategy.

[0091] Optionally, the preset start conditions include: the number of MFRL training iterations meets the preset number of iterations and the persistence of the loss of the current real anti-interference strategy is greater than the preset first loss threshold.

[0092] S104. When the virtual jammer model training strategy is started, the jammer-side data in the current real interaction trajectory data is used to train the preset supervised model to obtain the virtual jammer model.

[0093] Optionally, prior to S105, the following are also included:

[0094] When the loss value of the virtual jamming machine model continues to exceed the preset second loss threshold, the current real anti-jamming strategy is directly adopted as the final anti-jamming strategy.

[0095] It should be noted that the pre-defined supervised model can be artificially constructed according to actual needs, or an existing supervised learning model can be used. This embodiment does not limit this.

[0096] S105. A virtual anti-jamming strategy is generated using a virtual jammer model and radar.

[0097] Optionally, S105 may specifically include:

[0098] When the loss value of the virtual jammer model is continuously less than the preset second loss threshold, the virtual jammer model interacts with the radar to generate virtual interaction trajectory data.

[0099] A virtual anti-jamming strategy is obtained by using radar-side data from virtual interactive trajectory data for model-free reinforcement learning (MFRL) training.

[0100] It should be noted that in S105, the loss value of the virtual interference machine model and the preset second loss threshold need to be judged within a preset number of iterations.

[0101] S106. The virtual anti-jamming strategy is used to counter the real jammer, generating an anti-jamming interaction trajectory.

[0102] S107. Based on the adversarial interaction trajectory and the most recent current real interaction trajectory data, determine the final anti-interference strategy.

[0103] The final anti-jamming strategy is used to distinguish target signals from jamming signals in radar echo signals.

[0104] Optionally, S107 may specifically include:

[0105] Compare the radar anti-jamming capability reward in the adversarial interaction trajectory with the radar anti-jamming capability reward in the most recent current real interaction trajectory data.

[0106] When the radar anti-jamming capability reward in the adversarial interaction trajectory is greater than the radar anti-jamming capability reward in the most recent current real interaction trajectory data, the virtual anti-jamming strategy is used as the final anti-jamming strategy.

[0107] When the radar anti-jamming capability reward in the adversarial interaction trajectory is less than the radar anti-jamming capability reward in the most recent current real interaction trajectory data, the current real anti-jamming strategy is used as the final anti-jamming strategy.

[0108] This invention provides a method for generating radar anti-jamming strategies based on model-dependent reinforcement learning, comprising: acquiring current real interaction trajectory data; the current real interaction trajectory data is the interaction data during online confrontation between the radar and a real jammer; performing model-free reinforcement learning (MFRL) training using radar-side data in the current real interaction trajectory data to obtain the current real anti-jamming strategy; determining whether to activate a virtual jammer model training strategy based on the current real anti-jamming strategy; when the virtual jammer model training strategy is activated, training a preset supervised model using jammer-side data in the current real interaction trajectory data to obtain a virtual jammer model; generating a virtual anti-jamming strategy through the virtual jammer model and the radar; engaging the virtual anti-jamming strategy against a real jammer to generate an adversarial interaction trajectory; determining the final anti-jamming strategy based on the adversarial interaction trajectory and the most recent current real interaction trajectory data; the final anti-jamming strategy is used to distinguish target signals and jamming signals in radar echo signals. In this invention, a virtual anti-jamming strategy is generated through a virtual jammer model and the radar, and an adversarial interaction trajectory is generated by engaging the virtual anti-jamming strategy against a real jammer. This process essentially expands the training dataset by generating a large amount of virtual interaction data based on real interaction data, thereby reducing the dependence of existing reinforcement learning on real interaction data. Secondly, by pitting the virtual anti-jamming strategy against the real jammer, the effectiveness of the virtual anti-jamming strategy can be verified, and the strategy can be further optimized based on the adversarial results, improving the robustness and adaptability of the final anti-jamming strategy. Furthermore, the final anti-jamming strategy is dynamically adjusted based on the judgment results of the adversarial interaction trajectory and the real interaction trajectory, enabling the final anti-jamming strategy to adapt to changes in the jammer in the real environment and maintaining its effectiveness. These improvements collectively enhance the sample efficiency in radar anti-jamming strategy acquisition, solving the problem of low sample efficiency in existing solutions.

[0109] To verify the effectiveness of the radar anti-jamming strategy generation method based on model-dependent reinforcement learning provided in this invention, simulation experiments were also conducted.

[0110] This embodiment includes a set of experiments:

[0111] This study experimentally investigates the countermeasures between an intra-pulse frequency-agile radar and a narrowband targeted self-defense jammer. The radar-related parameter settings are shown in Table 1.

[0112] Table 1. Radar-related parameter settings during the experiment.

[0113]

[0114] While both are narrowband targeted jamming, the differences between different jamming strategies mainly lie in the jammer's interception duration, jamming duration, and selection of jamming frequencies. Jammers with DRFM capabilities will store the intercepted radar subpulse frequency information in their own memory. Obviously, the more information the memory can store, the stronger the jammer's jamming capability. However, due to hardware limitations, the jammer's memory cannot be infinitely large. Therefore, the experiment assumes that the jammer will clear the memory once after each CPI (Continuous Frequency Injection). Furthermore, considering that the operating bandwidth of the jamming subpulse is generally greater than that of the radar subpulse, and that the jammer can simultaneously apply jamming to multiple intercepted frequencies, the experiment assumes that the frequency band of the jamming subpulse will cover the last 15 frequencies stored in the memory, as well as the 10 frequencies above and below them (this can also be understood as the jammer simultaneously transmitting 15 jamming subpulses with different carrier frequencies, each jamming subpulse's bandwidth being 20 times the minimum frequency hopping interval of the radar subpulse carrier frequency). For ease of subsequent description, the experimental design defines the jamming strategy's transmit / receive mode as "receive m, transmit n": the jammer's interception time equals the duration of m radar sub-pulses, and the jamming time equals the duration of n sub-pulses. The switching time between the jammer's two operating modes is negligible.

[0115] Based on the above assumptions, the experiment used five different interference strategies with different transmit and receive modes: receive 1 and transmit 1, receive 1 and transmit 4, receive 2 and transmit 3, receive 4 and transmit 6, and receive 5 and transmit 5.

[0116] Since each radar subpulse can arbitrarily select a frequency from the available frequency set, the radar's action set contains 20... 10 The radar has a very large action space, which is too large for a neural network to enumerate. Therefore, the experiment treats the radar action space as a continuous action space, and the neural network directly outputs the probability distribution of the frequency selection of each radar sub-pulse. That is, the 10 neurons in the network output layer correspond to 10 radar sub-pulses. Figure 3 An exemplary diagram of the network model structure corresponding to the anti-interference strategy is shown. Furthermore, the anti-interference strategy here can be either a virtual anti-interference strategy or a real anti-interference strategy. For example... Figure 3 As shown, the anti-jamming strategy corresponds to a network model structure with three hidden layers. The first two layers are convolutional neural networks (CNNs), with the first layer having 9 output channels and the second layer having 18 output channels. The third layer is a fully connected network with 64 neurons. The activation function for all three layers is the ReLU function. The output layer is divided into two parts, each consisting of 10 neurons, corresponding to the mean and variance of 10 radar sub-pulse frequency points. A Gaussian distribution can be constructed using the mean and variance to sample the radar sub-pulse frequency points. Since the sampling results may not be integers, discretization is required to obtain the accurate radar action. The discretization method used in the experiment is as follows:

[0117]

[0118] Where round(·) represents the floor function, f con It is a continuous value obtained from sampling, f dis Here, f represents the discrete radar subpulse frequency points, and N is the number of available radar frequency points; in this experiment, N = 20. It is not difficult to see that f... dis ∈{0,1,…,N-1}.

[0119] To utilize the aforementioned strategy network structure, experiments were conducted using the structure of the image data and a third-order tensor to represent the environmental state. Figure 4 An exemplary diagram of a radar state slice is shown. Since the experiment treats the "radar-jamming" actions of the past k=3 pulses as the environmental state, the state "image" contains three layers of matrices, each constructed based on the jammer action of the corresponding pulse. For example... Figure 4 As shown, in the matrix, '0' indicates that the jamming subpulse does not cover this frequency point (the jammer may be intercepting the signal); '1' indicates that the jamming subpulse covers this frequency point. Adding '2' to the radar subpulse frequency point position in the jammer's action matrix will result in '2' appearing in the state tensor, and possibly '3'. '2' indicates that the corresponding radar subpulse is not jammed, and '3' indicates that the corresponding radar subpulse is jammed. In summary, the numbers that may appear in the state tensor are: '0', '1', '2', and '3'.

[0120] In this experiment, S102 and S105 were trained online using the PPO algorithm to develop anti-interference strategies. The hyperparameters involved are shown in Table 2.

[0121] Table 2 shows the hyperparameters corresponding to the online training of the anti-interference strategy.

[0122]

[0123] In this experiment, the radar will construct an environment model and train a virtual anti-jamming strategy offline when the number of training rounds is a multiple of 10 and the non-obstruction rate is less than 90%. If the offline-trained virtual anti-jamming strategy significantly outperforms the online-trained real anti-jamming strategy—specifically, the former generates a signal with a 10% higher non-obstruction rate than the latter—then the radar will adopt the former as the new anti-jamming strategy. Parameters related to the virtual jammer construction process are shown in Table 3.

[0124] Table 3. Parameters related to the construction process of the virtual jammer

[0125]

[0126]

[0127] Simulation content and result analysis:

[0128] Please see Figure 5 , Figure 5 An illustrative diagram illustrating the differences between a virtual jammer model and a real jammer is provided. Figure 5 Specifically, it demonstrates the real interaction trajectory diagrams collected by the radar when training anti-jamming strategies against virtual jammer models or real jammers using a 1-receive, 4-transmit method, as well as the simulated interaction trajectory diagrams generated by the environment model, and the differences between the two. Figure 5 Figure (a) is a schematic diagram of the actual interaction trajectory collected by the radar against a real jammer. Figure 5 Figure (b) is a schematic diagram of the simulated interactive trajectory collected by the radar against the virtual jammer model. Figure 5 Figure (c) is a schematic diagram illustrating the difference between the real interaction trajectory and the simulated interaction trajectory. Figure 5 In the diagram, the yellow block corresponding to number '2' represents the actual operating frequency of the radar subpulse that is not interfered with; the green block corresponding to number '1' represents the frequency that is not the actual operating frequency of the radar subpulse but is obscured by the interfering subpulse; and the red block corresponding to number '3' represents the actual operating frequency of the radar subpulse that is obscured by the interfering subpulse. The comparison shows that the virtual jamming strategy (virtual jammer model) built based on supervised learning is generally similar to the real jamming strategy (real jammer), with deviations only at a few subpulses. The reason for these deviations is that the labeled data used to build the virtual jammer was sampled from the initial stage of the radar's online training of its anti-jamming strategy. At this time, the radar's anti-jamming capability was weak, and many operating frequencies were obscured by the interfering signal. Therefore, the virtual jammer trained based on this data will "overestimate" the jamming capability of the real jamming strategy.

[0129] Figure 6 An exemplary diagram of an anti-jamming strategy generated based on a virtual jammer model is shown. Figure 6 Figure (a) exemplarily illustrates the anti-jamming strategy obtained by training the jamming strategy generated for the virtual jammer model offline (1 transmit, 4 receive). Figure 6 Figure (b) exemplarily illustrates the target Figure 6 The results of online testing of the strategy corresponding to Figure (a). From Figure 6 As can be seen, although there are certain discrepancies between virtual and real jamming strategies, the anti-jamming strategy trained by the radar against the former, which has slightly stronger jamming capabilities, can still play a good role in real "radar-jamming" confrontation scenarios.

[0130] Figure 7 An exemplary diagram shows the comparison results of the learning curves between the method of the present invention and model-free reinforcement learning. Figure 7Figure (a) exemplarily shows a comparison of the learning curves of the model-dependent reinforcement learning method of the present invention with those of existing model-free reinforcement learning when facing a 1-to-1 transmit / receive interference strategy. Figure 7 Figure (b) exemplarily shows the comparison of the learning curves of the model-dependent reinforcement learning method of the present invention with those of the existing model-free reinforcement learning when facing interference strategy 1 transmits 4. Figure 7 Figure (c) exemplarily shows a comparison of the learning curves of the model-dependent reinforcement learning method of the present invention and the existing model-free reinforcement learning when facing the interference strategy of transmit 2 and receive 3. Figure 7 Figure (d) exemplarily shows the comparison of the learning curves of the model-dependent reinforcement learning method of the present invention with those of the existing model-free reinforcement learning when facing interference policy (transmit 4, receive 6). Figure 7 Figure (e) exemplarily illustrates the comparison of the learning curves of the model-dependent reinforcement learning method of the present invention with those of existing model-free reinforcement learning when facing interference strategies. Figure 7 The horizontal axis represents the number of rounds of online interaction and training for the radar against real jamming strategies. Since this experiment assumes that the speed of offline training of anti-jamming strategies by the radar far exceeds the speed of online collection of real interaction trajectories, the "model-dependent reinforcement learning" curve ignores the time spent building the environment model and offline training of anti-jamming strategies. Clearly, as the number of online training rounds increases, the unmasked rate of both curves gradually increases and slowly approaches 1 (i.e., the theoretical upper limit); the "model-dependent reinforcement learning" curve undergoes a jump during training (corresponding to the offline-trained anti-jamming strategy being transferred to the real electromagnetic game scenario), and converges before the "model-free reinforcement learning" curve. Therefore, it can be concluded that the method provided in this invention, compared with existing radar anti-jamming strategy acquisition methods based on conventional model-free reinforcement learning algorithms, can significantly reduce the number of real samples required for radar to acquire anti-jamming strategies and improve sample efficiency.

[0131] In summary, this invention provides a radar anti-jamming strategy acquisition method based on model-dependent reinforcement learning. First, by deeply mining the jammer's behavioral logic implicit in real interaction trajectories, this method significantly improves the radar's efficiency in utilizing real interaction trajectories and reduces the number of online training rounds required to acquire effective anti-jamming strategies. Second, since the interaction between the radar and the virtual jammer does not affect the real electromagnetic environment, this method allows the radar to repeatedly explore the optimal anti-jamming strategy offline, effectively mitigating the risk of the anti-jamming strategy getting trapped in local optima during online training and further enhancing the radar's anti-jamming capability. Furthermore, in acquiring effective anti-jamming strategies, this invention does not impose special constraints on the specific algorithms used to construct the virtual jammer or for offline or online radar training, exhibiting strong compatibility with existing work in related fields. Finally, this invention also considers the characteristics of simultaneous radar and jammer actions and multi-round interactions, and models the jammer's actions in the form of a time-frequency matrix, making the description of the radar-jammer interaction process more realistic.

[0132] The method provided in this embodiment of the invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc., and this embodiment of the invention does not limit the application to such devices.

[0133] Based on the same inventive concept, embodiments of the present invention also provide a radar anti-jamming strategy generation device based on model-dependent reinforcement learning. Figure 8 A schematic diagram of a radar anti-jamming strategy generation device based on model-dependent reinforcement learning provided in an embodiment of the present invention is shown below. Figure 8 As shown, it includes: an acquisition unit 601, a training unit 602, a judgment unit 603, a generation unit 604, an adversarial unit 605, and a determination unit 606;

[0134] The acquisition unit 601 is used to: acquire the current real interaction trajectory data; the current real interaction trajectory data is the interaction data between the radar and the real jammer during the online confrontation process;

[0135] Training unit 602 is used to: perform model-free reinforcement learning (MFRL) training using radar-side data from the current real interactive trajectory data to obtain the current real anti-jamming strategy;

[0136] The judgment unit 603 is used to: determine whether to start the virtual jamming machine model training strategy based on the current real anti-jamming strategy;

[0137] Training unit 602 is also used to: when the virtual jammer model training strategy is started, use jammer-side data in the current real interaction trajectory data to train the preset supervised model to obtain the virtual jammer model;

[0138] The generation unit 604 is used to: generate virtual anti-jamming strategies through a virtual jammer model and radar;

[0139] The countermeasure unit 605 is used to: counter the virtual anti-jamming strategy against the real jammer and generate an anti-jamming interaction trajectory;

[0140] The determining unit 606 is used to: determine the final anti-jamming strategy based on the adversarial interaction trajectory and the most recent current real interaction trajectory data; the final anti-jamming strategy is used to distinguish the target signal and the jamming signal in the radar echo signal.

[0141] Figure 9 A schematic diagram of a radar anti-jamming strategy generation device based on model-dependent reinforcement learning, provided in an embodiment of the present invention, includes: a processor 710, a storage medium 720, and a bus 730. The storage medium 720 stores machine-readable instructions executable by the processor 710. When the radar anti-jamming strategy generation device based on model-dependent reinforcement learning is running, the processor 710 communicates with the storage medium 720 via the bus 730, and the processor 710 executes the machine-readable instructions to perform the steps of the above-described method embodiment. The specific implementation and technical effects are similar and will not be described in detail here.

[0142] The storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0143] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0144] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0145] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0146] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0147] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the inventive concept, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A radar anti-jamming strategy generation method based on model-dependent reinforcement learning, characterized in that, include: Obtain the current actual interaction trajectory data; The current real interaction trajectory data refers to the interaction data between the radar and the real jammer during the online confrontation process; The current real anti-jamming strategy is obtained by using radar-side data from the current real interaction trajectory data to perform model-free reinforcement learning (MFRL) training. Based on the current real anti-interference strategy, determine whether to activate the virtual jamming machine model training strategy; When the virtual jammer model training strategy is activated, the jammer-side data in the current real interaction trajectory data is used to train the preset supervised model to obtain the virtual jammer model. A virtual anti-jamming strategy is generated using the virtual jammer model and the radar. The virtual anti-jamming strategy is used to counter the real jamming machine, generating an adversarial interaction trajectory. Based on the adversarial interaction trajectory and the most recent current real interaction trajectory data, the final anti-interference strategy is determined. The final anti-jamming strategy is used to distinguish between target signals and jamming signals in radar echo signals; The step of determining the final anti-interference strategy based on the adversarial interaction trajectory and the most recent current real interaction trajectory data includes: The radar anti-jamming capability reward in the adversarial interaction trajectory is compared with the radar anti-jamming capability reward in the most recent current real interaction trajectory data. When the radar anti-jamming capability reward in the adversarial interaction trajectory is greater than the radar anti-jamming capability reward in the most recent current real interaction trajectory data, the virtual anti-jamming strategy is used as the final anti-jamming strategy. When the radar anti-jamming capability reward in the adversarial interaction trajectory is less than the radar anti-jamming capability reward in the most recent current real interaction trajectory data, the current real anti-jamming strategy is used as the final anti-jamming strategy.

2. The radar anti-jamming strategy generation method based on model-dependent reinforcement learning according to claim 1, characterized in that, The current real-time interaction trajectory data includes: radar status, radar actions, jammer status, jammer actions, and radar anti-jamming capability reward. The radar status, radar action, and radar anti-interference capability reward are radar-side data in the current real interaction trajectory data; The jammer's state and the jammer's movement are considered jammer-side data in the current real-time interaction trajectory data.

3. The radar anti-jamming strategy generation method based on model-dependent reinforcement learning according to claim 1, characterized in that, The step of determining whether to activate the virtual jamming machine model training strategy based on the current real anti-jamming strategy includes: When the preset activation conditions are met, the virtual jamming machine model training strategy is activated. If the preset start conditions are not met, the virtual jamming machine model training strategy will not be started, and the current real anti-jamming strategy will be directly adopted as the final anti-jamming strategy. The preset activation condition is based on the loss value of the current real anti-interference strategy.

4. The radar anti-jamming strategy generation method based on model-dependent reinforcement learning according to claim 3, characterized in that, The preset start conditions include: the number of MFRL training iterations meets the preset number of iterations and the loss persistence of the current real anti-interference strategy is greater than the preset first loss threshold.

5. The radar anti-jamming strategy generation method based on model-dependent reinforcement learning according to claim 1, characterized in that, Before generating the virtual anti-jamming strategy using the virtual jammer model and the radar, the method further includes: When the loss value of the virtual jamming machine model continues to exceed the preset second loss threshold, the current real anti-jamming strategy is directly adopted as the final anti-jamming strategy.

6. The radar anti-jamming strategy generation method based on model-dependent reinforcement learning according to claim 5, characterized in that, The generation of a virtual anti-jamming strategy through the virtual jammer model and the radar includes: When the loss value of the virtual jammer model is continuously less than a preset second loss threshold, the virtual jammer model interacts with the radar to generate virtual interaction trajectory data. The virtual anti-jamming strategy is obtained by using radar-side data from the virtual interactive trajectory data for model-free reinforcement learning (MFRL) training.

7. A radar anti-jamming strategy generation device based on model-dependent reinforcement learning, characterized in that, The radar anti-jamming strategy generation device based on model-dependent reinforcement learning includes: an acquisition unit, a training unit, a judgment unit, a generation unit, an adversarial unit, and a determination unit; The acquisition unit is used to: acquire current real interaction trajectory data; the current real interaction trajectory data is the interaction data between the radar and the real jammer during online confrontation; The training unit is used to: perform model-free reinforcement learning (MFRL) training using radar-side data in the current real interaction trajectory data to obtain the current real anti-jamming strategy; The judgment unit is used to: determine whether to start the virtual jamming machine model training strategy based on the current real anti-interference strategy; The training unit is also used to: when the virtual jammer model training strategy is started, use the jammer-side data in the current real interaction trajectory data to train the preset supervised model to obtain the virtual jammer model; The generation unit is used to: generate a virtual anti-jamming strategy using the virtual jammer model and the radar; The countermeasure unit is used to: counter the virtual anti-jamming strategy against the real jamming machine and generate an anti-jamming interaction trajectory; The determining unit is used to: determine a final anti-jamming strategy based on the adversarial interaction trajectory and the most recent current real interaction trajectory data; the final anti-jamming strategy is used to distinguish between target signals and jamming signals in radar echo signals; The determining unit is specifically used to: compare the radar anti-jamming capability reward in the adversarial interaction trajectory with the radar anti-jamming capability reward in the most recent current real interaction trajectory data; When the radar anti-jamming capability reward in the adversarial interaction trajectory is greater than the radar anti-jamming capability reward in the most recent current real interaction trajectory data, the virtual anti-jamming strategy is used as the final anti-jamming strategy. When the radar anti-jamming capability reward in the adversarial interaction trajectory is less than the radar anti-jamming capability reward in the most recent current real interaction trajectory data, the current real anti-jamming strategy is used as the final anti-jamming strategy.

8. A radar anti-jamming strategy generation device based on model-dependent reinforcement learning, characterized in that, include: The device includes a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the radar anti-jamming strategy generation device based on model-dependent reinforcement learning is running, the processor communicates with the storage medium via the bus. The processor executes the machine-readable instructions to perform the steps of the radar anti-jamming strategy generation method based on model-dependent reinforcement learning as described in any one of claims 1-6.

Images