Normalized deep reinforcement learning radar waveform design method

Abstract

The invention discloses a normalized deep reinforcement learning radar waveform design method, which comprises the following steps: modeling a radar confrontation environment based on a Markov decision process model, expressing a radar anti-interference problem as a Markov decision process with a limited field-of-view state space, determining a state, an action and a conversion dynamic state, and designing a normalized deep reinforcement learning radar waveform. Setting a reward function in the existing deep reinforcement learning algorithm; the beneficial effects of the invention are that the system stability and the learning efficiency are improved through the enhancement of the reward normalized discount; a radar environment is modeled as a Markov decision process, so that an agent can optimize a waveform strategy through interaction with a state space, an action space and a reward function; compared with an LFM method, a non-normalized D3QN method and the like, the method has the advantages that the signal to interference and noise ratio of a radar receiver end is increased, the target detection probability is increased, interference and clutter interference are effectively reduced, and an effective way is provided for automation and intelligentization of radar waveform generation in an anti-environment.

Description

Technical Field

[0001] The present invention belongs to the field of radar and communication technology, and specifically relates to a normalized deep reinforcement learning radar waveform design method. Background Art

[0002] Radar plays a key role in modern defense, surveillance, and environmental monitoring, providing support for target detection, tracking, and situational awareness. However, the development of electronic warfare technology has posed severe challenges to radar. Noise suppression jamming and deception jamming have reduced the effectiveness of radar, and existing anti-interference measures have many shortcomings.

[0003] Spatial anti-interference technology can only suppress sidelobe interference and cannot deal with self-defense interference; the frequency selection of frequency agile technology is blind and lacks specificity; the anti-interference performance of phase-coded signal agile technology is limited by the number of codes and orthogonality, and these agile technologies cannot make optimal decisions based on real-time changes in the environment.

[0004] Cognitive radar designs waveforms based on the target reflection frequency domain characteristics and clutter response to weaken clutter interference, but models and algorithms need to be improved when applied to anti-interference. The waveform design method based on game models is limited in its anti-interference performance by model selection. Radar waveform design based on reinforcement learning is either limited to specific tracking scenarios or has too high requirements for parameter debugging due to unstable reward function values, making it difficult to adapt to dynamic interference environments. Summary of the Invention

[0005] The purpose of this invention is to provide a normalized deep reinforcement learning radar waveform design method to reduce variance during training, enabling faster and more stable model convergence; ensure that the waveform design method remains robust and adaptable in complex electromagnetic environments; and improve radar performance and reduce anti-interference vulnerability.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a normalized deep reinforcement learning radar waveform design method, comprising the following steps:

[0007] Modeling the radar countermeasure environment based on the Markov decision process model, formulating the radar anti-interference problem as a Markov decision process with a finite horizon state space, determining the state, action, and transition dynamics, and setting the reward function in the existing deep reinforcement learning algorithm;

[0008] A normalized deep reinforcement learning algorithm is introduced to normalize rewards based on the signal-to-interference-noise ratio (SIR). The normalization parameters are adaptively updated. The Dueling Network structure is used to improve action selection, Double Q-Learning is used to reduce bias, and prioritized experience replay is used to improve sample efficiency.

[0009] The performance improvement is evaluated using the signal-to-interference-noise ratio and detection probability metrics.

[0010] As a preferred technical solution of the present invention, in the radar countermeasure environment modeling based on the Markov decision process model, the state is composed of interference power (J), clutter interference (C), and radar cross section (RCS). At any time i, the state is expressed as Statei = {J i =[j i,1 ,j i,2 ,j i,3 ,…,j i,10 ],C i =[c i,1 ,c i,2 ,c i,3 ,…,c i,10 ], RCS i =[rcs i,1 ,rcs i,2 ,rcs i,3 ,…,rcs i,10 ]}, the system assumes short-term stationarity, and the agent's action is to continuously allocate power among N subbands to maximize the signal-to-interference-noise ratio.

[0011] As a preferred technical solution of the present invention, the reward function in the existing deep reinforcement learning algorithm uses the signal-to-interference-noise ratio of all allocated frequency bands as the reward, and the formula is:

[0012]

[0013] RCS k is the target reflection capability in the kth frequency band, A k is the radar signal amplitude, C k is the clutter response amplitude of the kth frequency band, J k is the interference power of the k-th frequency band, and noise is the noise power of the k-th frequency band.

[0014] As a preferred technical solution of the present invention, it also includes solving the high difference in the reward based on the signal-to-interference-noise ratio based on normalization, using zero-mean unit variance reward normalization, and the calculation method is

[0015] Where μ is the running mean of the observed reward, σ 2 is the running variance, ε is a constant to ensure numerical stability, and R is the reward value.

[0016] As a preferred technical solution of the present invention, the normalization parameter is adaptively updated according to the formula

[0017]

[0018] Where n represents the total number of rewards observed so far, μ new is the latest value of the reward running average, μ old is the running average of the reward before the update, The latest value of the reward variance, is the reward variance before update.

[0019] As a preferred technical solution of the present invention, in the Dueling Network structure, the Dueling DoubleDeep Q-Network architecture decomposes the action value function into:

[0020] Where V(s) is the state value function, A(s,a) is the action function, a represents the radar action, s represents the interference signal, a' represents the radar action at the next moment, and s' represents the interference signal at the next moment.

[0021] As a preferred technical solution of the present invention, in Double Q-Learning, the formula is used

[0022] y=R normalised +γQ rarget (s,argmax a′ Q potiey (s′, a′)),

[0023] where R normalised is the normalized reward value, Q target is the Q value obtained by the target network, Q policy is the Q value obtained by the policy network.

[0024] As a preferred technical solution of the present invention, in the priority experience replay, the conversion is based on probability sampling, and the formula is

[0025] where δ i It represents the time difference error amplitude, quantifying the degree of surprise or the amount of information of the transition. α represents the advantage function network parameter, which determines how many weights are used. If α = 0, uniform random sampling is used.

[0026] As a preferred technical solution of the present invention, in evaluating the performance improvement using the signal-to-interference-noise ratio and detection probability indicators, the detection probability is expressed by the formula

[0027] Among them, P FA is the false alarm rate, and

[0028]

[0029] Where τ is the radar detection threshold, d2 is the threshold coefficient.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] Enhanced by reward normalization discounting, the system stability and learning efficiency are improved; the radar environment is modeled as a Markov decision process, enabling the agent to optimize the waveform strategy by interacting with the state space, action space, and reward function; the challenge of scaling rewards under different operating conditions is addressed, ensuring stable and consistent training while maintaining robust performance in real-time scenarios; by normalizing the rewards, the risk of training instability is reduced, especially in high-variance environments, and the ability of the agent to converge to the optimal strategy is enhanced;

[0032] This normalized deep reinforcement learning ensures the optimal waveform spectrum power allocation strategy. After training, the algorithm can directly adapt to different environmental conditions.

[0033] Compared with methods such as LFM and non-normalized D3QN, this method improves the signal-to-interference-and-noise ratio at the radar receiver, increases the probability of target detection, effectively reduces interference and clutter, and provides an effective way to automate and intelligentize radar waveform generation in a countermeasure environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a normalized reward discount flow chart of the present invention;

[0035] Figure 2 This is a flowchart of combining a deep reinforcement learning algorithm with normalized reward discount using the D3QN algorithm as an example of the present invention;

[0036] Figure 3 This is a signal-to-interference-and-noise ratio diagram when the radar transmit power is changed with a fixed interference power according to the present invention;

[0037] Figure 4 The detection probability diagram of the present invention when the radar transmission power is changed with a fixed interference power;

[0038] Figure 5 This is the signal-to-interference-noise ratio diagram of the present invention when the fixed radar transmission power changes the interference power

[0039] Figure 6 This is a detection probability diagram when the radar transmission power is fixed and the interference power is changed according to the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] Example 1

[0042] See also Figures 1-6 , which is the first embodiment of the present invention, provides a normalized deep reinforcement learning radar waveform design method, comprising the following steps:

[0043] Modeling radar countermeasure environments based on the Markov Decision Process (MDP) model. This model formulates the radar anti-jamming problem as an MDP with a finite state space, determines the state, action, and transition dynamics, and sets the reward function in existing deep reinforcement learning algorithms.

[0044] A normalized deep reinforcement learning algorithm is introduced to normalize rewards based on the signal-to-interference-and-noise ratio (SJNR), adaptively update normalization parameters, use a Dueling Network structure to improve action selection, employ Double Q-Learning to reduce bias, and improve sample efficiency through prioritized experience replay.

[0045] The performance improvement is evaluated using the signal-to-interference-and-noise ratio (SJNR) and detection probability metrics.

[0046] In this embodiment, the radar anti-interference problem is preferably formulated as an MDP with a finite horizon state space consisting of N frequency bands (taking 10 frequency bands as an example); at each time step, the agent observes the environment through three key parameters:

[0047] Interference power (J): different interference levels applied by hostile systems;

[0048] Clutter (C): Environmental reflections and obstacles that affect radar echoes;

[0049] Radar cross section (RCS): The reflectivity characteristics of a target, which affects detection capabilities;

[0050] In the radar countermeasure environment modeling based on the Markov decision process model, the state is composed of interference power (J), clutter interference (C), and radar cross section (RCS). At any time i, the state is expressed as State i = {J i =[j i,1 ,j i,2 ,j i,3 ,…,j i,10 ],C i=[c i,1 ,c i,2 ,c i,3 ,…,c i,10 ], RCS i =[rcs i,1 ,rcs i,2 ,rcs i,3 ,…,rcs i,10 Crucially, the system assumes short-term stationarity—all environmental parameters remain fixed during an event, making state transitions deterministic. This stationarity assumption reflects realistic operational scenarios where interference and noise characteristics persist for meaningful durations while significantly simplifying the learning problem. The agent's actions consist of continuous power allocation among the N subbands, aiming to maximize the signal-to-interference-and-noise ratio (SJNR) while ensuring efficient resource utilization, allowing for flexible and adaptive responses to adversarial threats, as opposed to static allocation approaches.

[0051] In this embodiment, preferably, the reward function in the existing deep reinforcement learning algorithm uses the signal-to-interference-noise ratio of all allocated frequency bands as a reward, and the formula is:

[0052] RCS k is the target reflection capability in the kth frequency band, A k is the radar signal amplitude, C k is the clutter response amplitude of the kth frequency band, J k is the interference power of the k-th frequency band, and noise is the noise power of the k-th frequency band; this expression ensures that the reinforcement learning agent learns a global power allocation strategy instead of focusing on isolated frequency bands, thereby mitigating the narrowband interference vulnerability; however, since the SJNR value varies greatly under different environmental conditions, the original reward signal is usually unstable, resulting in inefficient learning updates; in view of this problem, this application introduces a normalization method in the reward link of the deep reinforcement learning algorithm to improve system stability and efficiency.

[0053] In this embodiment, it is preferred to further include normalization to address the high variance in rewards based on the signal-to-interference-and-noise ratio. In reinforcement learning, the size and variability of rewards play a crucial role in stabilizing policy updates. Excessive fluctuations in reward values ​​can lead to unstable learning behavior, hinder convergence, and reduce overall performance. This problem is particularly evident in radar anti-interference scenarios because the signal-to-interference-and-clutter ratio (SJNR) is highly sensitive to changes in power distribution, interference, and environmental noise and clutter. Therefore, reward distribution may exhibit large variance, leading to unstable gradient updates and inefficient learning.

[0054] To address this instability, zero-mean unit-variance reward normalization is introduced to ensure that the reward remains within a controlled numerical range; the normalization process uses the running mean and variance to normalize the reward, converting it into a stable distribution with zero mean and unit variance; the normalized reward is calculated as

[0055] Where μ is the running mean of the observed reward, σ 2 is the running variance, ε is a constant with a small value to ensure numerical stability, and R is the reward value. This transformation ensures that the reward maintains a consistent scale between training events, thereby preventing disproportionately large updates that may hinder convergence; by reducing the impact of extreme reward values, normalization improves learning efficiency and enables the agent to generalize more effectively under different interference conditions.

[0056] In this embodiment, preferably, in order to maintain numerical stability throughout the training process, the running mean and variance of the reward are iteratively updated; given a new reward observed at a certain step, the update will follow:

[0057]

[0058] Where n represents the total number of rewards observed so far, μ new is the latest value of the reward running average, μ old is the running average of the reward before the update, The latest value of the reward variance, is the reward variance before updating; this adaptive update mechanism ensures that the normalization parameters are dynamically adjusted to adapt to the changing reward distribution, maintaining stability when the agent encounters new environmental conditions; unlike static normalization methods that rely on pre-computed statistics, this method continuously optimizes mean and variance estimates, so that it can smoothly adapt to different interference scenarios; therefore, the reinforcement learning agent benefits from more stable training dynamics, reduces the risk of divergence, and improves policy convergence in complex radar environments.

[0059] In this embodiment, preferably, the key links of the normalized deep reinforcement learning algorithm are given by combining the reward normalized D3QN as an example. In order to effectively approximate the Q value, the Dueling Double Deep Q-Network (D3QN) architecture decomposes the action value function into:

[0060] Where V(s) is the state value function, A(s,a) is the action function, a represents the radar action, s represents the interference signal, a' represents the radar action at the next moment, and s' represents the interference signal at the next moment; this decomposition allows the agent to separate the value of being in a certain state from the advantage of a specific action, thereby achieving more effective generalization in states where the choice of power distribution is less important.

[0061] In this embodiment, preferably, taking D3QN in the deep reinforcement learning algorithm as an example, in order to reduce the overestimation bias in Q-learning, this method can use Double Q-learning, in which action selection and evaluation are decoupled using separate networks:

[0062] y=R normalised +γQ target (s′,argmax a′ Q policy (s, a′)),

[0063] where R normalised is the normalized reward value, Q target is the Q value obtained by the target network, Q policy This mechanism provides more stable value updates for the Q-values ​​obtained by the policy network, thereby reducing the possibility of suboptimal policy divergence.

[0064] In this embodiment, preferably, in order to further improve learning efficiency, Prioritized Experience Replay (PER) is adopted, wherein the conversion is based on the probability sampling shown in the following formula:

[0065] where δ i represents the temporal difference (TD) error amplitude, quantifying the degree of surprise or information content of the transition, α represents the advantage function network parameter, which determines how many weights to use. If α = 0, uniform random sampling is used; this priority sorting speeds up learning by focusing updates on high-value experiences, while importance sampling correction ensures fair updates.

[0066] In this embodiment, the signal-to-noise ratio is preferably a direct measure of the radar's performance under interference and is used as the agent's reward function to evaluate the tangible impact of the anti-interference strategy on the radar's operation and provide feedback on how the agent optimizes the reward value. In order to convert it into combat efficiency, the target detection probability can be introduced as a key performance indicator by solving the Neyman-Pearson theorem.

[0067]

[0068] Among them, P FA is the false alarm rate, and

[0069]

[0070] Where τ is the radar detection threshold, d 2 is the threshold coefficient.

[0071] Example 2

[0072] See also Figures 1-6 , which is the second embodiment of the present invention, is based on the previous embodiment, except that:

[0073] Consider the radar as an intelligent agent, the interference signal in electromagnetic space as state information, the radar's transmitted signal as the agent's action, and the radar signal's SJNR as the reward function. The following uses D3QN as an example to describe the normalized deep reinforcement learning algorithm process for radar waveform design.

[0074] First, the reward values ​​for taking different actions under different states are continuously calculated, and the corresponding state, action, and reward information are stored for training the neural network. Second, the neural network analyzes, calculates, and selects the radar anti-interference waveform strategy with the highest Q value. Finally, the selected optimal strategy is output to the radar, which synthesizes the time domain signal and transmits it. The specific working principle is as follows:

[0075] 1) Initialization phase

[0076] (1) Create a D3QN agent: Initialize the Dueling Double DQN (D3QN) agent, set the state and action dimensions, discount factor, learning rate, experience replay buffer capacity, and training batch size;

[0077] (2) Initialize the neural network: Build a policy network and a target network with the same structure (including feature extraction layer, value stream, and advantage stream). Initially, the target network copies the parameters of the policy network and is set to evaluation mode.

[0078] (3) Initialize the optimizer and memory: Use the Adam optimizer to create an experience replay buffer and reward normalizer;

[0079] (4) Set exploration parameters: initialize the starting value, ending value, and decay rate of the ε-greedy strategy;

[0080] 2) Training loop

[0081] (1) Environment reset: At the beginning of each training round, the environment is reset to obtain the initial state;

[0082] (2) Interactive loop:

[0083] Action selection: Based on the current state, an ε-greedy strategy is used to select an action; a log-normal power allocation is randomly generated with probability ε (exploration), otherwise an action is generated through the policy network (exploitation);

[0084] Execute action: execute the selected action in the environment and obtain the next state, reward and termination flag;

[0085] Storing experience: storing the current state, action, reward, next state, and termination flag as experience tuples into the replay buffer;

[0086] Training the network: When the number of experiences in the buffer is sufficient, randomly sample a batch for training;

[0087] (3) Training steps:

[0088] Calculate the current Q value: Use the policy network to calculate the Q value of the selected action in the current state;

[0089] Calculate the target Q value: Use the target network to calculate the maximum Q value of the next state, and calculate the target Q value by combining the reward and discount factor;

[0090] Calculate the loss: Calculate the loss between the current Q value and the target Q value through the mean square error (MSE);

[0091] Backpropagation: Perform gradient descent to update the parameters of the policy network;

[0092] (4) Update the target network: Regularly copy the parameters of the policy network to the target network to stabilize the training process;

[0093] (5) Decaying the exploration rate: gradually reducing the value of ε, reducing the exploration probability and increasing the utilization probability;

[0094] 3) Support functions

[0095] Reward normalization: Dynamically adjust the mean and variance of rewards and normalize the rewards to improve training stability;

[0096] State processing: Flatten the input state dictionary (containing interference, clutter response and target RCS) into a one-dimensional array for neural network processing;

[0097] Random action generation: Use log-normal distribution to generate power allocation, ensuring total power constraints;

[0098] Network action generation: The Q value output by the policy network is processed by Softmax to generate power allocation;

[0099] 4) Training ends

[0100] Termination condition: training stops after reaching a predetermined number of training rounds or performance threshold;

[0101] Model saving: Save the trained policy network parameters for subsequent evaluation or deployment;

[0102] 5) Key Points

[0103] Dual network structure: the policy network is used for action selection and training, and the target network provides a stable Q-value target;

[0104] Experience replay: By storing and randomly sampling experiences, it breaks data correlation and improves training efficiency;

[0105] Reward normalization: Dynamically adjust reward distribution to accelerate convergence and improve stability;

[0106] Exploration and Exploitation Balance: Gradually transition from exploration to exploitation through the ε-greedy strategy.

[0107] Although the embodiments of the present invention have been shown and described, as detailed above, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A normalized deep reinforcement learning radar waveform design method, characterized by: The steps include: Modeling the radar countermeasure environment based on the Markov decision process model, formulating the radar anti-interference problem as a Markov decision process with a finite horizon state space, determining the state, action, and transition dynamics, and setting the reward function in the existing deep reinforcement learning algorithm; A normalized deep reinforcement learning algorithm is introduced to normalize rewards based on the signal-to-interference-noise ratio (SIR). The normalization parameters are adaptively updated. The Dueling Network structure is used to improve action selection, Double Q-Learning is used to reduce bias, and prioritized experience replay is used to improve sample efficiency. The performance improvement is evaluated using the signal-to-interference-noise ratio and detection probability metrics.

2. The normalized deep reinforcement learning radar waveform design method according to claim 1, characterized in that: In the radar countermeasure environment modeling based on the Markov decision process model, the state is composed of interference power (J), clutter interference (C), and radar cross section (RCS). At any time i, the state is expressed as Statei = {J i =[j i,1 ,j i,2 ,j i,3 ,…,j i,10 ],C i =[c i,1 ,c i,2 ,c i,3 ,…,c i,10 ], RCS i =[rcs i,1 ,rcs i,2 ,rcs i,3 ,…,rcs i,10 ]}, the system assumes short-term stationarity, and the agent's action is to continuously allocate power among N subbands to maximize the signal-to-interference-noise ratio.

3. The normalized deep reinforcement learning radar waveform design method according to claim 1, characterized in that: The reward function in the existing deep reinforcement learning algorithm uses the signal-to-interference-noise ratio of all allocated frequency bands as the reward, and the formula is: RCS k is the target reflection capability in the kth frequency band, A k is the radar signal amplitude, C k is the clutter response amplitude of the kth frequency band, J k is the interference power of the kth frequency band, noise k is the noise power of the kth frequency band.

4. The normalized deep reinforcement learning radar waveform design method according to claim 1, characterized in that: It also includes a normalization based solution to the high variance in the SNR based rewards, using a zero mean unit variance reward normalization, calculated as Where μ is the running mean of the observed reward, σ 2 is the running variance, ε is a constant to ensure numerical stability, and R is the reward value.

5. The normalized deep reinforcement learning radar waveform design method according to claim 1, characterized in that: Normalization parameters are adaptively updated according to the formula Where n represents the total number of rewards observed so far, μ new is the latest value of the reward running average, μ old is the running average of the reward before the update, The latest value of the reward variance, is the reward variance before update.

6. The normalized deep reinforcement learning radar waveform design method according to claim 1, characterized in that: In the Dueling Network structure, the Dueling Double Deep Q-Network architecture decomposes the action value function into: Where V(s) is the state value function, A(s,a) is the action function, a represents the radar action, s represents the interference signal, a' represents the radar action at the next moment, and s' represents the interference signal at the next moment.

7. The normalized deep reinforcement learning radar waveform design method according to claim 1, characterized in that: In Double Q-Learning, use the formula y=R normalised +γQ target (s′,argmax a ·Q policy (s′,a′)), where R normalised is the normalized reward value, Q target is the Q value obtained by the target network, Q policy is the Q value obtained by the policy network.

8. The normalized deep reinforcement learning radar waveform design method according to claim 1, characterized in that: In the priority experience replay, the conversion is based on probability sampling, the formula is where δ i It represents the time difference error amplitude, quantifying the degree of surprise or the amount of information of the transition. α represents the advantage function network parameter, which determines how many weights are used. If α = 0, uniform random sampling is used.

9. The normalized deep reinforcement learning radar waveform design method according to claim 1, characterized in that: When evaluating performance improvement using the signal-to-interference-noise ratio and detection probability indicators, the detection probability is calculated using the formula Among them, P FA is the false alarm rate, and Where τ is the radar detection threshold, d 2 is the threshold coefficient.

Images