Radar intelligent jamming decision method and system of double-layer R-D3QN

Abstract

The application discloses a radar intelligent interference decision-making method and system based on double-layer R-D3QN, and the method comprises the following steps: S1, initializing the size and memory pointer of a working mode memory bank and a signal parameter memory bank, initializing the current value network and target value network of double-layer R-D3QN, a learning rate, a discount factor, a greed factor, a working mode rule bank and a signal parameter rule bank; S2, setting a training sample threshold value, a network parameter update interval, a training sample extraction interval and a network update counter, initializing a training step number x, and setting a total training number; S3, training; S4, updating the working mode rule bank and the signal parameter rule bank; S5, returning to S3 until the x reaches the total training number, and obtaining the current value network of the two trained R-D3QN models; and S6, using the current value network to select the interference strategy and parameters with the highest value for the identified radar working mode and signal parameters. The application improves the radar interference decision-making capability, and can select appropriate interference for the working mode of the radar.

Description

Technical Field

[0001] This invention belongs to the field of radar cognitive electronic warfare technology, specifically involving a two-layer R-D3QN (Rule-Dueling Double Deep Q-network) radar intelligent jamming decision-making method and system. It proposes a new radar jamming scenario modeling method and can be used in the field of cognitive electronic warfare. Background Technology

[0002] Simon Haykin proposed the concept of "cognitive radar" in 2006. The maturity of phased array antenna technology and digital array radar technology has driven the development of radar technology. Multifunctional radar (MFR), as a representative of this technology, has gradually replaced dedicated radar and become a key component of radar systems in various countries. MFR can change its operating parameters in real time according to the battlefield environment and mission requirements, and has strong anti-jamming capabilities. However, due to the continuous development of radar technology, traditional jamming decision-making methods can no longer meet the complex and ever-changing operating modes of new radar systems. Therefore, an intelligent jamming decision-making method is needed to counter the threats posed by MFR.

[0003] Reinforcement learning, a subfield of machine learning, specifically studies how agents make decisions in complex, dynamic environments to maximize gains. Unlike supervised learning, reinforcement learning learns from experience and optimizes policies through interaction with the environment, without requiring explicit datasets or large amounts of prior knowledge. Combining reinforcement learning with deep learning techniques forms Deep Reinforcement Learning (DRL), which possesses powerful environmental awareness and continuously optimizing decision-making capabilities. Jamming decision-making techniques based on deep reinforcement learning can perceive changes in the battlefield electromagnetic environment, dynamically adjust jamming strategies, and effectively counter complex electromagnetic situations. Currently, reinforcement learning algorithms such as DQN (Deep Q-Network), DDQN (Double Deep Q-Network), and D3QN have been applied to intelligent jamming decision-making for multi-functional radars, achieving good results.

[0004] Based on the above situation, this invention proposes a two-layer R-D3QN radar jamming decision-making method and system based on deep reinforcement learning theory, aiming to address the challenges posed by multi-functional radars. This invention can dynamically perceive the battlefield situation, adjust jamming strategies in real time, and incorporates knowledge from a jamming decision-making database, thus possessing significant military and theoretical implications. Summary of the Invention

[0005] To improve radar jamming decision-making performance in complex electromagnetic environments, this invention proposes a radar jamming decision-making method and system based on a two-layer R-D3QN.

[0006] The following simulation scenario was established: A single radar operates in various modes, including search, search-while-tracking, and search-plus-tracking. The radar signal parameters f0, pw, pri, bw, and f under each mode were simulated. c The parameters (carrier frequency, pulse width, pulse repetition interval, bandwidth, modulation frequency) have specific selection ranges and can be expressed as:

[0007]

[0008] Among them, c i This represents the i-th operating mode of a single radar, where N is the number of operating modes. These are the parameters of the radar signal in the current mode.

[0009] The jamming patterns of a single jammer include radio frequency noise, smart jamming, and range spoofing. Each jamming signal has parameters B, P, PRI, and P0. n F c (Bandwidth, pulse width, pulse repetition interval, noise power, center frequency) have specific selection ranges, which can be expressed as:

[0010]

[0011] Where, j i This represents the i-th jamming pattern of a single jammer, where M is the number of jamming patterns. These are the parameters of the current interference signal.

[0012] From the perspective of interference signals, the possible changes after interference are divided into two layers, and reward functions are given for each layer.

[0013] 1. Radar Operating Mode Changes: Effective jamming can disrupt the normal operation of a radar, causing it to gradually switch from a high-threat mode to a low-threat mode. The specific high-threat and low-threat modes are determined by the radar's needs in performing different operational missions. The reward function r1 for radar operating mode changes is as follows:

[0014]

[0015] 2. Radar signal parameter changes: When a radar is jammed, its transmitted signal parameters are usually altered. This may involve employing anti-jamming measures such as frequency agility, switching the operating frequency to a band not covered by interference noise, and setting parameters like Δf0, Δpw, Δpri, Δbw, and Δf. c This is a threshold for the change in radar signal parameters; changes exceeding this value are considered significant fluctuations in radar signal parameters. The reward function r2 for the change in radar signal parameters is as follows:

[0016]

[0017] The radar operating state c1 transition matrix P is shown below:

[0018]

[0019] In the formula, p(c m |c1,j n () indicates that under the condition that the radar is operating in mode c1, the jamming strategy j is adopted. n The working mode has been switched to C. m The probability is the same for other operating modes. Under the condition that the radar operating state is determined, the transition matrix of the radar signal parameters is also similar.

[0020] This invention provides a radar jamming decision-making method based on a two-layer R-D3QN, comprising the following steps:

[0021] S1. Initialize the working mode memory and signal parameter memory size mem, memory pointer index, initialize the two-layer R-D3QN current value network and target value network, learning rate α, discount factor γ, greedy factor ε, working mode rule base, and signal parameter rule base;

[0022] S2. Set the training sample threshold N th Network parameter update interval time, training sample extraction interval N time (N time <N th The network update counter count is set to 0, the number of training steps x is initialized to 1, and the total number of training iterations N is set.

[0023] S3. Training: The jammer selects jamming patterns and jamming parameters based on the obtained radar operating mode and signal parameters to obtain jamming feedback rewards, and stores the jamming samples in the operating mode memory bank and the signal parameter memory bank. When the number of samples is sufficient, samples are extracted from them and the corresponding rule values ​​are selected in the operating mode rule bank and the signal parameter rule bank for network training.

[0024] S4. Update the working mode rule base and signal parameter rule base;

[0025] S5. Repeat step S3 until x reaches the total number of training iterations N, and obtain the current value network of the two trained R-D3QN models.

[0026] S6. Using the current value network, select the most valuable jamming strategy and jamming parameters based on the identified radar operating mode and signal parameters.

[0027] Furthermore, in step S1, the greed factor ε is a hyperparameter of the ε-greedy algorithm. The fully greedy algorithm selects the strategy with the highest value at every moment, which is pure exploitation without exploration. The ε-greedy algorithm adds noise to this, randomly selecting a strategy with a probability of ε each time, and selecting the strategy with the highest value with a probability of 1-ε, thus balancing the relationship between exploitation and exploration.

[0028]

[0029] This invention further improves upon the previous one by estimating the rewards for each action more accurately as the number of explorations increases, at which point exploration ceases. Therefore, in the specific implementation of the ε-greedy algorithm, ε decays over time, meaning the probability of exploration continuously decreases. This is represented as follows:

[0030]

[0031] Furthermore, step S3 is as follows:

[0032] S31. The interfering party inputs the radar operating mode c at time t. m Signal parameters Selecting interference strategy j using the current value network n and interference parameters The interfering party gains r1 t+1 , And detect the radar's operating mode at the next moment. k With signal parameters

[0033] S32, Interference experience samples

[0034] <c m ,j n ,r1 t+1 ,c k >

[0035]

[0036] Store it in memory, and increment the pointer index by 1;

[0037] S33, index>N th At that time, N are randomly selected from the working mode memory. time Given several empirical samples, calculate the current Q-value Q(c) for each sample. m ,j n |θ,ω A ,ω V And retrieve the corresponding rule value τ from the work mode rule base. mn The target Q value is used to calculate the optimization target Y = r1 t+1 +γQ(ck argmax j Q(c k ,j|θ,ω A ,ω V )|θ - ,ω A ,ω V )+τ mn Minimize the mean square error L(θ) = 1 / N constructed from the optimization objective and the current Q value. time ∑(YQ(c i ,j i |θ,ω A ,ω V )) 2 This process is repeated to train the current Q-value network, while incrementing the count by 1. The update process for the signal parameter network is similar, where the Q-value network is constructed as follows:

[0038] Q(c,j|θ,ω A ,ω V )=A(c,j|θ,ω A )+V(c|θ,ω V ), where A(c,j|θ,ω) A ) is the dominance function, V(c|θ,ω) V Let be the state value function. The schematic diagram of the R-D3QN algorithm is as follows: Figure 2 As shown.

[0039] S34. When the counter count is an integer multiple of time, assign the parameters of the current Q-value network to the target value network.

[0040] S35. This training session ends when the radar reaches the termination mode, x increments by 1; (The termination mode is the state in which the radar poses the lowest threat to the jammer during a certain combat mission).

[0041] Furthermore, in step S4, the working mode rule base is updated: τ ij (t+1)=τ ij (t)+Δτ ij (t)

[0042]

[0043] Where, τ ij Let Δτ be the rule value for the i-th interference pattern on the j-th radar operating mode. ij (t) is the change in the rule value, which is calculated using the reward value (rewards) obtained in each cycle and the number of iterations (count). The update of the signal parameter rule base is done in the same way.

[0044] This invention also discloses a radar intelligent jamming decision system based on a two-layer R-D3QN, which, based on the above method, includes the following modules:

[0045] Initialization module: Initializes the size mem of the working mode memory and signal parameter memory, the memory pointer index, and initializes the two-layer R-D3QN current value network and target value network, learning rate α, discount factor γ, greedy factor ε, working mode rule base, and signal parameter rule base;

[0046] Settings module: Set the training sample threshold N th Network parameter update interval time, training sample extraction interval N time The network update counter `count` is set to 0, the initial training step count `x` is set to 1, and the total number of training iterations `N` is set. time <N th ;

[0047] Training module: The jammer selects jamming patterns and jamming parameters based on the obtained radar operating mode and signal parameters to obtain jamming feedback rewards, and stores the jamming samples in the operating mode memory bank and the signal parameter memory bank. When the number of samples reaches the set value, samples are extracted from them and the corresponding rule values ​​are selected in the operating mode rule bank and the signal parameter rule bank for network training.

[0048] Update module: Updates the working mode rule base and signal parameter rule base;

[0049] Iteration module: Continue training until x reaches the total number of training iterations N, and obtain the current value network of the two trained R-D3QN models;

[0050] Decision module: Using the current value network, selects the most valuable jamming strategy and jamming parameters based on the identified radar operating mode and signal parameters.

[0051] Compared with the prior art, the advantages of the present invention are as follows:

[0052] (1) The first R-D3QN model selects the interference strategy and the second R-D3QN model selects the interference parameters. This reduces the dimensionality of the search space and improves the learning efficiency of the model, which can help the agent make more accurate and better decisions.

[0053] (2) This invention utilizes the rule information in the interference decision library to improve the convergence rate and sample utilization of the algorithm.

[0054] (3) The R-D3QN algorithm models the action-state value function by decomposing it into the sum of a state value function and an advantage function. This allows the agent to better handle states that are less correlated with actions. In some cases, the agent only focuses on the value of the state and is less concerned about the differences caused by different actions. By modeling the state value function and advantage function separately, this situation can be estimated more effectively, and relatively stable estimates can be obtained, thereby accelerating the convergence speed of the algorithm. In short, the R-D3QN algorithm, by decomposing the action-state value function, enables the agent to better handle states that are less correlated with actions and improves the convergence efficiency of the algorithm. Attached Figure Description

[0055] Figure 1 This is a flowchart of a preferred embodiment of the dual-layer R-D3QN radar jamming decision-making method of the present invention;

[0056] Figure 2 This is a schematic diagram of the R-D3QN algorithm.

[0057] Figure 3 This is a flowchart illustrating the interaction between the jammer and the radar.

[0058] Figure 4 This is a performance comparison chart of the two-layer R-D3QN algorithm and the DQN algorithm in the example;

[0059] Figure 5 This is a performance comparison chart of the two-layer R-D3QN algorithm and the DDQN algorithm in the example;

[0060] Figure 6 This is a performance comparison chart of the two-layer R-D3QN algorithm and the D3QN algorithm in the example;

[0061] Figure 7 This is a block diagram of a preferred embodiment of the present invention based on a dual-layer R-D3QN radar jamming decision system. Detailed Implementation

[0062] The following describes the implementation process of the present invention through preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0063] See Figure 1 This embodiment provides a radar intelligent jamming decision-making method based on a two-layer R-D3QN, which specifically includes the following steps:

[0064] S1. Initialize the size mem of the working mode memory and the signal parameter memory, the memory pointer index, and initialize the two-layer R-D3QN current value network and target value network, learning rate α, discount factor γ, greedy factor ε, working mode rule base, and signal parameter rule base;

[0065] S2. Set the training sample threshold N th Network parameter update interval time, training sample extraction interval N time (N time <N th The network update counter count is set to 0, the number of training steps x is initialized to 1, and the total number of training iterations N is set.

[0066] S3. In a training process, the details are as follows:

[0067] S31. The interfering party inputs the radar operating mode c at time t. m Signal parameters Selecting interference strategy j using the current value network n and interference parameters The interfering party gains r1 t+1 , And detect the radar's operating mode at the next moment. k With signal parameters

[0068] S32, Interference experience samples

[0069] <c m ,j n ,r1 t+1 ,c k >

[0070]

[0071] Store it in memory, and increment the pointer index by 1;

[0072] S33, index>N th At that time, N are randomly selected from the working mode memory. time Given several empirical samples, calculate the current Q-value Q(c) for each sample. m ,j n |θ,ω A ,ω V And retrieve the corresponding rule value τ from the work mode rule base. mn The target Q value is used to calculate the optimization target Y = r1 t+1 +γQ(c k argmax j Q(c k ,j|θ,ωA ,ω V )|θ - ,ω A ,ω V )+τ mn Minimize the mean square error L(θ) = 1 / N constructed from the optimization objective and the current Q value. time ∑(YQ(c i ,j i |θ,ω A ,ω V )) 2 This process is repeated to train the current Q-value network, while incrementing the count by 1. The update process for the signal parameter network is similar. The Q-value network is constructed as follows:

[0073] Q(c,j|θ,ω A ,ω V )=A(c,j|θ,ω A )+V(c|θ,ω V ), where A(c,j|θ,ω) A ) is the dominance function, V(c|θ,ω) V Let be the state value function. The schematic diagram of the R-D3QN algorithm is as follows: Figure 2 .

[0074] S34. When the counter count is an integer multiple of time, assign the parameters of the current Q-value network to the target value network.

[0075] S35. This training session ends when the radar reaches the termination mode, x increments by 1; (The termination mode is the state in which the radar poses the lowest threat to the jammer during a certain combat mission).

[0076] S4. Update the working mode rule base: τ ij (t+1)=τ ij (t)+Δτ ij (t)

[0077]

[0078] Where, τ ij Let Δτ be the rule value for the i-th interference pattern on the j-th radar operating mode. ij (t) is the change in its rule value, which is calculated using the sum of the reward values ​​obtained in each cycle and the number of iterations count. The update of the signal parameter rule base is similar.

[0079] Training continues in S5 and S3 until x reaches the total number of training iterations N, resulting in the current value networks of the two trained R-D3QN models.

[0080] S6. Using the current value network, select the jamming strategy and jamming parameters with the highest value in the Q-value network for the identified radar operating mode and signal parameters.

[0081] In step S1 of this embodiment, the greed factor ε is a hyperparameter of the ε-greedy algorithm. The fully greedy algorithm selects the strategy with the highest value at each time step, which is pure exploitation without exploration. The ε-greedy algorithm adds noise to this and randomly selects a strategy with a probability of ε each time, while selecting the strategy with the highest value with a probability of 1-ε, thus balancing the relationship between exploitation and exploration.

[0082]

[0083] This invention further improves upon the previous one by estimating the rewards for each action more accurately as the number of explorations increases, at which point exploration ceases. Therefore, in the specific implementation of the ε-greedy algorithm, ε decays over time, meaning the probability of exploration continuously decreases. This is represented as follows:

[0084]

[0085] The jammer receives a negative reward for changes in the radar's operating mode or signal parameters during jamming, indicating jamming failure. A simulation experiment was conducted using 100 radars with three operating modes and three selectable parameters for each mode, and one jammer with three jamming patterns and three selectable parameters for each mode. The jammer's failure rate was calculated over 1500 time steps (one time step is added with each iteration of reinforcement learning). The interaction flowchart between the jammer and the radar is shown below. Figure 3 .

[0086] To verify the effectiveness of this invention, a comparative experiment was conducted using the dual-layer R-D3QN algorithm against the DQN, DDQN, and D3QN algorithms. The simulation results of the interference failure rate of the dual-layer R-D3QN and DQN algorithms over time are shown below. Figure 4 The experimental simulation results of the interference failure rate of the two-layer R-D3QN algorithm and the DDQN algorithm changing with time are as follows: Figure 5 The experimental simulation results of the interference failure rate of the two-layer R-D3QN algorithm and the D3QN algorithm changing over time are as follows: Figure 6 Simulation results show that although the failure rate of the two-layer R-D3QN algorithm is higher than that of other algorithms in the early stage of training, the convergence speed of the two-layer R-D3QN algorithm is better than that of other algorithms as the number of training rounds increases. Eventually, the interference failure rate of the two-layer R-D3QN algorithm converges to about 10%, while that of other algorithms is higher than 10%. This demonstrates the superiority of this invention over traditional reinforcement learning algorithms in the field of radar interference decision-making.

[0087] like Figure 7 As shown, this embodiment discloses a radar intelligent jamming decision-making system based on a two-layer R-D3QN. Based on the above method embodiment, it includes the following modules:

[0088] Initialization module: Initializes the size mem of the working mode memory and signal parameter memory, the memory pointer index, and initializes the two-layer R-D3QN current value network and target value network, learning rate α, discount factor γ, greedy factor ε, working mode rule base, and signal parameter rule base;

[0089] Settings module: Set the training sample threshold N th Network parameter update interval time, training sample extraction interval N time The network update counter `count` is set to 0, the initial training step count `x` is set to 1, and the total number of training iterations `N` is set. time <N th ;

[0090] Training module: The jammer selects jamming patterns and jamming parameters based on the obtained radar operating mode and signal parameters to obtain jamming feedback rewards, and stores the jamming samples in the operating mode memory bank and the signal parameter memory bank. When the number of samples reaches the set value, samples are extracted from them and the corresponding rule values ​​are selected in the operating mode rule bank and the signal parameter rule bank for network training.

[0091] Update module: Updates the working mode rule base and signal parameter rule base;

[0092] Iteration module: Continue training until x reaches the total number of training iterations N, and obtain the current value network of the two trained R-D3QN models;

[0093] Decision module: Using the current value network, selects the most valuable jamming strategy and jamming parameters based on the identified radar operating mode and signal parameters.

[0094] Other aspects of this embodiment can be found in the above method embodiments.

[0095] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A radar intelligent jamming decision-making method based on dual-layer R-D3QN, characterized in that, Includes the following steps: S1. Initialize the working mode memory and signal parameter memory size mem, memory pointer index, initialize the two-layer R-D3QN current value network and target value network, and learning rate. Discount Factor Greed factor Operating mode rule base, signal parameter rule base; S2. Set the training sample threshold N th Network parameter update interval time, training sample extraction interval N time And the network update counter count is set to 0, the number of training steps x is initialized to 1, and the total number of training iterations N is set; ; S3. Training: The jammer selects jamming patterns and jamming parameters based on the obtained radar operating mode and signal parameters to obtain jamming feedback rewards, and stores the jamming samples in the operating mode memory bank and the signal parameter memory bank. When the number of samples reaches the set value, samples are extracted from them and the corresponding rule values ​​are selected in the operating mode rule bank and the signal parameter rule bank for network training. S4. Update the working mode rule base and signal parameter rule base; S5. Return to the training in step S3 until x reaches the total number of training iterations N, and obtain the current value network of the two trained R-D3QN models. S6. Using the current value network, select the most valuable jamming strategy and jamming parameters based on the identified radar operating mode and signal parameters. Step S3 is as follows: S31. The interfering party inputs the radar operating mode at time t. Pulse repetition interval Using the current value network to select the interference strategy and interference parameter bandwidth The interfering party gains benefits. And detect the radar's operating mode in the next moment. With signal parameters ; S32, Interference experience samples Store it in memory, and increment the pointer index by 1; S33, At that time, N are randomly selected from the working mode memory. time Given several empirical samples, calculate the current Q-value for each sample. And retrieve the corresponding rule values ​​from the work mode rule base. Calculate the optimization target using the target Q value Minimize the mean squared error constructed from the optimization objective and the current Q value. The current Q-value network is trained, and the count is incremented by 1. During the update process of the signal parameter network, samples are drawn from the signal parameter memory to calculate the current Q-value, and the optimization target is calculated using the rule values ​​in the signal parameter rule base and the target Q-value to minimize its mean square error. The Q-value network is constructed as follows: ,in, For the dominant function, The state value function; S34. When the counter count is an integer multiple of time, assign the parameters of the current Q-value network to the target value network. S35. Training ends when the radar reaches termination mode, x increments by 1.

2. The radar intelligent jamming decision-making method based on dual-layer R-D3QN according to claim 1, characterized in that, In step S1, the greedy factor In a greedy algorithm, let Decay over time is represented as follows: 。 3. The radar intelligent jamming decision-making method based on dual-layer R-D3QN according to claim 1 or 2, characterized in that, In step S4, update the working mode rule base: Where rewards are the rewards obtained from the interference. Let be the rule value for the i-th interference pattern in the j-th radar operating mode. It is the change in the rule value, calculated using the reward value obtained in each cycle and the number of iterations. In the signal parameter rule base update, the change in the rule value is obtained by dividing the interference gain value by the counter value, and then updated to the signal parameter rule base.

4. A radar intelligent jamming decision-making system based on a dual-layer R-D3QN, wherein the method described in any one of claims 1-3 is characterized in that, Includes the following modules: Initialization module: Initializes the working mode memory and signal parameter memory size (mem), memory pointer (index), and initializes the two-layer R-D3QN current value network and target value network, as well as the learning rate. Operating mode rule base, signal parameter rule base; Settings module: Set the training sample threshold N th Network parameter update interval time, training sample extraction interval N time And the network update counter count is set to 0, the number of training steps x is initialized to 1, and the total number of training iterations N is set; ; Training module: The jammer selects jamming patterns and jamming parameters based on the obtained radar operating mode and signal parameters to obtain jamming feedback rewards, and stores the jamming samples in the operating mode memory bank and the signal parameter memory bank. When the number of samples reaches the set value, samples are extracted from them and the corresponding rule values ​​are selected in the operating mode rule bank and the signal parameter rule bank for network training. Update module: Updates the working mode rule base and signal parameter rule base; Iteration module: Continue training until x reaches the total number of training iterations N, and obtain the current value network of the two trained R-D3QN models; Decision module: Using the current value network, selects the most valuable jamming strategy and jamming parameters based on the identified radar operating mode and signal parameters.

Images