A multi-jammer intelligent frequency domain jamming decision method for networked radar countermeasure

Abstract

The application discloses a kind of multi-interferometer intelligent frequency domain interference decision methods for networking radar countermeasure, constructs multi-interferometer intelligent cooperative interference model, and establishes the Markov decision process of multi-agent reinforcement learning for countermeasure process, and establishes the whole system of countermeasure using centralized learning paradigm;According to the idea of hierarchical reinforcement learning, the total task of frequency domain interference parameter decision maker is decomposed, the strategy of each subtask is obtained by the task level of division, the strategies of multiple subtasks are combined to form an effective global strategy;Three space modules of decision engine are defined respectively, PER-DDQN is used as the network basic structure, and the decision strategy is output;Priority auxiliary experience playback is introduced in network optimization module, SumTree data structure is used as the experience pool of sample playback, and the parameters of deep neural network are updated according to function and parameter update formula.The application uses deep reinforcement learning algorithm with priority experience playback to improve the training speed of network.

Description

Technical Field

[0001] This invention belongs to the field of radar countermeasures technology, and in particular relates to a multi-jammer intelligent frequency domain jamming decision-making method for countering networked radar systems. Background Technology

[0002] Compared to traditional single-radar systems, networked radar offers advantages such as multi-radar data integration and resource sharing. Different radars within a network possess varying frequency bands, airspace coverage, and operating modes, resulting in superior anti-jamming performance that makes traditional one-to-one jamming methods ineffective. As intelligent radar systems, such as multi-functional radars, continue to evolve and mature into networked systems, intelligent jamming decision-making technologies to counter them have emerged. The key to countering intelligent jamming decision-making by networked radar lies in the optimal allocation of limited jamming resources to form a high-performance jamming architecture, thereby enhancing the overall countermeasure capability.

[0003] Traditional radar jamming decision-making methods typically require substantial prior knowledge to support the decision-making process. To overcome this dependency, reinforcement learning-based jamming decision-making methods have emerged. However, current reinforcement learning-based jamming decision-making methods suffer from the following problems: First, in adversarial networked radar problems, the high complexity of the jamming waveform parameter space leads to long convergence times for reinforcement learning, making it difficult for the jammer to determine the optimal parameters within a limited time. Second, in one-to-one jamming scenarios, jamming actions usually only consider the physical layer jamming patterns. In many-to-many adversarial scenarios, the focus is more on the allocation relationship between the radar and the jammer, lacking specific jamming parameter decision-making methods for intelligent jamming techniques targeting networked radars. Third, especially when radar employs anti-jamming measures such as frequency agility and frequency diversity, how can multiple jammers achieve optimal jamming effects with minimal jamming force, and realize intelligent cooperative jamming decision-making in the frequency domain without consuming excessive electromagnetic spectrum resources?

[0004] The patent "DQN Radar Jamming Decision-Making Method and Device Based on Priority Importance Sampling Fusion" (application number: CN202210605946.5) only selects jamming patterns to form adversarial examples and does not involve decision-making jamming waveform parameters, which differs from this invention patent; the patent "A Radar Suppression Jamming Decision-Making Method Based on Reinforcement Learning" (application number: CN202210343419.1) includes jamming patterns and jamming parameters in the jamming strategy generation, but the adversarial model is established in a "one-to-one" mode, which differs from this invention patent; the patent "A Multi-Jerker Cooperative Suppression Resource Scheduling Method for Radar Systems" (application number: CN202210930783.8) selects jamming patterns and jamming parameters from multiple pairs of jammers to form adversarial examples .... The paper presents methods designed from multiple adversarial angles, but uses a particle swarm optimization algorithm to allocate resources for jammer beam pointing and transmission power, which differs from this invention patent. The paper "An Intelligent Decision-Making Algorithm for Communication Counter-Interference Resource Allocation" addresses the challenge of frequency-hopping jamming decision-making, employing a hierarchical reinforcement learning model to determine the jamming frequency band and bandwidth. However, it utilizes an experience replay mechanism based on bootstrapping expert trajectories to optimize the algorithm. Its application background is communication signal counter-interference, with different signal characteristics and types, which differs from this invention patent. The paper "A Collaborative Communication Jamming Decision-Making Algorithm Based on Deep Reinforcement Learning" designs a decision-making algorithm under the Actor-Critic algorithm architecture, belonging to an adversarial strategy targeting communication signals, which differs from this invention patent. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-jammer intelligent frequency domain jamming decision-making method for countering networked radar systems.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A multi-jamming intelligent frequency domain jamming decision-making method for countering networked radar systems comprises the following steps:

[0008] Step 1: Construct a multi-jammer intelligent cooperative jamming model;

[0009] The adversarial process is established as a Markov decision process based on multi-agent reinforcement learning, and the decision tuple element representation is given under a preset number of agents; a centralized learning paradigm is adopted to establish the overall adversarial system, and mutual communication and cooperation between the jammers are realized through a shared experience pool and value function;

[0010] Step 2: Decompose the overall task of the frequency domain interference parameter decision-maker according to the idea of ​​hierarchical reinforcement learning;

[0011] The reinforcement learning structure is divided into layers according to the timeline, and sub-tasks for each layer are defined; a dual interference decision engine with interference frequency band and interference bandwidth is deployed; a global decision-making process is established, and the execution flow of each sub-task is given according to the timeline.

[0012] Step 3: Define the three spatial modules of the decision engine;

[0013] Define the state space module for each decision engine; establish an interference effect evaluation module, i.e., a reward space module that provides immediate rewards at each time step; combine the output sub-policies to form a global policy and establish an action space module.

[0014] Step 4: Introduce priority-assisted experience replay in the network optimization module;

[0015] The SumTree data structure is used as the experience pool for sample replay. The training samples extracted from PER-DDQN are stored in the structure, and the tree structure is retrieved from top to bottom. The parameters of the deep neural network are updated according to the function and parameter update formula. The overall algorithm decision-making process based on PER-DDQN is given.

[0016] Furthermore, if step 1 selects I jammers, then the Markov decision process with I agents can be represented as follows: This includes the set of state spaces s∈S for all agents and the set of joint action spaces. The state transition probability function P and the shared reward function R; γ∈(0,1) is the discount factor; the state transition probability function is... This represents the probability distribution of transitioning to the next state by taking a joint action from all agents in the current state; at this point, the agents receive the corresponding reward value. r ti This represents the immediate reward received by the i-th agent after performing an action.

[0017] Furthermore, step 2 decomposes the Markov decision process M of multi-agent reinforcement learning into a first-layer subtask "selecting the attack target" and a second-layer subtask "determining frequency domain interference parameters" in chronological order. The second-layer structure deploys two interference decision engines, which control the selection of interference frequency band and interference bandwidth at each time step. The interference parameters are combined as follows: carrier frequency and bandwidth. The interference frequency band decision has a constraint on the selection of interference bandwidth parameters, and the interference bandwidth decision assists in the optimization of parameters for frequency band selection.

[0018] Furthermore, step 2 establishes a global decision-making process, and the subtask execution flow is as follows:

[0019] (1) Initialization preparation process;

[0020] The bandwidth candidate parameters in the preset bandwidth library divide all target frequency points of the attack target into sub-bands; after the frequency points are arranged in frequency order, if the frequency difference is greater than the preset maximum bandwidth, the two frequency points are divided into two different sub-bands, until all frequency points are divided into each sub-band.

[0021] (2) The global decision-making process is represented as a decomposition of multiple sub-tasks {M0, M... 1t M 2t M 3t ,…,M 1T M 2T M 3T}; where M0 is the root task, which outputs the global strategy for multi-jammer cooperative jamming frequency domain parameters and has the highest level hierarchy; the remaining subtasks M ik (i = 1, 2, 3) represents the strategy that the corresponding task needs to form at the current time k; M1 represents the attack target subtask, located in the first layer of the hierarchical structure; M2 represents the decision interference frequency band; M3 represents the decision interference bandwidth, located in the second layer of the hierarchical structure.

[0022] (3) Each subtask in its respective layer is performed sequentially according to the time step in its hierarchical structure;

[0023] First, select the attack target, then decide on the interference frequency band and interference bandwidth; completing a predetermined action within each time step will result in an immediate reward, until the network training and optimization results reach the optimal level.

[0024] Furthermore, in step 3, the number of sub-bands is set to M, and the interference target is determined based on the threat coefficient; the state space of the interference frequency band decision-maker can be represented as:

[0025] S cf =[target,U]

[0026] U = [U1, U2, ..., U M [] indicates the number of frequency points of the target contained in each sub-band;

[0027] The state space of the interference bandwidth decision-maker can be represented as:

[0028]

[0029] A cf This is the output action of the interference frequency band determiner; The number of frequency points of the interfering targets contained in the selected sub-band; The number of frequency points for all targets within the selected sub-band;

[0030] The reward value r of the interference band decision-maker cf By r cfmain rcfadd1 and r cfadd2 It consists of three parts; the reward value of each part is expressed as follows:

[0031]

[0032] When a target that has been successfully interfered with becomes an attack target, the reward value is α times the target's threat coefficient; otherwise, the reward value is reward1.

[0033]

[0034] Where φ is the number of frequency points blocked in the selected sub-band, Φ is the total number of frequency points of all targets, and I is the number of jammers;

[0035]

[0036] If the number of blocking frequency points in the selected sub-band is greater than the average number of frequency points covered by all jammers, a reward value of reward2 is obtained; otherwise, the reward value will be reward3. The two are balanced to maintain the reward value, satisfying |reward2|=|reward3|. If multiple jammers repeatedly select a certain frequency band, a reward value of reward4 is obtained.

[0037] r cf =r cfmain +r cfadd1 +r cfadd2

[0038] Where h is the target of successful interference, t is the threat coefficient of h, and α is the weighting factor of the threat coefficient;

[0039] The reward value r of the interference bandwidth decision-maker bw By r bwmain and r bwadd Composition; represented as follows:

[0040]

[0041] The reward value obtained from the selected interference bandwidth increases proportionally with the number of interference frequency points within the current frequency range;

[0042]

[0043] B extra This represents the size of the redundant bandwidth that is not covered at frequency points; when the redundant bandwidth exceeds the preset bandwidth quantization interval ρ, a reward of size reward6 is obtained.

[0044] r bw =r bwmain +r bwadd

[0045] d sub D represents the number of interference frequency points within the selected bandwidth. band This indicates the total number of interfering frequency points within the current frequency band;

[0046] The actions output by the two decision-makers are combined to form the output decision strategy, denoted as P = [A1, A2]; the interference action A1 is output by the interference frequency band determiner, indicating that the interference sub-frequency band has been selected; the interference bandwidth determiner outputs the interference action A2, indicating that the bandwidth is selected from the preset bandwidth library.

[0047] Furthermore, step 4 uses a SumTree data structure as the experience pool for sample replay; the leaf nodes store the training samples extracted from PER-DDQN, and their values ​​represent the priority of the samples. The larger the value, the greater the probability of extracting samples from that interval length during uniform sampling; the steps of searching the tree structure from top to bottom are as follows:

[0048] (1) A random number x is drawn on average from each subinterval, and m subintervals will yield m uniformly distributed random numbers;

[0049] (2) Start searching from the topmost root node downwards; if x is less than the left child node, select the left node and continue searching downwards;

[0050] (3) If x is greater than the left child node, the value of x needs to be updated to x - the value of the left node, and then the right node is selected to search downwards;

[0051] (4) Following the rules of steps 2-3, continue until the lowest-level leaf node is found, which stores the quadruple. t ,a t ,r t ,s t+1 > This is the sample to be drawn from the interval. Repeat this process to obtain m samples.

[0052] The probability of each sample being sampled can be represented by its priority as follows:

[0053]

[0054] In the formula, P i p represents the probability that sample i will be sampled. i This indicates the priority of sample i, and k represents the sample index, i.e., p. k Indicates the priority of sample k;

[0055] In each iteration, the priority of the samples is updated based on the time difference error. Taking sample i as an example, the priority p i Represented as:

[0056] p​i =|y i -Q(s i ,a i ;θ i )|

[0057] In the formula, y i Let Q(s) be the target value of the sample. i ,a i ;θ i To evaluate the network's performance on the currently updated weight parameters θ i The state-action pair (s) i ,a i The action value function.

[0058] Furthermore, the overall algorithm decision-making process of step 4, PER-DDQN, is as follows:

[0059] (1) Initialize the SumTree data structure for storing training samples in the experience pool;

[0060] (2) Initialize the weight parameters θ and θ' of the evaluation network and the target network, and randomly generate the parameters, but the two must be consistent;

[0061] (3) Reconnoiter the frequency domain status information of the networked radar, initialize the bandwidth library, set the alternative bandwidth size, and divide the sub-frequency bands according to the preparatory process instructions;

[0062] (4) Determine the threat level of each radar and prioritize selecting radars with higher threat levels as attack targets. If the frequency points interfered with reach a preset proportion of the total number of targets, the threat level of that radar will be reduced.

[0063] (5) Determine the interference action of the dual decision engine based on the exploration rate ε. Randomly select the interference action with a probability of ε and select the best action to evaluate the network output with a probability of 1-ε, as expressed as:

[0064] a t =argmaxQ(s t ,a t )

[0065] (6) Obtain the instantaneous rewards of the two decision engines respectively from the interference effect evaluation module, and denote them uniformly as r. t ;

[0066] (7) Affected by the interference action, update the environmental state and check the new environmental state, denoted as s. t+1 ;

[0067] (8) Each sample t ,a t ,r t ,s t+1 ​Store the samples in the experience pool and update their priority based on the time difference error.

[0068] (9) When the number of stored samples reaches the capacity of the experience pool, select m samples according to the instructions for sampling from the experience pool. The approximation target value for each sample is expressed as:

[0069] y t =r t+1 +γQ(s t+1 ,argmaxQ(s t+1 ,a t+1 |θ t )|θ t ')

[0070] Where γ is the discount factor;

[0071] (10) Calculate the loss function, and based on the loss function, reduce the gap between the sample target and the current Q value, and update the network parameters;

[0072] (11) When the number of iterations reaches a multiple of the number of network assignments, the network weight parameters of the evaluation network are assigned to the target network weight parameters, and the two remain synchronized at this time.

[0073] (12) Repeat steps 4-11 until all jamming machine resources are used up, and then proceed to the next iteration until the number of iterations is terminated.

[0074] The beneficial effects of this invention are as follows:

[0075] This invention addresses the problem of excessively large interference parameter space when countering networked radars. It considers radar anti-jamming measures, proposes an intelligent interference decision-making method from the frequency domain perspective, and employs a deep reinforcement learning algorithm with priority experience replay to improve the network training speed.

[0076] The innovations of this invention are twofold: First, it establishes the adversarial process of multiple jammers as a Markov decision process of multi-agent reinforcement learning, effectively solving the large parameter space problem brought about by networked radar through the idea of ​​hierarchical reinforcement learning; second, it adopts the deep reinforcement learning PER-DDQN algorithm to further optimize network training, accelerate the training speed of the network, and effectively improve the learning efficiency of the network. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of the process of the present invention;

[0078] Figure 2 This is a schematic diagram illustrating how the overall task of the frequency domain interference parameter decision-maker is decomposed according to the idea of ​​hierarchical reinforcement learning in this invention.

[0079] Figure 3This is a flowchart illustrating the overall decision-making process of the PER-DDQN algorithm based on the present invention.

[0080] Figure 4 This is a schematic diagram illustrating the interference effect of a single decision-making process of an eight-unit jammer in an example of the present invention.

[0081] Figure 5(a) shows the error curve of the DDQN algorithm used in this invention;

[0082] Figure 5(b) shows the error curve of the PER-DDQN algorithm used in this invention. Detailed Implementation

[0083] The present invention will now be further described with reference to the accompanying drawings.

[0084] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0085] Reference Figure 1 This method includes: constructing a multi-jammer intelligent cooperative jamming model, establishing the adversarial process as a Markov decision process of multi-agent reinforcement learning, and establishing an overall adversarial system using a centralized learning paradigm; decomposing the overall task of the frequency domain jamming parameter decision-maker according to the idea of ​​hierarchical reinforcement learning, with the policy of each sub-task being learned from the divided task hierarchy, and combining the policies of multiple sub-tasks to form an effective global policy; defining three spatial modules of the decision engine, using PER-DDQN as the network infrastructure, and outputting the decision policy; introducing priority-assisted experience replay in the network optimization module, using the SumTree data structure as the experience pool for sample replay, and updating the parameters of the deep neural network according to the function and parameter update formula.

[0086] Construct a multi-jammer intelligent cooperative jamming model. The adversarial process is modeled as a multi-agent reinforcement learning Markov decision process with a centralized learning paradigm. If the number of jammers is selected as I, then the Markov decision process with I agents can be represented as follows: This includes the set of state spaces s∈S for all agents and the set of joint action spaces. The state transition probability function P and the shared reward function R are given. γ∈(0,1) is the discount factor. The state transition probability function is... This represents the probability distribution of transitioning to the next state by taking the joint action of all agents in the current state. At this point, the agents receive the corresponding reward value. r ti This represents the immediate reward received by the i-th agent after performing an action.

[0087] The overall task of the frequency domain interference parameter decision-maker is decomposed according to the idea of ​​hierarchical reinforcement learning. (Refer to...) Figure 2 The Markov decision process M of multi-agent reinforcement learning is decomposed along the time axis into two subtasks: a first-layer subtask of "selecting the attack target" and a second-layer subtask of "determining frequency domain interference parameters." The second-layer structure deploys two interference decision engines, which control the selection of interference frequency bands and bandwidths at each time step. The interference parameters are combined to form [carrier frequency, bandwidth]. The interference frequency band decision constrains the selection of interference bandwidth parameters, while the interference bandwidth decision assists in frequency band selection for parameter optimization; the two complement each other. A global decision process is established, and the subtask execution flow is as follows:

[0088] Step 1: Initialization preparation process. The bandwidth candidate parameters in the preset bandwidth library are used to divide all target frequency points of the attack target into sub-bands. After arranging consecutive frequency points in frequency order, if the frequency difference between them is greater than the preset maximum bandwidth, the two frequency points are divided into two different sub-bands, until all frequency points are divided into sub-bands.

[0089] Step 2: The global decision-making process is represented as a decomposition of multiple subtasks {M0, M... 1t M 2t M 3t ,…,M 1T M 2T M 3T}. M0 is the root task, which outputs the global strategy for multi-jammer cooperative jamming frequency domain parameters and has the highest level hierarchy. The remaining subtasks M ik (i = 1, 2, 3) represents the strategy that the corresponding task needs to form at the current time k. M1 represents the attack target subtask, located in the first layer of the hierarchical structure; M2 represents the decision jamming frequency band; and M3 represents the decision jamming bandwidth, located in the second layer of the hierarchical structure.

[0090] Step 3: Each subtask is performed sequentially within its respective layer structure, step by step. First, the attack target is selected, then the jamming frequency band and jamming bandwidth are decided. Completing a predetermined action within each time step will result in an immediate reward, until the network training and optimization results reach their optimal state.

[0091] PER-DDQN is adopted as the network infrastructure, and three spatial modules of the decision engine are defined. Assuming the number of sub-bands is M, the interference target is determined based on the threat coefficient. The state space of the interference band decision engine can be represented as:

[0092] S cf =[target,U] (1)

[0093] U = [U1, U2, ..., U M[] indicates the number of frequency points of the target contained in each subband.

[0094] The state space of the interference bandwidth decision-maker can be represented as:

[0095]

[0096] A cf This is the output action of the interference frequency band determiner; The number of frequency points of the interfering targets contained in the selected sub-band; This represents the number of frequency points for all targets within the selected sub-band.

[0097] The reward value r of the interference band decision-maker cf By r cfmain r cfadd1 and r cfadd2 It consists of three parts. The reward value of each part is expressed as follows:

[0098]

[0099]

[0100]

[0101] r cf =r cfmain +r cfadd1 +r cfadd2 (6)

[0102] Where h is the target successfully jammed, t is the threat coefficient of h, and α is the weighting factor of the threat coefficient. As shown in equation (3), when a target successfully jammed is an attack target, the reward value obtained is α times the target threat coefficient. Otherwise, the reward value is reward1. In equation (4), φ is the number of frequency points blocked in the selected subband, Φ is the total number of frequency points of all targets, and I is the number of jammers. Equation (4) means that if the number of blocked frequency points in the selected subband is greater than the average number of frequency points covered by all jammers, the reward value reward2 is obtained; otherwise, the reward value will be reward3. The two are to maintain the balance of reward values, satisfying |reward2|=|reward3|. In particular, if multiple jammers repeatedly select a certain frequency band, a reward value of reward4 is obtained. Because this additional reward value is used to regulate the problem of repeated selection among multiple jammers, the difference between reward4 and reward5 should be large enough to enable the jammers to quickly meet the training requirements.

[0103] The reward value r of the interference bandwidth decision-maker bw By r bwmain and r bwaddComposition. Represented as follows:

[0104]

[0105]

[0106] r bw =r bwmain +r bwadd (9)

[0107] d sub D represents the number of interference frequency points within the selected bandwidth. band This indicates the total number of interfering frequency points within the current frequency band.

[0108] As shown in equation (7), the reward value obtained from the selected interference bandwidth increases proportionally with the number of interference frequency points within the current frequency range. In equation (8), B extra This represents the amount of redundant bandwidth that is not covered at frequency points. When the redundant bandwidth exceeds the preset bandwidth quantization interval ρ, a reward of size reward6 is awarded.

[0109] The actions output by the two decision-makers are combined to form the output decision strategy, denoted as P = [A1, A2]. Interference action A1 is output by the interference frequency band determiner, indicating the selection of an interference sub-band. The interference bandwidth determiner outputs interference action A2, indicating the bandwidth selected from a preset bandwidth library.

[0110] Priority-assisted experience replay is introduced in the network optimization module. A SumTree data structure is used as the experience pool for sample replay. The leaf nodes store the training samples extracted from PER-DDQN, and their values ​​represent the priority of the sample; the larger the value, the greater the probability of extracting a sample from that interval during uniform sampling. The specific steps for searching the tree structure from top to bottom are as follows:

[0111] Step 1: Draw a random number x from each sub-interval on average. After drawing m sub-intervals, you will get m uniformly distributed random numbers.

[0112] Step 2: Start searching from the top-level root node downwards. If x is less than the left child node, select the left node and continue searching downwards;

[0113] Step 3: If x is greater than the left child node, the value of x needs to be updated to x - the value of the left node, and then the right node is selected to search downwards;

[0114] Step 4: Following the rules of Steps 2-3, continue until the lowest-level leaf node is found, which stores the quadruple. t ,a t ,r t ,s t+1 ​> This is the sample to be drawn from the interval. Repeat this process to obtain m samples.

[0115] The probability of each sample being sampled can be represented by its priority.

[0116]

[0117] In the formula, P i p represents the probability that sample i will be sampled. i This indicates the priority of sample i, and k represents the sample index, i.e., p. k This indicates the priority of sample k.

[0118] In each iteration, the priority of the samples is updated based on the time difference error. Taking sample i as an example, the priority p i Represented as:

[0119] p i =|y i -Q(s i ,a i ;θ i (11)

[0120] In the formula, y i Let Q(s) be the target value of the sample. i ,a i ;θ i To evaluate the network's performance on the currently updated weight parameters θ i The state-action pair (s) i ,a i The action value function.

[0121] Reference Figure 3 The overall algorithm decision-making process based on PER-DDQN in this embodiment is presented.

[0122] Step 1: Initialize the SumTree data structure for storing training samples in the experience pool;

[0123] Step 2: Initialize the weight parameters θ and θ' of the evaluation network and the target network. Randomly generate the parameters, but keep them consistent.

[0124] Step 3: Reconnoiter the frequency domain status information of the networked radar, initialize the bandwidth library, set the alternative bandwidth size, and divide the sub-frequency bands according to the preparatory process instructions;

[0125] Step 4: Determine the threat level of each radar, and prioritize radars with higher threat levels as attack targets. If the frequency points interfered with reach a preset proportion of the total number of targets, the threat level of that radar will be reduced.

[0126] Step 5: Determine the interference actions for the dual decision engines based on the exploration rate ε. Randomly select interference actions with a probability of ε, and select the best action to evaluate the network output with a probability of 1-ε, expressed as:

[0127] a t =argmaxQ(s t ,a t (12)

[0128] Step 6: Obtain the immediate rewards of the two decision engines respectively according to the interference effect evaluation module, and denote them uniformly as r. t ;

[0129] Step 7: Due to the interference action, update the environment state and check the new environment state, denoted as s. t+1 ;

[0130] Step 8: For each sample t ,a t ,r t ,s t+1 Store the samples in the experience pool and update their priority based on the time difference error.

[0131] Step 9: When the number of stored samples reaches the capacity of the experience pool, select m samples according to the sample extraction instructions for the experience pool. The approximation target value for each sample is expressed as:

[0132] y t =r t+1 +γQ(s t+1 ,argmaxQ(s t+1 ,a t+1 |θ t )|θ t ') (13)

[0133] Where γ is the discount factor.

[0134] Step 10: Calculate the loss function, and based on the loss function, reduce the gap between the sample target and the current Q value, and update the network parameters;

[0135] Step 11: When the number of iterations reaches a multiple of the number of network assignments, assign the network weight parameters of the evaluation network to the target network weight parameters, and keep the two in a synchronized state at this time;

[0136] Step 12: Repeat steps 4-11 until all jamming machine resources are used up, then proceed to the next iteration until the iteration count is terminated.

[0137] To verify the beneficial effects of the present invention, the following simulation experiments were conducted:

[0138] Constructing the networked radar environment detected by reconnaissance: ​

[0139] As shown in Table 1, the network radar frequency set is configured as follows:

[0140] Table 1

[0141]

[0142] The higher the threat level value, the greater the threat level; that is, Radar 1 has the highest threat level.

[0143] The preprocessing of simulation adversarial parameters is shown in Table 2:

[0144] Table 2

[0145]

[0146] Reference Figure 4 This paper demonstrates the jamming effect of eight jammers in a single decision-making process. Jamming is applied sequentially to radars 1, 2, and 3 according to their threat coefficients. Action is taken to jam the next target only when the proportion of jammed frequencies for radar 1 exceeds 80% of its total frequency range, satisfying the simulation design. Furthermore, radar 4's operating frequency band overlaps with radars 1 and 2; therefore, when the jammer is targeting in real-time, it will affect radar 4's frequency band, resulting in radar 4 being jammed first. Referring to Figures 5(a) and 5(b), the algorithm error curves for 3200 training steps recorded during 4000 training rounds are shown. It can be seen that the PER-DDQN algorithm has smaller errors, maintains relatively stable error fluctuations during most training processes, and its error curve converges significantly faster than the DDQN algorithm. It can quickly achieve optimal jamming decision-making effects while reducing the number of network training rounds, improving jamming decision-making efficiency and making it more suitable for radar countermeasures technology.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for multi-jammer intelligent frequency domain jamming decision against networked radar countermeasures, characterized in that: The specific steps are as follows: Step 1: Constructing a multi-jammer intelligent cooperative jamming model; The countermeasure process is established as a Markov decision process of multi-agent reinforcement learning, and the decision tuple element representation is given under the preset number of agents; The centralized learning paradigm is used to establish the overall countermeasure system, and the mutual communication and cooperation between each jammer is realized through the shared experience pool and value function; Step 2: According to the idea of hierarchical reinforcement learning, the total task of the frequency domain jamming parameter decision maker is decomposed; The structure level of reinforcement learning is divided according to the time axis, and the subtasks of each level are formulated; Deploy the jamming frequency band and jamming bandwidth double jamming decision engine; Establish the global decision process, and give the execution process of each subtask divided according to the time axis; The Markov decision process M of multi-agent reinforcement learning is decomposed into the first layer subtask "selecting attack target" and the second layer subtask "deciding frequency domain jamming parameters" according to the order of time axis; Two jamming decision engines are deployed in the second layer structure, which control the selection of jamming frequency band and jamming bandwidth at each time step, and the jamming parameters decided by the two are combined as: carrier frequency, bandwidth; The jamming frequency band decision has a constraint effect on the parameter selection of the jamming bandwidth, and the jamming bandwidth decision assists the frequency band selection in parameter optimization; Establish the global decision process, and the subtask execution process is: (1) Initialization preparation process; Preset the bandwidth candidate parameters in the bandwidth library, and divide all target frequency points of the attack object into subbands; After arranging the front and rear frequency points in frequency order, if the frequency difference is greater than the preset maximum bandwidth, the two frequency points are divided into two different subbands, and all frequency points are divided into each subband; (2) Global decision-making process is represented as decomposition of multiple sub-tasks {M0, M 1t ,M 2t ,M 3t ,…,M 1T ,M 2T ,M 3T}; wherein M0 is the root task, i.e. output multi-jammer cooperative jamming frequency domain parameter global strategy, with the highest level hierarchy; the rest of the sub-tasks M ik (i = 1, 2, 3) represent the strategy that needs to be formed at the current time k; M1 represents the attack target sub-task, located at the first layer of the hierarchical structure; M2 represents the decision-making jamming frequency band, M3 represents the decision-making jamming bandwidth, located at the second layer of the hierarchical structure; (3) Each layer subtask is carried out in the corresponding hierarchical structure according to the time step; First, select the attack target, then decide the jamming frequency band and the jamming bandwidth; Each time step completes a given action to get an instant reward, until the network training optimization result reaches the best; Step 3: Define three space modules of decision engine respectively; Define the state space module of each decision engine; Establish the jamming effect evaluation module, i.e. the reward space module, to give the instant reward at each time step; Combine the output sub-strategies to form the global strategy, and establish the action space module; Step 4: Introduce priority auxiliary experience replay in network optimization module; Use SumTree data structure as the experience pool for sample playback, store the training samples of PER-DDQN in the structure, and search the tree structure from top to bottom; Update the parameters of deep neural network according to the function and parameter update formula; Give the overall algorithm decision process based on PER-DDQN.

2. The multi-jammer intelligent frequency domain jamming decision method against networked radar countermeasures according to claim 1, characterized in that: The step 1 selects the number of jamming machines as I, and a Markov decision process with I agents can be expressed as where S is the state space set of all agents, A is the joint action space set of all agents, P is the state transition probability function, R is the shared reward function, and γ is the discount factor. The state transition probability function P and the shared reward function R; γ ∈ (0, 1) is the discount factor; the state transition probability function is represents the probability distribution of transitioning to the next state under the current state by taking the joint action of all agents; at this time, the agent gets the corresponding reward value r ti represents the immediate reward obtained by the i-th agent after performing the action.

3. The method of claim 1, wherein the method further comprises: determining a frequency of the jammer; and determining a frequency of the radar. In step 3, the number of subbands is M, and the jamming target target is determined according to the threat coefficient; The state space of the jamming frequency band decision maker can be represented as: S cf = [target, U] U = [U1, U2,..., U M ] denotes the number of frequency points containing the target in each sub-band; The state space of the jamming bandwidth decision maker can be represented as: A cf is an output action of the interference band decider; is the number of frequency points of the interference target target contained in the selected subband; is the number of frequency points of all targets within the selected subband; The reward value r of the interference band decision maker cf The reward value r of the interference band decision maker cfmain The reward value r of the interference band decision maker cfadd1 The reward value r of the interference band decision maker cfadd2 The reward value r of the interference band decision maker When a jammed target is the attack target, the reward value is α times the threat coefficient of the target; Otherwise, the reward value is reward1; Where φ is the number of frequency points blocked by the selected subband, Φ is the total number of target frequency points, and I is the number of jammers. If the number of jamming frequency points in the selected sub-band is greater than the average number of all jammer coverage frequency points, a reward value reward2 is obtained, otherwise, the reward value will be reward3; wherein both of them are to keep the balance of the reward value, and satisfy |reward2|=|reward3|; If multiple jammers repeatedly select a certain frequency band, a reward value with a size of reward4 is obtained; r cf ＝r cfmain +r cfadd1 +r cfadd2 Wherein, h is the target of successful interference, t is the threat coefficient of h, and a is the weight adjustment factor of the threat coefficient; The reward value r of the interference bandwidth decision-maker bw By r bwmain and r bwadd Composition; represented as follows: The reward value obtained by the selected jamming bandwidth increases in proportion to the number of jamming frequency points in the current frequency range; B extra represents the redundant bandwidth size that does not cover the frequency point; when the redundant bandwidth exceeds the preset bandwidth quantization interval p, a reward with a size of reward6 is obtained; r bw ＝r bwmain +r bwadd d sub D represents the number of frequency points of the interference within the selected bandwidth band D represents the total number of frequency points of the interference within the current frequency band The output decision strategy is formed by combining the actions output by the two decision makers, denoted as P=[A1, A2]; The jamming action A1 is output by the jamming frequency band determiner, indicating that the jamming sub-band is selected; The jamming bandwidth determiner outputs the jamming action A2, indicating that the bandwidth selected from the preset bandwidth library.

4. The multi-jammer intelligent frequency domain jamming decision method for networked radar countermeasures according to claim 1, characterized in that: The step 4 uses a SumTree data structure as the experience pool for sample playback; Wherein the leaf nodes store the training samples of the extracted PER-DDQN, and the numerical value represents the priority of the sample. The larger the value, the greater the possibility of extracting the sample from the interval length in uniform sampling; The steps of searching the tree structure from top to bottom are as follows: (1) Average extraction of a random number x in each sub-interval, and m sub-intervals obtain m uniformly distributed random numbers; (2) Start from the root node at the top layer and search down; If x is less than the left child node, select the left node to continue searching down; (3) If x is greater than the left child node, the value of x needs to be updated as x-left node value, and then the right node is selected to search down; (4) According to the rules of step (2) - step (3), until the bottom layer of leaf nodes are found, the four tuples <s t , a t , r t , s t+1 > stored in the node are the extracted samples of the interval, and the process is repeated to obtain m samples; Wherein the probability of sampling each sample can be represented by the priority as: where P i denotes the probability that sample i is sampled, p i denotes the priority of sample i, k denotes the sample number, i.e. p k denotes the priority of sample k; The priority of the samples is updated according to the time difference error in each round of iteration process, taking sample i as an example, the priority p i is represented as: p i = |y i - Q(s i , a i ; θ i ) | In the formula, y i Let Q(s) be the target value of the sample. i ,a i ;θ i To evaluate the network's performance on the currently updated weight parameters θ i The state-action pair (s) i ,a i The action value function.

5. The multi-jammer intelligent frequency domain jamming decision method for networked radar countermeasures according to claim 1, characterized in that: The overall algorithm decision process of the step 4 PER-DDQN is as follows: (1) Initialize the SumTree data structure for experience pool to store training samples; (2) Initialize the weight parameters θ and θ' of the evaluation network and the target network, and randomly generate parameters, but the two should be consistent; (3) The frequency domain state information of the reconnaissance networking radar is detected, the bandwidth library is initialized, the size of the selected bandwidth is set, and the sub-band is divided according to the preparation process; (4) Determine the threat level of each radar, and preferentially select the radar with high threat level as the attack target. If the number of jammed frequency points reaches a preset proportion of the total number of target, the threat level of the radar is reduced; (5) Determine the jamming action of the double decision engine according to the exploration rate ε. Select the jamming action randomly with a probability of ε, and select the best action output by the evaluation network with a probability of 1-ε, denoted as: a t = argmax Q(s t ,a t ) (6) According to the interference effect evaluation module, the real-time rewards of the two decision engines are obtained respectively, and are uniformly denoted as r t ; (7) The environment state is updated under the influence of the disturbed action, and the new environment state is noted as s t+1 ; (8) store each sample <s t , a t , r t , s t+1 > into the experience pool, and update the priority of the sample according to the time difference error; (9) When the number of stored samples reaches the capacity of the experience pool, select m samples according to the experience pool sample extraction instructions. The approximation target value of each sample is represented as: y t = r t+1 + γQ(s t+1 , argmaxQ(s t+1 , a t+1 | θ t ) | θ t ') Wherein γ is the discount factor; (10) Calculate the loss function, and reduce the gap between the sample target and the current Q value according to the loss function, and update the network parameters; (11) When the number of iterations reaches a multiple of the network assignment number, the network weight parameters of the evaluation network are assigned to the target network weight parameters, and the two are in a synchronous state at this time; (12) repeating steps 4-11 until all jammer resources are used up, and then entering the next iteration until the iteration number is terminated.

Images