A networking radar suppression jamming method based on hierarchical modular reinforcement learning

Abstract

The application discloses a networking radar suppression jamming method based on layered modular reinforcement learning, which is applied to the field of radar countermeasure, and aims at the short board of the existing networking radar adopting a standard layered reinforcement learning framework.The three-step decomposition method is adopted to decompose the joint optimization problem of interference resource allocation and trajectory planning into three sub-problems, including interference beam allocation, trajectory planning and interference power allocation.Further, in order to cope with the time sequence coupling and environmental uncertainty, a layered modular reinforcement learning framework is established to improve the sequence decision and generalization ability.In detail, three special modules based on the greedy strategy and the double-delay deep deterministic policy gradient method are designed to solve the three sub-problems.The suppression jamming efficiency on the networking radar is effectively improved, and the method is superior to other comparative methods, and exhibits better jamming performance and generalization ability in the unknown environment.

Description

Technical Field

[0001] This invention belongs to the field of radar countermeasures, and specifically relates to a networked radar detection, jamming swarm coordinated jamming, and reinforcement learning technology. Background Technology

[0002] In recent years, networked radar has become a key technology in the field of anti-jamming detection due to its superior detection, tracking, anti-jamming, and anti-damage capabilities. Traditional single-platform jamming faces significant challenges in its effectiveness against networked radar systems, while coordinated jamming has become a typical and effective solution. Coordinated jamming refers to networking and coordinating multiple jammers, and by uniformly planning parameters such as jamming timing, transmission power, jamming patterns, and spatial deployment of each jammer, a jamming combat system with swarm intelligence is constructed, thereby improving the overall electronic warfare capabilities of the jamming force.

[0003] Currently, adversarial techniques based on reinforcement learning algorithms have been widely applied in military electronic systems under complex environments. Especially in multi-dimensional and highly complex electronic warfare scenarios, reinforcement learning algorithms demonstrate strong real-time performance and environmental adaptability. Among them, hierarchical reinforcement learning, with its ability to hierarchically decompose complex adversarial tasks and its long-term decision optimization characteristics, exhibits excellent task decoupling and collaborative planning capabilities in high-dimensional and complex scenarios such as multi-jamming machine cooperative adversarial network radar. However, the standard hierarchical reinforcement learning framework has two inherent shortcomings: firstly, this framework relies on a shared global reward function, making it difficult to evaluate the actual contributions of different layers of networks, thus causing instability in the policy update process; secondly, the guiding role of the top-level network limits the diversified exploration of lower-level networks, resulting in insufficient model generalization ability. To solve these problems, there is an urgent need to design a modular framework that enables completely independent training of various policy networks. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a network radar suppression jamming method based on hierarchical modular reinforcement learning. To cope with temporal coupling and environmental uncertainty, a hierarchical modular reinforcement learning framework is established to improve sequence decision-making and generalization capabilities. This effectively enhances the jamming performance of jamming swarms against network radars, and the method exhibits superior jamming performance and generalization capabilities in unknown environments.

[0005] The technical solution adopted in this invention is: a network radar suppression and jamming method based on hierarchical modular reinforcement learning, applied to an offensive and defensive scenario between jammer swarms and radar networks. The attacking side includes fighter jets and multiple jammers, and the defending side includes multiple radar sites. The jamming method includes the following steps:

[0006] Based on the threat level of each radar station to the detection probability of the fighter jet, a greedy algorithm is used to sequentially assign the jammer that generates the maximum received jamming power to the radar station with the highest threat level to be assigned, forming an jamming beam allocation matrix between the jammer and the radar station.

[0007] Based on the jamming beam allocation matrix, each jammer independently plans its trajectory according to the radar it is assigned to. A single jammer trajectory planning model is constructed using a dual-delay, depth-deterministic policy gradient algorithm. The state space, action space, and reward function of the single jammer trajectory planning problem are input into the value network and policy network of reinforcement learning for evaluation and updating, and then deployed to all jammers. Each jammer shares the same network parameters and independently generates trajectory planning actions in parallel interaction with the environment. The interaction experience tuples are uniformly stored in a shared experience pool. Training is performed by randomly sampling from the shared experience pool, thereby achieving the autonomous generation of jammer swarm cooperative trajectory planning strategies in a networked radar adversarial scenario based on reinforcement learning.

[0008] The global state space and joint action space, which are derived from the interference beam allocation matrix and the prior information of the trajectory planning actions of each jammer, as well as the reward function defined by the detection probability of the networked radar, are input into the value network and policy network of reinforcement learning for evaluation and updating. Finally, the dual-delay deep deterministic policy gradient algorithm is used to realize the autonomous generation of the cooperative power allocation strategy of the jammer swarm in the networked radar confrontation scenario based on reinforcement learning.

[0009] The beneficial effects of this invention: This invention proposes a network radar suppression jamming method based on hierarchical modular reinforcement learning, used to suppress network radar systems. This method first employs a three-step decomposition method to break down the complex joint optimization problem into three sub-problems: jamming beam allocation, trajectory planning, and jamming power allocation, significantly reducing the problem dimensionality. Based on this, a hierarchical modular reinforcement learning framework is established, designing a jamming beam allocation module based on a greedy policy, a jamming trajectory planning module based on a dual-delay deep deterministic policy gradient algorithm, and a jamming power allocation module based on a dual-delay deep deterministic policy gradient algorithm. Each module optimizes independently and works collaboratively, overcoming the difficulties in contribution evaluation caused by globally shared rewards and the limited diversity exploration of lower-level networks under the guidance of the top-level network in traditional hierarchical reinforcement learning. Simulation results show that the method of this invention has excellent jamming effectiveness and generalization ability in randomly generated unknown environments, and can more effectively suppress the detection performance of network radar systems compared to comparative methods, exhibiting good robustness in emergency situations such as sudden maneuvers by fighter jets. Attached Figure Description

[0010] Figure 1This is a schematic diagram of a networked radar system for collaborative jamming suppression using the method provided by the present invention;

[0011] Figure 2 This is a flowchart of the reinforcement learning training process of the method provided by the present invention;

[0012] Figure 3 This is a training process diagram of the reward curves of the trajectory planning module and the interference power allocation module of the present invention;

[0013] Among them, (a) the average echo curve of the trajectory planning module, and (b) the average echo curve of the power distribution module;

[0014] Figure 4 This invention provides the combat strategy for fighter jets trained and generated along a predetermined trajectory.

[0015] Among them, (a) interference suppression effect, (b) trajectory planning strategy, and (c) interference power allocation strategy;

[0016] Figure 5 This invention provides a countermeasure strategy for fighter jets during sudden maneuvers, generated through training.

[0017] Among them, (a) interference suppression effect, (b) trajectory planning strategy, and (c) interference power allocation strategy;

[0018] Figure 6 This is a comparison of the detection performance of the training and generation strategy of this invention with existing strategies on networked radar systems;

[0019] Among them, (a) the fighter jet follows a predetermined trajectory during the penetration mission, and (b) the fighter jet suddenly maneuvers during the penetration mission;

[0020] Figure 7 This is the interference performance of the training and generation strategy of this invention under 10 different scene parameters;

[0021] Figure 8 This represents the interference performance of different strategies under 10 different scenario parameters. Detailed Implementation

[0022] To facilitate understanding of the technical content of this invention by those skilled in the art, the following terms are first explained:

[0023] Term 1: Cooperative interference

[0024] The jamming swarm utilizes the mutual verification feature of networked radars to allocate jamming resources and achieve the effect of jamming the networked radars.

[0025] Term 2: Greedy Algorithm

[0026] The jamming swarm calculates the detection probability of all radars to assess their threat level. The radar with the highest detection probability is designated as the priority jamming target, and the jammer capable of generating the maximum received jamming power is assigned to that radar.

[0027] Term 3: Interference Beam Assignment Matrix

[0028] In a networked radar system, under the scenario of multi-jamming machine cooperative penetration, based on the threat level of each radar detection probability, a greedy algorithm is used to sequentially assign the jammer with the maximum received jamming power to the radar with the greatest threat, thus forming an interference beam allocation matrix between the jammer and the radar.

[0029] Term 4: Interference Power Allocation Matrix

[0030] Based on the determined jamming beam allocation matrix, the jammer dynamically adjusts the transmit power of each beam according to its spatial relative position with the target radar through a jamming power allocation model based on reinforcement learning algorithm, and outputs a normalized jamming power allocation matrix.

[0031] Term 5: Value Network

[0032] During the confrontation, networked radars and jamming swarms will use environmental parameters and their own status and actions to predict the current value and guide their next move.

[0033] This invention is primarily verified using simulation experiments, and all steps and conclusions have been verified to be correct using Python 3.11. The following provides a further detailed description of the specific implementation methods.

[0034] Step 1: Training Scene Modeling

[0035] The present invention is as follows Figure 1 The scenario depicted is an attack and defense between a swarm of jammers and a radar network. The attacking side consists of an attack formation composed of fighter jets and multiple jammers, while the defending side is a networked radar system composed of multiple radar sites.

[0036] For the attacking side, fighter jets carrying weapons perform penetration missions, aiming to fly into enemy positions and attack designated areas. Networked radars coordinate to detect, track, and lock onto fighter jets within the airspace, improving target detection probability through data fusion. Jamming aircraft are deployed around the fighter jets, coordinating jamming resource allocation and flight path planning to create suppression jamming around them, thereby reducing the probability of the fighter jets being detected and tracked by the networked radars.

[0037] In combat operations, in order to effectively jam and protect fighter jets, jamming aircraft need to dynamically adjust their position according to the direction of the radar beam, maneuver to the coverage area of ​​the radar main lobe beam, and carry out effective drag jamming.

[0038] To address the interference challenges posed by networked radars changing their operating states according to a certain periodic pattern, jamming aircraft swarms need to adjust their flight trajectories in real time to reacquire radar beams and form a continuous and effective suppression jamming. This type of joint optimization problem involving jamming resource allocation and trajectory planning presents challenges due to its high dimensionality, temporal coupling, and environmental uncertainties. To reduce the problem's dimensionality, this invention employs a three-step decomposition method to break down the joint optimization problem of jamming resource allocation and trajectory planning into three sub-problems: jamming beam allocation, trajectory planning, and jamming power allocation.

[0039] Step 2: Optimize the interference beam assignment matrix

[0040] In this scenario, according to the radar equations, the power of the echo signal... It can be represented as:

[0041]

[0042] in It is radar The transmission power, It's the main lobe gain. It's the wavelength. It is the radar cross section (RCS) of a fighter jet. It is the first radar at each time step Distance to the fighter jet. The superscript 't' indicates transmission; the superscript 'r' indicates radar.

[0043] Assume that the jamming signals from different jammers are uncorrelated. Radar The total received interference power is expressed as:

[0044]

[0045] in It is the first The jamming power of the jammer m received by radar n at each time step. It is a jammer Antenna gain, It is the polarization mismatch factor. It is the first Time step jammer With radar The distance between them It is the first radar at each time step jammer Gain in the direction of interference beam It is the first Does the jammer m at each time step transmit jamming signals to radar n? It is the first The jamming power emitted by jammer m at each time step towards radar n;

[0046] The expression is as follows:

[0047]

[0048] The superscript J stands for Jammer.

[0049] This invention assumes that the radar employs coherent pulse accumulation technology. Under cooperative suppression, the radar... The interference-to-signal ratio relative to the fighter jet is expressed as:

[0050]

[0051] in This is the number of accumulated pulses. According to the Swerling I target model, the detection probability is approximately:

[0052]

[0053] in It is the detection threshold. It is radar The probability of detecting fighter jets.

[0054] Networked radar systems typically integrate data from multiple radar nodes through data fusion to improve detection performance and anti-jamming capabilities. This invention employs... Integration criteria. Specifically, let Indicates radar The partial judgment, in which This indicates that a target has been detected. This indicates that no target was detected. The fusion center generates a global decision vector based on the local decisions of all radars. The fusion rules for networked radar systems are defined as follows:

[0055]

[0056] in yes Based on the fusion threshold, the probability of a networked radar system detecting a fighter jet is calculated as follows:

[0057]

[0058] in, It is the set of all possible global decision vectors. Because the jammer does not know the network radar system's... Fusion thresholds employ a conservative strategy, namely, assuming... This ensures that suppressing any radar helps protect the fighter jet.

[0059] Based on the threat level of each radar detection probability, the radar with the highest detection probability against the fighter jet is designated as the most threatening radar. A greedy algorithm is then used to sequentially generate jammers with the highest received jamming power for each of the most threatening radars, resulting in an interference beam allocation matrix between the jammers and the radars. Here, "generating jammers with the highest received jamming power for each of the most threatening radars using a greedy algorithm" should be understood as assigning the currently unassigned jammers with the highest received jamming power to each radar in descending order of radar detection probability.

[0060] Step 3: Optimize the operating trajectory of the jamming aircraft group

[0061] Based on the prior information of the aforementioned jamming beam allocation matrix, the environmental input is converted into a single jammer trajectory planning state, and the acceleration and angular velocity of each jammer are optimized so that the jammer can maneuver to the coverage area of ​​the target radar main lobe beam, thereby improving jamming effectiveness.

[0062] (1) State-space design

[0063] Under ideal windless conditions, jamming machine The equations of motion are described as follows:

[0064]

[0065] in, jammer In the The position of each time step. and Their velocity and acceleration are respectively. and These are its heading angle and angular velocity, respectively. , , , These represent the rates of change of the x-coordinate, y-coordinate, velocity, and heading angle of the jammer m, respectively.

[0066] acceleration With angular velocity The following kinematic constraints must be met:

[0067]

[0068] in, and These are the minimum and maximum speeds, respectively. For maximum acceleration, For the maximum angular velocity, It represents the absolute value.

[0069] The state space can be represented as:

[0070]

[0071] in, Indicates the location of the fighter jet. Indicates the speed of the fighter jet. and For the jammed machine The radar position vector of the jamming.

[0072] (2) Action space design

[0073] Jammers use the speed of fighter jets to predict their future position and maneuver to remain within the main lobe coverage of the assigned radar, thus achieving effective jamming. trajectory planning action Defined as:

[0074]

[0075] The superscript T stands for Trajectory planning.

[0076] (3) Design of the reward function

[0077] The reward function consists of arrival reward The arrival reward provides the jammer with continuous feedback on its position and orientation as it approaches the radar's main lobe coverage area, while the path reward incentivizes the jammer to further explore its position within that area to maximize jamming effectiveness. The arrival reward is calculated as follows:

[0078]

[0079] in, L represents the detection probability of a radar being jammed by jammer m, and L represents the maximum number of beams of the jammer. This represents the detection probability of the nth radar. The superscript 'a' indicates arrival; 'p' indicates planning.

[0080] The path reward is calculated as follows:

[0081]

[0082] in, Indicates jammer Maximum gain direction and radar The angle between the directions of maximum gain.

[0083] If the jammer maneuvers to the radar's main lobe coverage area, the agent receives an arrival reward; otherwise, it receives a path reward. The trajectory planning reward function is expressed as follows:

[0084]

[0085] in, For radar The width of the main lobe, It is a group of jammed machines A collection of interfering radars, Let m be the reward function for trajectory planning of the jammer.

[0086] (4) Strengthen learning and training

[0087] The trajectory planning module employs a parallel training mechanism. After completing the jamming beam allocation, each jammer can focus on its assigned target radar for independent trajectory planning. This characteristic transforms the joint trajectory planning problem into multiple independent single-jammer trajectory planning problems. Finally, a transferable single-jammer trajectory planning model is constructed using a dual-delay deep deterministic policy gradient algorithm. The state space, action space, and reward function of the single-jammer trajectory planning problem are input into the reinforcement learning value network and policy network for evaluation and updating, respectively, and then deployed to all jammers. Each jammer shares the same network parameters and independently generates trajectory planning actions in parallel interaction with the environment, storing the interaction experience tuples in a shared experience pool. Training is performed by randomly sampling from the shared experience pool, thereby achieving the autonomous generation of collaborative trajectory planning strategies for jammer swarms in networked radar confrontation scenarios based on reinforcement learning.

[0088] The update formula for the dual-delay deep deterministic policy gradient algorithm is:

[0089]

[0090] in, Let i be the loss function updated for the i-th value network. Let i be the i-th value network parameter. The current state of the agent. For the actions of the intelligent agent, The reward value for the agent is calculated during training. , This represents the state of the agent in the next moment. Let y be the experience pool, and y be the target value. As a discount factor, For the i-th target value network parameters, The action performed by the target policy network. For the target policy network parameters, Mean 0 standard deviation Cut off Gaussian noise, This indicates the samples from the experience pool. The expected value for calculating the value error. Indicates the predicted state of the value network Next action value, Indicates the target value network prediction state Next action value, This represents the truncation function (used to limit the boundaries of noise), where l is the boundary for truncating Gaussian noise. For the policy gradient of the policy network, For policy network parameters, For state Based on strategy implement value, For state Based on strategy implement The value of the action gradient For state The following strategy For state The following policy network parameters The gradient; express From the target policy network in state The action to be performed.

[0091] yes The target network parameters, and The structures are identical, and they are copied at regular intervals. once:

[0092]

[0093] in, This represents the soft update factor.

[0094] Step 4: Optimize the interference power allocation matrix

[0095] Based on the aforementioned interference beam allocation matrix and trajectory planning action prior information, the environmental input is converted into an interference power allocation state, and the interference power allocation matrix is ​​optimized to minimize the final network radar detection probability.

[0096] (1) State-space design

[0097] The received jamming power is highly sensitive to the distance between the jammer and the radar. This is based on the jamming beam allocation matrix. In conjunction with trajectory planning, environmental input is converted into a disturbance power allocation state. The details are as follows:

[0098]

[0099] in, jammer The range vector between it and its target radar jammer The angle vector between the direction of maximum gain and the direction of maximum gain of the target radar.

[0100] (2) Action space design

[0101] Interference power allocation action is defined as:

[0102]

[0103] in, This refers to the power allocation action at time k, where the superscript P represents power. The pre-allocated interference power matrix, which should be understood as the direct output of the agent's policy network, is expressed as follows:

[0104]

[0105] The actual interference power is calculated by normalizing the pre-allocated interference power, as follows:

[0106]

[0107] in, Pre-allocate jammer m to radar Interference power, This represents the total interference power.

[0108] (3) Design of the reward function

[0109] To directly reflect the threat posed by networked radar systems to fighter jets, the jamming power allocation reward function is defined as the detection probability of the networked radar system:

[0110]

[0111] in, This represents the power allocation reward function at time k. This indicates the detection probability of the networked radar.

[0112] (4) Strengthen learning and training

[0113] The power allocation module employs a centralized training mechanism, taking the global state space and joint action space derived from the interference beam allocation matrix and trajectory planning action prior information, along with the reward function defined by the networked radar detection probability, as inputs. These inputs are then fed into the reinforcement learning value network and policy network for evaluation and updating. Finally, the dual-delay deep deterministic policy gradient algorithm is used to autonomously generate the cooperative power allocation strategy for the jamming swarm in a networked radar countermeasure scenario based on reinforcement learning. The update formula for the dual-delay deep deterministic policy gradient algorithm is the same as that in step three, the difference being in the state space and action space parameters.

[0114] like Figure 2 As shown, this framework employs a hierarchical modular reinforcement learning structure to jointly optimize jamming resource allocation and trajectory planning. The jamming beam allocation module uses a greedy algorithm to perform initial beam allocation based on multiple radar threat levels. The trajectory planning module transforms the environmental input into the local states of each jammer through a state transformer and generates jammer actions using a jammer trajectory planning model based on a dual-delay deep deterministic policy gradient algorithm. Based on the beam allocation and trajectory planning results, the jamming power allocation module generates a centralized power allocation state through a state transformer. The jamming power allocation model based on a dual-delay deep deterministic policy gradient algorithm outputs joint actions, and the interaction experience is stored in an independent buffer. Finally, the value networks of the two modules are centrally trained and their parameters updated through sampled experience, thereby realizing the generation of jamming strategies in networked radar countermeasure scenarios.

[0115] A comparative experiment was designed by replacing the modules in the hierarchical modular reinforcement learning framework. The method used is as follows:

[0116] Random Beam Allocation (RBA): This method is similar to the greedy strategy-based jamming beam allocation, where jamming beams are allocated sequentially according to the threat level of each radar. By randomly allocating jamming beams, the radar with the highest detection probability is jammed.

[0117] Fixed Trajectory Planning (FTP): This involves deploying jammers around the target aircraft to perform accompanying jamming, ensuring that the jammers are always within the main lobe coverage area of ​​each radar.

[0118] Average Power Allocation (APA): Distributes the interference power evenly across all interfering beams.

[0119] Random Beam Assignment-Fixed Trajectory Planning-Average Power Allocation (RBA-FTP-APA): This method combines RBA, FTP, and APA strategies to complete beam assignment, trajectory planning, and power allocation tasks, respectively.

[0120] Table 1 shows the operating parameters of the jammer and radar, and Table 2 shows the hyperparameter settings for model training; a reinforcement learning simulation model is established based on the data in Tables 1 and 2.

[0121] Table 1 Operating parameters of jammer and radar

[0122]

[0123] Table 2 Hyperparameters used in training

[0124]

[0125] like Figure 3 As shown in (a), the agent is in the environment exploration phase during the initial training stage, with a low average reward value. As training progresses, the agent accumulates more effective experience and learns an optimized trajectory planning strategy, resulting in a rapid increase in the average reward value and a gradual decrease in the fluctuation of the reward curve. After approximately 1.2 million training iterations, the curve begins to converge, indicating that the trajectory planning strategy tends to stabilize. Figure 3 As shown in (b), the agent failed to find an effective interference power allocation strategy in the early stages of training, resulting in a decrease in the average reward value. As training progressed, the agent accumulated experience and learned better strategies, and although the average reward value fluctuated, it showed an overall upward trend. However, after more than 200,000 training sessions, a downward trend appeared, indicating that overfitting began to emerge. Therefore, this invention terminated training at this point.

[0126] When the fighter jet follows a predetermined trajectory, the detection probability of each radar within the networked radar is as follows: Figure 4 As shown in (a), the detection probabilities of radars 1, 2, and 3 show a decreasing trend, while the detection probabilities of radars 4 and 5 gradually increase. It is noteworthy that the detection probabilities of all radars remain below 0.03, indicating effective suppression of the networked radar system. The jamming strategy results in this scenario are as follows... Figure 4 As shown in (b) and (c) in the figure. The trajectory planning results are as follows. Figure 4 As shown in (b): In the initial phase of the penetration mission, the jammer is deployed around the fighter jet; as the mission progresses, the jammer maneuvers towards the radar to maximize jamming effectiveness; in the later stages of the mission, the jammer gradually moves closer to the fighter jet and remains within the radar's main lobe coverage throughout the entire penetration process. The jamming resource allocation results are as follows: Figure 4As shown in (c), the left vertical axis represents the number of jamming beams, the right vertical axis represents the number of radars jammed by the jamming beams, and the grid color represents the normalized jamming power. The specific allocation strategy is as follows: jammer 1 jams radars 1 and 2, jammer 2 jams radars 4 and 5, and jammer 3 jams radars 1 and 3. As the penetration mission progresses, the relative spatial distribution between combat units gradually changes, causing changes in the echo power and received jamming power of each radar.

[0127] When a fighter jet suddenly maneuvers, the detection probability of each radar within the networked radar is as follows: Figure 5 As shown in (a), the sudden maneuvering of the fighter jet causes varying degrees of fluctuation in the detection probability of each radar. However, the method proposed in this invention can still maintain a low detection probability in the networked radar system under emergency conditions, and experimental results demonstrate that the method has excellent robustness. The jamming strategy results in this scenario are as follows: Figure 5 As shown in (b) and (c) in the figure. The trajectory planning results are as follows. Figure 5 As shown in (b): the fighter jet suddenly changes direction at time steps 5, 10, and 15. In response, the jammer dynamically adjusts its trajectory. In the initial phase, jammers 1 and 3 fly close to the radar. As the fighter jet's course changes, their trajectories oscillate between the fighter jet and the radar. In contrast, jammer 2 maintains a relatively smooth trajectory. The jamming resource allocation results are as follows: Figure 5 As shown in (c): Jammer 1 jams radars 1 and 2, jammer 2 jams radars 4 and 5, and jammer 3 jams radars 1 and 3. Jammers 1 and 3 dynamically adjust their power allocation strategies according to changes in the spatial relationships between combat units. Due to its stable trajectory, jammer 2 maintains a nearly constant power allocation because its relative distance to the fighter jets and radars 4 and 5 remains essentially constant.

[0128] When a fighter jet follows a predetermined trajectory, the detection probability of a networked radar system under different methods is as follows: Figure 6 As shown in (a) of the diagram, the random beam allocation method exhibits significant fluctuations, rising sharply in the [3,5] and [10,14] timeframes. The detection probabilities of the fixed trajectory planning method and the random beam allocation-fixed trajectory planning-average power allocation method both rise rapidly, significantly higher than other methods. The detection probabilities of the average power allocation method and the proposed method initially decrease but then slightly recover, and their values ​​are extremely close throughout the mission. Among all methods, the proposed method achieves the lowest detection probability for the networked radar system, demonstrating superior jamming effectiveness. Based on the above analysis, the influence of the three strategies on jamming effectiveness, from highest to lowest, is as follows: trajectory planning, jamming beam allocation, and jamming power allocation.

[0129] When a fighter jet suddenly maneuvers, the detection probability of a networked radar system under different methods is as follows: Figure 6As shown in (b), the detection probability of all methods fluctuates accordingly with changes in the flight trajectory. Specifically, the detection probability of the fixed trajectory planning method continuously increases before step 15, then begins to decrease; the detection probability of the random beam assignment-fixed trajectory planning-average power allocation method increases in the time intervals [1,5] and [11,15], and decreases in the time intervals [6,10] and [16,20]; the random beam assignment method maintains a low detection probability, with its fluctuations mainly concentrated in the time interval [5,10]; the average power allocation method and the method proposed in this invention exhibit similar trends to the random beam assignment method, but demonstrate superior performance in responding to sudden maneuvers. Overall, the method proposed in this invention can maintain the lowest possible detection probability for the networked radar system in emergency situations, and experimental results demonstrate its excellent robustness.

[0130] The detection probability of a networked radar system under different scenario parameters is as follows: Figure 7 As shown, in all scenarios, the detection probability of the networked radar system remained below 0.45, effectively preventing the networked radar from detecting the fighter jet.

[0131] Figure 8 The average detection probability of networked radar systems using different methods is presented in 10 scenarios. The proposed method achieves the lowest average detection probability in all scenarios, demonstrating consistently superior performance. Its performance is particularly outstanding in scenarios 3, 6, and 7, significantly outperforming all other methods. The average power allocation method also exhibits good jamming effectiveness, typically ranking second. In contrast, the random beam allocation-fixed trajectory planning-average power allocation method has the highest average detection probability but the worst performance, followed by the fixed trajectory planning method. Furthermore, the random beam allocation-fixed trajectory planning-average power allocation method and the fixed trajectory planning method show the greatest performance fluctuations, while the proposed method maintains the most stable performance in all scenarios. Overall, the jamming effectiveness ranking of these methods is: the proposed method > average power allocation method > random beam allocation method > fixed trajectory planning method > random beam allocation-fixed trajectory planning-average power allocation method. These results demonstrate the superior jamming effectiveness and stability of the proposed method in different scenarios.

[0132] Those skilled in the art will recognize that the embodiments described herein are for the purpose of helping to understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A network radar jamming suppression method based on hierarchical modular reinforcement learning, characterized in that, This method is applied to offensive and defensive scenarios between jamming swarms and radar networks. The attacking side includes fighter jets and multiple jamming aircraft, while the defending side includes multiple radar sites. The jamming method includes the following steps: Based on the threat level of each radar station to the detection probability of the fighter jet, a greedy algorithm is used to sequentially assign the jammer that generates the maximum received jamming power to the radar station with the highest threat level to be assigned, forming an jamming beam allocation matrix between the jammer and the radar station. Based on the jamming beam allocation matrix, each jammer independently plans its trajectory according to the radar it is assigned to. A single jammer trajectory planning model is constructed using a dual-delay, depth-deterministic policy gradient algorithm. The state space, action space, and reward function of the single jammer trajectory planning problem are input into the value network and policy network of reinforcement learning for evaluation and updating, and then deployed to all jammers. Each jammer shares the same network parameters and independently generates trajectory planning actions in parallel interaction with the environment. The interaction experience tuples are uniformly stored in a shared experience pool. Training is performed by randomly sampling from the shared experience pool, thereby achieving the autonomous generation of jammer swarm cooperative trajectory planning strategies in a networked radar adversarial scenario based on reinforcement learning. The global state space and joint action space, which are derived from the interference beam allocation matrix and the prior information of the trajectory planning actions of each jammer, as well as the reward function defined by the detection probability of the networked radar, are input into the value network and policy network of reinforcement learning for evaluation and updating. Finally, the dual-delay deep deterministic policy gradient algorithm is used to realize the autonomous generation of the cooperative power allocation strategy of the jammer swarm in the networked radar confrontation scenario based on reinforcement learning.

2. The network radar jamming suppression method based on hierarchical modular reinforcement learning according to claim 1, characterized in that, The process for calculating the detection probability of fighter jets at each radar station is as follows: Assuming the jamming signals from different jammers are uncorrelated, then the radar... The total received interference power is expressed as: ； in, It is the first The jamming power of the jammer m received by radar n at each time step. It is a jammer Antenna gain, It is the polarization mismatch factor. It is the first Time step jammer With radar The distance between them It is the first radar at each time step jammer Gain in the direction of interference beam It is the first Does the jammer m at each time step transmit jamming signals to radar n? It is the first The jamming power emitted by jammer m at each time step towards radar n; Assuming the radar employs coherent pulse accumulation technology, under coordinated suppression, the expression for calculating the interference-to-signal ratio (ISR) of each radar relative to the fighter jet based on the total interference power received by the radar is as follows: ； in, It is the number of pulse accumulations; Based on the interference-to-signal ratio of each radar relative to the fighter jet, the probability expression for each radar to detect the fighter jet is calculated as follows: ； in, It is the detection threshold. It is radar The probability of detecting fighter jets; Based on the detection probability of each radar for a fighter jet, the expression for the detection probability of the networked radar system for a fighter jet is as follows: ； in, Indicates the first The probability of a networked radar detecting a fighter jet at each time step. This indicates the fusion rules for the networked radar system, allowing Indicates radar The partial judgment, This indicates that a target has been detected. This indicates that no target was detected. This is the global decision vector. It is the set of all possible global decision vectors.

3. The network radar jamming suppression method based on hierarchical modular reinforcement learning according to claim 2, characterized in that, The state space representation of the value network and policy network corresponding to the jammer swarm cooperative trajectory planning strategy in reinforcement learning is as follows: ； in, The x-axis and y-axis coordinates represent the fighter jet's position. Indicates the speed of the fighter jet; jammer In the The x-axis and y-axis coordinates of the position at each time step; jammer In the The speed of each time step; jammer In the The heading angle at each time step; and For the jammed machine The radar position of the jamming radar is represented by its x-axis vector and y-axis vector.

4. The network radar jamming suppression method based on hierarchical modular reinforcement learning according to claim 3, characterized in that, The action space representation of the value network and policy network in reinforcement learning corresponding to the jammer swarm cooperative trajectory planning strategy is as follows: ； in, jammer trajectory planning actions, jammer In the Acceleration at each time step jammer In the Angular velocity at each time step.

5. The network radar jamming suppression method based on hierarchical modular reinforcement learning according to claim 4, characterized in that, The reward functions of the value network and policy network corresponding to the cooperative trajectory planning strategy of the jammer swarm are expressed as follows: ； in, Let m represent the reward function for trajectory planning of the jammer m. To reach the reward, For path rewards, Indicates jammer Maximum gain direction and radar The angle between the directions of maximum gain, For radar The width of the main lobe, It is a group of jammed machines A collection of interfering radars.

6. The network radar jamming suppression method based on hierarchical modular reinforcement learning according to claim 5, characterized in that, The update formulas for the value network and policy network corresponding to the cooperative trajectory planning strategy of the jammer swarm are as follows: ； in, Let i be the loss function updated for the i-th value network. This indicates the samples from the experience pool. The expected value for calculating the value error, where y is the target value. Indicates the predicted state of the value network Next action value, Let i be the i-th value network parameter. The current state of the agent. For the actions of the intelligent agent, The reward value for the agent, As a discount factor, Indicates the target value network prediction state Next action value, This represents the state of the agent in the next moment. For experience pool, For the i-th target value network parameters, The action performed by the target policy network. express From the target policy network in state The action to be performed For the target policy network parameters, Mean 0 standard deviation truncated Gaussian noise, This represents the truncation function, where l is the boundary for truncating Gaussian noise. For the policy gradient of the policy network, For policy network parameters, For the action gradient, The gradient of the policy network parameters.

7. The network radar jamming suppression method based on hierarchical modular reinforcement learning according to claim 6, characterized in that, The state space representation of the value network and policy network corresponding to the interference swarm cooperative power allocation strategy is as follows: ； in, jammer The range vector between it and its target radar , jammer The angle vector between the direction of maximum gain and the direction of maximum gain of the target radar.

8. The network radar jamming suppression method based on hierarchical modular reinforcement learning according to claim 7, characterized in that, The action space representation of the value network and policy network in reinforcement learning corresponding to the interference swarm cooperative power allocation strategy is as follows: ； in, This is the pre-allocated interference power matrix.

9. A network radar jamming suppression method based on hierarchical modular reinforcement learning according to claim 8, characterized in that, The reward functions of the value network and policy network in reinforcement learning corresponding to the interference swarm cooperative power allocation strategy are expressed as follows: 。

Images