A cognitive jamming decision method based on deep reinforcement learning and an application system thereof

Abstract

The application discloses a kind of cognitive interference decision-making method and its application system based on deep reinforcement learning, it is suitable for our side interference unmanned aerial vehicle when networked radar system in complex electromagnetic environment under confrontation completes intelligent interference decision-making task, based on interference decision-making algorithm DAR-PPO, to improve the performance of existing interference decision-making system in sample efficiency and strategy stability, interference decision-making algorithm DAR-PPO includes two big innovation modules: one is KTD-Clip module, feedback signal is constructed based on experience KL divergence, clipping threshold in strategy update is dynamically adjusted, to control strategy update step, so that it is always in stable and effective trust domain, significantly improve the adaptability of algorithm to environmental change;Second is RA-IS module, by constructing trajectory playback pool and introducing clipping importance sampling, realize the effective reuse of historical interaction samples, while designing weight entropy regularization mechanism, inhibit the strategy fluctuation caused by abnormal weight, thereby significantly improve training sample utilization and enhance algorithm robustness.

Description

Technical Field

[0001] This invention relates to the fields of electronic warfare and artificial intelligence, and in particular to a cognitive jamming decision-making method and its application system based on deep reinforcement learning. It is applicable to the intelligent jamming strategy learning and dynamic countermeasure optimization of multi-functional radars in radar countermeasures and electronic warfare systems, and belongs to the interdisciplinary research field of cognitive electronic warfare and intelligent countermeasure decision-making. Background Technology

[0002] Cognitive Electronic Warfare (CEW), a product of the deep integration of artificial intelligence and traditional electronic warfare technology, has become an important development direction in the future field of electronic warfare, attracting widespread attention from experts and scholars both domestically and internationally. The CEW concept originated from cognitive radio research, and since its inception in 1999, it has continuously driven the integration and evolution of related technologies. The introduction of the "cognitive radar" concept in 2006 further promoted the development of intelligent perception and countermeasure systems. However, limited by the early development levels of artificial intelligence and microelectronics technology, how to construct a countermeasure system with actual cognitive capabilities became a challenge. In recent years, with breakthroughs in sensing, computing, and control technologies, CEW has begun to show potential for engineering implementation. It can be said that the development of CEW technology has benefited from the progress and integration of numerous technologies. However, its effective application in practical combat still faces many challenges, and CEW urgently needs to address issues such as the construction of cognitive theoretical models and the application of methods.

[0003] Driven by artificial intelligence, radar systems are rapidly evolving towards intelligence. Modern radar systems with cognitive capabilities can dynamically adjust reconnaissance parameters based on battlefield environment feedback, exhibiting stronger mission adaptability and varying degrees of detection capabilities. Relying on advanced hardware platforms and software architectures, radar has evolved from a traditional single-mode system to a new type of multi-functional radar with multi-mode switching and flexible beam control capabilities. Multi-functional radars not only possess greater mobility and adaptability but can also learn from the "learning methods" of natural organisms, achieving a leap from passive perception to proactive response, significantly enhancing their survivability and combat capabilities in complex adversarial environments. These changes place higher demands on radar jamming decision-making. However, traditional jamming strategies mostly rely on pre-set rules and offline optimization, which cannot effectively adapt to multi-radar, multi-scenario, and dynamically evolving adversarial environments. This "static rules + static targets" approach struggles to cope with the high-frequency dynamic changes in radar strategies in adversarial environments, resulting in unstable jamming efficiency, low resource utilization, and poor countermeasure effectiveness.

[0004] In recent years, reinforcement learning (RL), as one of the core technologies of artificial intelligence, has emerged as a leading field. It continuously optimizes long-term reward policies through interaction with the environment, demonstrating superior learning and decision-making capabilities in uncertain and dynamic task environments. However, traditional reinforcement learning techniques often suffer from weak state representation, poor generalization, and low sample efficiency when facing high-dimensional state spaces and complex task objectives. To address this, researchers have introduced deep learning techniques into the reinforcement learning framework, proposing the Deep Reinforcement Learning (DRL) method. This method enhances state perception and policy expression capabilities through deep neural networks, effectively improving learning performance in complex, continuous, and nonlinear tasks. However, in electronic warfare scenarios, challenges such as complex state spaces, uncertain radar behavior, scarce training samples, and reward delays mean that directly applying traditional RL models often results in training instability, policy oscillations, and poor generalization. Therefore, there is an urgent need for a structurally innovative, mechanism-improved, and domain-adaptable deep reinforcement learning interference algorithm to achieve stable, efficient, and adaptive cognitive interference policy generation and online optimization.

[0005] With the development of the Actor-Critic framework in reinforcement learning, gradient-based proximal policy optimization (PPO) has become the preferred solution for many experts and scholars, as well as for interference systems, due to its advantages such as simple implementation and stable convergence. However, in practical applications, the PPO algorithm still has two major problems: 1. Low sample efficiency. PPO relies on on-policy policy updates, and old samples cannot be reused, resulting in low utilization of training data. This is especially true in scenarios where data acquisition costs are high or interaction is limited, significantly extending the training cycle; 2. Difficulty in tuning the pruning threshold. Using a fixed pruning threshold for policy updates cannot adapt to policy drift at different training stages. A threshold that is too small may lead to training stagnation, while a threshold that is too large may easily cause gradient explosion or policy collapse. Extensive grid searches are required to determine suitable hyperparameters. Summary of the Invention

[0006] To address the problems existing in the prior art, the present invention aims to provide a cognitive interference decision-making method and its application system based on deep reinforcement learning. It aims to achieve intelligent, dynamic, and efficient interference of multi-functional radar systems by jamming UAVs. By enhancing the adaptability, generalization ability, and adversarial performance of the interference strategy through the reinforcement learning framework, it overcomes the problems of unstable training, slow response, and poor environmental adaptability of traditional methods.

[0007] To achieve the above objectives, this invention provides a cognitive interference decision-making method based on deep reinforcement learning. This method is used to interfere with UAVs in a multi-functional networked radar jamming environment, and includes the following steps:

[0008] Step S1: Construct a radar jamming decision model, define a multi-agent cooperative jamming decision-making adversarial framework, and model the environment state space, jamming action space, and reward function;

[0009] Step S2: Initialize the parameters of the DAR-PPO perturbation decision algorithm at the initial time. The parameters include the policy network parameters. Value network parameters Learning rate Discount Factor and KL target Initialize the replay buffer to store trajectory data;

[0010] Step S3: Initialization The minimum and maximum values;

[0011] Step S4: Obtain the current state Make decisions based on the policy network: ;

[0012] Step S5: Perform the action Observe the reward at this step and the next step ;

[0013] Step S6: Calculate the temporal difference objective based on the Q-value output by the value network. and timing difference error ;

[0014] Step S7: According to the new Adjust strategy;

[0015] Step S8: Sample old trajectories from the replay pool for training;

[0016] Step S9: Proceed to the next round until the training ends.

[0017] Furthermore, the state space of the radar jamming decision model includes the radar's state parameters and the positional information between the jamming UAV and the radar.

[0018] Furthermore, the action space of the radar jamming decision model includes the target number and the selected jamming mode, which together constitute an ordered combination of jamming strategies.

[0019] Furthermore, the radar's state parameters include carrier frequency, pulse width, bandwidth, pulse repetition frequency, and amplitude.

[0020] Furthermore, in step S4, a reward mechanism is designed, and a reward function is used to determine the reward for this step. Continuously refine jamming strategies to minimize the target radar threat level.

[0021] Furthermore, the reward function is divided into two parts: intermediate reward. Rewards for reaching When a jamming drone performs a jamming action, if the radar threat level decreases, the jamming action is considered effective and a positive intermediate reward is given; conversely, if the radar threat level increases, the jamming is considered a failure and a negative intermediate reward is given.

[0022] Furthermore, if the radar switches from tracking to searching, or from identification to tracking, the threat level of the radar is determined to have decreased; if the radar switches from searching to tracking, or from tracking to identification, the threat level of the radar is determined to have increased.

[0023] Furthermore, the interference decision algorithm DAR-PPO introduces a KTD Clip dynamic pruning mechanism on the basis of the traditional PPO algorithm. It adaptively adjusts the pruning threshold by the difference between the empirical KL divergence and the target value, avoiding jumps in the policy update process while automatically converging to a suitable trust region.

[0024] Furthermore, the interference decision algorithm DAR-PPO also includes a RA-IS experience replay module, which stores recent trajectories in a replay buffer and combines importance sampling with outlier suppression weights to achieve... Correction to improve sample reuse rate.

[0025] On the other hand, the present invention also provides a cognitive interference decision-making application system based on deep reinforcement learning, the system being used to implement the above-mentioned cognitive interference decision-making method based on deep reinforcement learning.

[0026] The beneficial effects of this invention are as follows:

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] 1. Enhanced adaptability and generalization: Through the NR-CJM jamming decision model and the DAR-PPO jamming decision algorithm, rapid perception and adaptive adjustment of jamming modes are achieved for different radar behavior strategies.

[0029] 2. Higher training stability and adversarial efficiency: The adaptive KL regulator and importance sampling replay mitigate policy oscillation and reward sparsity issues, significantly improving training efficiency and decision robustness.

[0030] 3. Flexible environmental simulation capabilities: Through the NR-CJM jamming decision model, it supports modeling of multiple radars and multiple working modes, realizing the requirements of complex jamming and countermeasure scenarios with high dynamics and multiple targets. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the overall framework of the cognitive interference decision-making method based on deep reinforcement learning of this invention.

[0032] Figure 2 This is a diagram showing the switching of interference and countermeasure states;

[0033] Figure 3 This is a diagram illustrating radar status switching;

[0034] Figure 4 These are comparison charts of simulation experiments;

[0035] Figure 5 This is a schematic diagram of the interaction mechanism between an agent and its environment based on reinforcement learning. Detailed Implementation

[0036] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0038] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0039] The following combination Figures 1-5 Specific embodiments of the present invention will be described in detail below. It should be understood that the specific embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the present invention.

[0040] This invention relates to a cognitive jamming decision-making method and its application system based on deep reinforcement learning, belonging to the interdisciplinary field of electronic warfare, artificial intelligence, and intelligent control technology. It is particularly suitable for our jamming drones to complete intelligent jamming decision-making tasks when countering networked radar systems in complex and variable electromagnetic environments. This invention proposes a "lightweight enhancement-based" near-end policy optimization method, namely the jamming decision algorithm DAR-PPO (Dynamic Adjustment & Replay-aware PPO), to improve the performance of existing jamming decision-making systems in terms of sample efficiency and policy stability. The interference decision-making algorithm DAR-PPO includes two innovative modules: the first is the KTD-Clip module, which constructs feedback signals based on empirical KL divergence and dynamically adjusts the clipping threshold in policy updates to control the policy update step size, ensuring it remains within a stable and effective trust region, thus significantly improving the algorithm's adaptability to environmental changes; the second is the RA-IS module, which constructs a trajectory replay pool and introduces clipped importance sampling to achieve effective reuse of historical interaction samples. Simultaneously, it designs a weight entropy regularization mechanism to suppress policy fluctuations caused by abnormal weights, thereby significantly improving the utilization rate of training samples and enhancing the algorithm's robustness.

[0041] This invention simultaneously constructs a cognitive jamming decision model, NR-CJM (Networked Radar Cooperative Jamming Model). This model simulates the game-like confrontation between our multi-agent jamming platform (e.g., jamming UAVs) and the opponent's networked multi-functional radar system. It comprehensively considers the opponent's networked radar's multi-mode dynamic switching, adaptive signal parameter adjustment, and jamming feedback mechanism, characterizing the cognitive closed-loop of jamming strategies in complex adversarial environments. Based on the NR-CJM, our jamming agents continuously interact with the electromagnetic environment, autonomously learn and optimize their jamming strategies, thereby achieving continuous suppression of the target radar system.

[0042] like Figure 1As shown, the cognitive interference decision-making application system based on deep reinforcement learning according to the present invention includes interference drones and penetration drones. In the application scenario of a specific embodiment of the present invention, a swarm of four interference drones provides cover for a high-value penetration drone to perform a mission. The interference drones generate interference strategies based on the current observation state through the Actor module and carry out interference against the enemy's networked multi-functional radar deployed in the environment. The enemy's networked radar can autonomously adjust its reconnaissance parameters (e.g., operating carrier frequency, bandwidth, pulse width, etc.) according to changes in the electromagnetic environment. The trajectory memory, such as environmental state information (including radar parameters, interference effect feedback, etc.) and reward information during the execution of the interference round, is continuously stored in the experience replay pool on the drone for subsequent advantage function estimation and strategy optimization. The experience sampling adopts the RA-IS mechanism, which assigns importance weights to the trajectory memory, realizes the effective reuse of old samples, and suppresses the impact of outlier weights on training stability.

[0043] In the Cognitive Interference Decision Model (NR-CJM), within our UAV training network, the Actor (which selects actions from the current state, representing the agent's "policy") serves as the policy network, responsible for generating action decisions to interfere with the UAV under a given state. The Critic (which evaluates the quality of the current policy) serves as the value function network, used to estimate the value V(s) of the current state, assisting in evaluating the policy's merits and quantifying the benefit of the current action relative to the state value by calculating the advantage function, thus providing optimization direction for policy updates. Simultaneously, to address the difficulty in adjusting the clipping threshold in the PPO algorithm, this invention introduces the KTD-Clip module, utilizing empirical KL divergence as a feedback signal and dynamically adjusting the clipping threshold through a proportional controller. This ensures that policy updates remain within a reasonable trust region, achieving improved adaptability and generalization capabilities. Finally, the value function network (Critic) trains to minimize the loss function through an optimizer, continuously improving the effectiveness and robustness of the interference policy. In this study, a relatively stable electromagnetic environment is assumed, and the simulation scenario is constructed as an idealized scenario to facilitate evaluation of the algorithm's convergence performance, training efficiency, and real-time adversarial response capabilities.

[0044] Figure 2This is a diagram showing the switching of jamming and countermeasure states. In this embodiment, initially, the opposing multi-function radar is in state 1 (S1) and actively reconnoiters our jamming drones within the operational airspace. To effectively suppress its detection capabilities and achieve concealment and cover for our penetrating drones, the jamming drone matches its jamming strategy according to the current state of the opposing networked radar. At this time, it selects the first jamming mode (A1). As the electromagnetic environment dynamically evolves, when the multi-function networked radar senses a change in the external environment and switches from state 1 (S1) to a new state 2 (S2), our jamming drone can adjust its jamming strategy in real time, selecting a new jamming mode to adapt to the state changes of the opposing radar, thus achieving a continuous and effective suppression and jamming effect.

[0045] Figure 3 This diagram illustrates the radar's state transitions. The search state represents the lowest threat level, with the radar performing only wide-area scanning. The tracking state indicates a target lock (i.e., our drone). The identification state signifies the radar has entered the target classification and identification phase, further escalating the threat. Once in the guidance state, the target has been confirmed, and the radar begins guiding the fire control system for engagement, representing the highest threat level. The radar can freely switch between these three states (search, tracking, and identification) based on the battlefield situation. Once in the guidance phase, the radar state becomes irreversible and cannot be reverted to a lower level, reflecting the unrecoverable nature of high-threat situations.

[0046] According to the cognitive jamming decision-making method based on deep reinforcement learning of the present invention, the method is used to jam unmanned aerial vehicles (UAVs) in a multi-functional radar jamming countermeasure environment. It implements a jamming strategy against the radar by constructing a NetworkedRadar-Cooperative Jamming Model (NR-CJM). This model takes the state perception information of the radar in the environment and the relative position information of the UAV and the radar as inputs, constructs a state space, and outputs jamming actions adapted to the current mission environment, including the selection of the jamming radar and the selection of the jamming mode, as the policy function part of the reinforcement learning. Specifically, the method includes the following steps:

[0047] Step S1: Construct a radar jamming decision model, define a multi-agent cooperative jamming decision-making adversarial framework, and model the environment state space, jamming action space, and reward function;

[0048] Step S2: Initialize algorithm parameters, including policy network parameters, at the initial time. Value network parameters Learning rate Discount factor and KL target Initialize the replay buffer to store trajectory data;

[0049]

[0050] Step S3: Initialization Minimum and maximum values: , ; It is used to limit the magnitude of policy changes during each update, ensuring that the policy does not change drastically. Set to 0.05. Set to 0.3.

[0051] Step S4: Obtain the current state Make decisions based on the policy network: ; where the current state This includes the opponent's radar status parameters and the positional information between our jammer and the radar. The radar status parameters include carrier frequency, pulse width, bandwidth, pulse repetition frequency, and amplitude, which characterize the radar's physical properties and reflect its operational status. The positional information includes the coordinates of our jammer's position and the radar's position. (Policy network) Indicates the current state and current network parameters Next, select action.

[0052] Step S5: Perform the action Calculate the reward generated by this action. After the action is completed Afterwards, the environment changes, which is recorded as the next state. The action space includes the jamming target and the jamming mode to be executed. The jamming target is the enemy radar number to be jammed. The jamming pattern library has 8 jamming types: amplitude modulation noise jamming, frequency modulation noise jamming, noise modulation jamming, dense decoy table suppression jamming, range deception jamming, velocity deception jamming, angle deception jamming, and range-velocity deception jamming. The jammer selects a jamming pattern from the jamming pattern library.

[0053] Step S6: Based on the Q-value output by the value network (the Q-value includes...) , indicating that in a given state Below, the expected cumulative return after selecting action a is calculated, and the time-difference objective (TD objective) is calculated. And timing difference error (TD error) .

[0054] The calculation formula is:

[0055] ;

[0056] in, As a reward, This represents the predictive value of the next state (estimated via a Critic network).

[0057] TD error is used to measure the difference between the value estimated by the current value function network Critic and the true TD target. Its calculation formula is as follows:

[0058] ;

[0059] in, The TD error represents the predicted value of the current state. It tells the value function network Critic how much its estimation of the current state's value is off. A large error indicates that the Critic's estimation needs adjustment.

[0060] Finally, Store in the replay buffer and proceed to the next moment until the end of the current round;

[0061] Step S7: According to the new Adjust strategy; To limit the magnitude of policy changes during each update and ensure that the policy does not undergo significant alterations, a KTD Clip (KL Target Dynamic Clip) dynamic clipping mechanism is introduced to impose a clipping threshold. This avoids the learning instability that may result from a fixed threshold.

[0062] Step S8: Sample old trajectories from the replay pool for training, and calculate the importance sampling weights:

[0063] ;

[0064] in, Indicates the pre-policy in state Select action The probability, Indicates the old strategy in state Select action The probability is then used to calculate the weight value. (Regarding the weights...) Perform cropping to prevent excessive weighting of importance. This indicates a significant difference between the current and old strategies, potentially leading to unstable strategy updates. Excessive weighting... To impose penalties, add a penalty term to the loss function. This limits the drastic pace of policy updates.

[0065] Step S9: Proceed to the next round until the training ends.

[0066] The multi-functional networked radar space countermeasure mission scenario of this invention is as follows: A formation consisting of one penetrating UAV performing a combat mission and four jamming UAVs attempts to traverse a defense zone deployed with four ground-based multi-functional networked radars to complete a predetermined combat mission. The jamming UAVs fly in front of the penetrating UAV, undertaking the task of suppressing enemy radar, aiming to provide effective cover for the combat UAV (penetrating UAV) so that it can smoothly reach the target area along a predetermined trajectory and complete its mission. It is assumed that the location of the enemy networked radar is known, and the flight trajectories of the UAV formation are all within the effective detection range of the radar. If no jamming measures are taken, the combat UAV is very easily detected by the enemy radar system during flight.

[0067] During a training round, four jamming drones perceive changes in the environmental state in real time. This state includes the carrier frequency, pulse width, bandwidth, pulse repetition frequency, amplitude, and real-time position of each enemy network radar, as well as the drone formation's position. This state information serves as the input data for training; environmental state data is collected in real time and fed into the reinforcement learning model for processing.

[0068] After data collection is complete, the policy network determines the current state. Output an action distribution (usually a probability distribution), from which a specific jamming strategy is sampled. Each strategy corresponds to a jamming action, specifically selecting a jamming style from the radar jamming style library for a particular target.

[0069] After the selected action is performed, the environment will give a reward based on the effect of the action (e.g., whether the interference was successful). The reward reflects the merits of choosing that action in that state under the current strategy. For jamming missions, successfully suppressing enemy radar and protecting the combat mission aircraft might yield a higher reward, while failure would yield a lower reward. Then, based on the Q-value output of the value network (representing the performance of an aircraft in a given state), the reward is calculated. Next, select an action. The expected cumulative return is used to calculate the time-difference objective (TD objective). And timing difference error (TD error) The data is placed in the buffer, and one training round ends. The next training iteration is dynamically adjusted based on the KL divergence. To adjust and update the strategy, execute multiple rounds to load a large amount of data into the cache for training the value network.

[0070] The value network is trained using cached data, evaluates the effectiveness of the current policy, and helps the policy network improve the policy by optimizing the state-value function. The policy network optimizes and updates its policy based on the feedback provided by the value network, enabling future decisions to yield greater rewards. This process continues until an optimal perturbation decision model is trained.

[0071] The state space of the radar jamming decision model includes the radar's state parameters and the positional information between the jamming UAV and the radar. The radar's state parameters include carrier frequency (CF), pulse width (PW), bandwidth (BW), pulse repetition frequency (PRF), and amplitude (AP). These state parameters not only characterize the radar's physical transmission characteristics but also reflect its operating mode. The action space of the radar jamming decision model includes the target number and the selected jamming mode, i.e., the decision to implement a certain jamming pattern on a specific radar, together forming an ordered combination of jamming strategies.

[0072] The interference decision algorithm DAR-PPO of this invention introduces a KTD Clip (KLTarget Dynamic Clip) dynamic clipping mechanism on the basis of the traditional PPO algorithm. It adaptively adjusts the clipping threshold based on the difference between the empirical KL divergence and the target value, avoiding jumps during policy updates while automatically converging to a suitable trust region. KL divergence is used to measure the difference between the old and new policies. Constraints on KL divergence are added during policy updates to control the magnitude of policy updates. If the KL divergence is too large, it indicates that the policy change is too large and the update is too fast; therefore, the update step size needs to be reduced (usually by reducing...). (to control); conversely, if the KL divergence is too small, the update stride can be increased to further accelerate training.

[0073] The KTD Clip module dynamically adjusts the clipping threshold. This allows it to automatically adjust based on the difference between the KL divergence at each step and the target KL divergence. The calculation method is as follows:

[0074] ;

[0075] in, The KL divergence at the current moment is calculated as follows:

[0076] ;

[0077] This is the learning rate hyperparameter that controls the update magnitude; it is set to 0.005. It is a policy network function. Represents the old policy network. This represents the new strategy network. This represents the target KL divergence, set to a value of 0.01, and can be dynamically adjusted. The KTD Clip module can effectively control the range of policy changes, improving training efficiency and the stability of policy updates.

[0078] Furthermore, the interference decision-making algorithm DAR-PPO of this invention adds a RA-IS (Replay-Aware Importance Sampling) experience replay module to the traditional PPO algorithm. This module stores recent trajectories in a replay buffer and combines importance sampling with outlier suppression weights, thereby achieving stable, low-variance performance. Correction significantly improves sample reuse rate.

[0079] The specific algorithm of the method of this invention is shown below:

[0080] ;

[0081]

[0082] This invention introduces the interference decision algorithm DAR-PPO (Dynamic Adjustment Replay Aware PPO) based on the reinforcement learning algorithm PPO: aiming to balance sample efficiency and pruning robustness without increasing the complexity of the PPO main structure. The algorithm mainly consists of the following two modules:

[0083] (1) Module: Using empirical KL divergence as feedback signal, a proportional control mechanism is introduced to dynamically adjust... This ensures that policy updates always fall within a reasonable trust domain. Compared to fixed pruning, this method can adapt to changes in environmental complexity, significantly reducing hyperparameter search costs; the KTD Clip mechanism dynamically adjusts the PPO pruning threshold. This avoids problems such as learning instability or insufficient updates that may arise from a fixed threshold, thereby improving data utilization efficiency and training stability. By dynamically adjusting the pruning threshold... This allows it to automatically adjust proportionally to the difference between the KL divergence at each step and the target KL divergence.

[0084] ;

[0085] in, The KL divergence at the current moment is calculated as follows:

[0086] ;

[0087] This is the learning rate hyperparameter that controls the update magnitude; it is set to 0.005. It is a policy network function. Represents the old policy network. This represents the new strategy network. This represents the target KL divergence, set to a value of 0.01. It is dynamically adjusted. KTD Clip can effectively control the range of policy changes, improving training efficiency and the stability of policy updates.

[0088] (2) Module: The PPO algorithm typically uses only the most recently sampled data for training, resulting in low data utilization. The RA-IS mechanism introduces experience replay and importance sampling weights, enabling PPO to reuse older data while maintaining stability. During training, trajectories are cached in an experience replay pool, and trajectory samples are then retrieved from the cache during training, with importance sampling weights calculated.

[0089] ;

[0090] in, Indicates the pre-policy in state Select action The probability, Indicates the old strategy in state Select action The probability is used to calculate the weight value. For the weight... Perform cropping to prevent excessive weighting of importance. This indicates that the current policy is better than the old policy in terms of actions. A higher probability of success indicates a significant change in strategy. Rapidly updating the strategy can lead to instability, so it's necessary to suppress such updates and limit excessively large weights. To impose penalties, add a penalty term to the loss function. And by introducing an entropy regularization term This suppresses policy fluctuations caused by abnormal weights, improving sample reuse while maintaining training stability. Centropy is the entropy regularization coefficient used to control the contribution of the entropy regularization term to the total loss function.

[0091] ;

[0092] in, In state Next, take action The probability (usually a probability distribution output by the policy network, used to measure the diversity or uncertainty of the policy).

[0093] In step S4, a reward mechanism is designed, and a reward function is used to determine the reward for this step. By continuously refining the jamming strategy, the threat level of the target radar is minimized, thereby completing the mission of covering the penetration.

[0094] Specifically, the reward function is divided into two parts: intermediate reward. Rewards for reaching As shown in equation (1). After the jamming UAV performs a jamming action, if the radar threat level decreases (e.g., switching from tracking to searching), the jamming action is effective, and a positive intermediate reward is given. =0.3; conversely, if the radar's threat level increases (e.g., switching from search to track mode), the jamming is considered a failure, and a negative intermediate reward is given. =-1, if the threat level remains unchanged, then the intermediate reward... =0.01. A decrease in radar threat level refers to a radar switching from tracking to search mode, or from identification to tracking mode; an increase in radar threat level refers to a radar switching from search mode to tracking mode, or from tracking to identification mode.

[0095]

[0096] Whether the penetration drone successfully reaches the target area is a key indicator for evaluating mission completion. Therefore, this objective should be strongly guided, and a higher reward value should be assigned to this result. When the drone successfully reaches the target area, a higher positive arrival reward should be given. =10; If detected by the multi-function networked radar and enters guidance mode during flight, the mission fails and a negative arrival reward is given. As shown in equation (2):

[0097]

[0098] In summary, the overall form of the reward function set by this invention is shown in equation (3), which includes the intermediate cumulative reward during the round execution process. Rewards for reaching the end of the round :

[0099]

[0100] This invention also proposes an information entropy perception strategy adjustment method: based on the uncertainty estimation of radar behavior, the strategy exploration degree is dynamically adjusted to prevent the strategy from getting stuck in local optima and to enhance the adversarial generalization ability.

[0101] Figure 4This is a simulation experiment comparison graph, comparing the reward curves of the DAR-PPO interference decision algorithm of this invention and the classic PPO algorithm during reinforcement learning training. The horizontal axis represents the number of training rounds, and the vertical axis represents the reward obtained by the agent in the environment. As can be seen from the graph, the interference drone using the DAR-PPO interference decision algorithm of this invention reached a stable convergence state in 3792 rounds, while the classic PPO algorithm only stabilized after 4662 rounds. The convergence speed of the DAR-PPO interference decision algorithm was improved by 17.27% during training. In terms of convergence stability, the reward fluctuation of the DAR-PPO interference decision algorithm after convergence was significantly smaller than that of PPO, demonstrating stronger training stability and indicating that its policy optimization process is more robust. Furthermore, throughout the entire training process and the final convergence stage, the average reward of the DAR-PPO interference decision algorithm was significantly higher than that of PPO, with an improvement of approximately 13.64%, reflecting the superior policy performance of this method.

[0102] Figure 5 This study demonstrates the interaction mechanism between an agent and its environment based on reinforcement learning. It primarily includes five core elements: Agent, Environment, State, Action, and Reward, reflecting the basic process of policy learning and environmental feedback in cognitive interference decision-making tasks. At each decision time t, the agent, based on its current state... Select and execute an action This action acts on an electromagnetic interference environment, causing a change in the environmental state and generating a new state for the next step. and the corresponding reward signals After receiving sufficient states and rewards, the agent optimizes its strategy according to a pre-defined policy update mechanism, thereby gradually improving its decision-making performance in dynamic adversarial environments. The entire interaction process constitutes a closed-loop feedback system, allowing the agent to maximize long-term cumulative rewards through continuous interaction with the environment.

[0103] The following technical effects can be achieved by using this invention:

[0104] 1. Enhanced adaptability and generalization: Through the NR-CJM jamming decision model and the DAR-PPO jamming decision algorithm, rapid perception and adaptive adjustment of jamming modes are achieved for different radar behavior strategies.

[0105] 2. Higher training stability and adversarial efficiency: The adaptive KL regulator and importance sampling replay mitigate policy oscillation and reward sparsity issues, significantly improving training efficiency and decision robustness.

[0106] 3. Flexible environmental simulation capabilities: Through the NR-CJM jamming decision model, it supports modeling of multiple radars and multiple working modes, realizing the requirements of complex jamming and countermeasure scenarios with high dynamics and multiple targets.

[0107] Furthermore, the present invention has the following advantages:

[0108] (1) This invention proposes an interference decision algorithm with no structural redundancy and lightweight deployability to improve the learning efficiency of adversarial strategies;

[0109] (2) This invention introduces a dynamic tuning and sample replay mechanism to enhance the stability and sample utilization of deep reinforcement learning in interference tasks;

[0110] (3) The present invention establishes a realistic, multi-mode, multi-agent interference simulation platform to support the evaluation of intelligent interference strategies in complex electromagnetic environments.

[0111] Any process or method described in the flowcharts of this invention or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device. The computer-readable medium can be any medium containing a program for storage, communication, propagation, or transmission for use by the execution system, apparatus, or device, including read-only memory, magnetic disks, or optical disks.

[0112] In the description of this specification, references to terms such as "embodiment," "example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, those skilled in the art can combine or combine the different embodiments or examples described in this specification and the features therein without causing contradiction.

[0113] While embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and alterations to the above embodiments within the scope of the present invention.

Claims

1. A cognitive interference decision-making method based on deep reinforcement learning, characterized in that, The method is used to interfere with the jamming of UAVs against multi-functional networked radar in an environment, and includes the following steps: Step S1: Construct a radar jamming decision model, define a multi-agent cooperative jamming decision-making adversarial framework, and model the environment state space, jamming action space, and reward function; Step S2: Initialize the parameters of the DAR-PPO perturbation decision algorithm at the initial time. The parameters include the policy network parameters. Value network parameters Learning rate Discount Factor and KL target Initialize the replay buffer to store trajectory data; Step S3: Initialization Minimum and maximum values Step S4: Obtain the current state Make decisions based on the policy network: ; Step S5: Perform the action Observe the reward at this step and the next step ; Step S6: Calculate the temporal difference objective based on the Q-value output by the value network. and timing difference error ; Step S7: According to the new Adjust strategy; Step S8: Sample old trajectories from the replay pool for training; Step S9: Proceed to the next round until the training ends; Among them, policy network Indicates the current state and current network parameters Next, select the action. The value represents the expected cumulative reward after choosing an action in a given state; This indicates the clipping threshold.

2. The cognitive interference decision-making method based on deep reinforcement learning according to claim 1, characterized in that, The state space of the radar jamming decision model includes the radar's state parameters and the positional information between the jamming UAV and the radar.

3. The cognitive interference decision-making method based on deep reinforcement learning according to claim 1, characterized in that, The action space of the radar jamming decision model includes the target number and the selected jamming mode, which together constitute an ordered combination of jamming strategies.

4. The cognitive interference decision-making method based on deep reinforcement learning according to claim 2, characterized in that, The radar's status parameters include carrier frequency, pulse width, bandwidth, pulse repetition frequency, and amplitude.

5. The cognitive interference decision-making method based on deep reinforcement learning according to claim 1, characterized in that, In step S4, a reward mechanism is designed, and a reward function is used to determine the reward for this step. Continuously refine jamming strategies to minimize the target radar threat level.

6. The cognitive interference decision-making method based on deep reinforcement learning according to claim 5, characterized in that, The reward function consists of two parts: intermediate reward. Rewards for reaching When a jamming drone performs a jamming action, if the radar threat level decreases, the jamming action is considered effective and a positive intermediate reward is given; conversely, if the radar threat level increases, the jamming is considered a failure and a negative intermediate reward is given.

7. The cognitive interference decision-making method based on deep reinforcement learning according to claim 6, characterized in that, If the radar switches from tracking to searching, or from identification to tracking, the threat level of the radar is judged to have decreased. If the radar switches from search mode to track mode, or from track mode to identification mode, the threat level of the radar is considered to have increased.

8. The cognitive interference decision-making method based on deep reinforcement learning according to claim 1, characterized in that, The interference decision algorithm DAR-PPO introduces a KTD Clip dynamic pruning mechanism, which adaptively adjusts the pruning threshold based on the difference between the empirical KL divergence and the target value, avoiding jumps during policy updates while automatically converging to a suitable trust region.

9. The cognitive interference decision-making method based on deep reinforcement learning according to claim 8, characterized in that, The interference decision algorithm DAR-PPO also includes a RA-IS experience replay module, which stores recent trajectories in a replay buffer and combines them with importance sampling to suppress outlier weights, thereby achieving... Correction to improve sample reuse rate.

10. A cognitive interference decision-making application system based on deep reinforcement learning, characterized in that, The system is used to implement the cognitive interference decision-making method based on deep reinforcement learning according to any one of claims 1-9.

Images