An equipment intelligent decision-making method based on threat level classification

Abstract

The present application relates to radar electronic countermeasure technical field, especially to a kind of equipment intelligent decision-making method based on threat level classification.It includes: the electronic warfare decision state space including own defense equipment, enemy attack equipment and environment state is constructed;Select threat assessment factor, determine the weight of each factor by preset weight calculation method, quantify single enemy equipment and battlefield total threat value, at least two threat levels are divided according to threshold;The collaborative decision-making architecture including high-level threat assessment network, multiple low-level strategy network and timing state control module is constructed, high-level network outputs threat level and activates corresponding low-level network, low-level network outputs decision action, timing module calculates state change score to determine whether to trigger new decision;Network training optimization, through initialization parameter, partition management trajectory data, preset algorithm updates network, dynamically adjusts exploration rate.The advantages are: improve the pertinence of decision, real-time;Enhance the stability of complex electronic warfare scene decision.

Description

Technical Field

[0001] This invention relates to the field of radar electronic countermeasures technology, and in particular to an intelligent equipment decision-making method based on threat level classification. Background Technology

[0002] Reinforcement learning (RL), as a key branch of artificial intelligence, learns optimal strategies through interaction between intelligent agents and the environment, providing a feasible path for intelligent decision-making in electronic warfare equipment. Among them, the Proximal Policy Optimization (PPO) algorithm has become one of the mainstream Actor-Critic algorithms due to its high training stability and sample efficiency. Hierarchical reinforcement learning (HRL) shows potential in complex decision-making scenarios by decomposing complex tasks into high-level policy selection and low-level action execution. However, the electronic warfare environment is characterized by high dynamism, strong adversarial nature, and uncertainty, and existing reinforcement learning methods are still difficult to directly meet the actual decision-making needs of equipment.

[0003] Currently, the closest existing technical solutions mainly include three categories: (1) Electronic warfare equipment decision-making scheme based on standard PPO algorithm. This scheme adopts a single policy network. The input includes the electronic warfare equipment status (such as radar detection range and jammer power) and environmental information (such as enemy equipment location). The output is the decision action (such as jamming target selection and radar working mode switching). The PPO algorithm updates the policy parameters based on the reward signal from the environment (such as enemy equipment failure and friendly equipment successful protection); (2) Equipment decision-making scheme based on hierarchical reinforcement learning (HRL). It adopts a two-level decision structure of high-level "policy selection" and low-level "action execution". The high-level network selects pre-trained low-level strategies based on the macro-situation of the battlefield (such as the size of the enemy's equipment cluster and the stage of the operation). The low-level network generates specific equipment control commands accordingly and improves training efficiency through experience playback mechanism. This method has been applied to scenarios such as training of air combat agents and collaborative interference decision-making. (3) The PPO (PPO-RNN) scheme based on recurrent neural network enhancement introduces an RNN structure into the PPO algorithm in order to capture the temporal characteristics in the electronic warfare environment. It utilizes the ability to remember historical state sequences to model the evolution of battlefield state and improve the temporal coherence of decision-making. It is typically applied to scenarios such as radar anti-jamming frequency hopping.

[0004] However, the above-mentioned existing technical solutions still have many shortcomings: (1) For the solution based on the standard PPO algorithm, it is difficult to deal with the reward sparsity problem. In the electronic warfare scenario where the effective reward signal is extremely limited, the single policy network is difficult to extract effective decision patterns from a large number of unrewarded steps, resulting in slow convergence or getting stuck in local optima during the training process. At the same time, the decision frequency lacks reasonable control, and the necessary decision steps and redundant steps are not distinguished. Each step generates a decision action, which not only introduces noise interference to the training process, but also wastes computing resources and is difficult to meet the real-time decision-making needs of electronic warfare; (2) For the solution based on hierarchical reinforcement learning (HRL), it lacks a policy guidance mechanism based on threat level and does not quantify enemy threat factors (such as distance, speed, and damage capability). The high-level strategy selection relies solely on macro-level scene features and cannot generate differentiated decision-making strategies based on threat levels, affecting decision accuracy. Furthermore, it lacks the ability to respond to sudden threats and has limited perception of key state changes such as the sudden appearance of high-threat targets on the battlefield, making it difficult to trigger strategy switching in a timely manner and easily missing key defense windows. (3) For the PPO-RNN scheme, its temporal feature extraction mechanism is inefficient. Although the RNN structure has the ability to remember historical states, it does not distinguish the importance of different state changes, making it difficult to accurately identify key decision-making opportunities. There are still redundant decision-making problems. Moreover, the strategy network structure has not been optimized in a targeted manner. Using a single strategy network to deal with different threat level scenarios, it cannot achieve strategy adaptation in differentiated situations such as high threat and low threat, resulting in insufficient decision adaptability.

[0005] In summary, existing reinforcement learning techniques still face many unresolved technical problems in electronic warfare equipment decision-making applications: (1) Reward sparsity problem: Electronic warfare confrontation processes involve a large number of simulation steps, but reward signals that can reflect effective confrontation results (such as successfully jamming enemy equipment, intercepting missiles, etc.) appear infrequently, making it difficult for equipment to obtain effective learning signals under initial random strategies, resulting in low training efficiency. (2) Unreasonable decision frequency setting: In dynamic electronic warfare environments, it is not necessary to generate new decisions in every simulation step. Frequent execution of unnecessary decisions will introduce noisy data, affect training stability, and waste computing resources, making it difficult to meet the real-time decision-making requirements of electronic warfare. (3) Enemy threat assessment is not fully incorporated into the strategy selection mechanism: Existing hierarchical reinforcement learning methods do not fully consider the impact of dynamic changes in the battlefield threat situation on equipment decision-making, and lack the ability to formulate differentiated decision-making strategies according to different threat levels, resulting in insufficient decision accuracy and environmental adaptability. (4) Existing decision-making frequency control mechanisms are insufficient in responding to sudden threats: When new high-threat targets appear on the battlefield (such as the sudden appearance of enemy anti-radiation missiles), existing methods are unable to promptly stop current ineffective actions and trigger re-decision-making, which can easily lead to missed critical defense opportunities and reduced combat effectiveness. Therefore, it is urgent to propose improvement solutions to address the above-mentioned technical problems. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides an intelligent equipment decision-making method based on threat level classification.

[0007] The purpose of this invention is to provide an intelligent equipment decision-making method based on threat level classification, which specifically includes the following steps:

[0008] S1. Construct a decision-making state space in an electronic warfare scenario, including the state of friendly defensive equipment, the state of enemy attack equipment, and the environmental state;

[0009] S2. Select multiple threat assessment factors, determine the weight of each factor through a preset weight calculation method, quantify the threat value of a single enemy attack equipment and the total battlefield threat value based on the weight, and divide the battlefield threat level into at least two levels according to a preset threshold.

[0010] S3. Construct a threat level hierarchical strategy-self-attention collaborative decision-making network architecture, including a high-level threat assessment network, multiple low-level policy networks, and a temporal state control module; the high-level threat assessment network outputs the current threat level based on the total battlefield threat value and key state parameters and activates the corresponding low-level policy network, and the low-level policy network outputs the decision actions of electronic warfare equipment; the temporal state control module runs in parallel with the low-level policy networks, calculates the state change score by capturing the temporal correlation characteristics of the battlefield state, and determines whether to trigger a new decision;

[0011] S4. Perform network training and iterative optimization by initializing network parameters, partitioning and managing trajectory data, updating the low-level policy network with preset algorithms and optimizing the high-level threat assessment network, dynamically adjusting the training exploration rate and setting convergence conditions to improve network decision-making performance.

[0012] Preferably, in step S1, the friendly defense equipment includes early warning radar, electronic jammer, fire control radar, and core targets that need to be protected; the enemy attack equipment includes penetration aircraft, electronic warfare aircraft, reconnaissance aircraft, and various missiles.

[0013] The status of the friendly defense equipment includes the location coordinates, operating parameters, and operating status of the defense equipment. The operating parameters include the radar operating frequency band and the jammer output power. The operating status includes standby, running, and fault.

[0014] The status of the enemy attack equipment includes the position coordinates, motion parameters, damage capability level, and electronic jamming capability parameters of the attack equipment; the motion parameters include speed, heading, and altitude; the environmental status includes weather parameters, including visibility, rainfall intensity, and electromagnetic interference intensity.

[0015] Preferably, in step S2, the threat assessment factors include distance, heading angle, target speed, target damage capability, target electronic jamming capability, target combat status, and environmental factors; the preset weight calculation method includes the analytic hierarchy process (AHP), which calculates weights by constructing a judgment matrix, performing consistency checks, and using the eigenvector method.

[0016] Preferably, the battlefield threat levels include low threat, medium threat, and high threat, and the dividing thresholds are determined through electronic warfare scenario simulation and combat experience.

[0017] When quantifying the threat value of a single enemy attack device, the min-max normalization method is used to map the original data of each threat assessment factor to the [0,1] interval, and the threat value of a single enemy attack device is calculated using a weighted summation formula:

[0018] ;

[0019] In the formula: This represents the threat value of the i-th enemy attack equipment, with a value range of [0,1]. This represents the weight of the k-th type of factor; This represents the normalized value of the i-th blue team equipment in the k-th factor category;

[0020] The formula for the total battlefield threat value is as follows:

[0021] ;

[0022] In the formula, This represents the threat value of the i-th enemy attack equipment, with a value range of [0,1]. n This indicates the number of enemy attack equipment.

[0023] Preferably, the battlefield threat levels include low threat, medium threat, and high threat, and the dividing thresholds are determined through electronic warfare scenario simulation and combat experience.

[0024] In step S3, the high-level threat assessment network adopts a single hidden layer fully connected structure and outputs the threat level through the Softmax activation function; the hidden layer contains 64 neurons.

[0025] The low-level policy network is set up with three layers, corresponding to low threat, medium threat and high threat levels respectively. Each network has 128 neurons in its hidden layer. After being mapped by the Tanh activation function, it outputs four types of decision actions: jamming target selection, fire control radar target selection, jamming intensity adjustment and fire control radar mode switching.

[0026] Preferably, in step S3, the timing state control module is a self-attention action control module or a simple timing control module based on threshold judgment;

[0027] The self-attention action control module adopts a combination structure of self-attention layer and fully connected layer. The input is a tensor composed of 5-step historical state sequence and current state. The self-attention layer calculates the similarity score between states. After normalization, feature dimension averaging and mapping by the fully connected layer, the output is a scalar-type state change score S as a state change index.

[0028] The simple timing control module based on threshold judgment calculates the Euclidean distance or Manhattan distance between the five historical states and the current state, and uses the distance as an indicator of state change.

[0029] Preferably, when using the self-attention action control module, the similarity score is calculated using the following formula:

[0030] ;

[0031] in, , Represents any two states; W Q , W K These are the learnable queries and the key weight matrix, respectively. d k For key dimensions.

[0032] Preferably, the state change score S is compared with a preset threshold τ. When S>τ, a new decision is triggered, and when S≤τ, the previous decision is maintained. The preset threshold τ=2.2.

[0033] Preferably, the preset algorithm is the PPO algorithm, which uses a clipped objective function to update the low-level policy network. The objective function expression is:

[0034] ;

[0035] in, For strategy ratio, This is the estimated value of the dominance function. For the clip parameter.

[0036] Preferably, in step S3, the timing state control module is a self-attention action control module;

[0037] Step S4 specifically includes the following sub-steps:

[0038] S41. Design a multi-dimensional reward function: Construct a reward system that includes positive rewards and negative penalties, with the total reward being the sum of all components:

[0039] ;

[0040] In the formula: R1 is a reward for locking onto the fire control radar; R 2 is a reward for the effectiveness of interference; R 3 is the reward for destroying enemy equipment; R 4 indicates a penalty for protecting one's own defensive equipment; R 5 represents the task completion reward; the expression and reward mechanism are as follows: When the distance between the fire control radar and the target is less than its working range, When the distance between the jammer and the target is less than its operating range, A fixed reward is given for destroying enemy attack equipment. A penalty is imposed when friendly defensive equipment is destroyed. A global reward is given for successfully intercepting or preventing the launch of an enemy missile. ;

[0041] In the above formula: R radar For radar operating range, D target For the target distance, T target Threat level of the target N For the number of consecutive locks, α , β , These are the weighting coefficients. R jammer The operating range of the jammer. The penalty coefficient is... High reward coefficient;

[0042] S42. Initialize training parameters: The learnable parameters of the high-level threat assessment network, the low-level policy network, and the self-attention action control module are randomly initialized using the Xavier method;

[0043] S43. Iterative Training and Network Update: The agent interacts with the electronic warfare environment to generate "state-action-reward-next state" trajectory data, which is stored in an independent experience replay buffer according to threat level and updated using a first-in-first-out mechanism; the trajectory data is randomly sampled, and the low-level policy network is updated through the PPO algorithm clipped objective function. At the same time, a joint loss function is constructed based on the accuracy of threat level determination and the battlefield mission completion rate to optimize the parameters of the high-level threat assessment network.

[0044] S44. Dynamically adjust training strategy: Adjust the exploration rate in stages according to the training progress;

[0045] S45. Training Convergence Judgment: Set a convergence index system including battlefield mission completion rate, threat level judgment accuracy, and decision response delay. Training is terminated when all convergence indices are met for multiple consecutive rounds of training; otherwise, the hyperparameters are adjusted and training is restarted.

[0046] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0047] (1) Solve the problem of sparsity of rewards and improve decision-making orientation: By designing a multi-dimensional reward system that includes positive rewards such as fire control radar lock-on, jamming effectiveness, and destruction of enemy equipment, as well as negative penalties for the protection of friendly equipment, the core combat targets of electronic warfare are accurately associated, avoiding the problem of "no clear reward feedback" in traditional reinforcement learning, guiding the agent to prioritize learning high-value decision behaviors, and shortening the training convergence cycle.

[0048] (2) Optimize decision frequency and balance real-time performance and resource consumption: Relying on the time-series state control module (self-attention mechanism or threshold time-series control) to capture the time-series correlation characteristics of battlefield state, new decisions are triggered only when the state change reaches the key threshold, effectively filtering unnecessary decision switching, avoiding delays in response to sudden threats, reducing the occupation of ineffective computing resources, and adapting to the real-time combat needs of electronic warfare equipment.

[0049] (3) Strengthen threat assessment integration and improve decision-making targeting: Quantify the weight of threat factors through the hierarchical analysis method, accurately calculate the total threat value of individual enemy equipment and battlefield, and divide multiple threat scenarios; then combine the hierarchical strategy network to output exclusive decision actions, avoid the limitations of the traditional "single strategy to adapt to all scenarios", and realize the priority handling of high threats and efficient response to low threats.

[0050] (3) Enhance network adaptability and practicality: Provide threat assessment methods that can be flexibly selected according to electronic warfare scenarios (such as the difficulty of factor quantification and equipment resource constraints); through dynamic exploration rate adjustment and multi-index convergence judgment training optimization, ensure that the network can still maintain high decision accuracy and stability in complex and ever-changing electronic warfare environments. Attached Figure Description

[0051] Figure 1 This is an algorithm flowchart of an equipment intelligent decision-making method based on threat level classification provided in an embodiment of the present invention. Detailed Implementation

[0052] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0054] The core objective of this invention is to address the problems of reward sparsity, unreasonable decision frequency, insufficient integration of threat assessment, and delayed response to sudden threats in the application of existing reinforcement learning techniques in electronic warfare equipment decision-making. It provides an intelligent equipment decision-making method based on threat level classification, specifically including the following steps:

[0055] S1. Construct a decision-making state space for electronic warfare scenarios. The state space should include at least the state of friendly defensive equipment, the state of enemy attacking equipment, and the environmental state. The state of friendly defensive equipment includes its location, operating parameters, and operational status. The state of enemy attacking equipment includes its location, motion parameters, and combat capability-related parameters. The environmental state includes environmental parameters affecting equipment effectiveness. Specifically, it includes:

[0056] S11. Clearly define the main combatants in the electronic warfare scenario as friendly defensive equipment (red force equipment) and enemy attack equipment (blue force equipment). Friendly defensive equipment includes early warning radar, electronic jammers, fire control radar and core targets that need to be protected, while enemy attack equipment includes penetrating aircraft, electronic warfare aircraft, reconnaissance aircraft and various missiles.

[0057] S12. Define the three core data dimensions of the state space:

[0058] Status of friendly defense equipment: Record the location coordinates, operating parameters (such as radar operating frequency band, jammer output power) and operating status (standby, running, fault) of all friendly defense equipment;

[0059] Enemy attack equipment status: Record the location coordinates, motion parameters (speed, heading, altitude), damage level, and electronic jamming capability parameters (jamming power, frequency band coverage) of the detected enemy attack equipment.

[0060] Environmental conditions: Record weather-related parameters that affect the sensor's detection performance (such as visibility, rainfall intensity, and electromagnetic interference intensity).

[0061] S2. Select multiple threat assessment factors, use the analytic hierarchy process (AHP) to determine the weights of each factor, quantify the threat value of individual enemy attack equipment and the total battlefield threat value through weighted calculation, and classify the battlefield threat level into at least two levels based on preset thresholds; specifically including:

[0062] S21. Seven threat factors (C1-C7) are selected: distance, heading angle, target speed, target damage capability, target electronic jamming capability, target combat status, and environmental factors. A judgment matrix is ​​constructed using the Analytic Hierarchy Process (AHP), and the weights of each factor are calculated using the eigenvector method. The weights of distance, damage capability, and speed are 0.1823, 0.1715, and 0.1516, respectively. See Table 1 for a description of the threat factors.

[0063] Table 1 Threat Factor Description Table

[0064]

[0065] In some embodiments, values ​​are assigned according to the Saaty 1-9 scale (1 indicates equal importance, 3 indicates slightly important, 5 indicates significantly important, 7 indicates very important, 9 indicates extremely important, and the reciprocal indicates the opposite comparison result) to form a 7×7 judgment matrix; the judgment matrix is ​​subjected to a consistency test (the consistency index CI, the random consistency index RI, and the consistency ratio CR are calculated; if CR < 0.1, the test is passed; otherwise, the judgment matrix is ​​readjusted); the threat factor AHP judgment matrix is ​​shown in Table 2;

[0066] Table 2 Threat Factor AHP Judgment Matrix

[0067]

[0068] S22. Based on the weights of each factor and the normalized values ​​of the corresponding factors of the enemy attack equipment, the threat value of a single enemy attack equipment is obtained by weighted summation.

[0069] In some embodiments, the original data of the k-th type of threat factor of the i-th enemy attack equipment is normalized. The min-max normalization method is used to map the factor data of different dimensions to the [0,1] interval (for example, the distance factor is mapped according to the principle of "the closer, the higher the threat", and the speed factor is mapped according to the principle of "the faster, the higher the threat") to obtain the normalized value. Based on the weight of each factor and the corresponding normalized value, the threat value of a single enemy attack equipment is calculated by a weighted summation formula:

[0070] ;

[0071] In the formula: This represents the threat value of the i-th enemy attack equipment, with a value range of [0,1]. This represents the weight of the k-th type of factor; This represents the normalized value of the i-th blue team equipment in the k-th factor category;

[0072] S23. Sum the threat values ​​of all enemy attack equipment to obtain the total battlefield threat value, and divide the battlefield threat level into at least two levels according to a preset threshold;

[0073] In some embodiments, battlefield threat levels include low threat, medium threat, and high threat;

[0074] The total battlefield threat value is obtained by summing the threat values ​​of all n detected enemy attack equipment.

[0075] ;

[0076] In the formula, This represents the threat value of the i-th enemy attack equipment, with a value range of [0,1]. n Indicates the number of enemy attack equipment;

[0077] Threat levels are divided into three categories: "low threat," "medium threat," and "high threat" based on preset thresholds. These thresholds are determined through electronic warfare scenario simulations and operational experience. For example, the threshold separating low and medium threats is 0.35, and the threshold separating medium and high threats is 0.7. The weights or thresholds of each factor are dynamically adjusted according to specific operational mission requirements (such as when protecting key targets) to optimize the scenario adaptability of the threat level classification.

[0078] In some embodiments, in addition to using the Analytic Hierarchy Process (AHP) to quantify threat levels, the following alternatives may be used:

[0079] (1) Threat level assessment based on fuzzy comprehensive evaluation: By establishing the fuzzy membership function of the threat factors (such as the "near-medium-far" fuzzy set of distance), the weighted average operator is used to calculate the fuzzy comprehensive evaluation value of a single target threat, and the threat level is divided according to the principle of maximum membership. This scheme does not require the construction of a judgment matrix and is suitable for scenarios where the threat factors are difficult to describe quantitatively, but the accuracy is slightly lower than that of AHP (error of about 5%-8%).

[0080] (2) Threat level classification model based on machine learning: Threat factor data (such as enemy equipment location, speed, and damage records) in historical electronic warfare scenarios are used as training samples. Support vector machine (SVM) or random forest algorithm is used to construct a threat level classification model to replace the manually designed AHP weights. This scheme relies on ≥10,000 sets of labeled data, but it is more adaptable in complex multi-factor coupled scenarios.

[0081] S3. Construct a threat level hierarchical strategy-self-attention collaborative decision-making network architecture, including a high-level threat assessment network, multiple low-level policy networks, and a temporal state control module. The high-level threat assessment network outputs the current threat level based on key battlefield state parameters and activates the corresponding low-level policy networks. The low-level policy networks output the decision actions of electronic warfare equipment. The temporal state control module runs in parallel with the low-level policy networks, calculating state change scores by capturing the temporal correlation characteristics of battlefield states, and determining whether to trigger new decisions accordingly. This achieves filtering of unnecessary decisions and rapid response under critical states, balancing decision-making real-time performance and resource consumption. Specifically, it includes:

[0082] S31. Construct a threat level hierarchical network architecture consisting of one high-level threat assessment network and three low-level policy networks:

[0083] High-level threat assessment network: Input key battlefield status parameters, output the current threat level and activate the corresponding low-level strategy network. The key status parameters include enemy missile distance and friendly jamming device status.

[0084] Low-level strategy network: Each network corresponds to a threat level and adopts a double hidden layer fully connected structure. It takes the current battlefield state vector as input and outputs four types of decision actions: jamming target selection, fire control radar target selection, jamming intensity adjustment, and fire control radar mode switching.

[0085] In some embodiments, the input to the high-level threat assessment network is key state parameters (including enemy missile distance and friendly jamming aircraft operational status). It adopts a single hidden-layer fully connected structure (the hidden layer contains 64 neurons), outputs the current threat level (low / medium / high) after passing through the Softmax activation function, and triggers the activation of the corresponding low-level policy network. Three low-level policy networks are set up, corresponding to low, medium, and high threat levels respectively. Each network adopts a double hidden-layer fully connected structure (each hidden layer contains 128 neurons). The input is the current battlefield state vector (integrating friendly equipment status, enemy equipment status, and environmental status data). After being mapped by the Tanh activation function, it outputs four types of decision actions: jamming target selection (such as reconnaissance aircraft / penetrating aircraft), fire control radar target selection (such as missiles / penetrating aircraft), jamming intensity adjustment (0 / low / medium / high), and fire control radar mode switching (active / silent).

[0086] S32. Trajectory data partitioned storage: Trajectory data generated by the interaction between the agent and the electronic warfare environment (formatted as "state-action-reward-next state") is stored in three independent experience replay buffers according to the threat level corresponding to the currently activated low-level policy network, to avoid cross-interference of data under different threat scenarios;

[0087] S33. Policy Network Update: The clipped objective function of the Proximal Policy Optimization (PPO) algorithm is used to update the lower-level policy networks. The objective function expression is as follows:

[0088] ;

[0089] in, For strategy ratio, This is the estimated value of the dominance function. The clip parameter is used for the high-level threat assessment network. Simultaneously, the network optimizes the joint loss function of threat level determination accuracy and battlefield mission completion rate to improve the accuracy of threat level identification.

[0090] S34. Construct a temporal state control module; specifically, construct a self-attention action control module: the self-attention action control module is deployed in parallel with the low-level policy network, adopting a combined structure of "self-attention layer + fully connected layer"; the input is a 5×10 dimension tensor composed of "the previous 4 steps of historical state sequence + current state", the tensor contains temporal correlation features such as blue team threat value, reconnaissance update rate, electronic interference situation, step counter, policy network switching flag, weather condition index, etc.; the correlation weight matrix between each state is calculated through the self-attention layer, first for any two states , Calculate the similarity score:

[0091] ;

[0092] in, W Q , W K These are the learnable queries and the key weight matrix (both with a dimension of 10×10). d k For key dimensions;

[0093] The weight matrix is ​​then normalized using the Softmax function, and then the weight matrix is ​​averaged along the feature dimension to obtain a 1-dimensional state influence vector. This vector is then input into a single hidden fully connected layer (the hidden layer contains 32 neurons, and the activation function is ReLU), and finally mapped to a state change score S.

[0094] S35. Decision Triggering Logic Configuration: Compare the state change score S with a preset threshold. t Compare and set a threshold. t The optimal value was determined to be 2.2 after simulation verification in typical electronic warfare scenarios; if S > t If a "critical state transition" is identified (such as the sudden appearance of a high-threat target, a sudden change in threat level, or a drastic change in the enemy's equipment combat status), the current decision execution is immediately suspended, triggering the corresponding lower-level policy network to generate a new decision; if S≤ t If the previous decision is maintained, unnecessary decision switching will be avoided, which will introduce noisy data and resource consumption.

[0095] In some embodiments, alternative network architectures are as follows:

[0096] (1) Alternative to the self-attention action control module: Instead of the self-attention mechanism, calculate the Euclidean distance (or Manhattan distance) between the historical state and the current state. If the distance is greater than a preset threshold, a new decision is triggered. This solution has low computational complexity (70% reduction in computation), but it cannot distinguish the “importance” of state changes and is prone to misjudging non-critical state transitions (misjudgment rate of about 15%-20%).

[0097] (2) Alternatives to hierarchical policy networks:

[0098] Dynamic strategy network quantity adjustment mechanism: The number of strategy networks is dynamically increased or decreased according to the distribution density of battlefield threat levels (such as the proportion of high-threat scenarios). (For example, low-threat strategy networks are deleted when the proportion of low-threat scenarios is <10%). The computational resource consumption is optimized through network pruning technology (memory consumption is reduced by 15%-25%), which is suitable for small electronic warfare equipment with limited resources.

[0099] Multi-task learning (MTL) integrated policy network: The decision-making tasks of three threat levels are integrated into a single multi-task policy network. The independent low-level policy network is replaced by a shared feature extraction layer and a task-specific output layer. This approach reduces the number of network parameters (by 36%), but there is inter-task interference (e.g., the accuracy of high-threat tasks affects the accuracy of low-threat tasks by about 3%-5%).

[0100] S4. Conduct network training and iterative optimization. This involves initializing network parameters, managing trajectory data by threat level partitioning, updating the low-level policy network using the PPO algorithm, and optimizing the high-level threat assessment network based on the joint loss function. During training, the exploration rate is dynamically adjusted to adapt to different stage requirements. Simultaneously, the training convergence status is determined using task completion rate, threat level determination accuracy, and decision latency as core thresholds, continuously improving the network's decision-making performance and practical adaptability. Specifically, this includes:

[0101] S41. Design a multi-dimensional reward function (to ensure training guidance): Construct a reward system that includes positive rewards and negative penalties, with the total reward being the sum of all components:

[0102] ;

[0103] In the formula: R 1 is a reward for locking onto the fire control radar; R 2 is a reward for the effectiveness of interference; R 3 is the reward for destroying enemy equipment; R 4 indicates a penalty for protecting one's own defensive equipment; R 5 represents the task completion reward; the expression and reward mechanism are as follows: When the distance between the fire control radar and the target is less than its working range, When the distance between the jammer and the target is less than its operating range, A fixed reward is given for destroying enemy attack equipment. A penalty is imposed when friendly defensive equipment is destroyed. A global reward is given for successfully intercepting or preventing the launch of an enemy missile. ;

[0104] In the above formula: R radarFor radar operating range, D target For the target distance, T target Threat level of the target N For the number of consecutive locks, α , β , These are the weighting coefficients. R jammer The operating range of the jammer. The penalty coefficient is... High reward coefficient;

[0105] S42. Initialize training parameters: The learnable parameters of the high-level threat assessment network, the low-level policy network, and the self-attention action control module are randomly initialized using the Xavier method;

[0106] In some embodiments, the training configuration includes: setting the total number of iterations to 5000 rounds, with each round containing 100 training steps, an experience replay buffer capacity of 100,000 records, and using a first-in-first-out (FIFO) mechanism to manage data;

[0107] S43. Iterative Training and Network Update: The agent interacts with the electronic warfare environment to generate trajectory data in the format of "state-action-reward-next state". Based on the threat level corresponding to the currently activated low-level policy network, the trajectory data is stored in multiple independent experience replay buffers, and the buffer data is updated using a first-in, first-out (FIFO) mechanism. Trajectory data is randomly sampled from each experience replay buffer, and the clipped objective function of the PPO algorithm is used to update the parameters of the low-level policy network. Simultaneously, a joint loss function is constructed based on the accuracy of threat level determination and the battlefield mission completion rate to optimize the parameters of the high-level threat assessment network; details are as follows:

[0108] Trajectory data acquisition: The intelligent agent interacts with the electronic warfare environment to generate "state-action-reward-next state" trajectory data, which is stored in the corresponding experience replay buffer according to the threat level;

[0109] Low-level policy network update: Randomly sample several trajectory data (e.g., 32 trajectories) from the buffer, and optimize the parameters using the clipped objective function of the PPO algorithm;

[0110] High-level threat assessment network optimization: Adjust parameters using a joint loss function of "threat level determination accuracy + task completion rate";

[0111] S44. Dynamically adjust the training strategy: Adjust the exploration rate in stages according to the training process. Increase the exploration rate in the early stage of training to enrich the diversity of trajectory data, reduce the exploration rate in the middle stage of training to strengthen the learning of the optimal strategy, and further reduce the exploration rate in the later stage of training to optimize the stability of network decision-making.

[0112] In some embodiments, the exploration rate is adapted according to the training stage: the exploration rate is set to 0.3 in the early stage of training (first 1000 rounds) to encourage the agent to try different actions and enrich the diversity of data; the exploration rate is reduced to 0.1 in the middle stage of training (1001-3000 rounds) to strengthen the learning of the optimal strategy; and the exploration rate is adjusted to 0.05 in the later stage of training (3001-5000 rounds) to fine-tune parameters and improve decision stability.

[0113] S45. Training Convergence Determination: Set a convergence index system that includes battlefield mission completion rate, threat level determination accuracy, and decision response delay. When all convergence indices are met for multiple consecutive training rounds, training is determined to be converged and training is terminated. If the convergence conditions are not met, the hyperparameters are adjusted and the training process is restarted.

[0114] In some embodiments: when the "battlefield mission completion rate ≥ 90%, threat level determination accuracy ≥ 95%, and decision response delay ≤ 0.5s" conditions are met for 50 consecutive rounds, training is considered to have converged and training is terminated; if the convergence conditions are not met, the hyperparameters (such as learning rate and buffer capacity) are adjusted and training is retrained.

[0115] The key technical points of this invention are: (1) Seven types of threat factors are selected, including distance, heading angle, speed, damage capability, electronic jamming capability, combat status and environmental factors. A judgment matrix is ​​constructed and its weights are calculated. The total threat value of the battlefield is obtained by accumulating the threat values ​​of each target and is divided into three threat levels: low, medium and high. (2) The threat level hierarchical network includes a high-level threat assessment network and three low-level strategy networks. The high-level network activates the corresponding low-level network according to the total threat value of the battlefield. Each low-level network adopts a structure containing two hidden layers (128 neurons per layer) and outputs electronic warfare decision actions such as jamming target selection and radar mode switching. Each network has the ability to independently store and update experience data. (3) The 5×10-dimensional tensor composed of five-step historical state and current state is used as input. The state correlation weight is calculated by the self-attention layer and mapped to the state change score by the fully connected layer. It is compared with the preset threshold to determine whether a new decision is triggered. The ten-dimensional state parameters include key indicators such as blue force threat value, reconnaissance update rate and electronic warfare engagement rate.

[0116] Compared with the prior art, the advantages of the present invention are:

[0117] (I) Comparison with the standard PPO algorithm: In terms of sparse reward resolution, this invention, through a threat level hierarchical design, decomposes the globally sparse rewards in electronic warfare scenarios into local rewards under low, medium, and high threat levels, significantly improving the utilization rate of reward signals and increasing the training convergence speed by more than 11.23%. In contrast, the standard PPO algorithm uses a single-policy network, which cannot effectively decompose sparse rewards. During training, it is prone to stagnation due to a lack of effective reward feedback for a long time, resulting in a convergence speed far lower than that of this invention. In terms of decision frequency control, this invention relies on a self-attention action control module, which can accurately identify key decision nodes on the battlefield and trigger new decisions only when the state undergoes significant changes (such as the approach of high-threat missiles), reducing the decision frequency by 79.4% and controlling the single-step decision delay to within 40ms, fully meeting the real-time requirements of electronic warfare. The standard PPO algorithm lacks a decision frequency control mechanism, generating decision instructions for each simulation step, which not only introduces a large amount of redundant noise that interferes with training stability but also leads to excessive computational latency, making it unsuitable for the rapid response requirements of electronic warfare. Regarding mission completion rate, this invention achieves an electronic warfare mission completion rate of over 72% after 1000 training rounds through threat level-based targeted decision-making and precise decision frequency control. The standard PPO algorithm, due to its high decision randomness and poor strategy adaptability, consistently achieves a mission completion rate below 30%, making it difficult to meet actual combat requirements.

[0118] (II) Comparison with PPO-RNN algorithm:

[0119] In identifying critical decision nodes, the self-attention action control module of this invention can distinguish the "importance" of state changes and trigger new decisions only when critical state transitions occur on the battlefield (such as enemy electronic warfare aircraft starting jamming or friendly radar being locked), effectively avoiding redundant actions. The PPO-RNN algorithm only captures the temporal continuity of the state sequence through RNN and cannot identify the importance of state changes. It will still generate invalid decisions in non-critical steps, and the proportion of redundant actions is more than 55% higher than that of this invention.

[0120] Regarding threat scenario adaptability, this invention configures dedicated policy networks for low, medium, and high threat levels, with each network trained specifically for its corresponding scenario, maintaining high decision-making accuracy across different threat scenarios. In contrast, the PPO-RNN algorithm uses a single policy network to address all threat scenarios, requiring the network to adapt to the decision-making needs of multiple scenarios simultaneously. This results in significant fluctuations in decision-making accuracy under differentiated scenarios such as high and low threats, making its adaptability far lower than that of this invention.

[0121] In terms of task completion rate, the present invention achieves a completion rate of over 72%, while the PPO-RNN algorithm, due to redundant decision-making and insufficient scenario adaptability, achieves a completion rate of only about 50%. Furthermore, in complex multi-threat coupled scenarios, the performance gap between the two further widens.

[0122] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An intelligent equipment decision-making method based on threat level classification, characterized in that: Specifically, the steps include the following: S1. Construct a decision-making state space in an electronic warfare scenario, including the state of friendly defensive equipment, the state of enemy attack equipment, and the environmental state; S2. Select multiple threat assessment factors, determine the weight of each factor through a preset weight calculation method, quantify the threat value of a single enemy attack equipment and the total battlefield threat value based on the weight, and divide the battlefield threat level into at least two levels according to a preset threshold. S3. Construct a threat level hierarchical strategy-self-attention collaborative decision-making network architecture, including a high-level threat assessment network, multiple low-level policy networks, and a time-series state control module; The high-level threat assessment network outputs the current threat level based on the total battlefield threat value and key status parameters, and activates the corresponding low-level strategy network. The low-level strategy network outputs the decision actions of electronic warfare equipment. The timing status control module runs in parallel with the low-level strategy network, and calculates the status change score by capturing the timing correlation characteristics of the battlefield status to determine whether to trigger a new decision. The timing state control module is a self-attention action control module or a simple timing control module based on threshold judgment; S4. Perform network training and iterative optimization by initializing network parameters, partitioning and managing trajectory data, updating the low-level policy network with a preset algorithm, and optimizing the high-level threat assessment network. Dynamically adjust the training exploration rate and set convergence conditions to improve network decision-making performance. The preset algorithm is the PPO algorithm, which uses a clipped objective function to update the low-level policy network.

2. The equipment intelligent decision-making method based on threat level classification according to claim 1, characterized in that: In step S1, the friendly defense equipment includes early warning radar, electronic jammer, fire control radar, and core targets that need to be protected; the enemy attack equipment includes penetration aircraft, electronic warfare aircraft, reconnaissance aircraft, and various missiles. The status of the friendly defense equipment includes the location coordinates, operating parameters, and operating status of the defense equipment. The operating parameters include the radar operating frequency band and the jammer output power. The operating status includes standby, running, and fault. The status of the enemy attack equipment includes the position coordinates, motion parameters, damage capability level, and electronic jamming capability parameters of the attack equipment; the motion parameters include speed, heading, and altitude; the environmental status includes weather parameters, including visibility, rainfall intensity, and electromagnetic interference intensity.

3. The equipment intelligent decision-making method based on threat level classification according to claim 1, characterized in that: In step S2, the threat assessment factors include distance, heading angle, target speed, target damage capability, target electronic jamming capability, target combat status, and environmental factors; the preset weight calculation method includes the analytic hierarchy process (AHP), which calculates weights by constructing a judgment matrix, performing consistency checks, and using the eigenvector method.

4. The equipment intelligent decision-making method based on threat level classification according to claim 3, characterized in that: The battlefield threat levels include low threat, medium threat, and high threat, and the threshold for these levels is determined through electronic warfare scenario simulation and combat experience. When quantifying the threat value of a single enemy attack device, the min-max normalization method is used to map the original data of each threat assessment factor to the [0,1] interval, and the threat value of a single enemy attack device is calculated using a weighted summation formula: ； In the formula: This represents the threat value of the i-th enemy attack equipment, with a value range of [0,1]. Indicates the weight of the k-th type of factor; This represents the normalized value of the i-th blue team equipment in the k-th factor category; The formula for the total battlefield threat value is as follows: ； In the formula, This represents the threat value of the i-th enemy attack equipment, with a value range of [0,1]. n This indicates the number of enemy attack equipment.

5. The equipment intelligent decision-making method based on threat level classification according to claim 1, characterized in that: The battlefield threat levels include low threat, medium threat, and high threat, and the threshold for these levels is determined through electronic warfare scenario simulation and combat experience. In step S3, the high-level threat assessment network adopts a single hidden layer fully connected structure and outputs the threat level through the Softmax activation function; the hidden layer contains 64 neurons. The low-level policy network is set up with three layers, corresponding to low threat, medium threat and high threat levels respectively. Each network has 128 neurons in its hidden layer. After being mapped by the Tanh activation function, it outputs four types of decision actions: jamming target selection, fire control radar target selection, jamming intensity adjustment and fire control radar mode switching.

6. The equipment intelligent decision-making method based on threat level classification according to claim 1, characterized in that: The self-attention action control module adopts a combination structure of self-attention layer and fully connected layer. The input is a tensor composed of 5-step historical state sequence and current state. The self-attention layer calculates the similarity score between states. After normalization, feature dimension averaging and mapping by the fully connected layer, the output is a scalar-type state change score S as a state change index. The simple timing control module based on threshold judgment calculates the Euclidean distance or Manhattan distance between the five historical states and the current state, and uses the distance as an indicator of state change.

7. The equipment intelligent decision-making method based on threat level classification according to claim 6, characterized in that: When the self-attention action control module is used, the similarity score is calculated using the following formula: ； in, , Represents any two states; W Q , W K These are the learnable queries and the key weight matrix, respectively. d k For key dimensions.

8. The equipment intelligent decision-making method based on threat level classification according to claim 6, characterized in that: The state change score S is compared with a preset threshold τ. If S>τ, a new decision is triggered; if S≤τ, the previous decision is maintained. The preset threshold τ=2.

2.

9. The equipment intelligent decision-making method based on threat level classification according to claim 1, characterized in that: The objective function expression for clipped is: ； in, For strategy ratio, This is the estimated value of the dominance function. For the clip parameter.

10. The equipment intelligent decision-making method based on threat level classification according to claim 9, characterized in that: In step S3, the timing state control module is a self-attention action control module; Step S4 specifically includes the following sub-steps: S41. Design a multi-dimensional reward function: Construct a reward system that includes positive rewards and negative penalties, with the total reward being the sum of all components: ； In the formula: R 1 is a reward for locking onto the fire control radar; R 2 is a reward for the effectiveness of interference; R 3 is the reward for destroying enemy equipment; R 4 indicates a penalty for protecting one's own defensive equipment; R 5 represents the task completion reward; the expression and reward mechanism are as follows: When the distance between the fire control radar and the target is less than its working range, When the distance between the jammer and the target is less than its operating range, A fixed reward is given for destroying enemy attack equipment. A penalty is imposed when friendly defensive equipment is destroyed. A global reward is given for successfully intercepting or preventing the launch of an enemy missile. ; In the above formula: R radar For radar operating range, D target For the target distance, T target Threat level of the target N For the number of consecutive locks, α , β , These are the weighting coefficients. R jammer The operating range of the jammer. The penalty coefficient is... High reward coefficient; S42. Initialize training parameters: The learnable parameters of the high-level threat assessment network, the low-level policy network, and the self-attention action control module are randomly initialized using the Xavier method; S43. Iterative Training and Network Update: The agent interacts with the electronic warfare environment to generate "state-action-reward-next state" trajectory data, which is stored in an independent experience replay buffer according to threat level and updated using a first-in-first-out mechanism; the trajectory data is randomly sampled, and the low-level policy network is updated through the PPO algorithm clipped objective function. At the same time, a joint loss function is constructed based on the accuracy of threat level determination and the battlefield mission completion rate to optimize the parameters of the high-level threat assessment network. S44. Dynamically adjust training strategy: Adjust the exploration rate in stages according to the training progress; S45. Training Convergence Judgment: Set a convergence index system including battlefield mission completion rate, threat level judgment accuracy, and decision response delay. Training is terminated when all convergence indices are met for multiple consecutive rounds of training; otherwise, the hyperparameters are adjusted and training is restarted.

Images