A network attack active defense strategy optimization method based on deep reinforcement learning

Abstract

The application discloses a network attack active defense strategy optimization method based on deep reinforcement learning, which comprises the following steps: collecting multi-source data of network environment, performing feature clipping and white list feature reservation; performing normalization and coding processing to generate a security posture vector; constructing a multi-index reward function to generate an instant reward value and an event-level reward value; executing a double closed-loop mechanism through an improved PPO model to respectively output an instant strategy instruction and a long-term strategy parameter; performing multi-source evidence deliberation on the instant strategy instruction and the security posture vector, judging a key evidence loss condition, and obtaining an execution token; inputting a risk budget pool to perform resource quota checking, anti-jittering and cooling control; and optimizing parameters of the multi-index reward function through a causal account book. The application can realize rapid response and continuous optimization of various attack behaviors, balance defense effect and resource utilization rate, reduce false alarm and missed alarm rates, and improve the adaptability and stability of the network defense system.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to an optimization method for proactive network attack defense strategies based on deep reinforcement learning. Background Technology

[0002] With the increasing complexity of cyberspace threats and the diversification of attack methods, traditional passive defense and rule-driven intrusion detection methods have significant limitations in the face of high-frequency, cross-stage, and hybrid attacks. Existing technologies typically rely on static feature matching or defense decisions based on single indicators. These methods are prone to both false positives and false negatives when dealing with zero-day attacks, stealthy penetration, and multi-hop attack paths, resulting in a lack of targeted and dynamic adaptability in defense strategy execution. Furthermore, multi-source data suffers from format heterogeneity, timestamp misalignment, and feature redundancy during collection. Existing data preprocessing and fusion methods have limited effectiveness in ensuring temporal consistency and semantic integrity, easily causing distortion of situational awareness vectors, thus affecting the accuracy and stability of strategy generation. For reward signal construction, traditional methods often use single-objective or static weights for evaluation, making it difficult to balance the contradiction between short-term threat handling and long-term system stability. Moreover, the lack of causal correlation analysis during weight adjustment leads to slow optimization convergence and a tendency to get trapped in local optima. On the other hand, existing strategies update mechanisms often lack the ability to coordinate immediate response and long-term optimization. When the network situation changes rapidly, the model's adjustment of strategy distribution lags behind, failing to balance the timeliness of defensive actions with the efficiency of resource utilization.

[0003] Therefore, how to provide an optimization method for proactive defense strategies against network attacks based on deep reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose a network attack proactive defense strategy optimization method based on deep reinforcement learning. This invention fully utilizes multi-source security data fusion, dynamic multi-index reward function construction, and an improved PPO model. It details how to generate multi-source datasets in complex network environments through feature pruning and whitelist feature retention, construct security posture vectors using Z-Score standardization and one-hot encoding, and execute strategy generation and parameter updates based on a dual closed-loop mechanism of immediate reward values ​​and event-level reward values. Simultaneously, it combines multi-source evidence consensus to generate action credibility scores and performs risk budgeting and cooling-off control. Finally, it optimizes the parameters of the multi-index reward function through a causal ledger, possessing advantages such as high attack interception rate, low false positive rate, strong resource scheduling controllability, and high strategy convergence stability.

[0005] An optimization method for proactive network attack defense strategy based on deep reinforcement learning according to an embodiment of the present invention includes the following steps:

[0006] Step 1: Collect multi-source data from the network environment, perform feature cropping and whitelist feature retention to form a multi-source dataset;

[0007] Step 2: Normalize and encode the multi-source dataset to generate a security posture vector;

[0008] Step 3: Construct a multi-index reward function based on the security situation vector to generate immediate reward values ​​and event-level reward values;

[0009] Step 4: Based on the immediate reward value and the event-level reward value, execute a dual closed-loop mechanism through the improved PPO model; the dual closed-loop mechanism includes a rapid processing loop and a strategy optimization loop, which respectively output immediate policy instructions and long-term policy parameters;

[0010] Step 5: Perform multi-source evidence deliberation on the real-time policy instruction and the security situation vector, generate an action credibility score, determine the loss of key evidence, and obtain an execution token;

[0011] Step Six: Input the execution token into the risk budget pool for resource quota verification, perform anti-jitter and cooling control, and output the action execution command;

[0012] Step 7: Optimize the parameters of the multi-indicator reward function through causal ledger.

[0013] Optionally, step one specifically includes:

[0014] Collect multi-source data covering host systems, network boundaries, and service traffic. The multi-source data includes communication metadata, log recording data, session behavior data, and alarm feedback data.

[0015] The multi-source data is subjected to feature trimming to remove feature fields that are weakly correlated with the attack or highly redundant, while retaining the core feature set.

[0016] Perform a whitelist feature retention operation on the feature-trimmed data. By comparing with a predefined whitelist rule set, filter feature instances that conform to the trusted behavior pattern and generate tag labels.

[0017] The tags and retained features are combined to form structured data items, which are then organized in a unified format according to time sequence and source modules to form a multi-source dataset. The retained feature combination includes attack fields filtered through pruning operations and trusted fields that conform to the whitelist rule set definition.

[0018] Optionally, step two specifically includes:

[0019] The data of various types in the multi-source dataset are segmented according to a unified time window, and the numerical features and categorical features are normalized and encoded respectively using the Z-Score standardization and one-hot encoding method to construct standardized feature vectors.

[0020] Based on the standardized feature vectors, a multi-dimensional security situation representation model describing the dynamic changes in the network environment is constructed. Nested mapping operations are performed on the normalized data from each source module to generate a set of structured vectors.

[0021] The structured vector set is input into the feature aggregation module, and cross-modal fusion and temporal arrangement operations are performed to generate a security posture vector representing the overall network security status.

[0022] Optionally, step three specifically includes:

[0023] Based on the security situation vector, network status feature indicators are extracted to construct an indicator vector set; the indicator vector set includes resource utilization indicators, alarm response delay indicators, attack interception rate indicators, false alarm rate indicators, and handling action coverage indicators.

[0024] A multi-indicator reward function is constructed based on the set of indicator vectors. The multi-indicator reward function includes an immediate reward sub-function and an event-level reward sub-function, which are used to quantify the impact of strategy actions on short-term handling effects and long-term system stability, respectively.

[0025] In the instant reward sub-function, the attack interception rate improvement and false alarm rate suppression magnitude are introduced as the main feedback items. A first set of weighting coefficients is set, and the weighted summation of each feedback item is performed to output the instant reward value.

[0026] In the event-level reward sub-function, the system resource utilization balance, the average response delay, and the coverage of the handling action are introduced as long-term evaluation items. A second set of weighting coefficients is set, and the evaluation items are normalized and aggregated to output the event-level reward value.

[0027] Optionally, the multi-metric reward function includes an immediate reward layer and an event-level reward layer, specifically:

[0028] The instant reward layer measures the balance between the suppression effect and resource cost of a single defense action in the threat handling process; based on the attack interception effect, false alarm cost, resource consumption and business impact, an instant feedback item set is constructed, a first weighting coefficient set is set, a weighted accumulation operation is performed on each feedback item, and the instant reward value is output.

[0029] The event-level reward layer measures the cumulative defense performance of continuous defense actions within the life cycle of an attack event; a long-term evaluation item set is constructed based on response latency, action jitter frequency, evidence gain and compliance conflict, a second weighting coefficient set is set, a normalization aggregation operation is performed on each evaluation item, and the event-level reward value is output.

[0030] The weights of each indicator in the multi-indicator reward function are dynamically updated based on the attribution analysis results of defense actions and defense effects constructed from the causal ledger and the state changes of the security situation vector. The smoothing function is used to suppress short-term fluctuations of the weighting coefficients. The smoothing function is any one of the following: exponential smoothing function, weighted moving average function, or Gaussian smoothing function.

[0031] Optionally, step four specifically includes:

[0032] A dual closed-loop mechanism is constructed based on instant reward values ​​and event-level reward values ​​to execute the process of generating policy instructions and updating policy parameters.

[0033] By using the rapid response loop to match the real-time policy trajectory in the policy model with the security situation vector and the real-time reward value, the optimal action sequence is selected and real-time policy instructions are generated.

[0034] By using the strategy optimization loop, the deviation between the target strategy distribution and the strategy distribution is calculated based on the event-level reward value and historical action feedback records, and the pruning target deviation is obtained.

[0035] Evaluate the state value difference between action choices and environmental responses, and obtain the state value residuals;

[0036] Calculate the information entropy of the policy model output distribution to obtain the policy distribution entropy;

[0037] Construct a joint loss function that weights and combines the pruning target bias, state value residual, and policy distribution entropy;

[0038] Perform the backpropagation process of the joint loss function to update the weight parameters in the policy model and generate long-term policy parameters.

[0039] Optionally, the improved PPO model is specifically as follows:

[0040] Construct a model structure that includes a state encoder, an action sampler, and a policy updater;

[0041] The state encoder receives a security situation vector, performs multi-level nested mapping operations, and outputs a state representation tensor.

[0042] The action sampler generates a set of candidate actions based on the state representation tensor and the policy probability distribution, and selects the optimal action based on the immediate reward value to form an action sequence;

[0043] The policy updater receives event-level reward values, historical action feedback records, and action sequences to construct the target policy distribution.

[0044] Calculate the KL divergence between the target policy distribution and the policy distribution corresponding to the action sequence to obtain the pruning target bias;

[0045] The residual between the environmental state response value corresponding to the execution result of the action sequence and the expected state value is evaluated to obtain the state value residual.

[0046] The information entropy is calculated by performing an operation on the probability distribution of the action sequence in the policy model to obtain the policy distribution entropy.

[0047] Construct a joint loss function that is a weighted combination of pruning target bias, state value residual, and policy distribution entropy;

[0048] Perform the backpropagation process of the joint loss function, update the weight parameters of each module in the policy model, and output the long-term policy parameters.

[0049] Optionally, step five specifically includes:

[0050] Receive real-time policy instructions and security posture vectors, and construct policy action requests;

[0051] Retrieve multi-source evidence information associated with the policy action request; the multi-source evidence information includes communication log information, behavioral feature records, interception feedback data, attack path structure and environmental context parameters;

[0052] A multi-dimensional evidence support matrix is ​​constructed based on multi-source evidence information, and a support degree vector covering each type of evidence is extracted.

[0053] The support level vector is input into the credibility aggregation module, and weighted summation and bias correction operations are performed according to the preset weight set to generate an action credibility score.

[0054] Based on the evidence support matrix and the support level threshold, it is determined whether there are missing, ambiguous or conflicting states in the key evidence dimensions on which the immediate strategy command depends. If they exist, they are marked as key evidence loss states; otherwise, they are marked as key evidence complete states.

[0055] A joint judgment is made based on the action credibility score and the status of key evidence loss to generate an execution token; if the action credibility score is greater than the preset credibility threshold and the key evidence is intact, a valid execution token is generated; otherwise, a risk warning token is generated.

[0056] Optionally, step six: inputting the execution token into the risk budget pool for resource quota verification, performing anti-jitter and cooling control, and outputting action execution instructions, specifically:

[0057] The execution token is entered into the risk budget pool, and resource status data and historical policy usage records are used to build a resource quota assessment model.

[0058] Based on the strategy action level, action credibility score and risk warning type contained in the execution token, the corresponding resource budget limit and resource tension coefficient are extracted from the resource quota assessment model to construct resource quota constraint items;

[0059] The resource quota constraints are matched and screened with the execution token. If they meet the resource allocation conditions, they are marked as valid execution tokens; otherwise, they are marked as resource rejection tokens.

[0060] For a valid execution token, the action frequency control program is invoked to read the action frequency information recorded in the policy action type and security situation vector, and to determine whether there is an oscillation trend in the action execution frequency.

[0061] If there is a trend of action oscillation, the cooldown time management program is invoked, a cooldown interval threshold is set, and the time interval between the same action command and the previous one is calculated. If the time interval is less than the cooldown interval threshold, the execution time marker is delayed; otherwise, the execution status is updated.

[0062] Integrate resource quota screening results, motion oscillation control results, and cooldown time management results to generate motion execution instructions;

[0063] The action execution instruction includes an action identifier, an action execution timestamp, strategy path source information, resource scheduling label, and action feedback channel identifier.

[0064] Optionally, step seven specifically includes:

[0065] Record the causal path between action execution instructions and environmental responses during strategy execution, and construct a causal ledger;

[0066] Statistically analyze the causal relationship information between each strategy action and the attack mitigation effect in the causal ledger, and extract the corresponding reward bias;

[0067] Three types of target factors are set: risk indicators, response latency indicators, and resource consumption indicators, and initial weight coefficients are assigned to each of them.

[0068] Construct a multi-indicator reward function, and generate a candidate reward function parameter set by weighting the return deviation and the target factor weight coefficient.

[0069] Perform parameter partitioning on the candidate reward function parameter group, evaluate the fitting error between the policy behavior and the target factor under different parameter combinations, and form a reward function parameter mapping relationship;

[0070] The parameter set with the smallest fitting error is selected as the final reward function parameters, and the parameters of the multi-index reward function in the strategy model are updated.

[0071] The beneficial effects of this invention are:

[0072] This invention addresses the problems of heterogeneous multi-source data formats, misaligned timestamps, high feature redundancy, and policy response lag in existing proactive defense against network attacks by synergistically applying a PPO model constructed and improved through multi-source data fusion and dynamic multi-index reward function. It employs feature pruning and whitelist feature retention to achieve high-precision screening of attack-related and trusted fields. In the data preprocessing stage, it constructs a cross-modal consistent security posture vector through unified time window segmentation, Z-Score standardization, and one-hot encoding. In the reward construction stage, it introduces multi-dimensional indicators such as attack interception rate, false alarm rate, response latency, and resource utilization balance to form instantaneous reward values ​​and event-level reward values ​​that take into account both short-term handling effects and long-term system stability. In the strategy execution phase, a dual closed-loop mechanism is constructed, comprising a rapid response loop and a strategy optimization loop. The rapid response loop matches the optimal action sequence based on immediate reward values, while the strategy optimization loop combines event-level reward values ​​and historical feedback to perform backpropagation of the joint loss function to update long-term strategy parameters. In the credibility assessment and resource management phase, action credibility scores are generated through multi-source evidence deliberation. Combined with the integrity judgment of key evidence and the verification of resource quotas in the risk budget pool, anti-jitter and cooling control are implemented to avoid system instability caused by resource conflicts or high-frequency actions. In the strategy optimization phase, a causal ledger is introduced to record the correlation path between actions and defense effects, dynamically updating the weight parameters of the multi-index reward function to achieve adaptive optimization of the strategy in complex network environments. Ultimately, a comprehensive defense effect is achieved, with a significant increase in attack interception rate, an effective reduction in false positive rate, enhanced controllability of resource scheduling, and improved strategy convergence stability. Attached Figure Description

[0073] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0074] Figure 1 This is a flowchart of a network attack proactive defense strategy optimization method based on deep reinforcement learning proposed in this invention;

[0075] Figure 2 This is a structural diagram of the improved PPO model proposed in this invention;

[0076] Figure 3 This is a diagram of the causal ledger feedback optimization structure proposed in this invention. Detailed Implementation

[0077] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0078] refer to Figure 1-3 A method for optimizing proactive defense strategies against network attacks based on deep reinforcement learning includes the following steps:

[0079] Step 1: By collecting multi-source data including network traffic logs, device communication status, intrusion detection alarms and system call trajectories, perform feature trimming operations to remove low-frequency redundant fields, while retaining whitelisted feature fields that are strongly correlated with attack behavior, and construct a set of data vectors with semantic labels to form a multi-source dataset;

[0080] Step 2: Perform normalization processing on the multi-source dataset. For continuous fields, use the Z-Score normalization method and for discrete fields, perform one-hot encoding transformation. After mapping the multimodal data to a unified embedding space, splice and fuse them to form a security situation vector representing the network state.

[0081] Step 3: Construct a multi-index reward function based on the security situation vector, select attack interception rate, response latency and resource consumption as target factors, assign initial weight coefficients to each, evaluate the execution consequences of the policy output actions, and calculate the immediate reward value and event-level reward value respectively.

[0082] Step 4: Based on the immediate reward value and the event-level reward value, execute the rapid response loop and the strategy optimization loop respectively. In the rapid response loop, match the security situation vector with the immediate policy trajectory recorded in the policy model, select the optimal action sequence, and output the immediate policy instruction. In the strategy optimization loop, construct the target policy distribution based on the event-level reward value and historical action feedback, calculate the state value residual, the pruning target bias, and the policy distribution entropy, construct the joint loss function, update the weight parameters of the policy model, and generate long-term policy parameters.

[0083] Step 5: Input the real-time policy instructions and security posture vector into the credibility assessment mechanism. Based on the similarity, consistency and redundancy among multi-source evidence, generate an action credibility score. Combine the completeness of evidence information to determine whether key evidence is missing. Output an execution token containing a unique execution number, credibility label and key loss marker.

[0084] Step 6: Input the execution token into the risk budget pool, determine whether the resource request of the action exceeds the preset risk budget threshold. If it does not exceed the limit, apply anti-jitter adjustment and cooldown period constraints to the action execution instruction to ensure the safety of resource usage, and output the final action execution instruction.

[0085] Step 7: During long-term training, record the policy actions and environmental state transition sequences, construct a causal ledger, extract the contribution of target factors in the multi-index reward function through correlation analysis, reconstruct the reward function parameters, dynamically adjust the weight coefficients of each target factor, and improve the behavioral sensitivity of reward feedback and the stability of policy convergence.

[0086] In this embodiment, step one specifically includes:

[0087] Collect multi-source data covering host systems, network boundaries, and service traffic. The multi-source data includes communication metadata, log recording data, session behavior data, and alarm feedback data.

[0088] A unified acquisition format is constructed for the multi-source data, and the data from each source is timestamped, data type is normalized and encoded through a standardized interface to generate a unified structured raw data block;

[0089] The structured raw data block is subjected to feature trimming processing to construct a feature evaluation matrix. The features of each dimension of communication metadata, log record data, session behavior data and alarm feedback data are separated and encoded, and the discriminative index of each feature field is calculated.

[0090] Discriminant metrics include three metrics: information gain ratio, feature variance, and feature correlation coefficient. The discriminant evaluation value of each feature is obtained by fitting the information gain ratio and feature variance, and the feature redundancy evaluation value is obtained by fitting the feature correlation coefficient.

[0091] Set a minimum discriminative threshold and a maximum redundancy threshold, and prune feature fields whose evaluation value is lower than the discriminative threshold or whose redundancy is higher than the redundancy threshold, removing redundant feature fields with high feature value missing rate, strong correlation with other fields, or no independent contribution to attack classification.

[0092] Retain the core feature fields that meet the discriminative index requirements, have low information redundancy, and have discriminative contribution, and output the set of data features as input to the deep reinforcement learning model.

[0093] Perform a whitelist feature retention operation on the cropped data feature set, call the preset feature whitelist rule set, filter feature fields that conform to the trusted behavior pattern, and generate trustworthy label tags;

[0094] The feature whitelist rule set is built on existing network security defense strategies and experience knowledge base, and includes key feature fields such as system asset status, access control policies, historical attack path identifiers, highly sensitive operation tags and key node topology and their legal value ranges.

[0095] Perform semantic matching and value alignment on the retained cropped feature fields and the feature whitelist set. Perform a mark retention operation on feature fields that match exactly or have the same semantics, and output a whitelist set of trusted fields.

[0096] Remove low-sensitivity fields or suspicious fields without traceability that are not in the whitelist set;

[0097] The attack-related fields retained after trimming are combined with the whitelist trusted fields to form structured feature items;

[0098] By combining credibility markers, feature field source types, and collection time order, a unified data format is generated. Structured feature items are categorized and integrated according to source modules, and the output is a multi-source dataset.

[0099] In this embodiment, step two specifically includes:

[0100] The communication metadata, log data, session behavior data and alarm feedback data in the multi-source dataset are aligned and segmented according to a unified time window. A standardized set of time segments is constructed by performing time contour mapping and sliding window segmentation operations on the original timestamps.

[0101] Z-Score standardization is performed on numerical feature fields within each time segment, and standardized values ​​are obtained by fitting the mean and standard deviation. One-hot encoding is performed on categorical feature fields, and multidimensional discrete vectors are obtained by fitting the category set mapping.

[0102] The standardized results are concatenated with the one-hot encoding results to generate a basic standardized feature vector;

[0103] The standardized feature vectors are grouped and mapped according to the data source modules to construct initial state sub-vectors describing the behavior patterns of each source module;

[0104] Introduce source label encoding in each initial state subvector to mark its corresponding data type and module number, thereby enhancing the semantic discriminability of subsequent fusion stages;

[0105] The initial state sub-vectors of each source module are input into the multidimensional nested mapping module, and feature compression and dimension normalization operations are performed. The structured nested vector set is obtained by fitting through the convolutional self-attention mechanism and the position embedding mechanism.

[0106] The structured nested vector set is input into the feature aggregation module to perform cross-modal feature fusion processing. During the fusion process, a modal attention mechanism is introduced to weight the feature contributions of each source module. The fusion weight distribution is obtained by fitting the modal correlation map.

[0107] Perform nested vector concatenation operations based on the fusion weight distribution to generate a cross-modal fusion representation of the complete structure;

[0108] The cross-modal fusion representation is temporally rearranged, and stacked and concatenated according to the time segment index to generate a serialized structure vector.

[0109] The serialized structure vector is input into the state alignment module, which performs dimension matching and redundant feature removal operations to output the final security posture vector representing the dynamic evolution of network security.

[0110] This implementation improves the consistency of multi-source datasets in both time and feature dimensions by using unified time window segmentation and multi-type feature preprocessing; it compresses feature distribution differences and enhances the model's stability in perceiving abnormal states by using Z-Score normalization and one-hot encoding; it enhances the model's ability to semantically distinguish between different source modules by introducing source labels and nested mapping mechanisms; it improves the accuracy of cross-modal information integration and avoids weak modal interference by constructing a fusion weight distribution through modal attention mechanisms; and it enhances the ability of the security posture vector to express the global security state by final serialization structure rearrangement and state alignment.

[0111] In this embodiment, step three specifically includes:

[0112] Based on the feature information of each source module contained in the security situation vector, structured state parameters reflecting the security status of the network environment are extracted, and an index vector set is constructed.

[0113] The set of indicator vectors includes resource utilization indicators, alarm response latency indicators, attack interception rate indicators, false positive rate indicators, and action coverage indicators. The resource utilization indicators include CPU utilization, memory utilization, and network bandwidth utilization. The alarm response latency indicator is obtained by fitting the time interval between the occurrence of a security event and the first response. The attack interception rate indicator and the false positive rate indicator are obtained by comparing and fitting historical attack labels with model output labels. The action coverage indicator is obtained by fitting the hit ratio of the executed action sequence in the life cycle of the attack event.

[0114] The set of indicator vectors is input into the reward construction module, and a multi-indicator reward function is constructed based on the reward feedback mechanism. The multi-indicator reward function includes an immediate reward sub-function and an event-level reward sub-function, which correspond to the short-term strategy evaluation channel and the long-term strategy adjustment channel, respectively.

[0115] In the instant reward sub-function, the attack interception rate improvement and the false alarm rate suppression magnitude are introduced as the main feedback terms. The attack interception rate improvement is obtained by fitting the difference with the historical average interception rate, and the false alarm rate suppression magnitude is obtained by fitting the current false alarm rate with the historical threshold error.

[0116] Set a first set of weighting coefficients, normalize the increase in attack interception rate and the reduction in false positive rate, and then perform weighted summation based on the weight vector to output the instant reward value.

[0117] In the event-level reward sub-function, system resource utilization balance, mean response latency, and coverage of handling actions are introduced as long-term evaluation items. The system resource utilization balance is obtained by fitting the variance of multi-dimensional resource utilization. The mean response latency is obtained by fitting the aggregated alarm response time within the time window. The coverage of handling actions is obtained by fitting the overlap ratio between attack path nodes and the scope of handling actions.

[0118] A second set of weighting coefficients is set, and after standardization processing and aggregation operation are performed on the long-term evaluation items, the event-level reward value is output.

[0119] The combination of immediate reward values ​​and event-level reward values ​​forms a complete reward feedback structure, providing feedback for strategy generation and strategy updates in the subsequent dual-loop mechanism.

[0120] This implementation method comprehensively covers the short-term handling effect and long-term security stability performance in the network environment by introducing a multi-dimensional index vector set. The immediate reward sub-function focuses on reflecting the immediate effect of the model output on attack interception capability and false alarm control, thereby improving rapid response capability. The event-level reward sub-function focuses on the rationality of long-term resource allocation, handling delay and policy distribution, effectively guiding the policy optimization process away from short-term bias. The multi-index reward function constructed through mechanisms such as standardization, normalization and difference comparison has robustness and adaptability, and can be adapted to security policy modeling scenarios in different network environments.

[0121] In this embodiment, the multi-metric reward function includes an immediate reward layer and an event-level reward layer, specifically:

[0122] The instant reward layer measures the balance between the suppression effect and resource cost of a single defense action in the threat handling process; based on the attack interception effect, false alarm cost, resource consumption and business impact, an instant feedback item set is constructed, a first weighting coefficient set is set, a weighted accumulation operation is performed on each feedback item, and the instant reward value is output.

[0123] The event-level reward layer measures the cumulative defense performance of continuous defense actions over the lifecycle of an attack event. A long-term evaluation item set is constructed based on response latency, action jitter frequency, evidence gain, and compliance conflict. A second weighting coefficient set is set, and a normalization aggregation operation is performed on each evaluation item to output the event-level reward value. The action jitter frequency is determined by the statistical results of action execution frequency output from the anti-jitter and cooling management stage. Evidence gain is jointly determined by the evidence credibility score and re-collection success rate output from the multi-source evidence deliberation stage. Compliance conflict is calculated by the matching deviation between the defense action execution process and the preset compliance restriction rules.

[0124] The weights of each indicator in the multi-indicator reward function are dynamically updated based on the attribution analysis results of defense actions and defense effects constructed from the causal ledger and the state changes of the security situation vector. The smoothing function is used to suppress short-term fluctuations of the weighting coefficients. The smoothing function is any one of the following: exponential smoothing function, weighted moving average function, or Gaussian smoothing function.

[0125] In this embodiment, step four specifically includes:

[0126] A dual closed-loop mechanism based on instant reward value and event-level reward value is constructed. The dual closed-loop mechanism includes two sub-modules: a rapid processing loop and a strategy optimization loop.

[0127] In the rapid response loop, the security situation vector and the instant reward value are received as inputs. The instant policy trajectory library in the policy model is called. The historical trajectory with the highest similarity to the security situation vector is selected by the policy trajectory matching module. The similarity is obtained by fitting the cosine distance of the feature space.

[0128] The action sequences contained in the matched historical trajectory are evaluated and scored, sorted according to the criterion of maximizing immediate reward value, and the action sequence with the highest score is output as the current action candidate set.

[0129] The action filtering module is invoked to filter actions in the candidate set that conflict with each other or have duplicate effects on the target, and a sequential execution queue is built based on the action dependencies to generate immediate policy instructions.

[0130] The policy model comprises an encoding module, an action generation module, a value assessment module, a trajectory memory module, and a parameter update module. It is constructed using a multi-layer Transformer architecture and has the capability to input multimodal state vectors and output action probability distributions and state value assessment results. The encoding module receives a security situation vector and performs a multi-head attention mechanism encoding operation to generate a context-enhanced representation. The action generation module predicts the action distribution of this representation and compares it with historical trajectories stored in the trajectory memory module, supporting trajectory retrieval and matching. The trajectory memory module stores high-frequency defense paths in a graph structure, enabling dynamically scalable action trajectory playback. The value assessment module outputs the action value in the current state, used to optimize the objective function during the training phase. The parameter update module performs weight adjustments based on the joint loss function to maintain stable convergence of the policy model.

[0131] In the strategy optimization loop, the event-level reward value and historical action feedback record are received, the environmental response distribution in the historical action is extracted, and the strategy distribution evaluation module is called to construct the target strategy distribution.

[0132] KL divergence calculation is performed on the target policy distribution and the current policy model output distribution to obtain the pruning target deviation between the policy distributions. The pruning target deviation is obtained by fitting the result by minimizing the difference in expected action probabilities.

[0133] Call the state value assessment module, input the policy execution path and environmental feedback trajectory, construct the state-action value function, obtain the expected contribution value of action selection to environmental response through the reward backtracking process, and compare it with the empirical value to fit the state value residual.

[0134] The entropy change is calculated based on the strategy output distribution, and the strategy distribution entropy is obtained through the entropy control module. The strategy distribution entropy reflects the degree of certainty and exploratory nature of strategy selection.

[0135] Construct a joint loss function, set weighting coefficients for each function, and then perform a linear weighted combination to form the final optimization objective. Use the joint loss function as the basis for backpropagation, pass it into the policy model weight update module, perform multiple rounds of gradient backpropagation and parameter correction operations, and generate long-term policy parameters.

[0136] The structure enables the strategy model to dynamically adjust learning weights between immediate response and long-term optimization, thereby improving the robustness and adaptability of defensive actions.

[0137] In this embodiment, the improved PPO model is specifically as follows:

[0138] A policy model structure is constructed that includes a state encoder, an action sampler, and a policy updater. The modules are connected by nested neural networks to form an end-to-end trainable structure with dynamic response and continuous optimization capabilities.

[0139] The state encoder receives an input feature matrix generated from a security situation vector, performs nested mapping operations based on multi-layer linear transformation and nonlinear activation functions, and generates a state representation tensor. The mapping process is obtained through a state feature fitting network.

[0140] Based on the state representation tensor, the action sampler uses a parameterized strategy distribution to generate an initial action distribution map and constructs a candidate action set based on multiple samplings. During the sampling process, a probability ratio control and temperature parameter adjustment mechanism are used to dynamically balance exploration and stability.

[0141] The instant feedback evaluation module is invoked to evaluate the instant reward value of each action in the candidate action set. The optimal action combination is selected according to the principle of maximizing the instant reward value, and the corresponding action sequence is output as a policy execution candidate.

[0142] The strategy updater receives historical action feedback records, the current action sequence, and event-level reward values. It constructs a target strategy distribution through the reward integration module. The target strategy distribution is generated by the strategy target fitter by combining action frequency, execution effect, and system stability indicators.

[0143] Calculate the KL divergence between the target policy distribution and the policy distribution corresponding to the action sequence. The KL divergence is obtained by fitting the mean policy probability offset and outputs the pruning target bias.

[0144] The state value assessment module is invoked to predict the system response state corresponding to the action sequence, and the deviation is fitted with the expected state value to obtain the state value residual. The deviation fitting is generated using the cumulative expected return difference modeling method.

[0145] Based on the probability distribution of action sequences in the current policy model, the information entropy value of the policy distribution is calculated. The policy distribution entropy is used to measure the determinism and exploration intensity of the policy. The entropy value calculation is completed by the policy entropy measurement module.

[0146] A joint loss function is constructed, which consists of three terms: pruning target bias, state value residual, and policy distribution entropy. Weighting coefficients are set for each term, and a linear weighted combination is performed to form the final loss target. The weighting coefficients are dynamically adjusted according to the model training phase.

[0147] The formula for the joint loss function is:

[0148] L total (θ)=w1L clip (θ)+w2L v (θ)-w3H(π θ );

[0149] Among them, L total (θ) represents the joint loss function; Lclip (θ) represents the cutting target deviation; L v (θ) represents the state value residual; H(π) θ ) represents the policy distribution entropy; w1, w2, and w3 correspond to the weighting coefficients of the three parts; θ represents the parameter vector of the policy model.

[0150] The formula for the cutting target deviation is:

[0151] L clip (θ)=E t [min(r t (θ)A t ,clip(r t (θ), 1-ε, 1+ε)A t )];

[0152]

[0153] Among them, L clip (θ) represents the cutting target deviation; r t (θ) represents the strategy probability ratio; A t Indicates the value of the dominant function; π θ (a t |s t ) indicates that the current policy is in state s t Choose action a t The probability of; This indicates that the old policy is in state s. t Choose action a t The probability; clip(·) represents a function truncated along the interval [1-ε, 1+ε]; ε represents the clipping threshold; E t [·] represents the expectation operator at the time step.

[0154] The formula for state value residual is:

[0155]

[0156] Among them, L v (θ) represents the state value residual; V θ (s t ) represents the value function estimate, indicating the policy model's response to state s. t Value prediction; The target reward can be obtained by weighting the immediate reward value and the event-level reward value; E t [·] represents the expectation operator at the time step.

[0157] The formula for strategy distribution entropy is:

[0158] H(π θ ) = E t[-∑ a π θ (a∣s t logπ θ (a∣s t )];

[0159] Wherein, H(π) θ ) represents the policy distribution entropy; π θ (a∣s t ) indicates that the policy model is in state s t The probability distribution of the next action 'a'; 'a' represents any defensive action in the action space; E t [·] represents the expectation operator at the time step.

[0160] The backpropagation process of the joint loss function is executed, and the loss target is input into the weight update module. Through gradient backpropagation and optimizer iteration, all weight parameters in the state encoder, action sampler and policy updater are updated, and finally the long-term policy parameters are generated.

[0161] While ensuring strategy convergence, it improves adaptability to complex security situations, supports the coordinated adjustment of defense strategies to achieve immediate response and long-term optimization under multi-scale timeliness, and significantly enhances the execution effect and robustness of proactive network attack defense strategies.

[0162] In this embodiment, step five specifically includes:

[0163] Receive the real-time policy instructions generated by the rapid response ring and the current security situation vector, and construct a policy action request. The policy action request includes the target node identifier, action type identifier and action context parameters of the action to be executed.

[0164] The multi-source data fusion module is invoked to retrieve communication log information, behavioral feature records, interception feedback data, attack path structure, and environmental context parameters corresponding to the policy action request, forming a multi-source evidence information set. The communication log information includes the source port, target address, and communication frequency; the behavioral feature records include process operations, system call sequences, and abnormal behavior scores; the interception feedback data includes the interception labels and confidence levels returned by the detection tool; the attack path structure includes the associated nodes and edge weights in the attack graph; and the environmental context parameters include system load, network congestion, and alarm level.

[0165] A multidimensional evidence support matrix is ​​constructed based on the multi-source evidence information set. Each row of the multidimensional evidence support matrix represents an evidence type, each column represents a strategy action attribute, and the elements in the matrix represent the degree of support of the corresponding evidence for the current action attribute. The degree of support is obtained by fitting a fuzzy matching degree model.

[0166] Extract support vectors covering each type of evidence. The support vectors represent the strength of support for the correctness of the strategy action by different types of evidence. Input the support vectors into the credibility aggregation module. The credibility aggregation module adopts a weighted summation and bias correction mechanism. By introducing a preset set of weights, different importance levels are assigned to the support of each type. After performing a linear combination operation, a confidence bias correction term is introduced for correction to generate an action credibility score.

[0167] In the evidence evaluation module, based on the constructed multidimensional evidence support matrix and the evidence support level threshold, the integrity of the key evidence dimensions on which the strategy action depends is judged. If the support level of a certain key evidence dimension is lower than the support level threshold or there are missing data, ambiguous values ​​or contradictory information, the action is marked as a key evidence loss state; otherwise, it is marked as a key evidence integrity state.

[0168] The action credibility score and the key evidence integrity status are input into the token generation module to build a joint judgment mechanism. The joint judgment mechanism sets a credibility threshold as the basis for execution. If the action credibility score is higher than the credibility threshold and the key evidence integrity status is established, a valid execution token is generated. If the action credibility score is lower than the credibility threshold or the key evidence loss status is established, a risk warning token is generated. This effectively avoids the risk of erroneous decision-making or information misleading in action execution and improves the reliability and controllability of strategy action triggering.

[0169] In this embodiment, step six: inputting the execution token into the risk budget pool for resource quota verification, performing anti-jitter and cooling control, and outputting action execution instructions, specifically includes:

[0170] The execution token is input into the risk budget pool, and the resource status database and the strategy behavior history are called to build a resource quota evaluation model. The resource status database includes the current available resources of the system, resource scheduling priority and risk resource occupancy ratio, and the strategy behavior history includes the resource consumption trajectory, execution frequency and risk level distribution of various strategy actions.

[0171] Based on the strategy action level, action credibility score and risk warning type contained in the execution token, the resource budget limit, risk resource coefficient and resource tension weight corresponding to the current action are extracted from the resource quota assessment model to construct a resource quota constraint term. The resource quota constraint term is obtained by jointly fitting the resource budget mapping function and the credibility offset function, which is used to characterize the resource consumption elasticity range of the strategy action.

[0172] The resource quota constraints are matched with the execution token content item by item. If the resources requested by the execution token are lower than the resource budget limit and the credibility multiplied by the resource shortage weight is higher than the allocation threshold, it is marked as a valid execution token; otherwise, it is marked as a resource rejection token.

[0173] For a valid execution token, the action frequency control program is invoked to extract the policy action type and the corresponding historical execution time sequence. Combined with the action trigger frequency recorded in the current security situation vector, an execution frequency change curve within a sliding window is constructed. The change gradient is used to fit whether there is an action oscillation trend.

[0174] If the action oscillation trend is determined to be valid, the cooldown time management program is invoked. The corresponding cooldown interval threshold is set according to the action type and oscillation amplitude. The execution timestamp of the last same action instruction is read and the time interval between it and the current request is calculated. If the interval is less than the cooldown interval threshold, a delayed execution flag is added to the current action execution request. Otherwise, the execution status label of the action is updated to schedulable.

[0175] The resource quota screening results, action oscillation control results, and cooldown time management results are jointly integrated and input into the action instruction generation module to generate the final action execution instruction. The action execution instruction includes the action identifier, action execution timestamp, strategy path source information, resource scheduling tag, and action feedback channel identifier.

[0176] This implementation method constructs an execution instruction generation mechanism with resource controllability and response stability by screening multi-dimensional resource constraints, suppressing action execution frequency, and managing dynamic cooling. This effectively avoids system instability caused by resource conflicts or high-frequency execution of policy behaviors, and improves the dynamic adaptability of network defense execution strategies and system security.

[0177] In this embodiment, step seven specifically includes:

[0178] Record the causal path between action execution instructions and environmental responses during the strategy execution process. Construct a causal ledger through time series alignment and strategy trajectory backtracking. The causal ledger records strategy actions, response changes, and execution times in the form of triples, covering multiple time scales such as immediate response, cumulative effect, and delayed feedback.

[0179] In the causal ledger, the causal relationship between each type of strategy action and its corresponding attack mitigation effect is statistically analyzed. The causal influence factor extraction program is called to extract the reward deviation vector based on three dimensions: mitigation timeliness, blocking persistence, and attack suppression rate. The reward deviation is obtained by jointly fitting the causal tracking graph and the attack event sequence, and is used to quantify the contribution of strategy actions to environmental improvement.

[0180] Risk indicators, response latency indicators, and resource consumption indicators are set as three types of target factors in the reward function, and initial weight coefficients are set for each to control the relative importance of each target factor in the overall optimization. The weight coefficients can be adaptively adjusted according to the strategy deployment objectives and task types.

[0181] A multi-index reward function model is constructed, and the reward deviation vector in the causal ledger is weighted and combined with the weight coefficients of each target factor to form a candidate reward function parameter set. The weighted combination is obtained by joint fitting of a weighted aggregation operator and a nonlinear normalization function to ensure that it has relatively consistent performance under different index dimensions.

[0182] Perform parameter partitioning on all candidate reward function parameter groups, input the parameter groups into the policy behavior simulator, generate corresponding policy action trajectories, and calculate the fitting error of each parameter group under each target factor. The fitting error is obtained by fitting the difference between the policy behavior trajectory and the actual response of the environment, and is used to evaluate the fitting ability of the reward function parameters to the control effect of the target factor.

[0183] A reward function parameter mapping graph is constructed based on all candidate parameter groups and their corresponding fitting errors. The parameter group with the smallest fitting error is selected as the final reward function parameter, and the parameter update operation of the multi-index reward function embedded in the strategy model is performed. This realizes the adaptive evolutionary adjustment of the reward function parameters, effectively improving the convergence accuracy, adaptability and long-term optimization effect of the strategy model when facing multi-target defense tasks.

[0184] Example 1:

[0185] To verify the feasibility of this invention in practice, it was applied to a network security protection system upgrade project for a financial data center. This center processes over 200 million transaction records daily, covering highly sensitive business scenarios such as securities trading, bank clearing, and cross-border settlement. Its network environment is complex and faces multiple threats, including distributed denial-of-service attacks, persistent attack vector (APT) intrusions, malicious script injection, and unauthorized internal access. Traditional defense systems based on static rules and feature matching often suffer from insufficient attack interception rates, high false positive rates, and delays in defense actions due to resource scheduling conflicts when facing high-frequency and diverse attacks, directly impacting business continuity and transaction security.

[0186] In the application of this invention, multi-source data covering host systems, network boundaries, and service traffic are first collected. Redundant features are removed through feature pruning while retaining high-value, trustworthy fields. Then, Z-Score normalization and one-hot encoding are used to uniformly process the multi-modal data, generating a security posture vector representing the real-time network state. Based on this vector, a multi-index reward function is constructed, considering both short-term effects such as attack interception rate and response latency, and long-term steady-state indicators such as resource utilization balance and action jitter frequency. During the policy execution phase, the rapid response loop can match the optimal defense action sequence within milliseconds, while the policy optimization loop continuously optimizes long-term policy parameters through historical feedback and event-level reward values. Before execution, all actions undergo multi-source evidence review and credibility assessment, and resource quotas and anti-jitter management are implemented in the risk budget pool, effectively avoiding system overload and resource conflicts.

[0187] After three months of continuous operation and testing, the deployment effect of the method of the present invention in this data center was significantly improved. To facilitate the quantification of the effect, the key performance indicators of the method of the present invention during the three months of continuous operation were recorded, as shown in Table 1.

[0188] Table 1 Performance test results of the method of the present invention

[0189] month Attack interception rate (%) False alarm rate (%) Average response latency (ms) Resource utilization rate (%) January 96.8 1.9 142 78.5 February 97.2 1.7 138 80.3 March 96.5 1.8 145 79.1 average 97.0 1.6 140 81.0

[0190] As shown in Table 1, the method of this invention exhibits stable and excellent attack interception rate, with a minimum of 96.5% and a maximum of 97.2%, fluctuating by only 0.7 percentage points. This demonstrates that the defense strategy maintains a high level of threat blocking capability against attacks of different types and intensities, and exhibits strong strategy stability. The false positive rate remains consistently low at 1.6%-1.9%, with latency reduced by more than 35% compared to the average response time of approximately 215ms for traditional defense systems. This means that defense actions can be triggered rapidly in the early stages of an attack, thereby better blocking threat propagation. Regarding resource utilization, the three-month average remains between 79%-81%, indicating that the system can rationally allocate resources such as CPU, memory, and network bandwidth while executing high-frequency defense strategies, without resource overload due to frequent strategy switching or action execution. This also reflects the crucial role played by the risk budget pool and anti-jitter mechanism in maintaining resource stability. In summary, the data in Table 1 not only demonstrates the significant improvement in defense effectiveness of the method of this invention but also reflects its synergistic optimization capabilities in stability, real-time performance, and resource management, providing a solid technical guarantee for high-security scenarios.

[0191] To further compare the differences between the method of this invention and the traditional defense system, performance indicators of the two methods were collected under the same period, and the results are shown in Table 2.

[0192] Table 2. Performance Comparison between the Invention Method and Traditional Methods

[0193] index Method of the present invention Traditional methods Attack interception rate (%) 97.0 88.4 False alarm rate (%) 1.6 4.8 Average response latency (ms) 140 215 Success rate of defensive actions (%) 98.7 91.2 Number of resource conflicts 2 15

[0194] As shown in Table 2, the method of this invention achieves an attack interception rate 8.6 percentage points higher than traditional methods, significantly improving threat blocking capabilities; the false alarm rate is reduced by 3.2 percentage points, a relative reduction of over 66%, effectively reducing invalid alarms; the average response latency is shortened by 75ms, enabling defensive actions to take effect within an earlier time window; the success rate of defensive action execution is increased by 7.5 percentage points, reflecting more stable implementation of defensive actions under optimized resource management mechanisms; the number of resource conflicts is reduced by 13 times, a decrease to 86.7%, fully demonstrating the advantages of this invention in resource management and the stability of defense strategies.

[0195] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing a network attack active defense strategy based on deep reinforcement learning, characterized in that, The method comprises the following steps: Step one: collect multi-source data of network environment, perform feature clipping and whitelist feature reservation, and form a multi-source data set; Step two: normalize and encode the multi-source data set to generate a security posture vector; Step three: construct a multi-index reward function based on the security posture vector to generate an immediate reward value and an event-level reward value; Step four: based on the immediate reward value and the event-level reward value, execute a double closed-loop mechanism through an improved PPO model; the double closed-loop mechanism includes a rapid disposal loop and a policy optimization loop, which respectively output immediate policy instructions and long-term policy parameters; The step four is specifically: Based on the immediate reward value and the event-level reward value, a double closed-loop mechanism is constructed to execute a policy instruction generation and policy parameter update process; Through the rapid disposal loop, the immediate policy trajectory in the policy model is matched according to the security posture vector and the immediate reward value, the optimal action sequence is screened, and the immediate policy instruction is generated; Through the policy optimization loop, the deviation between the target policy distribution and the policy distribution is calculated according to the event-level reward value and the historical action feedback record, and the clipping target deviation is obtained; The state value difference between the action selection and the environment response is evaluated to obtain the state value residual error; The information entropy of the policy model output distribution is calculated to obtain the policy distribution entropy; A joint loss function is constructed to weight and combine the clipping target deviation, the state value residual error, and the policy distribution entropy; The joint loss function back propagation process is executed to update the weight parameters in the policy model and generate long-term policy parameters; Step five: multi-source evidence deliberation is performed on the immediate policy instruction and the security posture vector to generate an action credibility score, and the key evidence loss condition is judged to obtain an execution token; Step six: the execution token is input into a risk budget pool for resource quota review, and anti-jitter and cooling control are performed to output an action execution instruction; Step seven: optimize the parameters of the multi-index reward function through a causal ledger; The step seven is specifically: Record the causal path between the action execution instruction and the environment response in the policy execution process to construct a causal ledger; Statistical causal association information between each policy action and attack mitigation effect in the causal ledger is extracted to extract the corresponding return deviation; Set risk indicators, response time delay indicators, and resource consumption indicators as three types of target factors, and respectively assign initial weight coefficients; A multi-index reward function is constructed to weight and combine the return deviation and the target factor weight coefficient to generate a candidate reward function parameter group; Parameter division operation is performed on the candidate reward function parameter group to evaluate the fitting error between the policy behavior and the target factor under different parameter combinations to form a reward function parameter mapping relationship; Select the parameter group with the smallest fitting error as the final reward function parameter to update the multi-index reward function in the policy model.

2. The network attack active defense strategy optimization method based on deep reinforcement learning according to claim 1, characterized in that, The step one is specifically: Collect multi-source data covering host systems, network boundaries, and business traffic, including communication metadata, log record data, session behavior data, and alarm feedback data; Perform feature clipping on the multi-source data to filter out features with weak attack association or high redundancy, and reserve a core feature set; The white list feature retention operation is performed on the data after feature pruning, and by comparing a pre-defined white list rule set, feature instances conforming to a trusted behavior mode are screened, and a marked label is generated; The marked label is combined with the retained features to form a structured data item, which is organized into a unified format in time sequence and source module to form a multi-source data set; the retained feature combination includes attack fields screened by pruning operation and trusted fields defined by white list rule set.

3. The method of claim 2, wherein, The step two is specifically: Each type of data in the multi-source data set is divided according to a unified time window, and numerical value type features and category type features are normalized and encoded respectively by using a method based on Z-Score standardization and one-hot encoding to construct a standardized feature vector; Based on the standardized feature vector, a multi-dimensional security posture representation model describing the dynamic change state of the network environment is constructed, and a nested mapping operation is performed on the normalized data of each source module to generate a structured vector set; The structured vector set is input into a feature aggregation module to perform cross-modal fusion and time sequence arrangement operation to generate a security posture vector representing the overall network security state.

4. The network attack active defense strategy optimization method based on deep reinforcement learning according to claim 3, characterized in that, The step three is specifically: Based on the security posture vector, network state feature indicators are extracted to construct an indicator vector set; the indicator vector set includes resource utilization rate indicators, alarm response delay indicators, attack interception rate indicators, false positive rate indicators and disposal action coverage rate indicators; A multi-indicator reward function is constructed based on the indicator vector set; the multi-indicator reward function includes an immediate reward sub-function and an event-level reward sub-function, which are respectively used to quantify the influence of policy action on short-term disposal effect and long-term system stability; In the immediate reward sub-function, the attack interception rate improvement and the false positive rate suppression amplitude are introduced as the main feedback items, a first set of weighting coefficients is set, and the weighted accumulation is performed on each feedback item to output the immediate reward value; In the event-level reward sub-function, the system resource utilization balance degree, the response delay average and the disposal action coverage range are introduced as long-term evaluation items, a second set of weighting coefficients is set, and the normalized aggregation is performed on the evaluation items to output the event-level reward value.

5. The network attack active defense strategy optimization method based on deep reinforcement learning according to claim 4, characterized in that, The multi-indicator reward function includes an immediate reward layer and an event-level reward layer, which are specifically: The immediate reward layer measures the balance between the suppression effect and the resource cost of a single defense action in the threat disposal process; an immediate feedback item set is constructed based on attack interception effect, false positive cost, resource consumption and business impact, a first set of weighting coefficients is set, and a weighted cumulative operation is performed on each feedback item to output an immediate reward value; The event-level reward layer measures the cumulative defense performance of continuous defense actions in the attack event life cycle; A long-term evaluation item set is constructed based on response time delay, action jitter frequency, evidence gain and compliance conflict, a second set of weighting coefficients is set, and a normalized aggregation operation is performed on each evaluation item to output an event-level reward value; The weight of each index in the multi-index reward function is dynamically updated based on the causal ledger constructed defense action and defense effect attribution analysis results and the state change of the security posture vector, and the short-term fluctuations of the weighting coefficients are suppressed through a smoothing function; the smoothing function is any one of an exponential smoothing function, a weighted moving average function or a Gaussian smoothing function.

6. The network attack active defense strategy optimization method based on deep reinforcement learning according to claim 5, characterized in that, The improved PPO model is specifically: A model structure including a state encoder, an action sampler and a policy updater is constructed. The state encoder receives the security posture vector, performs a multi-layer nested mapping operation, and outputs a state representation tensor. The action sampler generates a candidate action set based on the state representation tensor and the policy probability distribution, and filters the optimal action based on the immediate reward value to form an action sequence. The policy updater receives the event-level reward value, the historical action feedback record and the action sequence, and constructs a target policy distribution. The KL divergence between the target policy distribution and the policy distribution corresponding to the action sequence is calculated to obtain a clipped target bias. The residual between the environment state response value corresponding to the execution result of the action sequence and the expected state value is evaluated to obtain a state value residual. The information entropy of the probability distribution corresponding to the action sequence in the policy model is calculated to obtain a policy distribution entropy. A joint loss function is constructed to combine the clipped target bias, the state value residual and the policy distribution entropy. The joint loss function is backpropagated to update the weight parameters of each module in the policy model, and the long-term policy parameters are output.

7. The method of claim 6, wherein the method further comprises: The step five is specifically: Receive the immediate policy instruction and the security posture vector to construct a policy action request; Retrieve multi-source evidence information associated with the policy action request; the multi-source evidence information includes communication log information, behavior feature record, interception feedback data, attack path structure and environment context parameter; A multi-dimensional evidence support matrix is constructed based on the multi-source evidence information, and a support degree vector covering each evidence type is extracted; The support degree vector is input into the credibility aggregation module, and weighted summation and bias correction operations are performed according to the preset weight set to generate an action credibility score; Based on the evidence support matrix and the support degree threshold, it is judged whether there is a missing, ambiguous or conflict state in the key evidence dimension relied on by the immediate policy instruction, if there is, it is marked as a key evidence loss state, otherwise it is marked as a key evidence complete state; A joint judgment is performed according to the action credibility score and the key evidence loss state to generate an execution token; If the action credibility score is greater than the preset credibility threshold and the key evidence complete state is established, an effective execution token is generated, otherwise a risk prompt token is generated.

8. The network attack active defense strategy optimization method based on deep reinforcement learning according to claim 7, characterized in that, The step six: input the execution token into the risk budget pool for resource quota review, and perform anti-jitter and cooling control to output an action execution instruction, specifically: Input the execution token into the risk budget pool, call the resource state data and the historical policy usage record to construct a resource quota evaluation model; According to the policy action level, the action credibility score and the risk prompt type contained in the execution token, the corresponding resource budget limit and resource tension coefficient are extracted from the resource quota evaluation model to construct a resource quota constraint term; The resource quota constraint item is matched with the execution token, if the resource allocation condition is met, the valid execution token is marked, otherwise the resource rejection token is marked; For the valid execution token, the action frequency control program is called to read the action frequency information recorded in the policy action type and the security posture vector, and to judge whether the action execution frequency has a shock trend; If there is a shock trend of action, the cooling time management program is called to set the cooling interval threshold, calculate the time interval between the same action instruction and the last time, if the time interval is less than the cooling interval threshold, the execution time marker is delayed, otherwise the execution state is updated; The action execution instruction is generated by integrating the resource quota screening result, the action shock control result and the cooling time management result; The action execution instruction includes an execution action identifier, an action execution timestamp, policy path source information, a resource scheduling label and an action feedback channel identifier.

Images