An intelligent security audit method, system and device based on hierarchical reinforcement learning

Abstract

The present invention relates to the technical field of network security auditing and automated penetration testing, and specifically to an intelligent security auditing method, system, and device based on hierarchical reinforcement learning, specifically as follows: the security auditing process is modeled as a hierarchical Markov decision process, the server generates an attack sequence and sends it to the client, the attack sequence is generated based on hierarchical reinforcement learning, then the client performs an attack action according to the received attack sequence and collects feedback, and then sends the feedback to the server, the server updates the hierarchical reinforcement learning according to the feedback, and optimizes and iterates, when the set iteration conditions are met, the security auditing process ends, otherwise the security auditing operation continues until the iteration stop conditions are met and the security auditing process ends. The present invention improves the detection capability of coping with complex attack scenarios through automated hierarchical learning and adaptation mechanisms, reduces manual intervention, and thus obtains more efficient and accurate security threat identification.

Description

Technical Field

[0001] The present invention relates to the technical field of network security auditing and automated penetration testing, and in particular to an intelligent security auditing method, system and equipment based on hierarchical reinforcement learning. Background Art

[0002] As network environments and application systems continue to grow in complexity, traditional security auditing methods are lacking in efficiency, coverage, and diverse attack scenarios. Manual audits are time-consuming and rely heavily on expert experience, making them difficult to adapt to the evolving threat landscape. Consequently, reinforcement learning methods have been studied and applied in recent years in areas such as automated security auditing and penetration testing. The goal is to improve detection capabilities for complex attack vectors and reduce manual intervention through automated learning and adaptation mechanisms. However, traditional reinforcement learning often faces challenges such as excessively large search spaces, low training efficiency, and unstable policy convergence, making it difficult to fully meet the needs of actual network security audits.

[0003] Therefore, the present invention proposes an intelligent security audit method, system and device based on hierarchical reinforcement learning to solve the above problems. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention develops an intelligent security audit method, system and equipment based on hierarchical reinforcement learning. The present invention improves the detection capability of responding to complex attack scenarios through automated hierarchical learning and adaptation mechanisms, reduces manual intervention, and thus obtains more efficient and accurate security threat identification.

[0005] The technical solution to the technical problem solved by the present invention is an intelligent security audit method based on hierarchical reinforcement learning, which is as follows:

[0006] The security audit process is modeled as a hierarchical Markov decision process. The server generates an attack sequence and sends it to the client to perform security audit operations. The attack sequence is generated based on hierarchical reinforcement learning, which includes macro-strategies and micro-strategies. The client then performs attack actions based on the received attack sequence and collects feedback, which is then sent to the server. The server updates the macro-strategy and micro-strategy based on the feedback, iterates and optimizes the macro-strategy and micro-strategy, and sets an iteration stop condition. If the iteration condition is met, the security audit process ends. Otherwise, the security audit operation continues according to the updated macro-strategy and micro-strategy until the iteration stop condition is met.

[0007] In a specific embodiment, hierarchical reinforcement learning includes a macro strategy and micro-strategies , macro strategy For high-level, micro-strategies in hierarchical reinforcement learning It is the lower layer in hierarchical reinforcement learning, where the higher layer defines the subtasks. and high-level goals , , executable micro-actions defined by the lower layer , , Indicates an attack action;

[0008] The status of macro strategy in the system Select subtask , expressed as , micro-strategy in subtasks and the status of the system Select Micro Actions , expressed as ,state , Indicates the network environment, system configuration information, and current attack progress.

[0009] In a specific implementation, the macro strategy selection subtask The process is as follows:

[0010] Macro strategies analyze the current system status through threat intelligence information and historical audit records of the system , dynamically determine the high-level goals that need to be executed , then based on macro strategy Status in the system From high-level goals Select the appropriate subtask ;

[0011] Threat intelligence information includes data sources from network detection systems, intelligence sharing platforms, and security vulnerability databases;

[0012] High-level goals A series of subtasks composition, , Representing high-level goals Middle subtasks, Representing high-level goals The number of neutron tasks, ;

[0013] The macro strategy calculates the macro value generated by the immediate reward through the macro value function and arranges the execution order and timing of the subtasks. The specific process is as follows:

[0014] The formula for calculating macro value is as follows:

[0015] ,

[0016] in, express The discount factor at the moment, Indicates the expected value, Indicates the preset planning time steps, Indicates the status of the system Next subtask The macro value of Indicates The status of the system at all times Execute subtasks The immediate rewards generated after

[0017] Then, according to the macro value, the priority is determined and the execution order and timing of subtasks are arranged. Specifically, the priority threshold is set. , filter out Subtasks As candidate subtasks, the candidate subtasks are then sorted according to The values ​​are sorted in descending order to obtain the priority sequence of the subtasks, among which the top ranked Higher values ​​have higher execution priority;

[0018] Finally, according to the priority sequence of subtasks and the current resource status of the system, the system status is guaranteed. Under the premise of allowing subtasks to be executed, each candidate subtask is started in sequence, and updated in real time according to task feedback during execution .

[0019] In a specific implementation, a micro-strategy selects a micro-action. The process is as follows:

[0020] In subtask and the status of the system Based on micro-strategy Select micro-actions , , attack action Including command execution, port and traffic detection, vulnerability exploitation, privilege escalation, and resource usage and concealment related actions;

[0021] Micro-strategy by defining micro-value In each subtask Evaluate different micro-actions The execution value of the system is continuously updated with real-time feedback. The specific process is as follows:

[0022] During the system initialization phase, for each subtask and all possible micro-movements , micro value are assigned random initial values;

[0023] Status in the system Next, execute the subtask When the micro-strategy is based on the current micro-value The value of selects a micro action Execute and then use - Greedy strategy to balance exploration and exploitation, based on probability Select The action with the largest value, according to the probability Randomly select other actions, micro actions After being executed, the state of the system is determined by Transformed into , and receive instant feedback rewards After multiple iterations and continuous updates of immediate feedback rewards, the micro-strategy is in the state of the system and subtasks According to micro value The micro-actions selected by the value of tend to be optimal.

[0024] In a specific implementation, the client's execution and feedback:

[0025] (1) In At this moment, the client receives the micro-action sent by the server , and based on System status at any moment and corresponding subtasks Carry out specific attack operations. During the execution of specific operations, the client performs micro-actions The instructions execute attack actions on the audit object;

[0026] (2) After the client completes the execution, the system status is Transformed into , Indicates execution of an action The latest status after the attack, including network topology and port changes, system configuration and resource usage, current attack progress and subtask completion information, whether there are security anomalies or defense mechanism triggering abnormal events;

[0027] (3) The client calculates the immediate feedback reward of the micro-strategy based on the update of the system status , the calculation formula is as follows:

[0028] ,

[0029] Among them, there are five reward items, namely 、 、 、 and , It represents the degree of success of a quantitative attack operation. represents the time cost of the attack operation, Indicates the security risks brought by attack operations. Indicates the system resources consumed by the attack operation. represents the state-action exploration reward, 、 、 、 and Respectively in the state Next reward 、 、 、 and Dynamic weight coefficient of

[0030] State-Action Exploration Reward The calculation formula is as follows:

[0031] ,

[0032] in, represents the exploration reward base coefficient, Indicates the total number of attack actions executed in the history. express The index of Indicates the number of historical executions An attack action, Indicates the current attack action and The state-action space distance between represents the distance sensitivity adjustment factor;

[0033] Determined through a multi-level security priority assessment model and security status entropy weight method , , An index representing the number of reward items, , The calculation formula is as follows:

[0034] ,

[0035] in, It also represents the index of the number of reward items. Indicates the The default basic weight of each reward item is Indicates status No. Dynamic adjustment factor for each reward item, Indicates status The state entropy of Indicates the The default basic weight of each reward item is Indicates status No. Dynamic adjustment factors for each reward item;

[0036] state The calculation formula of the state entropy is as follows:

[0037] ,

[0038] in, Indicates the number of system status categories, Indicates status Belong to the probability of the class state;

[0039] state Dynamic adjustment factor The calculation formula is:

[0040] ,

[0041] in, Indicates status Next An important indicator of the award, represents the normalization constant, and represents two different adjustment coefficients, Indicates the Adjustment coefficient of reward items , Indicates the Adjustment coefficient of reward items , Indicates the current attack progress indicator, Indicates the The historical success rate of each reward item;

[0042] Historical success rate The calculation formula is as follows:

[0043] ,

[0044] in, Indicates that as of The total number of attack actions executed at the moment, express The index of Indicates the The set of related actions for each reward item, represents the indicator function, if ,but ,otherwise , Indicates the judgment Attack action Is it successful? If successful, If it fails, ;

[0045] (4) The client will update the latest status , instant rewards , subtask completion indication information and auxiliary diagnosis information are sent back to the server in the form of data packets.

[0046] In a specific implementation, the iteration and optimization of macro and micro strategies:

[0047] (1) Update micro-strategy:

[0048] The server responds to the data packet from the client. Perform strategy iteration and use value-based methods to analyze micro-values To update, the calculation process is as follows:

[0049] ,

[0050] in, Represents the learning rate used to control the ratio of fusion of new and old values, represents the discount factor, Indicates that all micro actions get the maximum Candidate actions for values, express The subtask selected at the moment, The value of attack actions representing the optimal micro-strategy;

[0051] (2) Update macro strategy:

[0052] When the service segment completes or switches subtasks, it iterates the strategy based on the benefit and risk information of the macro strategy, and the immediate reward of the macro strategy , the calculation formula is as follows:

[0053] ,

[0054] in, Indicates the degree of coverage improvement of unknown network nodes, ports and potential vulnerability areas. It represents the degree of success of a quantitative attack operation. Indicates the risk of measuring attack actions at the macro level, Represents the cost of performing an attack action;

[0055] When completing or switching subtasks, the macro value is updated based on the immediate rewards of the macro strategy;

[0056] (3) Set the iteration stop condition:

[0057] Set the value function change threshold, when the micro-strategy and macro strategies When the update amplitude is lower than the given threshold, the micro-strategy layer is considered stable, as follows:

[0058] Define update amplitude thresholds for micro strategies and macro strategies respectively. Set micro thresholds for the micro strategy layer. , set macro thresholds for macro strategies ;

[0059] In each iteration, the status of all systems , subtasks and micro-movements , calculate the maximum absolute change before and after the micro value update , the calculation formula is as follows:

[0060] ,

[0061] in, Indicates based on Calculate the maximum value, represents the updated micro value, Indicates the micro value before updating;

[0062] At the same time, for all system states , calculate the maximum absolute change before and after the macro value update , the calculation formula is as follows:

[0063] ,

[0064] in, Indicates based on Calculate the maximum value, represents the updated macro value, Indicates the macro value before the update;

[0065] If in continuous In all iterations, and , it is considered that the strategy update has stabilized and the iteration is terminated;

[0066] The training process is also terminated when the main security audit objectives set have been achieved or the available resources are insufficient to continue to perform large-scale audits.

[0067] The present invention also provides an intelligent security audit method based on hierarchical reinforcement learning, including a module for executing processing instructions of each step in the intelligent security audit method based on hierarchical reinforcement learning.

[0068] The present invention also provides an electronic device that executes an intelligent security audit method based on hierarchical reinforcement learning. The device includes: a memory, a processor, and a program stored in the memory and capable of running the intelligent security audit method based on hierarchical reinforcement learning on the processor.

[0069] The present invention also provides a memory on which a program of an intelligent security audit method is stored. When the program of the intelligent security audit method is executed by a processor, an intelligent security audit method based on hierarchical reinforcement learning is implemented.

[0070] The effects provided in the summary of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0071] The hierarchical reinforcement learning framework splits security audit tasks at different levels, narrowing the search space and improving learning efficiency. Macro strategies are responsible for global planning and risk control, while micro strategies further refine the execution and effectiveness evaluation of single-step attack actions, better adapting to the needs of complex multi-step and multi-stage network environments.

[0072] By building a layered reinforcement learning system between the server and client, we can combine macro and micro audit tasks. This not only enables more effective discovery of system vulnerabilities, but also adapts to various scenarios and dynamically manages risks, significantly improving the scalability and practical value of network security audits and automated penetration testing.

[0073] Through the hierarchical design of macro-strategies and micro-strategies, the problem of state-action space explosion in traditional single-layer reinforcement learning in complex security audit scenarios is effectively solved, and intelligent security auditing in complex network environments is realized.

[0074] The macro layer of the present invention is responsible for subtask planning and high-level goal decomposition, while the micro layer focuses on the execution and optimization of specific attack actions. The two work together to significantly narrow the search space, improve the learning convergence speed and strategy quality, and further introduce a multi-dimensional adaptive reward mechanism based on state entropy and dynamic adjustment factors. The state-action exploration reward can effectively avoid the local optimal problem commonly seen in reinforcement learning. The dynamic mapping relationship between the system state and the reward weight is established through the security state entropy weight method, thereby realizing intelligent risk control in the security audit process.

[0075] In summary, the present invention can automatically learn and optimize attack sequences, greatly improve vulnerability discovery efficiency and coverage, reduce dependence on professional security personnel, reduce enterprise security audit costs, and has the ability to adapt to network environments of different complexities. It has a strong ability to discover new threats and unknown vulnerabilities, providing more proactive and forward-looking technical support for network security protection, and has important theoretical value and practical significance for improving the security protection capabilities of critical information infrastructure. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0077] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0078] In order to clearly illustrate the technical features of this solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for realizing different structures of the present invention.

[0079] Example 1

[0080] An intelligent security audit method based on hierarchical reinforcement learning is as follows:

[0081] The security audit process is modeled as a hierarchical Markov decision process. The server generates an attack sequence and sends it to the client to perform security audit operations. The attack sequence is generated based on hierarchical reinforcement learning, which includes macro-strategies and micro-strategies. The client then performs attack actions based on the received attack sequence and collects feedback, which is then sent to the server. The server updates the macro-strategy and micro-strategy based on the feedback, iterates and optimizes the macro-strategy and micro-strategy, and sets an iteration stop condition. If the iteration condition is met, the security audit process ends. Otherwise, the security audit operation continues according to the updated macro-strategy and micro-strategy until the iteration stop condition is met.

[0082] In a specific embodiment, hierarchical reinforcement learning includes a macro strategy and micro-strategies , macro strategy For high-level, micro-strategies in hierarchical reinforcement learning It is the lower layer in hierarchical reinforcement learning, where the higher layer defines the subtasks. and high-level goals , , executable micro-actions defined by the lower layer , , Indicates an attack action;

[0083] The status of macro strategy in the system Select subtask , expressed as , micro-strategy in subtasks and the status of the system Select Micro Actions , expressed as ,state , Indicates the network environment, system configuration information, and current attack progress.

[0084] In a specific implementation, the macro strategy selection subtask The process is as follows:

[0085] Macro strategies analyze the current system status through threat intelligence information and historical audit records of the system , dynamically determine the high-level goals that need to be executed , then based on macro strategy Status in the system From high-level goals Select the appropriate subtask ;

[0086] Threat intelligence information includes data sources from network detection systems, intelligence sharing platforms, and security vulnerability databases;

[0087] High-level goals A series of subtasks composition, , Representing high-level goals Middle subtasks, Representing high-level goals The number of neutron tasks, ;

[0088] Each subtask can be considered as a high-level audit objective, such as: information collection subtask, vulnerability detection subtask, authority escalation subtask, persistence and lateral movement subtask, and risk assessment subtask;

[0089] The macro strategy calculates the macro value generated by immediate rewards through the macro value function to measure the advantages and disadvantages of different subtasks in terms of long-term benefits and risk control, and arranges the execution order and timing of subtasks. The specific process is as follows:

[0090] The formula for calculating macro value is as follows:

[0091] ,

[0092] in, express The discount factor at the moment, Indicates the expected value, Indicates the preset planning time steps, Indicates the status of the system Next subtask The macro value of Indicates The status of the system at all times Execute subtasks The immediate reward generated after the transaction reflects the comprehensive indicators of benefits and risk control;

[0093] Then, according to the macro value, the priority is determined and the execution order and timing of subtasks are arranged. Specifically, the priority threshold is set. , filter out Subtasks As candidate subtasks, the candidate subtasks are then sorted according to The values ​​are sorted in descending order to obtain the priority sequence of the subtasks, among which the top ranked Higher values ​​have higher execution priority;

[0094] Finally, according to the priority sequence of subtasks and the current resource status of the system, the system status is guaranteed. Under the premise of allowing subtasks to be executed, each candidate subtask is started in sequence, and updated in real time according to task feedback during execution , and then adjust the priority and execution timing of subsequent tasks.

[0095] In a specific implementation, a micro-strategy selects a micro-action. The process is as follows:

[0096] In subtask and the status of the system Based on micro-strategy Select micro-actions , , attack action Including command execution, port and traffic detection, vulnerability exploitation, privilege escalation, and resource usage and concealment related actions;

[0097] Micro-strategy by defining micro-value In each subtask Evaluate different micro-actions The execution value of the audit process is continuously updated with real-time feedback, so that the action selection in the audit process gradually tends to be optimal. The specific process is as follows:

[0098] During the system initialization phase, for each subtask and all possible micro-movements , micro value are assigned random initial values;

[0099] Status in the system Next, execute the subtask When the micro-strategy is based on the current micro-value The value of selects a micro action Execute and then use - Greedy strategy to balance exploration and exploitation, based on probability Select The action with the largest value, according to the probability Randomly select other actions, micro actions After being executed, the state of the system is determined by Transformed into , and receive instant feedback rewards After multiple iterations and continuous updates of immediate feedback rewards, the micro-strategy is in the state of the system and subtasks According to micro value The micro-actions selected by the value of tend to be optimal.

[0100] In a specific implementation, the client's execution and feedback:

[0101] (1) In At this moment, the client receives the micro-action sent by the server , and based on System status at any moment and corresponding subtasks Carry out specific attack operations. During the execution of specific operations, the client performs micro-actions The instructions execute attack actions on the audit object;

[0102] (2) After the client completes the execution, the system status is Transformed into , Indicates execution of an action The latest status after the attack, including network topology and port changes, system configuration and resource usage, current attack progress and subtask completion indication information. When the completion conditions are met, the subtask completion flag is set to 1, otherwise it is set to 0. Whether there is a security anomaly or a defense mechanism triggers an abnormal event;

[0103] (3) The client calculates the immediate feedback reward of the micro-strategy based on the update of the system status , the calculation formula is as follows:

[0104] ,

[0105] Among them, there are five reward items, namely 、 、 、 and , It represents the degree of success of a quantitative attack operation. represents the time cost of the attack operation, Indicates the security risks brought by attack operations. Indicates the system resources consumed by the attack operation. represents the state-action exploration reward, 、 、 、 and Respectively in the state Next reward 、 、 、 and Dynamic weight coefficient of

[0106] State-Action Exploration Reward The calculation formula is as follows:

[0107] ,

[0108] in, represents the exploration reward base coefficient, Indicates the total number of attack actions executed in the history. express The index of Indicates the number of historical executions An attack action, Indicates the current attack action and The state-action space distance between represents the distance sensitivity adjustment factor;

[0109] Determined through a multi-level security priority assessment model and security status entropy weight method , , An index representing the number of reward items, , The calculation formula is as follows:

[0110] ,

[0111] in, It also represents the index of the number of reward items. Indicates the The default basic weight of each reward item is Indicates status No. Dynamic adjustment factor for each reward item, Indicates status The state entropy of Indicates the The default basic weight of each reward item is Indicates status No. Dynamic adjustment factors for each reward item;

[0112] state The calculation formula of the state entropy is as follows:

[0113] ,

[0114] in, Indicates the number of system status categories, Indicates status Belong to the probability of the class state;

[0115] state Dynamic adjustment factor The calculation formula is:

[0116] ,

[0117] in, Indicates status Next An important indicator of the award, represents the normalization constant, and represents two different adjustment coefficients, Indicates the Adjustment coefficient of reward items , Indicates the Adjustment coefficient of reward items , Indicates the current attack progress indicator, Indicates the The historical success rate of each reward item;

[0118] Historical success rate The calculation formula is as follows:

[0119] ,

[0120] in, Indicates that as of The total number of attack actions executed at the moment, express The index of Indicates the The set of related actions for each reward item, represents the indicator function, if ,but ,otherwise , Indicates the judgment Attack action Is it successful? If successful, If it fails, ;

[0121] (4) The client will update the latest status , instant rewards , subtask completion indication information and auxiliary diagnosis information are sent back to the server in the form of data packets.

[0122] In a specific implementation, the iteration and optimization of macro and micro strategies:

[0123] (1) Update micro-strategy:

[0124] The server responds to the data packet from the client. Perform strategy iteration and use value-based methods to analyze micro-values To update, the calculation process is as follows:

[0125] ,

[0126] in, Represents the learning rate used to control the ratio of fusion of new and old values, represents the discount factor, Indicates that all micro actions get the maximum Candidate actions for values, express The subtask selected at the moment, The value of attack actions representing the optimal micro-strategy;

[0127] (2) Update macro strategy:

[0128] When the service segment completes or switches subtasks, it iterates the strategy based on the benefit and risk information of the macro strategy, and the immediate reward of the macro strategy , the calculation formula is as follows:

[0129] ,

[0130] in, Indicates the degree of coverage improvement of unknown network nodes, ports and potential vulnerability areas. It represents the degree of success of a quantitative attack operation. Indicates the risk of measuring attack actions at the macro level, Represents the cost of performing an attack action;

[0131] When completing or switching subtasks, the macro value is updated based on the immediate rewards of the macro strategy;

[0132] (3) Set the iteration stop condition:

[0133] Set the value function change threshold, when the micro-strategy and macro strategies When the update amplitude is lower than the given threshold, the micro-strategy layer is considered stable, as follows:

[0134] Define update amplitude thresholds for micro strategies and macro strategies respectively. Set micro thresholds for the micro strategy layer. , set macro thresholds for macro strategies ;

[0135] In each iteration, the status of all systems , subtasks and micro-movements , calculate the maximum absolute change before and after the micro value update , the calculation formula is as follows:

[0136] ,

[0137] in, Indicates based on Calculate the maximum value, represents the updated micro value, Indicates the micro value before updating;

[0138] At the same time, for all system states , calculate the maximum absolute change before and after the macro value update , the calculation formula is as follows:

[0139] ,

[0140] in, Indicates based on Calculate the maximum value, represents the updated macro value, Indicates the macro value before the update;

[0141] If in continuous In all iterations, and ,set up , it is considered that the strategy update has become stable and the iteration can be terminated;

[0142] The training process may also be terminated when the main security audit objectives have been achieved or when the available resources are insufficient to continue with the large-scale audit;

[0143] If the audit policy performance is still not significantly improved after exceeding the maximum number of iterations, you can choose to exit or switch to manual review mode based on specific needs.

[0144] Example 2

[0145] An intelligent security audit method based on hierarchical reinforcement learning includes a module for executing processing instructions of each step in the intelligent security audit method based on hierarchical reinforcement learning.

[0146] Example 3

[0147] An electronic device executes an intelligent security audit method based on hierarchical reinforcement learning. The device comprises: a memory, a processor, and a program stored in the memory and capable of running the intelligent security audit method based on hierarchical reinforcement learning on the processor.

[0148] Example 4

[0149] A memory stores a program of an intelligent security audit method. When the program of the intelligent security audit method is executed by a processor, an intelligent security audit method based on hierarchical reinforcement learning is implemented.

[0150] Example 5

[0151] To verify that the proposed method has better performance than traditional rule-based security audit methods and basic single-layer reinforcement learning methods, an enterprise internal network security simulation environment was selected for experimental verification. This simulation environment constructed a virtual enterprise internal network, which includes elements such as servers, workstations, network devices, and firewalls. It also integrated network monitoring, data collection, and logging modules to simulate the security status and operation of the enterprise internal network.

[0152] The experimental goal is to deploy the intelligent security audit system based on hierarchical reinforcement learning proposed in this invention, realize automated attack sequence optimization, and improve vulnerability discovery efficiency and coverage; in addition, under the same background, verify the performance of traditional rule-based security audit methods and basic single-layer reinforcement learning methods, and compare them with the method proposed in this invention in terms of vulnerability discovery efficiency, location threat detection rate, average audit cost, critical vulnerability coverage and corresponding APT attack time.

[0153] The process of deploying the method of the present invention in the virtual enterprise internal network is as follows:

[0154] (1) Build the server and client:

[0155] Server:

[0156] Macro Strategy Engine: Generates high-level objectives (e.g., "obtain database permissions") based on threat intelligence (e.g., the MITRE ATT&CK framework);

[0157] Micro-strategy optimizer: dynamically adjusts attack actions (such as SQL injection and port scanning);

[0158] Feedback analysis module: calculates the reward function and updates the strategy;

[0159] Client:

[0160] Attack execution agent: deployed in each subnet, executes actions and collects status feedback.

[0161] Security status sensor: monitors network topology and defense mechanism triggering in real time.

[0162] The functions of the relevant core components in the server and client are shown in Table 1;

[0163] Table 1 Core component function comparison table

[0164]

[0165] (2) Implementation process and dynamic optimization

[0166] (2-1) Macro strategy generation:

[0167] Input: Current status s_t (e.g., "Web service exposes ports 22 / 80 / 443, defense mechanism not triggered").

[0168] Output: subtask sequence G={ :Port Scan, :Vulnerability Exploitation, :Privilege escalation}, press (s, ) sort.

[0169] (2-2) Micro-strategy execution:

[0170] Subtasks Under (Port Scan), select Action (For example, "Using Nmap for SYN stealth scanning").

[0171] Exploration-exploitation balance: ϵ=0.2, 80% probability of selection The action with the highest value is used, and 20% randomly tries new actions (such as "ICMP stealth scan").

[0172] (2-3) Feedback and reward calculation:

[0173] Example of reward function:

[0174] ,

[0175] : Scan found 2 open ports → +0.4;

[0176] : Firewall alarm triggered → -0.3;

[0177] ;

[0178] Dynamic weight adjustment: If the state entropy High (complex network status), (Exploration Reward) Weight increased to 0.15.

[0179] (2-4) Strategy Iteration Optimization:

[0180] Micro-strategy update: ,

[0181] Macro Strategy Update: If Subtask Completed and (s, ) is increased by 10%, the priority of subsequent tasks will be increased.

[0182] (3) Similarly, the traditional rule-based security audit method and the basic single-layer reinforcement learning method were deployed on the internal network of the virtual enterprise, and the comparative results of the three methods were obtained. The test period was one month, as shown in Table 2.

[0183] Table 2 Evaluation and comparison results of various methods

[0184]

[0185] From the data in Table 2, it can be seen that the traditional rule-based security audit method relies on fixed rules, so its unknown threat detection rate and critical vulnerability coverage in a dynamic environment are low, and the average audit cost and response time to APT attacks are high; the basic single-layer reinforcement learning method has a certain degree of adaptability, but in a complex network environment, the vulnerability discovery efficiency, unknown threat detection rate and critical vulnerability coverage are still insufficient; the layered reinforcement learning-based method proposed in this application has significantly improved the vulnerability discovery efficiency, unknown threat detection rate and critical vulnerability coverage, and the number of iterative convergence times is also greatly reduced, indicating that this method can perform security audits on the target system faster and more accurately, thereby comprehensively improving the effectiveness and efficiency of security detection.

[0186] Although the above describes the specific implementation methods of the invention in conjunction with the accompanying drawings, it does not limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. An intelligent security audit method based on hierarchical reinforcement learning, characterized by: The security audit process is modeled as a hierarchical Markov decision process. The server generates an attack sequence and sends it to the client to perform security audit operations. The attack sequence is generated based on hierarchical reinforcement learning, which includes macro-strategies and micro-strategies. The client then executes attack actions based on the received attack sequence and collects feedback. The feedback is then sent to the server. The server updates the macro-strategy and micro-strategy based on the feedback, iterates and optimizes the macro-strategy and micro-strategy, and sets an iteration stop condition. If the iteration condition is met, the security audit process ends. Otherwise, the security audit operation continues according to the updated macro-strategy and micro-strategy until the iteration stop condition is met. Client execution and feedback: (1) In At this moment, the client receives the micro-action sent by the server , and based on System status at any moment and corresponding subtasks Carry out specific attack operations. During the execution of specific operations, the client performs micro-actions The instructions execute attack actions on the audit object; (2) After the client completes the execution, the system status is Transformed into , Indicates execution of an action The latest status after the attack, including network topology and port changes, system configuration and resource usage, current attack progress and subtask completion information, whether there are security anomalies or defense mechanism triggering abnormal events; (3) The client calculates the immediate feedback reward of the micro-strategy based on the update of the system status , the calculation formula is as follows: ， Among them, there are five reward items, namely 、 、 、 and , It represents the degree of success of a quantitative attack operation. represents the time cost of the attack operation, Indicates the security risks brought by attack operations. Indicates the system resources consumed by the attack operation. represents the state-action exploration reward, 、 、 、 and Respectively in the state Next reward 、 、 、 and Dynamic weight coefficient of State-Action Exploration Reward The calculation formula is as follows: ， in, represents the exploration reward base coefficient, Indicates the total number of attack actions executed in the history. express The index of Indicates the number of historical executions An attack action, Indicates the current attack action and The state-action space distance between represents the distance sensitivity adjustment factor; Determined through a multi-level security priority assessment model and security status entropy weight method , , An index representing the number of reward items, , The calculation formula is as follows: ， in, It also represents the index of the number of reward items. Indicates the The default basic weight of each reward item is Indicates status No. Dynamic adjustment factor for each reward item, Indicates status The state entropy of Indicates the The default basic weight of each reward item is Indicates status No. Dynamic adjustment factors for each reward item; state The calculation formula of the state entropy is as follows: ， in, Indicates the number of system status categories, Indicates status Belong to the probability of the class state; state Dynamic adjustment factor The calculation formula is: ， in, Indicates status Next An important indicator of the award, represents the normalization constant, and represents two different adjustment coefficients, Indicates the Adjustment coefficient of reward items , Indicates the Adjustment coefficient of reward items , Indicates the current attack progress indicator, Indicates the The historical success rate of each reward item; Historical success rate The calculation formula is as follows: ， in, Indicates that as of The total number of attack actions executed at the moment, express The index of Indicates the The set of related actions for each reward item, represents the indicator function, if ,but ,otherwise , Indicates the judgment Attack action Is it successful? If successful, If it fails, ; (4) The client will update the latest status , instant rewards , subtask completion indication information and auxiliary diagnosis information are sent back to the server in the form of data packets.

2. The intelligent security audit method based on hierarchical reinforcement learning according to claim 1 is characterized by: Hierarchical reinforcement learning including macro strategies and micro-strategies , macro strategy For high-level, micro-strategies in hierarchical reinforcement learning It is the lower layer in hierarchical reinforcement learning, where the higher layer defines the subtasks. and high-level goals , , executable micro-actions defined by the lower layer , , Indicates an attack action; The status of macro strategy in the system Select subtask , expressed as , micro-strategy in subtasks and the status of the system Select Micro Actions , expressed as ,state , Indicates the network environment, system configuration information, and current attack progress.

3. The intelligent security audit method based on hierarchical reinforcement learning according to claim 2 is characterized in that: Macro strategy selection subtask The process is as follows: Macro strategies analyze the current system status through threat intelligence information and historical audit records of the system , dynamically determine the high-level goals that need to be executed , then based on macro strategy Status in the system From high-level goals Select the appropriate subtask ; Threat intelligence information includes data sources from network detection systems, intelligence sharing platforms, and security vulnerability databases; High-level goals A series of subtasks composition, , Representing high-level goals Middle subtasks, Representing high-level goals The number of neutron tasks, ; The macro strategy calculates the macro value generated by the immediate reward through the macro value function and arranges the execution order and timing of the subtasks. The specific process is as follows: The formula for calculating macro value is as follows: ， in, express The discount factor at the moment, Indicates the expected value, Indicates the preset planning time steps, Indicates the status of the system Next subtask The macro value of Indicates The status of the system at all times Execute subtasks The immediate rewards generated after Then, according to the macro value, the priority is determined and the execution order and timing of subtasks are arranged. Specifically, the priority threshold is set. , filter out Subtasks As candidate subtasks, the candidate subtasks are then sorted according to The values ​​are sorted in descending order to obtain the priority sequence of the subtasks, among which the top ranked Higher values ​​have higher execution priority; Finally, according to the priority sequence of subtasks and the current resource status of the system, the system status is guaranteed. Under the premise of allowing subtasks to be executed, each candidate subtask is started in sequence, and updated in real time according to task feedback during execution .

4. The intelligent security audit method based on hierarchical reinforcement learning according to claim 3 is characterized in that: Micro-strategy selection micro-actions The process is as follows: In subtask and the status of the system Based on micro-strategy Select micro-actions , , attack action Including command execution, port and traffic detection, vulnerability exploitation, privilege escalation, and resource usage and concealment related actions; Micro-strategy by defining micro-value In each subtask Evaluate different micro-actions The execution value of the system is continuously updated with real-time feedback. The specific process is as follows: During the system initialization phase, for each subtask and all possible micro-movements , micro value are assigned random initial values; Status in the system Next, execute the subtask When the micro-strategy is based on the current micro-value The value of selects a micro action Execute and then use - Greedy strategy to balance exploration and exploitation, based on probability Select The action with the largest value, according to the probability Randomly select other actions, micro actions After being executed, the state of the system is determined by Transformed into , and receive instant feedback rewards After multiple iterations and continuous updates of immediate feedback rewards, the micro-strategy is in the state of the system and subtasks According to micro value The micro-actions selected by the value of tend to be optimal.

5. The intelligent security audit method based on hierarchical reinforcement learning according to claim 4 is characterized in that: Iteration and optimization of macro and micro strategies: (1) Update micro-strategy: The server responds to the data packet from the client. Perform strategy iteration and use value-based methods to analyze micro-values To update, the calculation process is as follows: ， in, Represents the learning rate used to control the ratio of fusion of new and old values, represents the discount factor, Indicates that all micro actions get the maximum Candidate actions for values, express The subtask selected at the moment, The value of attack actions representing the optimal micro-strategy; (2) Update macro strategy: When the service segment completes or switches subtasks, it iterates the strategy based on the benefit and risk information of the macro strategy, and the immediate reward of the macro strategy , the calculation formula is as follows: ， in, Indicates the degree of coverage improvement of unknown network nodes, ports and potential vulnerability areas. It represents the degree of success of a quantitative attack operation. Indicates the risk of measuring attack actions at the macro level, Represents the cost of performing an attack action; When completing or switching subtasks, the macro value is updated based on the immediate rewards of the macro strategy; (3) Set the iteration stop condition: Set the value function change threshold, when the micro-strategy and macro strategies When the update amplitude is lower than the given threshold, the micro-strategy layer is considered stable, as follows: Define update amplitude thresholds for micro strategies and macro strategies respectively. Set micro thresholds for the micro strategy layer. , set macro thresholds for macro strategies ; In each iteration, the status of all systems , subtasks and micro-movements , calculate the maximum absolute change before and after the micro value update , the calculation formula is as follows: ， in, Indicates based on Calculate the maximum value, represents the updated micro value, Indicates the micro value before updating; At the same time, for all system states , calculate the maximum absolute change before and after the macro value update , the calculation formula is as follows: ， in, Indicates based on Calculate the maximum value, represents the updated macro value, Indicates the macro value before the update; If in continuous In all iterations, and , it is considered that the strategy update has stabilized and the iteration is terminated; The training process is also terminated when the set security audit objectives have been achieved or the available resources are insufficient to continue performing large-scale audits.

6. An intelligent security audit system based on hierarchical reinforcement learning, characterized by: It includes a module for executing the processing instructions of each step in the intelligent security audit method based on hierarchical reinforcement learning described in any one of claims 1-5.

7. An electronic device that executes the intelligent security audit method based on hierarchical reinforcement learning as described in any one of claims 1 to 5, characterized in that: The device includes: a memory, a processor, and a program stored in the memory and capable of running on the processor an intelligent security audit method based on hierarchical reinforcement learning.

8. A memory, characterized in that: The memory stores a program of an intelligent security audit method, and when the program of the intelligent security audit method is executed by the processor, an intelligent security audit method based on hierarchical reinforcement learning as described in any one of claims 1 to 5 is implemented.

Images