Multi-stage penetration agent training method based on a verifiable penetration range environment

Abstract

The application relates to a multi-stage penetration agent training method based on a verifiable penetration target range environment, applied to the field of network security, comprising the following steps: configuring a target range container in a target range container environment, and starting a target range instance; according to a first flag randomly generated, executing an operation instruction issued by a teacher agent to perform target range instantiation verification on the target range container environment; and disconnecting the target range container from the target range network; in a temporary container environment, executing a vulnerability exploitation script submitted by the teacher agent to obtain a script output result; the temporary container environment is injected with a second flag different from the first flag; when it is determined that the script output result contains the second flag, the penetration trajectory generated by the teacher agent is determined based on a preset environment verification strategy in the temporary container environment; and the penetration trajectory is used as a supervised fine-tuning data set to train the penetration agent, so that objective verification of the correctness of the teacher agent answer is realized.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning, and in particular to a multi-stage penetration agent training method based on a verifiable penetration range environment. Background Technology

[0002] A Large Language Model (LLM) is a deep learning model trained on a large amount of text data. This model can generate natural language text or understand the meaning of language text. To further improve the cybersecurity of LLMs, penetration testing is required to simulate attack behavior and detect vulnerabilities. This is typically done using a penetration testing agent.

[0003] Traditional methods primarily rely on demonstration trajectories generated by teacher agents as training data to supervise and fine-tune the penetration testing agent. However, the training data may contain seemingly reasonable trajectories that do not actually penetrate the target area, causing the model to only imitate surface patterns after training on such data and fail to learn real vulnerability exploitation capabilities. Therefore, existing penetration testing strategies lack objective verification methods to validate the correctness of the teacher agent's answers.

[0004] There is currently no effective solution to the problem that penetration testing strategies in related technologies lack objective verification methods to verify the correctness of teacher agents' answers. Summary of the Invention

[0005] This embodiment provides a multi-stage penetration agent training method based on a verifiable penetration test environment to address the problem in related technologies that there is a lack of objective verification methods to verify the correctness of teacher agent answers in penetration testing strategies.

[0006] Firstly, this embodiment provides a multi-stage penetration agent training method based on a verifiable penetration range environment, the method comprising:

[0007] In the target range container environment, start the target range instance and randomly generate a first flag; according to the first flag, execute the operation instructions issued by the teacher agent to perform target range instantiation verification on the target range container environment; and disconnect the target range container from the target range network.

[0008] In the temporary container environment, the vulnerability exploit script submitted by the teacher agent is executed, and the script output is obtained; the configuration method of the temporary container environment is the same as that of the target range container environment; a randomly generated second flag is injected into the temporary container environment; the string corresponding to the second flag is different from that of the first flag;

[0009] If the script output contains the second flag, the penetration trajectory generated by the teacher agent is determined based on the preset environment verification strategy in the temporary container environment.

[0010] The penetration trajectory is used as a supervised fine-tuning dataset to train the penetration agent in both the target range container environment and the temporary container environment.

[0011] In some embodiments, executing the operation instructions issued by the teacher agent based on the first flag to perform range instantiation verification of the range container environment includes:

[0012] Inject the first flag into the target range container environment;

[0013] The system receives and executes operation instructions issued by the teacher's intelligent agent through a pre-defined unified penetration interaction interface, and outputs structured results; the structured results are associated with the type of the operation instructions.

[0014] When the type of the operation instruction is flag submission, the flag string in the structured result is obtained;

[0015] The flag string in the structured result is compared with the string corresponding to the expected flag to obtain a binary verification result; the expected flag is generated after the first flag is injected into the target range container environment.

[0016] Based on the binary verification results, verify whether the target range container environment is isolated.

[0017] In some embodiments, determining the penetration trajectory generated by the teacher agent based on a preset environment verification strategy in the temporary container environment, when it is determined that the script output contains the second flag, includes:

[0018] Based on the exploit script and the script output, a binary deterministic signal is generated; the binary deterministic signal is used to characterize whether the exploit script has cross-flag generalization capability.

[0019] If the exploit script does not involve the second flag, but the script output includes the second flag, a binary deterministic signal is generated; the binary deterministic signal includes the binary deterministic signal.

[0020] Based on a preset environment verification strategy, the penetration trajectory is obtained by filtering all trajectories generated by the teacher agent according to the binary determination signal.

[0021] In some embodiments, the step of filtering all trajectories generated by the teacher agent based on a preset environment verification strategy and the binary determination signal to obtain the penetration trajectory includes:

[0022] Obtain the round-by-round penetration trajectory of the teacher agent in the target range container environment for each penetration interaction, and the cumulative number of interaction rounds corresponding to each round-by-round penetration trajectory;

[0023] Based on a preset environment verification strategy, the penetration trajectory generated by the teacher agent in each round is filtered to obtain a penetration trajectory that conforms to the environment verification result; the preset environment verification strategy includes the binary verification result, the cumulative number of interaction rounds, and the binary deterministic signal.

[0024] In some embodiments, step S110, configuring the range container in the range container environment and starting the range instance, includes:

[0025] The range data in the range warehouse is parsed to obtain range data entries; the range data entries include the task information and service port information of each range; the task information includes task description text;

[0026] Launch the target range instance corresponding to the target range data entry.

[0027] In some embodiments, the penetration trajectory is used as a supervised fine-tuning dataset to train the penetration agent in both the target range container environment and the temporary container environment, including:

[0028] The penetration trajectory is converted into training samples in a preset format, with the task description text as input text and the reasoning text and tool call text in the penetration interaction process as output text, to construct a supervised fine-tuning dataset;

[0029] Based on a preset loss function, the penetration agent is subjected to supervised fine-tuning on the supervised fine-tuning dataset to obtain a preheating strategy model.

[0030] The preheating strategy model is loaded as the initial strategy, and strategy deduction is performed in the target range container environment and the temporary container environment respectively to obtain the deduction results; the parameters of the preheating strategy model are updated based on the deduction results.

[0031] In some embodiments, loading the preheating strategy model as an initial strategy, performing strategy deduction in the target range container environment and the temporary container environment respectively, and obtaining deduction results; updating the parameters of the preheating strategy model based on the deduction results includes:

[0032] The preheating strategy model is loaded as the initial strategy, and the strategy inference is performed in parallel for each sample in the supervised fine-tuning dataset under the same target range container environment and the same flag to obtain the inference results; the inference results include the binary verification results and the binary deterministic signal;

[0033] The binary verification result is assigned a first weight, and the binary deterministic signal is assigned a second weight.

[0034] If either the binary verification result or the binary deterministic signal reaches a preset value, the reward signal corresponding to the deduction result is scaled to obtain the final reward.

[0035] Based on the final reward, the parameters of the preheating strategy model are updated to update the penetration agent.

[0036] Secondly, this embodiment provides a multi-stage penetration agent training device based on a verifiable penetration range environment. The device includes: a first verification module, a second verification module, a screening module, and a training module.

[0037] The first verification module is used to start a target range instance in the target range container environment and randomly generate a first flag; according to the first flag, execute the operation instructions issued by the teacher agent to perform target range instantiation verification on the target range container environment; and disconnect the target range container from the target range network.

[0038] The second verification module is used to execute the vulnerability exploit script submitted by the teacher agent in a temporary container environment and obtain the script output result; the configuration method of the temporary container environment is the same as that of the target range container environment; a randomly generated second flag is injected into the temporary container environment; the string corresponding to the second flag is different from that of the first flag;

[0039] The filtering module is used to determine the penetration trajectory generated by the teacher agent based on a preset environment verification strategy in the temporary container environment, when it is determined that the script output contains the second flag.

[0040] The training module is used to train the penetration agent in the target range container environment and the temporary container environment, respectively, using the penetration trajectory as a supervised fine-tuning dataset.

[0041] Thirdly, this embodiment provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-stage penetration agent training method based on a verifiable penetration range environment described in the first aspect.

[0042] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the multi-stage penetration agent training method based on a verifiable penetration range environment described in the first aspect.

[0043] Compared with related technologies, this embodiment provides a multi-stage penetration agent training method based on a verifiable penetration range environment. This method achieves replayable verification mechanisms and flag matching under different isolation environments by performing target range instantiation verification in a target range container environment and verification including flag matching in a temporary container environment. Based on the verification results of replayable verification and flag matching, combined with environmental verification strategies, a supervised fine-tuning dataset for training the penetration agent is generated, enabling objective training of the penetration agent. Through verification strategies under different isolation environments, objective and accurate verification signals can be generated. These verification signals undergo objective quality verification within the target range environment, ensuring that every trajectory learned by the penetration agent has been tested by the objective environment, rather than solely based on the subjective output of the teacher agent.

[0044] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a flowchart of a multi-stage penetration agent training method based on a verifiable penetration range environment provided in this application embodiment;

[0047] Figure 2 This is a hardware structure block diagram of the terminal for the multi-stage penetration agent training method based on a verifiable penetration range environment provided in this embodiment.

[0048] Figure 3 This is a schematic diagram of the intelligent agent training system provided in this specific embodiment;

[0049] Figure 4 This is a flowchart of a multi-stage penetration agent training method based on a verifiable penetration range environment provided in this specific embodiment;

[0050] Figure 5 This is a flowchart of the replayable verification process provided in this specific embodiment;

[0051] Figure 6 This is a schematic diagram of reinforcement learning reward calculation provided in this specific embodiment;

[0052] Figure 7 This is a schematic diagram of the progressive learning process of the course difficulty provided in this specific embodiment;

[0053] Figure 8 This is a structural block diagram of the multi-stage penetration agent training device based on a verifiable penetration range environment in this embodiment.

[0054] Reference numerals: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 10, first verification module; 20, second verification module; 30, screening module; 40, training module. Detailed Implementation

[0055] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0056] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0057] In existing technologies, large language models serve as the "intelligent brain" of penetration testing agents, providing them with core intelligent driving forces such as task planning, reasoning, and tool invocation, enabling them to autonomously simulate hacker attacks and execute penetration tests. For example, they can simulate cybersecurity attacks, continuously and efficiently discover new vulnerabilities and generate reports, making up for the shortcomings of manual testing, which is infrequent and relies on experience.

[0058] However, training large language models to penetrate agents faces two fundamental difficulties: First, in the supervised fine-tuning stage, the model needs to learn demonstration trajectories generated by the teacher model. However, the "correct actions" corresponding to these trajectories lack objective verification methods, and may contain samples that appear reasonable but have not actually breached the target. If the model is trained on such data, it will only imitate the surface patterns of the trajectories and will not learn real vulnerability exploitation capabilities. Second, in the reinforcement learning stage based on expert feedback, an approximate reward model is usually trained to replace human judgment. However, this approximate model has an inherent defect of insufficient out-of-distribution generalization (i.e., its performance drops significantly in unknown environments outside the training data distribution because it learns "spurious associations" related to the data distribution, rather than stable causal features). This defect further allows policy models to obtain high scores by exploiting the bias of the approximate reward model, i.e., the phenomenon of "reward hacking," which causes the policy optimization direction to deviate from the true goal.

[0059] Therefore, in order to address the technical problem of unverifiable teacher trajectories during the training of large language model penetrating agents, which leads to the model merely imitating surface text patterns and approximating rewards through reinforcement learning, resulting in blind spots in the model and the "reward hacking" phenomenon caused by the model exploiting rule loopholes.

[0060] This embodiment provides a multi-stage penetration agent training method based on a verifiable penetration range environment. By introducing a dynamic flag comparison and replayable verification mechanism for vulnerability exploitation scripts in a capture-the-flag competition range, the objective, binary real environment feedback signal is used simultaneously as a data quality filter in the supervised fine-tuning stage and a reward signal in the reinforcement learning stage. This fundamentally eliminates the dependence on manual annotation and approximate reward models, effectively prevents the agent from memorizing flags and engaging in pseudo-success behavior, and achieves a robust improvement in the penetration agent's realistic and generalizable vulnerability exploitation capabilities. Figure 1 This is a flowchart of a multi-stage penetration agent training method based on a verifiable penetration range environment provided in this application embodiment, such as... Figure 1 As shown, the process includes the following steps:

[0061] Step S110: Configure the range container in the range container environment and start the range instance; randomly generate a first flag; execute the operation instructions issued by the teacher agent according to the first flag to verify the range instantiation of the range container environment; and disconnect the range container from the range network.

[0062] Since the capabilities of the penetrating agent corresponding to the small model are limited, direct training may be inefficient. Therefore, a teacher agent is first used, such as a closed-source large model, to generate trajectory data in the target range container. Then, the penetrating agent is supervised and fine-tuned through the trajectory data to obtain an initial direction before entering the real target range environment for reinforcement learning.

[0063] First, start the target instance in the target container environment and dynamically generate a unique random flag string, i.e., the first flag. The random flag generated each time the target instance in the target container environment is started must be different to eliminate the possibility of flag memory.

[0064] Furthermore, the first flag also satisfies the global uniqueness constraint. The first flag is synchronously written to two fields in the same target instance: the injection parameter field for writing the first flag to the target container, the reachable path from the first flag to the target container, and the truth value field of the reward model for writing the first flag to the current penetration round, so as to facilitate comparison during subsequent reward calculation.

[0065] After starting the range instance, the operation instructions sent by the teacher agent are executed in the isolated range container environment. Based on the structured results corresponding to the operation instructions, the range instantiation verification of the range container environment is then completed.

[0066] After completing this target range instantiation verification in the target range container environment, follow the reverse steps of the startup phase to disconnect the target range container from the target range network, so as to ensure that the target range container environments are completely isolated between different penetration rounds and will not affect each other.

[0067] Step S120: In the temporary container environment, execute the vulnerability exploit script submitted by the teacher agent and obtain the script output.

[0068] The configuration method of the temporary container environment is the same as that of the target range container environment. A randomly generated second flag is injected into the temporary container environment; the string corresponding to the second flag is different from that of the first flag.

[0069] After the penetration interaction in the target environment container is completed, the target instance in the target environment container environment needs to be stopped, and a new second flag, different from the first flag, needs to be used to rebuild the temporary container environment according to the same configuration method as the target environment container environment. Subsequently, replayable verification is performed in the temporary container environment based on the second flag to obtain the script output results. Based on the script output results, objective filtering signals are provided for the penetration trajectory generated by the teacher agent during the penetration interaction.

[0070] Furthermore, the reconstructed temporary container environment is completely separated from the original target range container environment at the flag value level, but is completely consistent at the network topology, service configuration, and vulnerability environment levels.

[0071] Step S130: If the script output contains the second flag, determine the penetration trajectory generated by the teacher agent based on the preset environment verification strategy in the temporary container environment.

[0072] When the script output contains a second flag, it indicates that the teacher agent has read and output the second flag using the target range vulnerability. This means that the vulnerability exploit script submitted by the teacher agent has the ability to generalize vulnerability exploitation across flags. At this point, the script output of the vulnerability exploit script and the preset environment verification strategy can be combined to filter the trajectory generated by the teacher agent during the penetration interaction process, so as to obtain a penetration trajectory that meets the requirements of vulnerability exploitation capability and conforms to the environment verification strategy.

[0073] This step, through the script output and the preset environment verification strategy, can act as a quality filter to select the penetration trajectory generated by the teacher agent during the penetration interaction process. Each penetration trajectory has been tested by the objective environment, rather than being the subjective output of the teacher agent, which further improves the accuracy of the teacher trajectory used as training data.

[0074] Step S140: Using the penetration trajectory as a supervised fine-tuning dataset, the penetration agent is trained in both the target range container environment and the temporary container environment.

[0075] Specifically, the penetration trajectory is used as a supervised fine-tuning dataset, and the flags and exploit scripts in the interaction process of the penetration agent are trained sequentially in the target range container environment and the temporary container environment.

[0076] Furthermore, the target range container is equipped with a unified penetration interaction interface, providing a unified target range context for the teacher agent to include the target host address, port, and task description. The target range container receives penetration instructions issued by the teacher agent, executes them in an isolated target range container, and returns structured results, enforcing the maximum number of interaction rounds and total time limits.

[0077] Through the above steps, a replayable verification mechanism and flag matching are implemented under different isolation environments by performing target range instantiation verification in a target range container environment and verification including flag matching in a temporary container environment. Then, based on the verification results of replayable verification and flag matching, combined with the environment verification strategy, a supervised fine-tuning dataset for training the infiltration agent is generated, achieving objective training of the infiltration agent. The verification strategy under different isolation environments generates objective and accurate verification signals, which are objectively quality-verified by the target range environment, ensuring that every trajectory learned by the infiltration agent has been tested by the objective environment, rather than solely based on the subjective output of the teacher agent.

[0078] In some embodiments, in step S120, according to the first flag, the operation instructions issued by the teacher agent are executed to perform range instantiation verification of the range container environment, including:

[0079] Inject the first flag into the target range container environment;

[0080] Through a pre-defined unified interactive interface, it receives and executes operation instructions issued by the teacher's intelligent agent, and outputs structured results; the structured results are associated with the type of operation instructions.

[0081] When the operation command type is flag submission, retrieve the flag string from the structured result;

[0082] The flag string in the structured result is compared with the string corresponding to the expected flag to obtain the binary verification result; the expected flag is generated after the first flag is injected into the target container environment.

[0083] Based on the binary verification results, verify whether the target range container environment is isolated.

[0084] After the target instance starts, the target container receives operation commands from the teacher agent through a unified penetration testing interface, and routes them to the corresponding execution logic based on the type of the operation command. Operation commands generally include command execution, flag submission, and exploit script submission.

[0085] When the instruction type is flag submission, the flag string submitted by the agent is recorded. And the interaction round number t at the submission time. At the same time, a strict equality comparison is performed between the flag string and the string corresponding to the expected flag to obtain the binary verification result; the expected flag here is generated after the first flag is injected into the parameter field of the target container environment.

[0086] Subsequently, based on the binary verification results, it is verified whether the target range container environment has been isolated to prevent subsequent agents from fraudulently obtaining rewards by submitting only part of the flag content.

[0087] After obtaining the binary verification result, disconnect the target container from the target network in the reverse order of the startup phase, stop each target container in the target container environment in turn, clear all states generated at the target container runtime level in this round, including but not limited to the container file system layer, memory state and process tree, and destroy the target dedicated isolation network and related network namespace resources.

[0088] In some embodiments, in step S130, if it is determined that the script output contains a second flag, the penetration trajectory generated by the teacher agent is determined based on a preset environment verification strategy in the temporary container environment, including:

[0089] Based on the exploit script and its output, a binary deterministic signal is generated; the binary deterministic signal is used to characterize whether the exploit script has cross-flag generalization capability.

[0090] If the exploit script does not involve a second flag, but the script output contains a second flag, a binary deterministic signal is generated; a binary deterministic signal includes a binary deterministic signal.

[0091] Based on a pre-defined environment verification strategy, and using binary deterministic signals, all trajectories generated by the teacher agent are filtered to obtain the penetration trajectory.

[0092] In the process of instantiating the teacher agent in the target environment and then replaying the penetration interaction trajectory of the teacher agent in the reconstructed temporary container environment, it is necessary to further replay the vulnerability exploit script submitted by the teacher agent and verify whether the script has the ability to generalize across flags based on the script output results of the vulnerability exploit script.

[0093] Based on the binary deterministic signal generated by the replayable verification, the quality filter for all trajectories that perform pervasive interactions as the teacher agent during the supervised fine-tuning stage is used. Only pervasive trajectories that pass the replayable verification are included in the training dataset, ensuring that every trajectory learned by the model has been tested by the objective environment.

[0094] In some embodiments, based on a preset environmental verification strategy and according to a binary determination signal, all trajectories generated by the teacher agent are filtered to obtain penetration trajectories, including:

[0095] Obtain the penetration trajectory of the teacher agent in each penetration interaction in the target range container environment, as well as the cumulative number of interaction rounds corresponding to each penetration trajectory;

[0096] Based on the preset environment verification strategy, the penetration trajectory generated by the teacher agent in each round is screened to obtain the penetration trajectory that meets the environment verification results; the preset environment verification strategy includes binary verification results, cumulative number of interaction rounds, and binary deterministic signals.

[0097] To ensure the quality of the supervised fine-tuning dataset, all trajectories generated by the collected teacher agents are sequentially filtered by flag verification, replayability of exploit scripts, interaction efficiency, and command legality. Trajectories that pass all of the above filtering criteria constitute a high-quality trajectory set, namely the supervised fine-tuning dataset.

[0098] Flag verification screening is mainly conducted by determining the consistency of flags based on the above binary verification results. Vulnerability exploit script replayability screening is conducted by checking whether the script output results represented by binary deterministic signals contain a second flag. Interaction efficiency screening is mainly conducted by the number of penetration interaction rounds. Command legitimacy screening is mainly conducted by checking whether the operation commands have out-of-bounds behavior.

[0099] In some embodiments, configuring a range container within a range container environment and launching a range instance includes:

[0100] The range data in the range warehouse is parsed to obtain range data entries; the range data entries include the task information and service port information of each range; the task information includes the task description text;

[0101] Start the target range instance corresponding to the target range data entry.

[0102] First, the capture-the-flag competition range repository is scanned, and the question descriptions, difficulty levels, and service port information of each range are parsed and converted into standardized range data entries. Each range data entry contains task description text and parameters required to start the range, and then the range instance is started.

[0103] Furthermore, each range data entry contains the following core fields: range unique identifier, mission description text, difficulty level, container orchestration configuration path, network configuration parameters, and resource quota limit, etc.

[0104] In some embodiments, the penetration trajectory is used as a supervised fine-tuning dataset to train the penetration agent in both a target range container environment and a temporary container environment, including:

[0105] The penetration trajectory is converted into training samples in a preset format, with the task description text as the input text and the reasoning text and tool call text in the penetration interaction process as the output text, to construct a supervised fine-tuning dataset.

[0106] Based on a preset loss function, the penetrating agent is subjected to supervised fine-tuning on a supervised fine-tuning dataset to obtain a preheating strategy model.

[0107] The preheating strategy model is loaded as the initial strategy, and the strategy simulation is performed in both the target range container environment and the temporary container environment to obtain the simulation results; the parameters of the preheating strategy model are updated based on the simulation results.

[0108] First, the validated trajectories are formatted into a supervised fine-tuning dataset and the target language model, i.e., the penetrating agent, is subjected to supervised fine-tuning to obtain a warm-up policy model. The supervised fine-tuning warm-up stage provides a high-quality initial policy for reinforcement learning, thereby alleviating the difficulty of reinforcement learning in exploring highly sparse reward environments.

[0109] Subsequently, the preheating strategy model was loaded as the initial strategy, and the strategy inference was performed in parallel for each sample in the supervised fine-tuning dataset under the same target range container environment and the same flag, to obtain the inference results; the inference results include binary verification results and binary deterministic signals;

[0110] The binary verification results are assigned a first weight, and the binary deterministic signal is assigned a second weight.

[0111] If either the binary verification result or the binary deterministic signal reaches a preset value, the reward signal corresponding to the deduction result is scaled to obtain the final reward.

[0112] Based on the final reward, update the parameters of the preheating strategy model to update the penetration agent.

[0113] Specifically, the reward signal corresponding to the inference result is scaled based on the efficiency adjustment coefficient; the efficiency adjustment coefficient is determined based on the actual number of interaction rounds and the maximum number of interaction rounds of the penetrating agent in the penetration interaction process.

[0114] Furthermore, based on the final reward, the relative advantage of the same group inference is calculated through the Group Relative Policy Optimization (GRPO) algorithm, and the parameters of the preheating policy model are updated to achieve the training of the penetration agent.

[0115] In some embodiments, the target range data items are stratified and sampled according to their difficulty level, dividing them into non-overlapping training and evaluation sets with consistent difficulty distribution. The training set is used in the training process described in the above embodiments.

[0116] The warm-up strategy model in the above training phase is periodically switched to the evaluation mode and subjected to independent blind testing on the evaluation set target range. The evaluation results are not included in the policy gradient update. The target range sampling distribution of the next round of reinforcement learning training is automatically adjusted according to the evaluation results, so that the training resources are concentrated in the weak direction. The cycle is iterated until the standardized score on the evaluation set reaches the preset threshold.

[0117] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 2 This is a hardware structure block diagram of the terminal for the multi-stage penetration agent training method based on a verifiable penetration range environment provided in this embodiment. (See diagram for example.) Figure 2 As shown, a terminal may include one or more ( Figure 2Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown are illustrated.

[0118] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multi-stage penetration agent training method based on a verifiable penetration range environment in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0119] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0120] The following describes and illustrates this embodiment through specific examples.

[0121] Figure 3 This is a schematic diagram of the intelligent agent training system provided in this specific embodiment, for reference. Figure 3The system comprises an agent and training layer, an interface layer, an orchestration and verification layer, and a container execution layer. The agent and training layer includes a teacher agent and a penetration agent, as well as a policy model. The teacher agent generates verifiable trajectories (penetration trajectories as described in the previous embodiments), and the penetration agent performs multiple rounds of penetration simulations based on these trajectories to achieve training. The interface layer includes a unified penetration interaction interface for interaction between the agent and the training layer, orchestration and verification layer, and container execution layer. The orchestration and verification layer includes a range repository and a range orchestration controller, as well as an evaluation / scheduler. The range orchestration controller is used to launch range instances, generate isolated networks, inject dynamic flags, perform periodic cleanup, perform precise flag comparison, provide replayable verification, and output binary verification signals, thus implementing the multi-stage penetration agent training method described in the previous embodiments. The container execution layer includes the range container (environment) group and temporary container environment described in the previous embodiments, as well as a penetration testing container. The range container environment executes agent commands, and the temporary container environment runs exploit scripts submitted by the agent.

[0122] Figure 4 This is a flowchart of a multi-stage penetration agent training method based on a verifiable penetration range environment provided in this specific embodiment. This method simulates cybersecurity attacks, and based on the trained penetration agent, continuously and efficiently discovers new vulnerabilities and generates reports, overcoming the shortcomings of low-frequency and experience-dependent manual testing. This method is applied to the aforementioned agent training system, refer to... Figure 4 The method includes:

[0123] Step S410, target range data preprocessing.

[0124] Specifically, the capture-the-flag competition range repository is first scanned, and the question descriptions, difficulty levels, and service port information of each range are analyzed and converted into standardized range data entries. Each entry contains a task description text and the parameters required to start the range instance. The range data entries are then stratified and sampled according to difficulty level, and divided into non-overlapping training sets and evaluation sets, with the difficulty distribution of the two sets remaining consistent.

[0125] Target range repositories are typically organized in a directory tree structure, with each subdirectory corresponding to a target range challenge. Each subdirectory contains a container orchestration configuration file, a challenge description file, a difficulty labeling file, and a service port configuration file. The scanning program recursively traverses the repository directory tree, extracting the following information for each qualifying target range entry:

[0126] The task description text includes background information about the target range, a description of the attack target, and necessary hints, used to convey the mission intent to the penetration agent.

[0127] Difficulty level labeling: A hierarchical labeling system is adopted, which is usually divided into beginner, intermediate, advanced and expert levels, for the target range stratification in the course learning stage;

[0128] Service port information: includes the service type, port number and transport layer protocol exposed by the target range, which is used to guide the penetration agent to determine the initial attack surface.

[0129] After being structured, the above information generates a corresponding range specification description file, which is then further converted into standardized range data entries. Each range data entry contains the following core fields: range unique identifier e, mission description text d, difficulty level l, container orchestration configuration path, network configuration parameters, and resource quota limit.

[0130] Complete all range data entries After construction, it is divided into training sets according to the principle of non-overlapping. With the evaluation set ,satisfy:

[0131] ;

[0132] When dividing the data, it is important to ensure that the two sets are balanced in terms of difficulty level distribution and vulnerability type distribution to prevent the evaluation results from being distorted due to differences in set distribution.

[0133] Preferably, for each difficulty level l, stratified random sampling is used to allocate a certain proportion of the target range entries to the evaluation set, with the remainder assigned to the training set. The training set is dedicated to providing the target range environment during the reinforcement learning policy optimization phase, while the evaluation set is dedicated to independent blind testing of the policy model's generalization ability. The two sets are strictly isolated at the target range content level; no target range in the evaluation set may appear in any form during the training phase to ensure that the evaluation results accurately reflect the policy model's generalization ability to unseen target ranges, rather than memorization or overfitting of the training target ranges.

[0134] Step S420 verifies the instantiation of the target range.

[0135] Specifically, a unique random flag string is dynamically generated each time a target instance is started. This flag is then synchronously injected into the target container environment and the verification truth field to ensure that the flag is different for each instantiation, thus eliminating the possibility of flag memory from a mechanism perspective.

[0136] Before each target range instance is started, a random string generator is called to generate a unique random flag f, which is the first flag in the aforementioned embodiment. Its format satisfies the global uniqueness constraint.

[0137] ;

[0138] Where u represents a random variable; ∼ indicates "follows a distribution of ..."; Uniform() indicates a uniform distribution. {0, 1} 128 This represents the set of all binary sequences (bit strings) of length 128.

[0139] The generated flag f is synchronously written to the following two fields during the same operation: 1. Parameter field for injecting the target range container. The target range container reads this parameter upon startup and stores f in the reachable path of the target range; secondly, it stores the truth field f* of the reward model for this round, for comparison during subsequent reward calculations. The above two writes originate from the same generation operation. This can be expressed by the formula:

[0140] ;

[0141] This constraint fundamentally eliminates the possibility of flag memory: since the flag f for each range instance is generated independently and randomly, any strategy that relies on remembering historical flag strings has a statistically close probability of success to zero.

[0142] Verifiable test range instances ensure the integrity of the test range environment and strict isolation between rounds by strictly managing the following four lifecycle phases.

[0143] 1. Startup Phase: The target container receives the target identifier e and dynamic flag f, and retrieves the corresponding container orchestration configuration file based on the target identifier. According to the container orchestration configuration file, it sequentially pulls the required container images and completes image verification, thereby constructing a dedicated isolated network for the target. It then starts each target container in the target container group according to the topological order of service dependencies. After the target container group has started, the flag f is injected into the runtime environment of the target container through the container runtime interface and written to the preset storage location of the target challenge. At this point, it is confirmed that the penetration testing attack container has completed initialization; it connects to the isolated network of the target through the container network interface, establishing network reachability between the penetration testing container and the target. After the startup phase is completed, it returns a ready status flag, the target host address h, and the service port p for use by the unified penetration testing interaction interface.

[0144] 2. Interaction Phase: The target container continuously receives operation commands issued by the teacher agent through the unified penetration interaction interface, and routes them to the corresponding execution logic according to the command type.

[0145] When the instruction type is command execution, the command is executed in the working directory of the penetration testing container, capturing the process exit code 'r' and the standard output stream content. and standard error stream content , the triple (r, , The structured return result is passed to the teacher agent.

[0146] When an exploit script is submitted, its content is persistently stored for later replay verification. A maximum interaction round limit is enforced during the penetration testing phase. Total time limit If either the maximum number of interaction rounds or the total time is reached, the current interaction will be automatically terminated and the reason for termination will be recorded.

[0147] When the instruction type is flag submission, the flag string submitted by the agent is recorded. And the interaction round number t at the submission time. At the same time, a strict equality comparison is performed between the flag string and the string corresponding to the expected flag to obtain the binary verification result; the expected flag here is generated after the first flag is injected into the parameter field of the target container environment.

[0148] 3. Evaluation Phase: The test range container will display the flags submitted by the agent. That is, the flag string in the aforementioned embodiments is compared precisely with the expected flag f* injected during the startup phase, and a binary verification result is output. :

[0149] ;

[0150] The comparison uses strict equivalence judgment and does not allow fuzzy matching or prefix matching to prevent strategy models from deceiving the reward by submitting only part of the flag content.

[0151] 4. Cleanup Phase: Following the reverse order of the startup phase, disconnect the target environment container environment from the target network, sequentially stop each container in the target container group, clear all states generated at the target container runtime level during this penetration round (including container file system layer, memory state, and process tree), and destroy the target dedicated isolation network and related network namespace resources. After the cleanup phase is completed, the host machine is restored to the clean state before the round started, ensuring complete isolation of the execution environment between different rounds, and any residual state from previous rounds will not affect subsequent rounds.

[0152] Step S430: Unify the penetration interaction interface.

[0153] For details, please refer to Figure 3 A unified penetration testing interface is set up to provide a unified target environment context for the penetration agent, including the target host address, port, and task description. The target environment container receives penetration commands issued by the penetration agent, executes them in an isolated target environment container environment, and returns structured results, enforcing a maximum number of interaction rounds and a total time limit. The penetration agent here includes the teacher agent in the above embodiments.

[0154] By providing a unified target context structure for the teacher agent, which is constructed by the orchestration layer and passed to the teacher agent at the start of each round, the target context contains the following fields: target identifier e, target host address h, and a list of service ports. The system includes the corresponding service type label, task description text (d), and working directory path. After receiving the target environment context, the teacher agent uses this as initial information to autonomously decide on the penetration path.

[0155] The reception and execution of penetration commands follow a unified interface specification. The teacher agent issues commands in a structured format, comprising an action type field and an action parameter field. After parsing the command, the interface layer routes it to the command execution logic, flag submission logic, or exploit script submission logic based on the action type field. All penetration commands are executed within the isolated network of the penetration testing container, i.e., the target range container in the aforementioned embodiments. The penetration testing container communicates with the target range target through its internal network, maintaining strict network-level isolation from the host machine and other containers operating in concurrent rounds.

[0156] The interface layer is also responsible for the collection and recording of telemetry data. The command content, execution time, return result length, and action type for each round of interaction are recorded as a round-level telemetry log for subsequent trajectory quality analysis and efficiency screening. When the cumulative number of interaction rounds for this round reaches... Or the cumulative execution time reaches When this happens, the interface layer forcibly interrupts the current round and returns a timeout notification to the agent to prevent the training process from being blocked indefinitely in a single round.

[0157] Step S440 can be replayed for verification.

[0158] Specifically, Figure 5 This is a flowchart of the replayable verification provided in this specific embodiment, for reference. Figure 5 After the penetration interaction ends, the current penetration phase target container group is first stopped, and the target container is rebuilt using a new random flag different from the penetration phase. Replayability verification is the core guarantee mechanism for verifying the reliability of verification signals in the penetration agent training method. The motivation for this mechanism is that flag matching alone is insufficient to completely eliminate the phenomenon of "false success," meaning that the penetration agent may submit the correct flag during the penetration process through a non-exploitation path, but has not truly mastered the generalizable exploit capability. Replayability verification verifies whether the exploit script submitted by the agent has cross-flag generalization capability by replaying the exploit script in a completely new flag environment, thus providing a second layer of objective verification signal for the quality screening of training data and the reward calculation of reinforcement learning.

[0159] During the target range reconstruction phase, a cleanup is triggered to completely stop and destroy the target range container group and its network resources used in the current penetration phase, ensuring that all states of the old target range instance have been cleared. Subsequently, a new random flag f2, independent of the penetration phase flag f, is dynamically generated (the second flag in the aforementioned embodiment), satisfying f2≠f. The target range container is rebuilt and started using the new flag f2, i.e., the temporary target range environment in the aforementioned embodiment is rebuilt. This involves re-executing the complete startup phase process according to the original target range specifications, building a completely new target range container group, and injecting f2 into the corresponding storage location of the target range runtime environment. This ensures that the rebuilt target range instance is completely separated from the original penetration environment at the flag value level, while being completely consistent with the original target range at the network topology, service configuration, and vulnerability environment levels.

[0160] If the exploit script The reason for the success in the original target range is that the specific string of f was remembered rather than the vulnerability was actually exploited. Therefore, the execution result in the reconstructed target range will contain f instead of f2, and thus be judged as failing by the verification logic.

[0161] During the isolation execution phase, the exploit script is executed in a restricted temporary container. An independent temporary container environment is created, located on the same isolated network as the target environment to ensure network reachability. The exploit script is then executed within this isolated environment. The file volume is mounted as a temporary container in read-only mode, and the target host address h for reconstructing the test range is used as the command-line argument to execute the script within the temporary container. To prevent scripts from abusing the execution environment during replay, the following resource and permission constraints are imposed on the temporary container: maximum memory usage. Processor time quota A read-only root file system (allowing only writing data to the temporary memory file system) and a network access whitelist (allowing access only to the target host within the target container group, prohibiting access to external networks). These constraints ensure that the replay process is performed in a controlled, isolated environment, while preventing malicious scripts from affecting the host machine or external systems.

[0162] During the output verification phase, capture the complete standard output after the exploit script has finished executing. Perform the following substring validation:

[0163] ;

[0164] Here, f2 represents a different, random new flag that is independent of the penetration phase flag f.

[0165] like This indicates that the script Without seeing the new flag f2, the script successfully read and output the new flag f2 by actually exploiting a vulnerability in the target range. This demonstrates that the script possesses the ability to generalize vulnerabilities across flags, and the verification is successful. This indicates that the script Unable to successfully extract the new flag in the new flag environment, failing to meet the criteria for generalizable exploitation, verification failed.

[0166] Replayable verification generates binary deterministic signals It plays a dual role in agent training methods:

[0167] During the supervised fine-tuning phase, this signal serves as a quality filter for the teacher's trajectories. Only trajectories that can be verified through replay are included in the training dataset, ensuring that every trajectory learned by the model has been tested by the objective environment, rather than relying solely on the subjective output of the teacher's model.

[0168] During the reinforcement learning phase, this signal is directly used as one of the core components of the reward function. Together with the flag matching signal, it constitutes the binary core reward, providing a deterministic gradient direction from the real environment for policy optimization without the need for any approximate reward model.

[0169] Step S450, supervised fine-tuning based on environmental validation.

[0170] The teacher agent generates penetration trajectories on the training set target range through a unified interactive interface. The penetration trajectories are then subjected to objective quality verification, including replayability verification. Only the verified trajectories are formatted as a supervised fine-tuning dataset and the target language model is then supervisedly fine-tuned to obtain the warm-up strategy model.

[0171] The training model first provides high-quality initial policies for reinforcement learning through a supervised fine-tuning warm-up phase, thereby alleviating the difficulty of exploration in highly sparse reward environments.

[0172] Select one or more closed-source large language models as teacher agents, and use a unified penetration interaction interface in the training set test field. The system performs multiple rounds of penetration testing, systematically collecting penetration patterns. Each round generates one penetration pattern. Formal representation:

[0173] ;

[0174] Where d represents the target range mission description. For the penetration command in round t, For the corresponding execution result, This involves analyzing the current penetration state and providing the rationale for instruction selection by the teacher agent before executing the t-th round of instructions. The final flag string to be submitted. This is the submitted exploit script.

[0175] To ensure the quality of the supervised fine-tuning dataset, the following four quality screening steps were performed sequentially on all collected trajectories:

[0176] Flag verification filtering: Only retain those that meet the requirements. The trajectory is the round in which the teacher agent successfully extracts the flag.

[0177] Exploitation scripts can be replayed and filtered: only those that meet the criteria are retained. The trajectory, that is, the exploit script submitted by the teacher agent, can independently reproduce the attack path on the reconstructed target range.

[0178] Interaction efficiency filtering: Eliminating interaction rounds The trajectory, as well as the trajectory containing consecutive repeated commands (i.e., the execution content of adjacent rounds is completely identical), is used to prevent inefficient trajectories from having a negative imitation learning effect on the policy model.

[0179] Command legitimacy screening: Eliminate commands containing dangerous system operation commands (such as deleting critical system files, modifying system configurations, etc.) or initiating network connections to non-target targets to ensure that the strategy model does not learn dangerous behaviors beyond the target range boundaries through imitation learning.

[0180] The trajectories selected using all the above filtering criteria constitute a high-quality trajectory set. .Will Each trajectory in The training samples are converted into instructions following a specific format, with the task description d as the input text and the inference process text for all rounds as the input text. With tool call text The output text is the sequence completely assembled in the order of interaction; a supervised fine-tuning dataset is then constructed. This refers to the penetration trajectory in the aforementioned embodiments. When multiple teacher agents are used, multiple successful trajectories generated by different teacher models for the same target range can be included. After deduplication, the data is fused to increase the diversity of training data and enhance the generalization ability of the preheating strategy model under different penetration paths.

[0181] The target language model corresponding to the penetration agent in the above embodiments. Supervised fine-tuning dataset Supervised fine-tuning is performed to minimize the standard autoregressive cross-entropy loss. :

[0182] ;

[0183] in, This indicates that the policy model is based on a given input X and a generated prefix. The probability of generating the t-th label under given conditions. After optimization, the preheating strategy model is obtained. , The expected function is represented; this model has initially acquired the basic ability to perform penetration operations in the target range environment and can be used as the initial strategy for the subsequent reinforcement learning stage.

[0184] It is important to emphasize that, since each sample in the supervised fine-tuning dataset has undergone objective quality verification in the target environment, the warm-up strategy model... The learned behavioral patterns have environmentally validated reliability, which is fundamentally different from traditional supervised fine-tuning methods where the model may learn unverified teacher output patterns.

[0185] Step S460, reinforcement learning based on objective environmental rewards.

[0186] Specifically, the preheating strategy model is loaded as the initial strategy, and strategy inference is performed on the target range of the training set; the flag matching results and replayable verification results directly generated in the target range environment are used as the reward signal for reinforcement learning. The reward signal is a binary deterministic signal based on the real state of the environment, and there is no need to train an approximate reward model; the strategy model parameters are updated based on the reward signal.

[0187] Specifically, the preheating strategy model Load as initial strategy In the training set The strategy is inferred on the target range. Specifically, for each training sample corresponding to a task description of a specific target range... Under the same target range and the same dynamic flags, G independent policy simulations are performed in parallel to obtain the output set. In each simulation, the initial strategy... The system interacts with the target instance through a unified penetration interface in multiple rounds to generate a complete penetration trajectory. The purpose of this parallel simulation is to provide multiple sampling results for comparison within the same group for subsequent group relative strategy optimization algorithms.

[0188] The scheme in this embodiment differs from the traditional reward function design that includes multiple auxiliary reward components; it adopts a simple binary core reward design. Figure 6 This is a schematic diagram illustrating the reinforcement learning reward calculation provided in this specific embodiment. (Reference) Figure 6 After acquiring the penetration range information, the penetration agent performs flag verification and replayability verification according to steps S410 to S440. The reward for each round of simulation consists of two components: flag submission reward and replayability verification reward. Both are binary deterministic signals based on the real state of the environment.

[0189] Among them, flag submission rewards Defined as:

[0190] ;

[0191] in The flag string submitted by the agent, where f* is the expected flag injected into the target instance. The flag submission reward is assigned the first weight w1, representing whether the penetrating agent successfully acquired the correct flag in the current round.

[0192] Replayable verification reward Defined as:

[0193] ;

[0194] in, f2 is the exploit script submitted by the agent, and f2 is a new random flag generated during the replayable verification phase. This is the standard output after the script is executed. The replayable verification reward is assigned a second weight w2, which indicates whether the exploit script submitted by the agent has the ability to generalize across flags.

[0195] Binary Core Rewards The base score is:

[0196] ;

[0197] Crucially, the total score cap is halved when either the flag submission reward or the replayable verification reward is zero. Formally, this halving constraint function is defined. for:

[0198] ;

[0199] in, Indicates a binary core reward. This indicates that the reward can be replayed to verify its validity. The flag submission reward is represented by w1, which represents the first weight granted to the flag submission reward, and w2 represents the second weight granted to the replayable verification reward.

[0200] The intention behind this halving constraint is: if an exploit agent only completes flag acquisition but fails to submit a generalizable exploit script (i.e., ... but If the agent submits a generalizable script but fails to submit the correct flag (i.e., ...), its total reward is limited to less than half of the maximum possible reward because it cannot be confirmed whether the agent has truly mastered the exploit capability. Conversely, if the agent submits a generalizable script but fails to submit the correct flag (i.e. ...), its total reward is limited to less than half of the maximum possible reward because it cannot be confirmed whether the agent has truly mastered the exploit capability. but This also limits the total reward cap because the agent has not completed the full penetration task in the current round. The agent can only receive the full reward when both verifications pass simultaneously. This design guides the strategy model to simultaneously pursue the generalization ability of flag acquisition and exploit scripts.

[0201] The efficiency adjustment coefficient is calculated based on the ratio of the actual number of interaction rounds t to the maximum number of interaction rounds T. :

[0202] ;

[0203] When an agent completes the penetration task with fewer interaction rounds, the value of t / T is smaller. When the value approaches 1, the reward is almost undiminished; as the agent consumes more interaction rounds, the value of t / T increases. The reward is reduced accordingly as a penalty. Coefficient The value range is (0.5, 1] ​​— even if the agent uses all the maximum allowed number of interaction rounds (t=T). The minimum value is 0.5, and the reward can be reduced by a maximum of half without becoming zero. This design ensures the gentleness of efficiency penalties: prioritizing positive incentives for successful task completion, and then encouraging penetration through fewer steps via efficiency adjustments.

[0204] The round reward is scaled using an efficiency adjustment factor to obtain the final reward value after efficiency adjustment. :

[0205] ;

[0206] in, This represents the efficiency adjustment coefficient, t represents the actual number of interaction rounds, and T represents the maximum number of interaction rounds. This represents the halving constraint function.

[0207] Based on the final reward value after efficiency adjustment The relative advantage of each group inference is calculated using the Group Relative Policy Optimization (GRPO) algorithm, and the policy model parameters are updated accordingly; this is known as GRPO training and updating. For the training samples... The G parallel simulations have the following reward set for each simulation: The relative advantage of normalization within the computational group. for:

[0208] ;

[0209] in, The average reward within the group. The standard deviation of the group's rewards. To prevent division by zero by small constants, the policy parameter update objective is to maximize the following pruning objective function. :

[0210] ;

[0211] in, Importance sampling ratio, Indicates training samples, This represents the output obtained after performing policy deduction on the training samples. As the current strategy, As a historical strategy, The cutoff factor is , Let KL divergence penalty coefficient be denoted as . Represents the expectation function, Indicates the relative advantage of normalization within the group. As a reference strategy (usually set as a preheating strategy model) ), This is the KL divergence between the policy model and the reference policy, used to prevent the policy from deviating too far from the reference policy during policy updates.

[0212] Step S470, training-evaluation closed loop.

[0213] The policy model during the training phase is periodically switched to the evaluation mode and subjected to independent blind testing on the evaluation set target range. The evaluation results are not included in the policy gradient update. The target range sampling distribution for the next round of reinforcement learning training is automatically adjusted according to the evaluation results, so that the training resources are concentrated in the weak direction. The cycle is iterated until the standardized score on the evaluation set reaches the preset threshold.

[0214] Periodically switch the policy model during the training phase to the evaluation mode, and use the evaluation set... The model undergoes independent blind testing at the target range. During the evaluation process, gradients are not calculated, model parameters are not updated, and the evaluation results are not used for policy gradient updates; they are only used to assess the true generalization ability of the policy model.

[0215] The independent blind test used a three-component standardized scoring system, with each component defined as follows:

[0216] Flag Submission Score Flags submitted by the agent A score of 1 is given if the match is an exact match to the expected flag f*, otherwise a score of 0 is given, expressed by the formula:

[0217] ;

[0218] Replayable verification score The exploit script submitted by the penetration testing agent scores 1 point if it passes the replayability verification, and 0 points otherwise. This can be expressed by the formula:

[0219] ;

[0220] Efficiency Score Based on the ratio of the actual number of interaction rounds t to the maximum number of interaction rounds T, normalized to the interval [0, 1], the fewer the number of interaction rounds, the closer the score is to 1, expressed by the formula:

[0221] ;

[0222] Summarizing the scores from the above three items, the following three indicators are calculated according to the difficulty level of the target range: (a) Target range problem-solving rate, i.e. (a) The average value at each difficulty level reflects the agent's overall ability to overcome targets at each difficulty level; (b) The average number of interaction rounds, i.e., the average number of actual interaction rounds t at each difficulty level, reflects the agent's penetration efficiency; (c) The generalization pass rate of the exploit script, i.e. The average value across all difficulty levels reflects the cross-flag generalization ability of the scripts submitted by the agent. A standardized evaluation report is output.

[0223] The strategy model trains at preset training steps. Switch to evaluation mode to perform independent blind testing. The evaluation results are output as a categorized score matrix M, based on the range difficulty level and range category. This represents the weighted average score of category j in difficulty level i.

[0224] Course learning scheduler, i.e. Figure 3 The evaluation / scheduler reads the classification score matrix M and marks categories with scores below a preset threshold in each difficulty level as weak categories. Then, it selects target ranges belonging to the weak categories from the training set and increases their sampling weights, while decreasing the sampling weights of categories that have already met the thresholds. The specific calculation for the weight adjustment is as follows: for the weak category c, its new sampling weight... It is directly proportional to the difference in scores for that category:

[0225] ;

[0226] in, The preset threshold for compliance. Adjust the sensitivity coefficient for weighting. This represents the weighted average score of category j in difficulty layer i. This formula ensures that categories with scores lower than the threshold receive a greater boost in sampling weight, automatically concentrating training resources towards weaker categories. The scheduler outputs the updated target range sampling distribution as input for the next round of reinforcement learning training.

[0227] The automatic termination condition for the training-assessment closed loop is: when the standardized scores of all categories at all difficulty levels on the assessment set reach the preset threshold for N consecutive rounds, and the variance of the scores in each round is lower than the preset variance threshold. When the closed loop terminates automatically, the requirement of meeting the standard for N consecutive rounds ensures the stability of the model performance rather than random fluctuations; the constraint of the variance threshold further ensures that the model performance has converged to a stable state.

[0228] Figure 7 This is a schematic diagram illustrating the progressive learning process of the course provided in this specific embodiment. (Reference) Figure 7 The course learning scheduler further implements a progressive difficulty advancement mechanism, enabling the strategy model to gradually challenge higher difficulty levels of the target range from easy to difficult.

[0229] Specifically, the training set target range is divided into m levels from L1 to Lm according to difficulty, where L1 is the lowest difficulty level and Lm is the highest difficulty level. Initialization is performed first, that is, reinforcement learning training uses only the lowest difficulty level, L1 level, for sampling training in the initial stage, and switches to evaluation mode periodically when there is no gradient update.

[0230] Subsequently, an independent blind test is conducted and a classification score matrix is ​​output, weak categories are marked and their sampling weights are increased.

[0231] When the comprehensive score of the Li level assessment exceeds the advancement threshold for that level for a predetermined number of consecutive times, i.e., when the assessment of the current level is continuously met, the target range of the next difficulty level, Li+1, is automatically included in the training data sampling pool, and the weight of the new target range increases linearly, as expressed by the formula:

[0232] ;

[0233] in, This represents the overall score of level Li in the nth round of evaluation. This is the promotion threshold for this level. This refers to the number of consecutive times the target must be met. The advancement threshold for each difficulty level is dynamically determined by the assessment reports within the training-assessment closed loop.

[0234] The newly included Li+1 level target range has a low initial weight in the sampling pool, which gradually increases linearly with the number of training steps until it reaches the same level as the Li level. Let s be the number of training steps after the Li+1 level is included in the sampling pool, and let the preset weight growth period be... Then the relative sampling weights of the Li+1 level for:

[0235] ;

[0236] in, Let s represent the relative sampling weights of the Li-th layer, and s represent the number of training steps. This indicates the preset weight growth period.

[0237] This linear growth strategy avoids the drastic fluctuations in the performance of the strategy model that may be caused by a sudden influx of training data into the target range at a new difficulty level, thus ensuring a smooth and orderly transition in difficulty.

[0238] This application's embodiments utilize the inherent verifiability of the Capture The Flag (CTF) test environment, namely, the flag matching and replayable verification mechanism can generate objective, binary, and deterministic verification signals. These verification signals are then used simultaneously as data quality filters in the supervised fine-tuning phase and reward signals in the reinforcement learning phase, fundamentally eliminating the dependence on approximate reward models.

[0239] It should be noted that the steps shown in the above process or in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions.

[0240] This embodiment also provides a multi-stage penetration agent training device based on a verifiable penetration range environment. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform predetermined functions. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0241] Figure 8 This is a structural block diagram of the multi-stage penetration agent training device based on a verifiable penetration range environment in this embodiment, as shown below. Figure 8 As shown, the device includes: a first verification module 10, a second verification module 20, a screening module 30, and a training module 40.

[0242] The first verification module 10 is used to configure the range container in the range container environment and start the range instance; randomly generate a first flag; execute the operation instructions issued by the teacher agent according to the first flag to perform range instantiation verification on the range container environment; and disconnect the range container from the range network.

[0243] The second verification module 20 is used to execute the vulnerability exploit script submitted by the teacher agent in a temporary container environment and obtain the script output results; the configuration method of the temporary container environment is the same as that of the target range container environment; a randomly generated second flag is injected into the temporary container environment; the string corresponding to the second flag is different from that of the first flag.

[0244] The filtering module 30 is used to determine the penetration trajectory generated by the teacher agent based on a preset environment verification strategy in the temporary container environment when the script output results contain the second flag.

[0245] Training module 40 is used to train the penetration agent in both the target range container environment and the temporary container environment, using the penetration trajectory as a supervised fine-tuning dataset.

[0246] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0247] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0248] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0249] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0250] S1. Configure the target range container in the target range container environment and start the target range instance; randomly generate the first flag; execute the operation instructions issued by the teacher agent according to the first flag to verify the target range instantiation of the target range container environment; and disconnect the target range container from the target range network.

[0251] S2, in the temporary container environment, execute the exploit script submitted by the teacher agent and obtain the script output; the configuration of the temporary container environment is the same as that of the target range container environment; a randomly generated second flag is injected into the temporary container environment; the string corresponding to the second flag is different from that of the first flag;

[0252] S3, if the script output contains the second flag, determine the penetration trajectory generated by the teacher agent based on the preset environment verification strategy in the temporary container environment;

[0253] S4 uses the penetration trajectory as a supervised fine-tuning dataset to train the penetration agent in both the target range container environment and the temporary container environment.

[0254] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0255] Furthermore, in conjunction with the multi-stage penetration agent training method based on a verifiable penetration range environment provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the multi-stage penetration agent training methods based on a verifiable penetration range environment described in the above embodiments.

[0256] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0257] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0258] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0259] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A multi-stage penetration agent training method based on a verifiable penetration range environment, characterized in that, The method includes: Configure the range container in the range container environment and start the range instance; randomly generate a first flag; execute the operation instructions issued by the teacher agent according to the first flag to perform range instantiation verification on the range container environment; and disconnect the range container from the range network. In the temporary container environment, the vulnerability exploit script submitted by the teacher agent is executed, and the script output is obtained; the configuration method of the temporary container environment is the same as that of the target range container environment; a randomly generated second flag is injected into the temporary container environment; the string corresponding to the second flag is different from that of the first flag; If the script output contains the second flag, the penetration trajectory generated by the teacher agent is determined based on the preset environment verification strategy in the temporary container environment. The penetration trajectory is used as a supervised fine-tuning dataset to train the penetration agent in both the target range container environment and the temporary container environment.

2. The multi-stage penetration agent training method based on a verifiable penetration range environment according to claim 1, characterized in that, The step of executing the operation instructions issued by the teacher agent according to the first flag to perform target range instantiation verification of the target range container environment includes: Inject the first flag into the target range container environment; The system receives and executes operation instructions issued by the teacher's intelligent agent through a pre-defined unified penetration interaction interface, and outputs structured results; the structured results are associated with the type of the operation instructions. When the type of the operation instruction is flag submission, the flag string in the structured result is obtained; The flag string in the structured result is compared with the string corresponding to the expected flag to obtain a binary verification result; the expected flag is generated after the first flag is injected into the target range container environment. Based on the binary verification results, verify whether the target range container environment is isolated.

3. The multi-stage penetration agent training method based on a verifiable penetration range environment according to claim 2, characterized in that, When it is determined that the script output contains the second flag, the penetration trajectory generated by the teacher agent is determined based on the preset environment verification strategy in the temporary container environment, including: Based on the exploit script and the script output, a binary deterministic signal is generated; the binary deterministic signal is used to characterize whether the exploit script has cross-flag generalization capability. If the exploit script does not involve the second flag, but the script output includes the second flag, a binary deterministic signal is generated; the binary deterministic signal includes the binary deterministic signal. Based on a preset environment verification strategy, the penetration trajectory is obtained by filtering all trajectories generated by the teacher agent according to the binary determination signal.

4. The multi-stage penetration agent training method based on a verifiable penetration range environment according to claim 3, characterized in that, The preset environment verification strategy, based on the binary determination signal, filters all trajectories generated by the teacher agent to obtain the penetration trajectory, including: Obtain the round-by-round penetration trajectory of the teacher agent in the target range container environment for each penetration interaction, and the cumulative number of interaction rounds corresponding to each round-by-round penetration trajectory; Based on a preset environment verification strategy, the penetration trajectory generated by the teacher agent in each round is filtered to obtain a penetration trajectory that conforms to the environment verification result; the preset environment verification strategy includes the binary verification result, the cumulative number of interaction rounds, and the binary deterministic signal.

5. The multi-stage penetration agent training method based on a verifiable penetration range environment according to claim 2, characterized in that, The step of configuring the range container in the range container environment and starting the range instance includes: The range data in the range warehouse is parsed to obtain range data entries; the range data entries include the task information and service port information of each range; the task information includes task description text; Launch the target range instance corresponding to the target range data entry.

6. The multi-stage penetration agent training method based on a verifiable penetration range environment according to claim 5, characterized in that, The step of using the penetration trajectory as a supervised fine-tuning dataset to train the penetration agent in both the target range container environment and the temporary container environment includes: The penetration trajectory is converted into training samples in a preset format, with the task description text as input text and the inference text and tool call text in the penetration interaction process as output text, to construct a supervised fine-tuning dataset; Based on a preset loss function, the penetration agent is subjected to supervised fine-tuning on the supervised fine-tuning dataset to obtain a preheating strategy model. The preheating strategy model is loaded as the initial strategy, and strategy deduction is performed in the target range container environment and the temporary container environment respectively to obtain the deduction results; the parameters of the preheating strategy model are updated based on the deduction results.

7. The multi-stage penetration agent training method based on a verifiable penetration range environment according to claim 6, characterized in that, The preheating strategy model is loaded as the initial strategy, and strategy deduction is performed in the target range container environment and the temporary container environment respectively to obtain the deduction results. The parameters of the preheating strategy model are updated based on the simulation results, including: The preheating strategy model is loaded as the initial strategy, and the strategy inference is performed in parallel for each sample in the supervised fine-tuning dataset under the same target range container environment and the same flag to obtain the inference results; the inference results include the binary verification results and the binary deterministic signal; The binary verification result is assigned a first weight, and the binary deterministic signal is assigned a second weight. If either the binary verification result or the binary deterministic signal reaches a preset value, the reward signal corresponding to the deduction result is scaled to obtain the final reward. Based on the final reward, the parameters of the preheating strategy model are updated to update the penetration agent.

8. A multi-stage penetration agent training device based on a verifiable penetration range environment, characterized in that, The device includes: a first verification module, a second verification module, a screening module, and a training module; The first verification module is used to start a target range instance in the target range container environment and randomly generate a first flag; according to the first flag, execute the operation instructions issued by the teacher agent to perform target range instantiation verification on the target range container environment; and disconnect the target range container from the target range network. The second verification module is used to execute the vulnerability exploit script submitted by the teacher agent in a temporary container environment and obtain the script output result; the configuration method of the temporary container environment is the same as that of the target range container environment; a randomly generated second flag is injected into the temporary container environment; the string corresponding to the second flag is different from that of the first flag; The filtering module is used to determine the penetration trajectory generated by the teacher agent based on a preset environment verification strategy in the temporary container environment, when it is determined that the script output contains the second flag. The training module is used to train the penetration agent in the target range container environment and the temporary container environment, respectively, using the penetration trajectory as a supervised fine-tuning dataset.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute the multi-stage penetration agent training method based on a verifiable penetration range environment as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-stage penetration agent training method based on a verifiable penetration range environment as described in any one of claims 1 to 7.

Images