Agent training method, penetration testing method, penetration testing system

Abstract

The application discloses an agent training method, a penetration testing method and a penetration testing system, and belongs to the technical field of computers. The method comprises the following steps: inputting a first to-be-tested information sample corresponding to a current reinforcement training round into a decision agent to generate a first task sequence corresponding to the first to-be-tested information sample; performing dependency relationship rationality evaluation on the first task sequence based on a dependency attribute of each subtask in the first task sequence, wherein the dependency attribute is used for indicating a parent task on which the subtask is dependent; updating a decision parameter of the decision agent according to an evaluation result, wherein the evaluation result comprises a dependency relationship rationality evaluation result; entering a next reinforcement training round and repeatedly performing the above steps until a first stop condition is met, and obtaining the decision agent after reinforcement training is completed.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an agent training method, a penetration testing method, and a penetration testing system. Background Technology

[0002] Penetration testing refers to assessing the security of a network or system by simulating malicious attacks in order to discover potential security vulnerabilities. Introducing large language models into penetration testing allows for the abstraction of test information, thereby supporting task planning. Therefore, improving the task planning capabilities of large language models in penetration testing is a crucial technical problem that needs to be solved. Summary of the Invention

[0003] This application provides an agent training method, a penetration testing method, and a penetration testing system, which can improve the task planning ability of large language models in penetration testing.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, a method for training an intelligent agent is provided. The method includes: inputting a first test information sample corresponding to the current reinforcement training round into a decision intelligent agent to generate a first task sequence corresponding to the first test information sample; evaluating the reasonableness of the dependency relationship of the first task sequence based on the dependency attributes of each subtask in the first task sequence, wherein the dependency attributes are used to instruct the subtask to execute the parent task it depends on; updating the decision parameters of the decision intelligent agent according to the evaluation result, wherein the evaluation result includes the reasonableness evaluation result of the dependency relationship; entering the next reinforcement training round and repeating the above steps until a first stopping condition is met to obtain the decision intelligent agent after reinforcement training is completed.

[0005] Secondly, a penetration testing method is provided, applied to a decision-making agent, which is trained based on the agent training method described in the first aspect. The method includes: acquiring target test information and target global state information corresponding to the target object to be tested; generating a target task sequence corresponding to the target test information based on the target test information and the target global state information; and obtaining the test result corresponding to the target object to be tested by executing each sub-task included in the target task sequence.

[0006] Thirdly, a penetration testing system is provided, comprising: a decision agent, configured to: generate a target task sequence based on target test information and target global state information corresponding to the target object to be tested; execute each subtask included in the target task sequence using a scheduling agent to obtain test results corresponding to the target object to be tested; a scheduling agent, configured to: receive the target task sequence sent by the decision agent; construct a task dependency graph based on the dependency attributes of each subtask in the target task sequence; concurrently execute the subtasks corresponding to nodes with an in-degree of 0 in the task dependency graph using an execution agent in each execution round until a preset stopping condition is met; and an execution agent, configured to: receive the subtasks sent by the scheduling agent; select a task execution tool based on the task description information corresponding to the subtasks and infer execution parameters based on the selected task execution tool; execute the subtasks based on the selected task execution tool and the inferred execution parameters to obtain execution results corresponding to the subtasks; update the target global state information based on the execution results; and return the execution results to the decision agent.

[0007] Fourthly, an agent training device is provided, comprising: a generation module, configured to input a first test information sample corresponding to the current reinforcement training round into a decision agent, generating a first task sequence corresponding to the first test information sample; an evaluation module, configured to evaluate the reasonableness of the dependency relationship of the first task sequence based on the dependency attributes of each sub-task in the first task sequence, wherein the dependency attributes are used to instruct the sub-task to execute the parent task it depends on; an update module, configured to update the decision parameters of the decision agent according to the evaluation results, wherein the evaluation results include the reasonableness evaluation results of the dependency relationship; and a first training module, configured to enter the next reinforcement training round and repeat the above steps until a first stopping condition is met, thereby obtaining the decision agent after reinforcement training is completed.

[0008] Fifthly, a penetration testing apparatus is provided, comprising: an acquisition module for acquiring target test information and target global state information corresponding to a target object to be tested; a generation module for generating a target task sequence corresponding to the target test information based on the target test information and the target global state information; and a testing module for obtaining test results corresponding to the target object to be tested by executing each sub-task included in the target task sequence.

[0009] A sixth aspect provides an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect, or implementing the steps of the method as described in the second aspect.

[0010] In a seventh aspect, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect, or implement the steps of the method described in the second aspect.

[0011] Eighthly, a computer program product is provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to perform the steps of the method described in the first aspect, or to perform the steps of the method described in the second aspect.

[0012] In this embodiment, by inputting the first test information sample corresponding to the current reinforcement training round into the decision agent, a first task sequence corresponding to the first test information sample is generated. Based on the dependency attributes of each subtask in the first task sequence, the rationality of the dependency relationship of the first task sequence is evaluated. The dependency attributes are used to instruct the subtask to execute the parent task it depends on. The decision parameters of the decision agent are updated according to the evaluation results, and the next reinforcement training round is entered and the above steps are repeated until the first stopping condition is met, resulting in a decision agent with completed reinforcement training. This can avoid the problem of logical errors in the decision agent. By evaluating the causal dependency of each subtask, accurate reward signals can be provided to the decision agent, guiding it to learn rigorous derivation paths, thereby significantly enhancing the task planning ability of the decision agent.

[0013] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0015] Figure 1 This invention illustrates a flowchart of an agent training method provided in an exemplary embodiment of this application. Figure 2 This application illustrates a graph structure provided by an exemplary embodiment. Figure 3 This application illustrates another graph structure provided by an exemplary embodiment; Figure 4 This illustration shows a schematic diagram of a model training stage provided by an exemplary embodiment of this application; Figure 5 This invention illustrates a flowchart of a penetration testing method provided in an exemplary embodiment of this application. Figure 6 This invention provides a schematic diagram of the structure of a penetration testing system according to an exemplary embodiment of the present application. Figure 7 This illustration shows an interactive diagram provided by an exemplary embodiment of this application; Figure 8 This application shows a schematic diagram of the architecture of a penetration testing system provided in an exemplary embodiment. Figure 9 This invention provides a schematic diagram of the structure of an agent training device according to an exemplary embodiment of the present application. Figure 10 This invention provides a schematic diagram of the structure of a penetration testing apparatus according to an exemplary embodiment of the present application. Figure 11 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0017] Figure 1 This illustration shows a flowchart of an agent training method provided in an exemplary embodiment of this application. The method can be executed by an electronic device, which may include a terminal device and a network-side device. In other words, the method can be executed by software or hardware installed on the electronic device, and the method may include the following steps: S110: Input the first test information sample corresponding to the current reinforcement training round into the decision agent to generate the first task sequence corresponding to the first test information sample.

[0018] The decision agent is built based on a large language model. The first test information sample includes feature description information of the test object. This feature description information may include the basic characteristics of the test system, such as the target domain name, IP address, known technology stack, authorized scope, test boundaries, etc. The first task sequence may include multiple sub-tasks, each sub-task corresponding to a task description information. For example, assuming that the feature description information of the first test information sample corresponding to the current reinforcement training round includes: {Target domain name: https: / / aaaaaa.com; IP address: aaa.aaa.aa.aa; Port: 443 (HTTPS); Technology stack: PHP 7.4 + MySQL 5.7 + Nginx 1.18; Allowed attack types: information gathering, vulnerability scanning, authentication bypass testing, SQL injection testing}, the first task sequence generated by the decision agent can be shown in Table 1. Table 1

[0019] In some embodiments, the subtasks included in the first task sequence may have the following data structure: {task identifier, parent task attribute, task description information}. Further, the data structure may also include fields such as status, priority, recommended tool, tool parameters, and execution result.

[0020] In other embodiments, the first sample of information to be tested is labeled with global state information. This target global state information may include, but is not limited to, the following: (1) Attack surface: used to characterize the testable entry points and basic attributes of the system under test. Such as address (IP / URL), list of open ports, etc.

[0021] (2) Technology stack: Used to describe the composition of the target environment and guide subsequent targeted testing. Such as application framework (e.g., Django 4.2), front-end library (e.g., React 18), database (e.g., MySQL 8.0), middleware (e.g., Redis 7.0), cloud platform identifier (e.g., AWS EC2), etc.

[0022] (3) Credentials: Identity authentication materials obtained during the testing process, for use in subsequent privilege escalation or lateral movement. Such as username, password, etc.

[0023] (4) Vulnerabilities: Structured records of confirmed or suspected security flaws are used to generate detection tasks.

[0024] (5) Session: Represents an established authentication context that enables the system to access protected resources without having to log in again.

[0025] (6) Other information that can be used for subsequent exploration or generation tasks.

[0026] S120: Based on the dependency attributes of each subtask in the first task sequence, evaluate the reasonableness of the dependency relationship in the first task sequence.

[0027] The dependency attribute is used to instruct the subtask to execute the parent task it depends on.

[0028] Understandably, in the generated first task sequence, each subtask can also have a dependency attribute, which indicates the parent task corresponding to the subtask; that is, when executing the subtask, its parent task must be executed first. For example, as shown in Table 2, each subtask has a dependency attribute.

[0029] Table 2

[0030] As shown in Table 2, the parent task attribute of Task2 indicates that the parent task is Task1, the parent task attribute of Task3 indicates that the parent task is Task2, and the parent task attribute of Task4 indicates that the parent task is Task3. Task1, as the first subtask, can have a null or NULL parent task attribute. Furthermore, the parent task attribute of other subtasks that are not the first subtask can also be a null or NULL.

[0031] During the training of the decision-making agent, if only the final generated first task sequence is used as supervision, logical errors and dependency errors between tasks cannot be identified. For example, as shown in Table 3, the generated first task sequence may contain the following dependency errors: Table 3

[0032] As shown in Table 3, both Task 3 and Task 4 are parent tasks of Task 2. Therefore, if Task 3 and Task 4 are executed concurrently after Task 2 completes, Task 4 will encounter an execution error because it depends on the result of Task 3, which has not yet finished. Therefore, it is necessary to perform dependency rationality judgment on the first task sequence generated by the decision-making agent to achieve logical supervision during the training phase, thereby improving the inference accuracy and robustness of the decision-making agent.

[0033] In some embodiments, S120 may include the following steps: S122: Generate a graph structure containing causal dependencies based on the dependency attributes of each subtask in the first task sequence.

[0034] S124: Evaluate the reasonableness of the dependency relationships of the first task sequence based on the graph structure.

[0035] Understandably, a directed graph (DAG) is constructed based on the parent task attributes of each task. For example, Figure 2 The graph structure containing causal dependencies corresponding to Table 2 above is shown. Figure 3 The graph structure containing causal dependencies corresponding to Table 3 above is shown, and then the reasonableness of the dependencies is evaluated based on the graph structure.

[0036] S130: Update the decision parameters of the decision-making agent based on the evaluation results.

[0037] The evaluation results include the evaluation results of the reasonableness of the dependency relationship.

[0038] The decision parameter can be a model parameter used to construct a large language model of a decision agent.

[0039] In some embodiments, the evaluation result can be represented by a reward score, and the decision parameters of the decision agent can be updated based on the reward score using the REINFORCE algorithm. The gradient calculation formula can be:

[0040] in, This is the first task sequence generated. It's a reward score.

[0041] S140: Proceed to the next reinforcement training round and repeat the above steps until the first stopping condition is met, and obtain the decision agent after reinforcement training is completed.

[0042] The first stopping condition may include: the model parameters converge or the maximum number of iterations is reached.

[0043] In this embodiment, by inputting the first test information sample corresponding to the current reinforcement training round into the decision agent, a first task sequence corresponding to the first test information sample is generated. Based on the dependency attributes of each subtask in the first task sequence, the rationality of the dependency relationship of the first task sequence is evaluated. The dependency attributes are used to instruct the subtask to execute the parent task it depends on. The decision parameters of the decision agent are updated according to the evaluation results, and the next reinforcement training round is entered and the above steps are repeated until the first stopping condition is met, resulting in a decision agent with completed reinforcement training. This can avoid the problem of logical errors in the decision agent. By evaluating the causal dependency of each subtask, accurate reward signals can be provided to the decision agent, guiding it to learn rigorous derivation paths, thereby significantly enhancing the task planning ability of the decision agent.

[0044] In some embodiments, the step of evaluating the reasonableness of the dependency relationship of the first task sequence based on the dependency attributes of each subtask in the first task sequence may include: (1) In response to the existence of a circular dependency among the multiple subtasks, a first penalty value is assigned to the first task sequence.

[0045] Circular dependency refers to a chain of dependencies between two or more tasks. For example, subtask A depends on subtask B, and subtask B depends on subtask A, thus forming a circular dependency.

[0046] (2) In response to the absence of the parent task indicated by the dependency attribute of the first subtask in the first task sequence, a second penalty value is assigned to the first task sequence.

[0047] For example, the dependency attribute of the first subtask indicates that the parent task is subtask A, but the first task sequence does not include subtask A, so the parent task indicated by the dependency attribute is invalid.

[0048] (3) In response to the fact that there is no logical dependency between the second subtask and the third subtask in the task sequence, a third penalty value is assigned to the first task sequence, wherein the parent task indicated by the dependency attribute of the second subtask is the third subtask.

[0049] For example, the second subtask is identifying SQL injection points in the login box, and the third subtask is enumerating domain names. Since there is no logical dependency between the two, a third penalty value needs to be assigned. In some embodiments, heuristic rules can be used to determine whether a logical dependency exists between the second and third subtasks.

[0050] It should be noted that the first penalty value, the second penalty value, and the third penalty value can be equal or unequal.

[0051] In some embodiments, each subtask corresponds to a task description information, and the evaluation result also includes a format evaluation result and / or a semantic clarity evaluation result; Before evaluating the reasonableness of dependencies in the first task sequence based on the dependency attributes of each subtask in the first task sequence, the method further includes at least one of the following: (1) Perform format structure verification on the task description information corresponding to each subtask in the first task sequence to obtain the format evaluation result, wherein the format structure verification includes field integrity verification, field type verification, enumeration field value out-of-bounds verification, and semantic parsing verification.

[0052] Among them, field integrity check is used to determine whether necessary fields are missing, field type check is used to determine whether a field matches its corresponding type, enumeration field value out-of-bounds check is used to determine whether the value of each enumeration field exceeds the predefined range, and semantic parsing check is used to determine whether the result output by the decision agent has been successfully parsed.

[0053] (2) Perform fuzzy representation recognition on the task description information corresponding to each subtask in the first task sequence to obtain the semantic clarity evaluation result.

[0054] In this embodiment, text can be scanned using a predefined set of regular expression rules to identify ambiguous references such as "target URL," "acquired credentials," and "management backend." Each instance of such a statement is considered a language ambiguity problem that reduces task executability.

[0055] In some embodiments, format structure verification, fuzzy representation identification, and dependency reasonableness assessment are independent of each other. In other embodiments, format structure verification can be performed first, followed by fuzzy representation identification to ensure that penalties are only imposed on tasks with valid structures but unclear semantics. Dependency reasonableness assessment is then performed after fuzzy representation identification.

[0056] In some embodiments, the evaluation result can be a score corresponding to the reward function:

[0057] in, The evaluation results can be represented by reward scores; The formatting evaluation results can be represented by a formatting error score. The results of semantic clarity assessment can be represented by a linguistic ambiguity score; The results of the dependency rationality assessment can be represented by dependency scores.

[0058] In some embodiments, before inputting the first test information sample corresponding to the current reinforcement training round into the decision agent to generate the first task sequence corresponding to the first test information sample, the method further includes: obtaining a pre-training sample set, wherein the pre-training sample set includes multiple second test information samples, each second test information sample being labeled with corresponding global state information and a second task sequence; inputting each second test information sample into the decision agent to generate multiple third task sequences; calculating a target loss based on the second task sequence corresponding to each second test information sample and the generated third task sequence; updating the decision parameters of the decision agent based on the target loss until a second stopping condition is met, thereby obtaining the pre-trained decision agent.

[0059] It is understandable that, before reinforcement training, the decision-making agent can undergo pre-training, i.e., supervised training, to enable it to learn task planning capabilities. In this embodiment, the pre-training sample set can be obtained through, but is not limited to, the following two methods: (1) Expert-annotated data.

[0060] Penetration testing experts annotate test data across several test scenarios: given the feature descriptions of the target object and its global state at a given moment, the experts provide a corresponding task queue and explicitly include all the context information required for execution when describing the tasks. For example, when a task requires login, the description should be "Log in to http: / / example.com / admin using credential admin / password123" instead of "Log in to the management backend using the obtained credentials." In some embodiments, the expert-annotated samples can be used as a high-quality seed dataset for subsequent training.

[0061] (2) Data generation based on historical trajectories.

[0062] Collect complete records of existing penetration tests, extract the global state and executed tasks at each time step, and form training samples. In some embodiments, the collected training samples can be augmented to address the issue of samples containing ambiguous references. In other embodiments, the augmentation method may be to use rules or templates to identify placeholders and references in task descriptions and replace them with specific values ​​from the global state. For example, upon identifying a statement such as "using discovered credentials," the corresponding credentials are queried from the global state and replaced with "using credentials {username} / {password}".

[0063] The loss function used in this pre-training phase can be the cross-entropy loss, which can be expressed by the following formula:

[0064] in, The input prompts contain feature descriptions and global states of the object being tested. Furthermore, the input prompts may also include a list of available tools. It is the first A generated token, These are the model parameters. This loss function helps the planning agent achieve consistency between the third task sequence generated and the labeled second task sequence.

[0065] During pre-training, the planning agent can learn to understand the structure of the global state, identify information such as credentials, sessions, and discovered endpoints, and correctly embed them into the task description. It also learns to generate results in JSON format, including the correct filling of fields such as task ID, parent task ID, description, and priority.

[0066] In this embodiment, supervised learning is first used to enable the planning agent to quickly learn basic task paradigms, standardized output formats, and basic logic from high-quality labeled data. Then, reinforcement training is performed to enable the decision-making agent to optimize output based on evaluation results, thereby further improving task planning capabilities. Furthermore, training enables the planning agent to explicitly write the necessary execution context (such as credentials, cookies, and specific URLs) into the task description when generating it. This allows subsequent task execution based solely on the task description, without needing to access the global state.

[0067] In some embodiments, after obtaining the decision agent with completed reinforcement training, the method further includes: labeling each first task sequence based on the execution result corresponding to each first task sequence to determine a negative sample set, wherein the labeling dimensions include: the rationality judgment dimension of the execution tool, the rationality judgment dimension of the task generation timing, the completeness judgment dimension of the global state information utilization, and the rationality judgment dimension of the task granularity; and performing negative sample reinforcement training on the decision agent based on the negative sample set.

[0068] Understandably, reinforcement training optimizes the format and basic quality, but the professionalism and rationality of the tasks still need further optimization. For example, the decision-making agent might choose an inefficient tool when a more efficient tool is available, or generate tasks that are inconsistent with the current testing phase, or ignore important information in the global state. These problems are difficult to judge through automated rules and require reinforcement training. In this embodiment, in each reinforcement training round, the generated first task sequence is executed to obtain the execution result corresponding to the first task sequence; then, based on the execution result, the first test information sample with quality problems and its corresponding first task sequence are added to the negative sample set as negative samples. The annotation dimensions of the negative samples include: (1) Dimension of judging the rationality of execution tools.

[0069] It's understandable that a suboptimal tool was chosen when a more suitable or efficient one existed. For example, if the object under test is running WordPress and should use the dedicated tool wpscan, but the general-purpose nikto scanner was chosen instead, then there is definitely an issue with the choice of execution tool.

[0070] (2) Dimension of judging the rationality of task generation timing.

[0071] In other words, the timing of task generation in the current testing phase needs to be assessed for its reasonableness. For example, if an exploit task is generated before the information gathering phase is complete and no vulnerabilities have been discovered, then the timing of task generation is deemed unreasonable.

[0072] (3) Global state information is judged using the integrity dimension.

[0073] In other words, it determines whether the generated task description information includes key information from the global status information. For example, the global status may indicate that an SQL injection point has been discovered, but subsequent tasks may not generate in-depth probing tasks targeting that injection point.

[0074] (4) Dimension for judging the rationality of task granularity For example, tasks that are too macro-level, such as "completing web application security testing," cannot be executed, while tasks that are too micro-level, such as "sending an HTTP GET request," are too trivial.

[0075] In this embodiment, negative sample reinforcement learning is used to learn only from errors without explicitly rewarding correct behaviors. That is, for complex decision-making tasks, there are many correct paths, and reinforcing a single correct path will lead to pattern collapse; while there are also many incorrect paths, by eliminating errors, the decision-making agent can automatically explore other possible correct paths based on its prior knowledge.

[0076] In some embodiments, for task plans labeled as negative samples We can calculate the negative sample enhancement loss to minimize the probability of generating negative samples:

[0077] The formula for gradient update can be:

[0078] At the token level, for each of the negative sample tasks The effect of the gradient on the model parameters is as follows:

[0079] Understandably, negative sample learning can reduce the probability of sampling incorrect tokens, while simultaneously increasing the probability of other tokens according to the decision-making agent's original prior probability distribution. This prevents the decision-making agent from blindly exploring, instead allocating probabilities to reasonable options based on its existing safety knowledge. This prior-based redistribution protects the decision-making agent's pre-trained knowledge and avoids the problem of indiscriminately increasing low-probability, meaningless options caused by simple entropy regularization.

[0080] In some embodiments, the step of performing negative sample reinforcement training on the decision agent based on the negative sample set includes: determining a positive sample set based on the labeling results of each first task sequence; setting the loss corresponding to the positive sample set as a first weight and the loss corresponding to the negative sample set as a second weight, wherein the second weight is greater than the first weight; and performing negative sample reinforcement training on the decision agent based on the positive sample set and the negative sample set.

[0081] Understandably, learning using only negative samples might result in a slow learning speed. Therefore, it's possible to incorporate positive samples for weighted reinforcement learning, assigning different weights to positive and negative samples.

[0082] in, For positive sample reinforcement loss, For negative sample enhancement loss, The first weight corresponding to the positive sample. This represents the second weight corresponding to the positive samples. The positive sample enhancement loss is used for this purpose. Defined as:

[0083] For example, , This means giving negative samples nine times the weight of positive samples. This asymmetric weighting can effectively avoid erroneous decisions, thereby improving training efficiency.

[0084] Based on the embodiments of the above-described agent training method, this application also provides a schematic diagram of the model training stage, as shown below. Figure 4As shown, the training process comprises three stages: Stage 1: Supervised fine-tuning of the decision-making agent; Stage 2: Reinforcement training of the decision-making agent; Stage 3: Negative sample reinforcement learning of the decision-making agent. The training objective is to enable the planning agent to possess the following capabilities: First, accurately generate task sequences: ensuring correct task format and inclusion of all required fields. Second, ensure the generated task queue conforms to the logical order of penetration testing: correctly representing dependencies between tasks through the parent task ID field, and setting reasonable priorities. Third, appropriately embed global state information in the task description information: when a task requires credentials, the task description information should include the specific username and password; when a task requires a session, the task description information should include the specific cookie string; when a task requires testing against discovered endpoints, the task description information should include the specific URL path. The following provides an illustrative explanation of each of the three training stages.

[0085] (1) First stage The goal of the first phase is to enable the decision-making agent to correctly extract necessary contextual information from the global state and generate JSON outputs that conform to a predefined format. This phase adopts a standard supervised fine-tuning paradigm.

[0086] (2) Second stage While the decision-making agent in the first stage can generate valid formats, it still suffers from three types of problems: format detail errors, ambiguity in the task description, and incorrect task dependencies. The second stage uses reinforcement learning to specifically address these issues. Training data for the second stage is obtained from the interaction between the decision-making agent and the testing environment: the agent trained in the first stage performs numerous tests in a virtual testing environment or sandbox, recording the task plan generated and the actual execution results for each test. This interactive data collection allows for the rapid accumulation of a large-scale sample. The reward function for the second stage can be:

[0087] in, For formatting errors, the raw task queue (usually a JSON string) output by the decision agent is parsed and converted into a Python object, then structured using a decision agent defined based on Pydantic. This decision agent follows a predefined JSON schema, containing all required fields and their precise types. Points are deducted for the following errors: 1) Missing required fields: deduct 'a' points for each missing required field.

[0088] 2) Field type error: deduct b points for each field with a mismatched type.

[0089] 3) Enumeration value out of bounds: deduct c points for each enumeration field value that exceeds the predefined range.

[0090] 4) JSON parsing failure: Deduct d points if the entire parsing fails.

[0091] For linguistic ambiguity scoring, natural language analysis is performed on each task description in the task queue before including it in the evaluation. Text is scanned using a predefined set of regular expression rules to identify ambiguous references such as "target URL," "acquired credentials," and "admin backend." Each such expression detected is considered a linguistic ambiguity issue that reduces task executability and is penalized. This error detection is independent of format validation and is performed after the task object has passed Pydantic validation, ensuring that only tasks with valid structure but unclear semantics are penalized.

[0092] After all tasks pass the format and language clarity checks, a directed graph (DAG) is constructed based on the parent task attributes of each task. Points are deducted based on the validity of the directed graph in the following dimensions.

[0093] 1) Circular dependency: If a circular dependency exists between tasks, deduct 0 points. 2) Dependence on a non-existent task: The parent task ID points to a non-existent task, deducting f points for each instance; 3) Unreasonable dependency relationships: There is no logical dependency between the parent task and the child task. Deduct h points for each instance by using heuristic rules. The REINFORCE algorithm is used for policy optimization. For each test scenario, the decision agent generates a task plan, calculates the reward score, and updates the model parameters based on the reward. The gradient calculation formula is:

[0094] in, It is the generated task plan trajectory. That is the total reward.

[0095] (3) Third stage The second stage optimized the format and basic quality, but the professionalism and rationality of the tasks still need improvement. For example, the decision-making agent might choose an inefficient tool when a more efficient one is available, generate tasks inconsistent with the current testing phase, or ignore important clues in the global state. These problems are difficult to identify through automated rules, so the third stage uses negative sample reinforcement learning to address these issues.

[0096] The third phase of training data comes from expert review of the actual trial run: the decision-making agent trained in the second phase performs numerous tests in a real or near-realistic testing environment, collecting all generated task plans and execution results. Experts review these task plans and label tasks with quality issues as negative samples.

[0097] The negative sample reinforcement learning used in this stage is only used to learn from errors without explicitly rewarding correct behavior: for complex decision-making tasks, there are many correct paths, and reinforcing a certain correct path will lead to pattern collapse; while there are also many incorrect paths, by eliminating errors, the decision-making agent can automatically explore other possible correct paths based on its own prior knowledge.

[0098] In this embodiment, a three-stage progressive strategy is adopted, combining supervised learning and reinforcement learning to gradually improve the planning ability of the decision-making agent: The first stage uses supervised fine-tuning to teach the agent to extract parameters from the global state and generate outputs that conform to a specific format. The second stage uses reinforcement learning to optimize format quality and context embedding, deducting points for format errors, linguistic ambiguity, and dependency errors through an automated reward function. The third stage uses negative sample reinforcement learning based on high-quality negative samples annotated by experts to optimize the professionalism and rationality of the task, while maintaining the diversity of planning and avoiding pattern collapse.

[0099] Figure 5 This illustration shows a flowchart of a penetration testing method provided in an exemplary embodiment of this application. The method can be executed by a decision agent in the above-described agent training method embodiment, and may include the following steps: S510: Obtain the target test information and target global state information corresponding to the object to be tested.

[0100] The object to be tested can be an application, which may include client applications and web systems. The target test information includes feature descriptions of the object to be tested, which may include basic characteristics of the system under test, such as the target domain name, IP address, known technology stack, authorized scope, and test boundaries. The target global state information may include, but is not limited to, the following: (1) Attack surface: used to characterize the testable entry points and basic attributes of the target system, such as addresses (IP / URL), lists of open ports, etc.

[0101] (2) Technology stack: Used to describe the composition of the target environment and guide subsequent targeted testing. Such as application framework (e.g., Django 4.2), front-end library (e.g., React 18), database (e.g., MySQL 8.0), middleware (e.g., Redis 7.0), cloud platform identifier (e.g., AWS EC2), etc.

[0102] (3) Credentials: Identity authentication materials obtained during the testing process, for use in subsequent privilege escalation or lateral movement. Such as username, password, etc.

[0103] (4) Vulnerabilities: Structured records of confirmed or suspected security flaws are used to generate detection tasks.

[0104] (5) Session: Represents an established authentication context that enables the system to access protected resources without having to log in again.

[0105] (6) Other information that can be used for subsequent exploration or generation tasks.

[0106] In some embodiments, the target global state information is obtained by the decision-making agent initializing a global state object based on the target test information.

[0107] S520: Based on the target test information and the target global state information, generate the target task sequence corresponding to the target test information.

[0108] For example, the sequence of target tasks can be referred to Table 1 above.

[0109] S530: By executing each subtask included in the target task sequence, the test result corresponding to the object to be tested is obtained.

[0110] In some embodiments, S530 may include: in response to the completion of each subtask, obtaining updated target global state information, wherein the target global state information is updated based on the execution results of the completed subtasks; adjusting the target task sequence based on the updated target global state information; and performing penetration testing based on the updated target task sequence.

[0111] Understandably, during execution, the global state information of the target will be dynamically updated after obtaining the execution results of the subtasks. Therefore, the decision-making agent needs to obtain the latest global state information and dynamically adjust the target task sequence based on the new global state information. For example, when a high-risk vulnerability is discovered, an exploit task is inserted, and when a new endpoint is discovered, a probe task is added. The newly generated task also contains the parent task attribute field, forming a complete dependency graph with the existing tasks. The updated task continues to enter the next iteration for execution. In this way, by adopting a structured global state (such as attack surface, technology stack, credentials, vulnerabilities, sessions, etc.) to uniformly model the penetration testing process, the planning agent dynamically generates or adjusts the task queue and dependencies based on the real-time updates of this state, realizing the leap from "linear or fixed process" to "state-aware adaptive planning".

[0112] In this embodiment, the decision-making agent acquires the target test information and target global state information corresponding to the object to be tested. Then, based on the target test information and the target global state information, it generates a target task sequence corresponding to the target test information. By executing each sub-task included in the target task sequence, the test result corresponding to the object to be tested is obtained, realizing the intelligent generation of test tasks and improving testing efficiency and accuracy. Furthermore, this decision-making agent is trained based on the embodiment of the agent training method described above. Therefore, the target task sequence generated by this decision-making agent has characteristics such as format completeness, semantic clarity, and reasonable dependency relationships, further improving the rationality and accuracy of the generated tasks, thereby enhancing the professionalism of penetration testing.

[0113] In some embodiments, obtaining the test result corresponding to the object under test by executing each subtask included in the target task sequence includes: transmitting the target task sequence to a scheduling agent and instructing the scheduling agent to perform the following steps: Step 1: Based on the dependency attributes of each subtask in the target task sequence, construct a task dependency graph, wherein the task dependency graph includes a set of nodes and a set of dependency edges. The set of nodes includes multiple nodes, each node corresponding to a task sequence. The set of dependency edges includes multiple dependency edges, each dependency edge connecting two nodes and having a directional property, used to indicate the execution dependency relationship between the two nodes.

[0114] In some embodiments, constructing a task dependency graph based on the dependency attributes of each subtask in the target task sequence includes: constructing a dependency relationship graph, which is a directed acyclic graph, based on the dependency attributes of each subtask in the target task sequence; and constructing the task dependency graph based on the dependency relationship graph.

[0115] Understandably, dependencies may exist between tasks in penetration testing. For example, a service version identification task depends on a port scanning task, because only by knowing which ports are open can the corresponding service version be probed. A vulnerability exploitation task depends on a vulnerability discovery task, because the existence of a vulnerability must be confirmed before it can be attempted. These dependencies determine the execution order of tasks, and violating dependencies will lead to task failure or invalid execution. In this embodiment, a dependency representation mechanism based on parent task attributes is adopted: each subtask contains an optional parent task attribute field. When this field is empty, it indicates that there are no prerequisites and it can be executed immediately; when this field points to the ID of another subtask, it indicates that the task must wait for the parent task to complete before it can be executed. Through the parent task attribute field, the task queue is naturally transformed into a directed acyclic graph structure, without the need for additional dependency identification or prompt word reasoning processes. For example, the task queue generated by the planning agent might look like this: Task A: id="task-1", parent_id=null, description="Scan all TCP ports on target 192.168.1.1"; Task B: id="task-2", parent_id="task-1", description="Identify the service versions of open ports on 192.168.1.1"; Task C: id="task-3", parent_id="task-2", description="Enumerate the directories of web services on 192.168.1.1". A dependency graph is constructed by parsing the parent task attribute fields. This dependency graph determines the topological order of the tasks: Task A has no parent task and can be executed immediately; Task B depends on Task A and must be executed after Task A is completed; Task C depends on Task B and must be executed after Task B is completed. Furthermore, many tasks in the graph can be executed concurrently. For example, if Task B and Task C both depend on Task A, but there is no dependency between B and C, then after Task A is completed, B and C can be executed simultaneously. Based on this, embodiments of this application construct a task dependency graph G=(V,E) to achieve concurrent execution of multiple subtasks. Wherein, It is a set of task nodes. It is a set of dependent edges. Each node There is an in-degree This indicates how many tasks must be performed. Completed previously.

[0116] Step 2: Add nodes with an in-degree of 0 in the task dependency graph to the ready queue, and concurrently execute the subtasks corresponding to the nodes included in the ready queue, wherein the in-degree represents the number of preceding nodes required to execute the subtask corresponding to the node.

[0117] Step 3: In response to the ready queue being empty, update the task dependency graph; Step 4: Return to the step of adding the node with an in-degree of 0 in the task dependency graph to the ready queue, until the third stopping condition is met.

[0118] The third cessation condition in this case may include: no tasks to be executed, or the number of times tasks are executed reaches the first threshold, or the number of executed tasks reaches the second threshold.

[0119] Understandably, the scheduling agent's processing includes an initialization phase and an iterative execution phase. In the initialization phase: the in-degree of all tasks is calculated; tasks with an in-degree of 0 are tasks that can be executed immediately and are added to the ready queue. During the iterative execution phase: extract at most [number] [items] from the ready queue. One task, of which This is the current concurrency limit. Concurrent execution of this... Each task, upon completion, updates the dependency graph and then returns to retrieving up to the maximum number of tasks from the ready queue. The steps of each task are repeated until there are no more subtasks to be executed.

[0120] In some embodiments, updating the dependency graph may include: for each completed task Find all dependencies Task ;Will In-degree minus 1: ;if ,Will Add to the ready queue.

[0121] In this embodiment, task dependencies are expressed through the parent task attribute field in the task object, and the task queue is represented as a directed acyclic graph. Then, the scheduling agent constructs a dependency graph based on the parent task attributes and performs topological sorting, identifies ready tasks with an in-degree of 0, and executes them concurrently at a set concurrency degree k, provided that the dependencies are satisfied. This achieves cross-stage, parallelizable task scheduling. Furthermore, the decision-making agent and the scheduling agent execute independently, which effectively improves the flexibility of penetration testing.

[0122] In some embodiments, the concurrent execution of the subtasks corresponding to each node in the ready queue includes: for each subtask, instructing the execution agent to perform the following steps via the scheduling agent: Step 1: Select a task execution tool based on the task description information corresponding to the subtask and infer the execution parameters based on the selected task execution tool.

[0123] In some embodiments, selecting a task execution tool based on the task description information corresponding to the subtask may include: receiving the subtask and a list of available tools; if the task description information corresponding to the subtask already indicates a task execution tool, then using that tool to execute the subtask; otherwise, selecting the most suitable tool from all available tools based on the task description information. In other embodiments, the selection process can be implemented by calling a large language model, whose input includes the task description information and a list of available tools, and whose output is the name of the selected tool and the reason for selection. For example, for the task "scan all open ports of target 192.168.1.1", the model would select the nmap tool because nmap is a standard tool for port scanning.

[0124] In some embodiments, inferring execution parameters based on the selected task execution tool may include: inferring the value of each parameter based on the parameter pattern of the selected task execution tool, whereby the parameter pattern defines which parameters the tool requires, the parameter types, whether they are required, and their value ranges. The inference process can also be implemented using a large language model, which needs to extract relevant information from the task description to populate the parameters. For example, for the port scanning task mentioned above, the model needs to infer that nmap's target parameter is "192.168.1.1" and the ports parameter is "1-65535" indicating a full port scan.

[0125] In some embodiments, the available tools may include, but are not limited to: (1) Information collection tools: nmap_scan (network port and service identification), whois_lookup (domain registration information query), dig_query (DNS record query), nikto_scan (web server security configuration scan), curl_request (custom HTTP request sending).

[0126] (2) Vulnerability scanning tools: sqlmap_scan (automatic detection and exploitation of SQL injection vulnerabilities), openvas_scan (scanning of vulnerabilities in multiple systems and applications), xsstrike_scan (intelligent detection of XSS vulnerabilities).

[0127] (3) Exploitation tools: msfconsole_command (Metasploit framework remote command execution), searchesploit (local ExploitDB vulnerability code retrieval), exploitdb_download (remote ExploitDB exploit script download).

[0128] (4) Post-exploitation tools: mimikatz_run (Windows memory credential extraction), meterpreter_command (Meterpreter in-session command execution), powersploit_run (PowerShell post-exploitation script invocation).

[0129] (5) Other tools: john_crack (offline password hashing), hydra_brute (multi-protocol online brute-force attack), wireshark_tshark (command-line network traffic capture and analysis), etc.

[0130] Step 2: Based on the selected task execution tool and the execution parameters obtained through reasoning, execute the sub-task and obtain the execution result corresponding to the sub-task.

[0131] In some embodiments, step 2 may include: the executing agent sending a request to the MCP server via a tool invoker, passing the tool name and parameter dictionary; handling details such as network communication, timeout control, and error handling through the tool invoker, and returning the execution result, which includes fields such as success flag, raw data, and error message. It is understood that tool invocation can use the MCP protocol, a standardized tool service interface protocol. Each tool provides description information and parameter patterns, and the executing agent can obtain a tool list, query tool details, invoke tools, and obtain results through an MCP client. This standardized interface allows for easy expansion of new tools during testing without modifying the core logic.

[0132] Step 3: Update the target global state information based on the execution result, and return the execution result to the decision-making agent.

[0133] Understandably, after obtaining the execution result, the executing agent parses the result and extracts key information. The execution result is typically unstructured or semi-structured text; for example, nmap's output is formatted text containing port numbers, service names, version information, etc. Therefore, the execution result needs to be parsed. In some embodiments, this can be done using a large language model to extract a structured list of discoveries. For example, extracting information like "Port 22 is open, running OpenSSH 7.4 service" from the nmap output. The extracted information is then converted into a global state update. In some embodiments, this update process can be implemented by the large language model by setting state update prompts. The large language model needs to determine which information is a new key discovery, which is credentials, which is a vulnerability, and which is session information, and organize them according to the global state data structure.

[0134] It should be understood that the training process of the intelligent agent involved in each embodiment of this application follows multiple legal and compliant principles, including legal data sources, compliant data content, compliant data governance, compliant training objectives and schemes, compliant training processes, compliant training environments and tools, and compliant ethical verification of training results. (1) Legality of data sources: The datasets used for training the intelligent agent were all obtained through legal means, covering three categories: publicly authorized data, data authorized by partners, and self-collected compliant data. Among them, the publicly authorized data comes from compliant data sources that follow open source protocols such as Apache 2.0, and the copyright ownership and scope of authorization have been fully marked. There is no unauthorized open source code or data reuse. The data authorized by partners has a formal data use agreement that clearly defines the scope of data use, the term and confidentiality obligations, and has a complete authorization chain. For the self-collected data, the informed consent procedure has been strictly followed for the parts involving personal information. The information that can identify personal identity has been removed through de-identification processing (including but not limited to field masking, feature anonymization and differential privacy technology application), which fully complies with the requirements of relevant laws and regulations such as the "Interim Measures for the Administration of Generative Artificial Intelligence Services" and the "Personal Information Protection Law".

[0135] (2) Data Content Compliance: The dataset of the intelligent agent undergoes multiple screenings and cleaning processes to remove all content that may violate social morality or harm public interests. It contains no obscene, pornographic, violent, discriminatory, or information that endangers national or public safety, nor does it involve the illegal acquisition or use of genetic resources. For data in sensitive fields (such as medical and financial fields), a privacy computing module (including federated learning and secure multi-party computation technology) is used to ensure that the data is "usable but not visible," avoiding compliance risks during the original data transmission process and ensuring that the data application scenarios and uses comply with public order and good morals and industry regulatory requirements.

[0136] (3) Data governance standardization: A complete data traceability system is established during the training process of the intelligent agent to automatically record the source, collection time, annotation process, cleaning rules and permission allocation of the training data, and generate traceable compliance reports to ensure that the data is verifiable throughout its entire life cycle. The dataset annotation process of the intelligent agent is completed by a professional human R&D team, which clarifies the proportion of human creative contributions, avoids relying on AI-generated data that has not been substantially modified by humans, and meets the examination requirements of "human main contribution" in AI patent applications.

[0137] (4) Compliance of training objectives and plans: The training plan and the final output results do not violate any mandatory provisions of laws and administrative regulations, do not harm the public interest or the legitimate rights and interests of others, and do not pose any potential risks of being used for illegal activities, privacy infringement, or public safety damage. The ethical principle of "intelligent for good" is strictly practiced.

[0138] (5) Compliance of training process: The training process is compliant and controllable by adopting a closed-loop training framework.

[0139] (6) Compliance of Training Environment and Tools: The intelligent agent training is implemented based on nationally licensed chips and a compliant training platform. All open-source frameworks and components used in the training process have obtained corresponding authorizations, and copyright statements and patent citation information are fully retained, with no infringement or reuse. The training environment is constructed using virtual devices (containers / virtual machines), with fixed random seeds and initial parameter configurations to ensure the reproducibility of the training process. At the same time, through access control, operation log recording, and other methods, risks such as data leakage and parameter tampering during the training process are prevented, ensuring the security and compliance of the training process.

[0140] (7) Ethical verification and compliance of training results: After the model is trained, it will be further verified through third-party ethical compliance assessment and algorithm filing review to ensure that the model output results do not violate social morality or harm public interests. For sensitive scenarios that may be involved (such as public services and intelligent decision-making), a special result verification mechanism will be established to ensure that the model always complies with Article 5 of the Patent Law and relevant laws and regulations in practical applications.

[0141] In summary, the data and training process used by the intelligent agent in this specification strictly comply with the relevant provisions of Article 5 of the Patent Law and the Patent Examination Guidelines (2023 Edition), and there are no violations of laws, social ethics, public interests, or illegal use of genetic resources. Therefore, it fully meets the compliance requirements for patent authorization.

[0142] Figure 6 An embodiment of the present application illustrates a test system that may include a decision agent 610, a scheduling agent 620, and an execution agent 630. The decision agent 610 is configured to: generate a target task sequence based on the target test information and target global state information corresponding to the test object; execute each subtask included in the target task sequence using a scheduling agent to obtain the test result corresponding to the test object; the scheduling agent 620 is configured to: receive the target task sequence sent by the decision agent; construct a task dependency graph based on the dependency attributes of each subtask in the target task sequence; concurrently execute the subtasks corresponding to nodes with an in-degree of 0 in the task dependency graph using an execution agent in each execution round until a preset stopping condition is met; the execution agent 630 is configured to: receive the subtask sent by the scheduling agent; select a task execution tool based on the task description information corresponding to the subtask and infer execution parameters based on the selected task execution tool; execute the subtask based on the selected task execution tool and the inferred execution parameters to obtain the execution result corresponding to the subtask; update the target global state information based on the execution result; and return the execution result to the decision agent.

[0143] The following is through Figure 7 The interactive diagram shown above serves as an example to illustrate the aforementioned test system. Figure 7 As shown, the following steps may be included: S701: Target the information to be tested to the decision-making agent.

[0144] S702: Initialize the target's global state information based on the target's test information.

[0145] S703: Based on the target test information, perform scene recognition and obtain the corresponding list of execution tools.

[0146] S704: Generate a sequence of target tasks based on the target test information, or generate a sequence of target tasks based on the target test information and a list of execution tools.

[0147] S705: Transmits the target task sequence to the scheduling agent.

[0148] S706: Construct a task dependency graph based on the dependency attributes of each subtask in the target task sequence.

[0149] S707: Extract executable subtasks through topological sorting.

[0150] S708: Transfers executable subtasks to the executing agent.

[0151] S709: Select the execution tool corresponding to each executable subtask and deduce the tool parameters of the selected execution tool.

[0152] S710: Invokes each selected execution tool to execute the corresponding subtask.

[0153] S711: Update the target's global state information based on the execution result.

[0154] S712: Return the execution result to the decision-making agent.

[0155] S713: Adjust the target task sequence based on the updated target global state information.

[0156] S714: Exit the loop in response to the absence of more tasks or the reaching of the task execution limit.

[0157] S715: Generate a test report based on the execution results of each subtask.

[0158] S716: Returns the test report to the user.

[0159] Understandably, during the initialization phase (S701-S704), the user submits target information, including the target address, type, and description. The system first initializes the global state object, then the planning agent performs scene recognition based on the target information, matches the corresponding skill set, and calls the large language model to generate an initial task sequence. The initial task sequence typically contains multiple high-priority information gathering tasks, such as port scanning, service identification, and directory enumeration. Each task contains a task ID and an optional parent task ID, naturally forming a task dependency graph through the parent task ID field. During the execution loop phase (S705-S712), the system enters the core iterative process. The task scheduler constructs a directed acyclic graph based on the parent task ID field in the task object, and identifies all tasks with satisfied dependencies through topological sorting; these tasks can be executed concurrently. The execution agent processes these tasks concurrently, with each task independently completing the process of tool selection, parameter inference, tool invocation, and result parsing. After task execution, the execution agent updates the newly discovered information to the global state, while simultaneously performing semantic deduplication to avoid redundancy. After the execution results are returned to the planning agent (S713), the planning agent obtains the latest global state and dynamically adjusts the task queue based on new discoveries. For example, when a high-risk vulnerability is discovered, an exploit task is inserted; when a new endpoint is discovered, a probe task is added. The newly generated tasks also contain a parent task ID field, forming a complete dependency graph with existing tasks. The updated plan continues into the next iteration. In the termination phase (S714-S716), when the task queue is empty or the preset execution limit is reached, the system generates a complete test report, including all discoveries, credentials, vulnerabilities, task execution details, etc., presented to the user in a structured format.

[0160] Furthermore, to achieve loose coupling between the agents, this application employs standardized message protocols and data structures. All communication between agents is based on message objects, with message types including PLAN messages for requesting a generation plan, TASK messages for assigning tasks, RESULT messages for returning execution results, and STATUS messages for querying status.

[0161] This penetration testing system forms a closed loop from target input to report output, continuously updating the state and adjusting tasks and strategies based on execution feedback. This supports iterative optimization and subsequent expansion, rather than a single, fixed script execution. Through structured global state and continuous reconstruction of the task DAG by a planning agent, adaptive planning—dynamically adjusting subsequent strategies based on preceding results—is achieved, replacing linear or fixed state machines and providing cross-stage decision-making capabilities. Furthermore, by decoupling planning and execution at two layers and standardizing messages, "planning and state evolution" are separated from "tool invocation and parsing," reducing context bloat and noise, minimizing illusions and repetitive calls, and making the decision path clearer and more traceable. Additionally, the DAG can be naturally obtained by expressing dependencies using only parent task attributes, without additional dependency parsing or cue word reasoning; combined with topological sorting and concurrent scheduling of the ready queue, parallelism is improved while maintaining order.

[0162] Figure 8 This illustration shows a schematic diagram of a penetration testing architecture provided in an embodiment of this application, such as... Figure 8 As shown, the system comprises an intelligent decision-making layer, a task scheduling layer, a tool execution layer, and a state management layer. This separation of decision-making and execution logic at different granularities allows each layer to focus on specific responsibilities while enabling inter-layer collaboration through standardized interfaces. The intelligent decision-making layer generates test tasks based on target information and the current state. A scenario skill library provides specialized knowledge and strategies for different types of targets. The core of this layer is reasoning and planning capabilities driven by a large language model. The task scheduling layer extracts executable tasks from the task queue, performs topological sorting based on dependencies between tasks, and achieves concurrent execution. The core of this layer is dependency identification and concurrency control algorithms. The tool execution layer handles the invocation of specific tools and result parsing. After receiving a task, the executing agent selects an appropriate tool and infers parameters. The tool invoker encapsulates communication with the MCP tool server, which provides standardized penetration testing tool interfaces. The core of this layer is understanding tool capabilities and automatic parameter inference. The state management layer is used for information persistence and retrieval. The global state stores all discoveries, credentials, vulnerabilities, sessions, and other information during the penetration testing process. The core of this layer is structured knowledge representation and an efficient query mechanism, which provides complete contextual information for planning intelligent agents.

[0163] Figure 9 This invention provides a schematic diagram of the structure of an agent training device according to an embodiment of the present application, as shown below. Figure 9 As shown, the intelligent agent training device 900 may include: a generation module 910, an evaluation module 920, an update module 930, and a first training module 940.

[0164] In this embodiment, the generation module 910 is used to input the first test information sample corresponding to the current reinforcement training round into the decision agent to generate a first task sequence corresponding to the first test information sample; the evaluation module 920 is used to evaluate the reasonableness of the dependency relationship of the first task sequence based on the dependency attribute of each sub-task in the first task sequence, wherein the dependency attribute is used to instruct the sub-task to execute the parent task it depends on; the update module 930 is used to update the decision parameters of the decision agent according to the evaluation result, wherein the evaluation result includes the reasonableness evaluation result of the dependency relationship; the first training module 940 is used to enter the next reinforcement training round and repeat the above steps until the first stopping condition is met, thereby obtaining the decision agent after reinforcement training is completed.

[0165] In some embodiments, the evaluation module 920 is specifically configured to: assign a first penalty value to the first task sequence in response to a circular dependency existing between the multiple subtasks; assign a second penalty value to the first task sequence in response to the absence of a parent task indicated by the dependency attribute of the first subtask in the first task sequence; and assign a third penalty value to the first task sequence in response to the absence of a logical dependency between the second subtask and the third subtask in the task sequence, wherein the parent task indicated by the dependency attribute of the second subtask is the third subtask.

[0166] In some embodiments, each subtask corresponds to a task description information, and the evaluation result further includes a format evaluation result and / or a semantic clarity evaluation result; the device further includes: a format structure verification module, used to perform format structure verification on the task description information corresponding to each subtask in the first task sequence to obtain the format evaluation result, wherein the format structure verification includes field integrity verification, field type verification, enumerated field value out-of-bounds verification, and semantic parsing verification; and / or, a fuzzy expression recognition module, used to perform fuzzy expression recognition on the task description information corresponding to each subtask in the first task sequence to obtain the semantic clarity evaluation result.

[0167] In some embodiments, the device further includes a second training module, configured to: acquire a pre-training sample set, wherein the pre-training sample set includes a plurality of second test information samples, each second test information sample being labeled with corresponding global state information and a second task sequence; input each second test information sample into the decision agent to generate a plurality of third task sequences; calculate a target loss based on the second task sequence corresponding to each second test information sample and the generated third task sequence; update the decision parameters of the decision agent based on the target loss until a second stopping condition is met, thereby obtaining the pre-trained decision agent.

[0168] In some embodiments, the device further includes a third training module, configured to: label each of the first task sequences based on the execution results corresponding to each of the first task sequences to determine a negative sample set, wherein the labeling dimensions include: a dimension for judging the rationality of the execution tool, a dimension for judging the rationality of the task generation timing, a dimension for judging the completeness of the use of global state information, and a dimension for judging the rationality of the task granularity; and perform negative sample reinforcement training on the decision agent based on the negative sample set.

[0169] In some embodiments, the third training module is specifically configured to: determine a positive sample set based on the annotation results of each first task sequence; set a first weight corresponding to the positive sample set and a second weight corresponding to the negative sample set, wherein the second weight is greater than the first weight; and perform negative sample reinforcement training on the decision agent based on the positive sample set and the negative sample set.

[0170] The intelligent agent training device provided in this application embodiment can realize the various processes implemented in the method embodiment of the intelligent agent training method shown above. To avoid repetition, it will not be described again here.

[0171] The intelligent agent training device in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in an electronic device. This application embodiment does not impose specific limitations.

[0172] The intelligent agent training device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0173] Figure 10 This invention provides a schematic diagram of the structure of a penetration testing apparatus according to an embodiment of the present application, as shown below. Figure 10 As shown, the penetration testing device 1000 may include: an acquisition module 1010, a generation module 1020, and a testing module 1030.

[0174] In this embodiment, the acquisition module 1010 is used to acquire the target test information and target global state information corresponding to the test object; the generation module 1020 is used to generate the target task sequence corresponding to the target test information based on the target test information and the target global state information; and the testing module 1030 is used to obtain the test result corresponding to the test object by executing each sub-task included in the target task sequence.

[0175] In some embodiments, the test module 1030 is specifically used to: transmit the target task sequence to the scheduling agent, and instruct the scheduling agent to perform the following steps: construct a task dependency graph based on the dependency attributes of each subtask in the target task sequence, wherein the task dependency graph includes a set of nodes and a set of dependency edges, the set of nodes includes multiple nodes, each node corresponds to a task sequence, the set of dependency edges includes multiple dependency edges, each dependency edge connects two nodes and has a directional property, used to indicate the execution dependency relationship between the two nodes; add nodes with an in-degree of 0 in the task dependency graph to a ready queue, and concurrently execute the subtasks corresponding to the nodes included in the ready queue, wherein the in-degree represents the number of preceding nodes required to execute the subtask corresponding to the node; update the task dependency graph in response to the ready queue being empty; return to the step of adding nodes with an in-degree of 0 in the task dependency graph to the ready queue, until a third stopping condition is met.

[0176] In some embodiments, the test module 1030 is specifically configured to: for each subtask, instruct the execution agent to perform the following steps through the scheduling agent: select a task execution tool based on the task description information corresponding to the subtask and infer execution parameters based on the selected task execution tool; execute the subtask based on the selected task execution tool and the inferred execution parameters to obtain the execution result corresponding to the subtask; update the target global state information based on the execution result and return the execution result to the decision agent.

[0177] The penetration testing apparatus provided in this application embodiment can realize the various processes implemented in the method embodiment of the penetration testing method described above. To avoid repetition, it will not be described again here.

[0178] The penetration testing device in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in an electronic device. This application embodiment does not impose specific limitations.

[0179] One penetration testing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0180] like Figure 11As shown, this application embodiment also provides an electronic device 1100, including a processor 1110 and a memory 1120. The memory 1120 stores a program or instructions that can run on the processor 1110. When the program or instructions are executed by the processor 1110, they implement the various processes of the embodiments of the above-described intelligent agent training method, or implement the various processes of the embodiments of the above-described penetration testing method, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0181] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the embodiments shown in the above-described agent training method or the various processes of the embodiments shown in the above-described penetration testing method, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0182] The processor mentioned above is the processor in the terminal described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

[0183] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the embodiments shown in the above-described intelligent agent training method, or to implement the various processes of the embodiments shown in the above-described penetration testing method, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0184] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0185] This application also provides a computer program / program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to implement the various processes of the embodiments shown in the above-described agent training method, or to implement the various processes of the embodiments shown in the above-described penetration testing method, and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0186] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0187] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes several instructions to cause the terminal or network-side device to execute the methods described in the various embodiments of this application.

[0188] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this application without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this application.

Claims

1. A method for training an intelligent agent, characterized in that, The method includes: Input the first test information sample corresponding to the current reinforcement training round into the decision-making agent to generate the first task sequence corresponding to the first test information sample; Based on the dependency attributes of each subtask in the first task sequence, the reasonableness of the dependency relationship of the first task sequence is evaluated, wherein the dependency attributes are used to indicate the parent task that the subtask depends on for execution; The decision parameters of the decision-making agent are updated based on the evaluation results, wherein the evaluation results include the evaluation results of the rationality of the dependency relationship; Enter the next reinforcement training round and repeat the above steps until the first stopping condition is met, and obtain the decision agent with completed reinforcement training.

2. The method according to claim 1, characterized in that, The step of evaluating the reasonableness of dependencies in the first task sequence based on the dependency attributes of each subtask in the first task sequence includes: In response to a circular dependency existing among the multiple subtasks, a first penalty value is assigned to the first task sequence; In response to the absence of the parent task indicated by the dependency attribute of the first subtask in the first task sequence, a second penalty value is assigned to the first task sequence; In response to the absence of a logical dependency between the second and third subtasks in the task sequence, a third penalty value is assigned to the first task sequence, wherein the parent task indicated by the dependency attribute of the second subtask is the third subtask.

3. The method according to claim 1, characterized in that, Each subtask corresponds to a task description information, and the evaluation results also include format evaluation results and / or semantic clarity evaluation results; Before evaluating the reasonableness of dependencies in the first task sequence based on the dependency attributes of each subtask in the first task sequence, the method further includes: The format structure verification is performed on the task description information corresponding to each subtask in the first task sequence to obtain the format evaluation result. The format structure verification includes field integrity verification, field type verification, enumerated field value out-of-bounds verification, and semantic parsing verification. And / or, The semantic clarity assessment result is obtained by performing fuzzy representation recognition on the task description information corresponding to each subtask in the first task sequence.

4. The method according to claim 1, characterized in that, Before inputting the first test information sample corresponding to the current reinforcement training round into the decision agent and generating the first task sequence corresponding to the first test information sample, the method further includes: Obtain a pre-training sample set, wherein the pre-training sample set includes multiple second test information samples, each of which is labeled with corresponding global state information and a second task sequence; Each second test information sample is input into the decision-making agent to generate multiple third task sequences; The target loss is calculated based on the second task sequence corresponding to each second test information sample and the generated third task sequence. The decision parameters of the decision agent are updated based on the target loss until the second stopping condition is met, thus obtaining the pre-trained decision agent.

5. The method according to claim 1, characterized in that, After the decision agent has completed its reinforcement training, the method further includes: Based on the execution results corresponding to each of the first task sequences, each of the first task sequences is labeled to determine the negative sample set. The labeling dimensions include: the rationality judgment dimension of the execution tool, the rationality judgment dimension of the task generation timing, the completeness judgment dimension of the global state information utilization, and the rationality judgment dimension of the task granularity. Based on the negative sample set, the decision-making agent is subjected to negative sample reinforcement training.

6. The method according to claim 5, characterized in that, The step of performing negative sample reinforcement training on the decision-making agent based on the negative sample set includes: Based on the annotation results of each of the first task sequences, a positive sample set is determined; Set a first weight corresponding to the positive sample set and a second weight corresponding to the negative sample set, wherein the second weight is greater than the first weight; Based on the positive sample set and the negative sample set, the decision-making agent is subjected to negative sample reinforcement training.

7. A penetration testing method, characterized in that, Applied to a decision-making agent, wherein the decision-making agent is trained based on the agent training method according to any one of claims 1-6, the method comprising: Obtain the target test information and target global state information corresponding to the object to be tested; Based on the target test information and the target global state information, a target task sequence corresponding to the target test information is generated; By executing each subtask included in the target task sequence, the test result corresponding to the object to be tested is obtained.

8. The method according to claim 7, characterized in that, The step of obtaining the test result corresponding to the object under test by executing each subtask included in the target task sequence includes: The target task sequence is transmitted to the scheduling agent, which is then instructed to perform the following steps: Based on the dependency attributes of each subtask in the target task sequence, a task dependency graph is constructed. The task dependency graph includes a set of nodes and a set of dependency edges. The set of nodes includes multiple nodes, each node corresponding to a task sequence. The set of dependency edges includes multiple dependency edges, each dependency edge connecting two nodes and having a directional property, used to indicate the execution dependency relationship between the two nodes. Nodes with an in-degree of 0 in the task dependency graph are added to the ready queue, and the subtasks corresponding to the nodes included in the ready queue are executed concurrently. The in-degree represents the number of preceding nodes required to execute the subtask corresponding to the node. In response to the ready queue being empty, update the task dependency graph; Return to the step of adding the node with an in-degree of 0 in the task dependency graph to the ready queue, until the third stopping condition is met.

9. The method according to claim 8, characterized in that, The concurrent execution of the subtasks corresponding to each node in the ready queue includes: For each of the subtasks, the scheduling agent instructs the execution agent to perform the following steps: Based on the task description information corresponding to the sub-task, a task execution tool is selected, and execution parameters are inferred based on the selected task execution tool; Based on the selected task execution tool and the execution parameters obtained through reasoning, the sub-task is executed to obtain the execution result corresponding to the sub-task; The global state information of the target is updated based on the execution result, and the execution result is returned to the decision-making agent.

10. A penetration testing system, characterized in that, include: A decision-making agent is used to: generate a target task sequence based on the target test information and target global state information corresponding to the test object; execute each sub-task included in the target task sequence using a scheduling agent; and obtain the test result corresponding to the test object. The scheduling agent is used to: receive the target task sequence sent by the decision agent; construct a task dependency graph based on the dependency attributes of each sub-task in the target task sequence; and in each execution round, concurrently execute the sub-tasks corresponding to the nodes with an in-degree of 0 in the task dependency graph using the execution agent until a preset stopping condition is met. An execution agent is configured to: receive the subtask sent by the scheduling agent; select a task execution tool based on the task description information corresponding to the subtask and infer execution parameters based on the selected task execution tool; execute the subtask based on the selected task execution tool and the inferred execution parameters; obtain the execution result corresponding to the subtask; update the target global state information based on the execution result; and return the execution result to the decision agent.

Images