Instruction verification method, device and system of large language model

Abstract

This invention discloses a method, apparatus, and system for verifying instructions from a large language model. The method includes: obtaining at least one candidate instruction generated by a large language model for a user task; filtering all candidate instructions based on a set of security policies to obtain a set of secure candidate instructions; performing a decision evaluation calculation on the set of secure candidate instructions to obtain a decision result, wherein the decision result includes an output instruction and corresponding authentication information, and the output instruction can only be executed after the authentication information has been verified. This invention can solve the technical problem of insufficient reliability of instructions generated by large language models.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of artificial intelligence security, trusted computing, intelligent agent control and automated system security technology, and in particular to instruction verification methods, devices and systems for large language models. Background Technology

[0002] In recent years, large language models have demonstrated strong capabilities in natural language understanding, text generation, tool invocation planning, and complex task decomposition, and are increasingly being used for task planning and execution control in automated systems. In a typical large language model-driven system, after the user inputs a task, the user-described task, historical context, system prompts, external tool return results, and environmental status are all input into the large language model. The large language model then generates subsequent operation instructions, execution plans, or tool invocation parameters, and the large language model-driven system executes the actual operation based on the output.

[0003] However, in actual deployment, this type of large language model-driven system has the following security issues.

[0004] First, the output of a large language model is highly dependent on the input context, which is typically maintained and dynamically assembled by the host system. If an attacker can manipulate user tasks, external documents, web page content, tool return values, intermediate state information, or system prompts, they may be able to influence the inference results of the large language model without directly altering the executable program, thereby inducing the large language model to generate insecure instructions. This type of problem is commonly referred to as a prompt injection attack or a context manipulation attack.

[0005] Second, existing large language model-driven systems typically trust the output of a single large language model by default, lacking verification records. After a large language model-driven system performs a high-risk operation, administrators often find it difficult to verify whether the operation originated from a legitimate task and whether the process was trustworthy.

[0006] Therefore, there is an urgent need for a technical solution that can reliably verify the instructions generated by large language models.

[0007] The background section described above is merely a description made by the inventor based on his understanding, and the above content should not be regarded as evidence of prior art disclosed before the filing date of this application. Summary of the Invention

[0008] This invention provides a method for verifying instructions in a large language model, addressing the technical problem of insufficient reliability of instructions generated by large language models, including: Obtain at least one candidate instruction generated by a large language model for the user task; All candidate instructions are filtered based on the set of security policies to obtain a set of secure candidate instructions; A decision evaluation calculation is performed on the set of security candidate instructions to obtain a decision result. The decision result includes an output instruction and corresponding authentication information. The output instruction can only be executed after the authentication information is verified.

[0009] Another aspect of the present invention provides an instruction verification device for a large language model to solve the technical problem of insufficient reliability of instructions generated by a large language model, comprising: The candidate instruction acquisition module is used to obtain at least one candidate instruction generated by a large language model for the user task. The security filtering module is used to filter all candidate instructions based on a set of security policies to obtain a set of secure candidate instructions. The decision evaluation module is used to perform decision evaluation calculations on the set of security candidate instructions to obtain decision results. The decision results include output instructions and corresponding authentication information. The output instructions can only be executed after the authentication information is verified.

[0010] Another aspect of the present invention provides an instruction verification system for large language models to solve the technical problem of insufficient reliability of instructions generated by large language models. The system includes an instruction verification device for large language models, an intelligent agent module, a task scheduling module, multiple large language models, and an execution control module. The intelligent agent module is used to: receive user tasks and generate task descriptions; The task scheduling module is used to: distribute task descriptions to at least one large language model, and send the received candidate instructions to the instruction verification device of the large language model. The large language model is used to: generate candidate instructions based on the received task description and return them to the task scheduling module; The execution control module is used to: verify the authentication information in the decision result output by the instruction verification device of the large language model, and execute the output instruction after the verification is successful.

[0011] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the instruction verification method for the large language model described above.

[0012] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described instruction verification method for a large language model.

[0013] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the instruction verification method for the large language model described above.

[0014] In this embodiment of the invention, at least one candidate instruction generated by a large language model for a user task is obtained; all candidate instructions are filtered based on a security policy set to obtain a set of secure candidate instructions; the set of secure candidate instructions is evaluated to obtain a decision result, which includes an output instruction and authentication information. The output instruction can only be executed after the authentication information is verified. Compared with existing technical solutions where the reliability of instructions generated by large language models is insufficient, filtering candidate instructions using a security policy set can effectively intercept instructions with security risks, avoiding system risks, data leakage risks, and operational security risks caused by harmful instructions generated by large language models from the source, and ensuring the security and reliability of the instruction execution environment. By evaluating the set of secure candidate instructions, output instructions that meet the user task requirements, are logically consistent, and have compliant content can be selected, and authentication information can be generated. Execution is only possible after authentication, further enhancing the controllability of instruction execution and avoiding mis-execution and illegal execution. Binding authentication information to output instructions ensures that only instructions that have passed security verification can be executed, and also enables traceability management of instructions through authentication information, facilitating subsequent investigation of problems in the instruction execution process and improving the regulatory oversight of the entire instruction processing flow. The methodology is clear and adaptable to the instruction generation and verification needs of large language models in different scenarios. It does not require large-scale modification of the large language model itself, and security control can be achieved solely through the verification of authentication information. It balances the flexibility of the large language model with the security of instruction execution, making it highly practical. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart of instruction verification for a large language model in an embodiment of the present invention; Figure 2 This is a schematic diagram of the instruction verification device for a large language model in an embodiment of the present invention; Figure 3 This is a schematic diagram of the instruction verification system for a large language model in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the deployment principle of the instruction verification device for the large language model in an embodiment of the present invention; Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0017] Figure 1 This is a flowchart of the instruction verification method for a large language model in an embodiment of the present invention, such as... Figure 1 As shown, the method includes: Step 101: Obtain at least one candidate instruction generated by a large language model for the user task; Step 102: Filter all candidate instructions based on the security policy set to obtain a set of secure candidate instructions; Step 103: Perform decision evaluation calculation on the set of security candidate instructions to obtain decision results. The decision results include output instructions and corresponding authentication information. The output instructions can only be executed after the authentication information is verified.

[0018] Depend on Figure 1 As illustrated in the process, in this embodiment of the invention, filtering candidate instructions using a set of security policies can effectively intercept instructions with security vulnerabilities. This avoids system risks, data leakage risks, and operational security risks caused by harmful instructions generated by large language models from the source, ensuring the security and trustworthiness of the instruction execution environment. By evaluating the set of secure candidate instructions, output instructions that meet user task requirements, are logically consistent, and have compliant content can be selected, and authentication information is generated. Execution is only permitted after successful authentication, further enhancing the controllability of instruction execution and preventing erroneous or illegal execution. Binding authentication information to output instructions ensures that only instructions that pass security verification can be executed, and also enables traceability management of instructions through authentication information, facilitating subsequent troubleshooting of problems during instruction execution and improving the overall regulatory oversight of the instruction processing flow. The method is clear and adaptable to the instruction generation and verification needs of large language models in different scenarios. It does not require large-scale modifications to the large language model itself; security control can be achieved solely through authentication information verification. It balances the flexibility of the large language model with the security of instruction execution, making it highly practical.

[0019] Figure 2 This is a schematic diagram of an instruction verification device for a large language model in an embodiment of the present invention, comprising: Candidate instruction acquisition module 201 is used to acquire at least one candidate instruction generated by a large language model for a user task; The security filtering module 202 is used to filter all candidate instructions based on the security policy set to obtain a set of secure candidate instructions; The decision evaluation module 203 is used to perform decision evaluation calculations on the set of security candidate instructions to obtain decision results. The decision results include output instructions and corresponding authentication information. The output instructions can only be executed after the authentication information is verified.

[0020] Figure 3 This is a schematic diagram of a large language model instruction verification system in an embodiment of the present invention. The system includes a large language model instruction verification device 301, an intelligent agent module 302, a task scheduling module 303, multiple large language models 304, and an execution control module 305. The intelligent agent module 302 is used to: receive user tasks and generate task descriptions; The task scheduling module 303 is used to: distribute task descriptions to at least one large language model, and send the received candidate instructions to the instruction verification device of the large language model. The large language model 304 is used to: generate candidate instructions based on the received task description and return them to the task scheduling module; The execution control module 305 is used to: verify the authentication information in the decision result output by the instruction verification device of the large language model, and execute the output instruction after the verification is successful.

[0021] The above-mentioned large language model instruction verification system can be applied to a variety of automated systems, including but not limited to: (1) large language model intelligent agent system; (2) intelligent agent system; (3) automated operation and maintenance system; (4) automated office system; (5) automated transaction system; (6) intelligent robot control system; (7) data access and query agent system; (8) industrial control auxiliary decision system; (9) multi-agent collaborative execution system.

[0022] The intelligent agent module can receive natural language tasks input by the user, complete task parsing and semantic understanding, remove invalid information and interference content in the task, extract the core execution goals and constraints, generate a standardized and structured task description T, clarify the task type, execution scope, risk level and expected results, and pass the task description T to the task scheduling module to complete the standardized conversion of the task entry point.

[0023] The task scheduling module receives the standardized task description output by the intelligent agent module, distributes the task description T to one or more large language models, records the model identifier, distribution time, and session ID, establishes the correspondence between tasks and model instances, waits for candidate instructions to be returned by the large language models, and marks the output of the large language model as invalid if no response is returned within a timeout period. All candidate instructions are summarized to form a candidate instruction set A={a1,a2,...,an}, which is then passed to the large language model for instruction verification.

[0024] Multiple large language models receive task descriptions T from the task scheduling module, and simultaneously load the context information and system constraints of the current session. Based on the task description, context, and constraint rules, they perform inference calculations to generate candidate instructions ai. The candidate instructions contain core information such as operation type, target object, execution parameters, triggering conditions, and risk level. Each candidate instruction is appended with a source model identifier and a timestamp to ensure traceability. The generated candidate instructions are then returned to the task scheduling module.

[0025] The instruction verification device for large language models, as a trusted decision layer added between "model generation instructions" and "system execution instructions", is deployed in a trusted entity or a distributed system composed of multiple trusted entities. The trusted entity includes, but is not limited to, a trusted execution environment, a hardware security module, an isolated virtual machine, a trusted coprocessor, and a trusted execution platform with security isolation, key protection and integrity proof capabilities.

[0026] In one embodiment, before filtering all candidate instructions based on a set of security policies, the method further includes: All candidate instructions are normalized into a structured representation to obtain a normalized candidate instruction set. The structured representation includes, but is not limited to, operation type, target object, operation parameters, execution conditions, risk level, required permissions, source model identifier, timestamp, and session identifier. All candidate instructions are filtered based on a set of security policies, including: The normalized candidate instruction set is filtered based on the set of security policies to obtain a set of secure candidate instructions.

[0027] In the above embodiments, due to differences in training data, prompting engineering, or internal architecture, different large language models may generate candidate instructions that vary greatly in expression, even for the same task. For example, one large language model might output "delete file / var / log / tmp", another might output "remove / var / log / tmp", and a third might output "rm -rf / var / log / tmp". If these raw candidate instructions are directly filtered and compared, it is easy to miss or misjudge them.

[0028] Therefore, before entering the security filtering stage, the system first "translates" all candidate commands into a unified, structured internal representation. This structured representation is similar to a standardized form, breaking down candidate commands into multiple fixed fields, including operation type (such as delete, modify, query), target object (such as file path, database table), operation parameters (such as recursive deletion flag), execution conditions (such as time limit), risk level, and required permissions. Simultaneously, metadata such as source model identifier, timestamp, and session identifier are attached to facilitate subsequent traceability and auditing. After this standardization process, the originally varied candidate commands become standardized data that can be processed uniformly. The security filtering module can then perform precise matching and checks on these structured fields based on a predefined set of security policies.

[0029] In one embodiment, the set of security policies includes, but is not limited to, candidate instructions for prohibiting access to sensitive directories, candidate instructions for prohibiting execution of dangerous levels greater than a level threshold, candidate instructions for restricting actions involving risks greater than a risk threshold, candidate instructions for restricting cross-system calls, candidate instructions for restricting access to preset types of data, candidate instructions that require the principle of least privilege to meet the requirements, candidate instructions that require manual confirmation, candidate instructions that require secondary verification or multiple consensus, and candidate instructions that prohibit the deletion of preset files.

[0030] In the above embodiments, the preset type data includes key configurations, key materials, authentication credentials, and database tables; the preset files include files in the system core directory, root directory, configuration directory, or certificate directory, etc. A set of security policies is used to intercept various dangerous instructions that the large language model may generate. These security policies review candidate instructions from multiple dimensions: The first category is access control policies, such as prohibiting access to sensitive directories (e.g., the system root directory, configuration file directory) and restricting access to preset types of data (e.g., key materials, authentication credentials, database tables) to prevent unauthorized access to core resources.

[0031] The second category is risk control strategies, such as prohibiting the execution of instructions with excessively high risk levels, restricting actions with risks exceeding thresholds, and prohibiting the deletion of preset files (such as system core files and certificate files) to avoid high-risk operations from damaging the system.

[0032] The third category is permission management strategies, such as requiring instructions to meet the principle of least privilege, ensuring that instructions only request the minimum permissions necessary for their execution, and preventing permission abuse.

[0033] The fourth category is process control strategies, such as instructions that require manual confirmation, instructions that require secondary verification or multiple consensuses, and setting up additional approval steps for high-risk operations.

[0034] These strategies can be combined to form a multi-layered defense system.

[0035] In one embodiment, a decision evaluation calculation is performed on the set of security candidate instructions to obtain a decision result, including: A consistency calculation is performed on the set of security candidate instructions to obtain the consistency matrix; Cluster the consistency matrix to obtain the output instructions; The candidate instructions generated by all large language models are concatenated and their hashes are calculated to obtain a summary of the candidate instruction set. Calculate the hash of the output instruction, generate an output instruction digest, and generate authentication information based on the output instruction digest; The decision result is constructed based on the output instruction, authentication information, and a summary of the candidate instruction set.

[0036] In one embodiment, a consistency calculation is performed on the set of security candidate instructions to obtain a consistency matrix, including: Each candidate instruction in the security candidate instruction set is split into multiple layers of feature vectors. The multiple layers of feature vectors include surface feature vectors, middle feature vectors, deep feature vectors, and meta-layer feature vectors. The surface feature vectors are used to represent the operation type and target object. The middle feature vectors are used to represent the operation parameters and execution conditions. The deep feature vectors are used to represent the required permissions and risk level. The meta-layer feature vectors are used to represent the source model identifier, timestamp, and session identifier. Construct a graph structure, wherein each security candidate instruction serves as a graph node, and multi-layer feature vectors serve as node features of the graph nodes; For any two graph nodes, calculate the similarity of the feature vectors at each layer. Based on the similarity, perform weighted fusion of all feature vectors to obtain the fusion similarity, and use the fusion similarity as the edge weight. The graph structure is normalized, and the normalized graph structure is input into the graph neural network for graph convolution to obtain the global consistency score of each graph node. All global consistency scores are combined to form a consistency matrix; Based on the consistency matrix, the average global consistency score of each security candidate instruction with other security candidate instructions is calculated. If the average global consistency score is lower than the consistency threshold, the element corresponding to the security candidate instruction is deleted from the consistency matrix.

[0037] The above embodiments describe how to accurately calculate the degree of consistency between different candidate instructions. Traditional simple comparison methods (such as direct string comparison) cannot capture the deep semantic consistency of candidate instructions. For example, "delete logs from 30 days ago" and "clean up records from one month ago" may express different meanings, but they have the same intent. First, each instruction is split into multiple layers of feature vectors. This hierarchical design considers the different dimensions of the instruction: Surface features focus on the "action" and "object" of the instruction, which is the most intuitive operational intent.

[0038] Mid-level features focus on "how to execute" and "when to execute," reflecting the specific parameters and constraints of the operation.

[0039] Deep features focus on "permission requirements" and "risk level," reflecting the security attributes of the operation.

[0040] Meta-level features focus on "source information," used to trace the source model and time of instructions.

[0041] Then, a graph structure is constructed, where each instruction is a graph node, and each graph node carries four layers of features. For any two graph nodes, their similarity across the four layers of features is calculated, and then weighted and fused according to a set weight to obtain the comprehensive similarity between the two candidate instructions. This similarity serves as the "edge weight" between graph nodes, representing the degree of association between the two candidate instructions.

[0042] Next, the graph structure is input into the graph neural network for graph convolution. The graph convolution process is similar to having each graph node "listen" to information from its neighboring graph nodes. Through multiple rounds of information transmission, each graph node eventually obtains a "global consistency score," which reflects the "grouping degree" of the candidate instruction in the entire instruction set.

[0043] Finally, a consistency matrix is ​​formed based on the global consistency scores of all nodes, and the average consistency score of each candidate instruction with other candidate instructions is calculated. If the average score of a candidate instruction is lower than a preset consistency threshold, it indicates that the candidate instruction differs significantly from other candidate instructions and belongs to an "outlier," so it is removed from the consistency matrix. The final set of retained candidate instructions is the safe candidate instruction with high consistency.

[0044] In one embodiment, a clustering formula is applied to the consistency matrix to obtain output instructions, including: Candidate instructions are clustered based on the consistency matrix to obtain a set of candidate clusters. Calculate a consensus score for each cluster in the candidate cluster set; The cluster with the highest consensus score is selected as the target cluster. If the consensus score of the target cluster is greater than the clustering threshold, an output instruction is generated based on the target cluster.

[0045] In the above embodiments, the formula for generating the candidate instruction set summary can be expressed as consensus_commit = H(action1 || action2 || ... || actionn).

[0046] The formula for generating the output instruction digest can be expressed as action_digest = H(a ).

[0047] Remote authentication information is generated upon system startup or session establishment. This remote authentication information may include: program measurements from the decision module, trusted computing carrier identity information, a public key or public key certificate, and software version, policy version, or configuration digest. Authentication information is generated using this remote authentication information. The execution control module can use this remote authentication information to verify whether the large language model's instructions are running on a trusted computing carrier, and whether its signing public key is bound to a specific trusted computing carrier instance. It should be noted that, besides using remote authentication information to sign the trusted computing carrier, other methods that can provide source authenticity and integrity authentication for output instructions are all within the scope of this invention. The authentication information generation formula can be expressed as auth_tag = Auth(action_digest, metadata), where metadata may include a timestamp, session identifier, security policy version, source model identifier, risk level, or execution context digest.

[0048] The clustering threshold can be a fixed threshold, an adaptive threshold, a risk-aware threshold, or a dynamic threshold associated with the task level. Based on the consistency matrix obtained in the previous step, the system performs cluster analysis on security candidate instructions. The clustering algorithm groups security candidate instructions with high consistency into the same cluster, forming multiple clusters. For example, if multiple security candidate instructions all point to the "delete log" operation, they will be clustered into one cluster; if other security candidate instructions point to the "backup data" operation, they will form another independent cluster. A consensus score is calculated for each cluster. This score comprehensively reflects the credibility and consistency strength of the instructions within the cluster. The cluster with the highest consensus score is called the target cluster, representing the group of security candidate instructions that the system considers most reliable. If the consensus score of the target cluster exceeds the preset clustering threshold (indicating that this group of instructions is sufficiently reliable), the system will generate the final output instruction based on this cluster. The output instruction can be a selection of a security candidate instruction within the target cluster, or a merging of multiple security candidate instructions within the target cluster (such as extracting common operation objects and conditions).

[0049] Simultaneously, calculate two key hash digests: Candidate instruction set summary: The hash value is calculated by concatenating all candidate instructions, which is equivalent to a snapshot of the concatenated candidate instructions.

[0050] Output instruction digest: Calculate the hash value of the final generated output instruction, which serves as the digital fingerprint of this output instruction.

[0051] Based on the output instruction digest, authentication information (such as a digital signature) is generated. The authentication information may include metadata such as timestamps and session identifiers to ensure the authenticity and integrity of the output instructions. Finally, the system constructs a decision result, comprising three parts: the output instruction, the authentication information, and a digest of the candidate instruction set.

[0052] In one embodiment, a consensus score is calculated for each cluster in the candidate cluster set, including: For each cluster, take the average of the weights of the large language model corresponding to all safe candidate instructions within that cluster; Calculate the ratio of the number of secure candidate instructions within each cluster to the total number of secure candidate instructions in the candidate cluster set; For each cluster, take the average of the global consistency scores between all pairs of security candidate instructions within that cluster; Calculate the success rate of historical candidate instructions for each major language model within each cluster; For each cluster, the consensus score is calculated based on the average and percentage of the weights of the large language model, the average global consistency score and the success rate, as well as the corresponding weights.

[0053] In the above embodiments, the consensus score is not a single indicator. For example, the consensus score can be calculated according to the weights of the above four dimensions, including weighted average, weighted summation, etc., and the weight of each dimension can be determined according to the actual situation, without any restrictions here.

[0054] The first dimension is the average model weights. Different weights can be pre-assigned to different large language models; for example, validated authoritative models have high weights, while experimental models have low weights. The higher the weight of the source models for all secure candidate instructions within a cluster, the higher the consensus score of that cluster.

[0055] The second dimension is the proportion of cluster size. The higher the proportion of secure candidate instructions contained in a cluster relative to the total number of secure candidate instructions, the more the majority of models "agree" with the scheme, and the higher the consensus score of that cluster. This reflects the principle of "majority consensus".

[0056] The third dimension is the consistency strength within the cluster. The average of the global consistency scores between all pairs of secure candidate instructions within a cluster is calculated. A higher score indicates stronger consistency among the instructions within the cluster, meaning they are more "in sync" with each other, and the higher the consensus score of that cluster.

[0057] The fourth dimension is the model's historical success rate. This considers the success rate of executing safe candidate instructions generated by various language models within the cluster in historical tasks. If a model has historically frequently generated incorrect instructions, its weight will be weakened in the consensus calculation.

[0058] Configure corresponding weight coefficients for each of these four dimensions (which can be adjusted according to the actual application scenario), and then sum them up by weight to obtain the final consensus score.

[0059] In one embodiment, generating output instructions based on the target cluster includes: According to preset rules, select safe candidate instructions from the target cluster, or merge all safe candidate instructions in the target cluster to generate output instructions.

[0060] In the above embodiments, two specific methods for generating output instructions are given, which can be flexibly selected according to the actual application scenario.

[0061] The first approach is "selection." When multiple candidate safety instructions within a target cluster are highly semantically consistent but slightly different in expression, one can be selected as the output instruction according to preset rules. For example, the instruction from the model with the highest weight can be selected, or the instruction with the most detailed and complete parameters can be selected, or the instruction with the lowest risk level assessment can be selected. This approach is suitable when the instructions are not fundamentally different, but only differ in format.

[0062] The second approach is "merging." When multiple security candidate instructions within a target cluster are functionally complementary, or when each security candidate instruction only expresses a part of a complete operation, they can be merged into a more comprehensive and precise output instruction. The merging process requires extracting common elements from all security candidate instructions and integrating the differences to form a final output instruction containing all necessary information. For example, if security candidate instruction A specifies the operation object, security candidate instruction B specifies the operation conditions, and security candidate instruction C specifies the operation parameters, merging them yields a complete output instruction.

[0063] In one embodiment, the method further includes: After concatenating all clusters, calculate the hash to obtain a summary of the clustering results; The decision chain digest is obtained by concatenating the user task, security policy set, output command, and security policy set version and then calculating the hash. Authentication information is generated based on at least one of the output instruction digest, clustering result digest, and decision chain digest.

[0064] In the above embodiments, the formula for generating cluster result summaries can be expressed as cluster_commit = H(C1 ||C2 || ... || Cm), where the candidate cluster set can be represented as C={C1,C2,...,Cm}, and the formula for generating decision chain summaries can be expressed as decision_commit = H(T || A' || a || policy_version), H represents a certain abstract method, T represents the task description, A' represents the set of security policies, a This indicates the output command, and `policy_version` indicates the security policy version.

[0065] In one embodiment, the method further includes: If the consensus score of the target cluster is less than or equal to the clustering threshold, a rejection decision is obtained, which means that the candidate instructions generated by the user task will not be executed.

[0066] In the above embodiments, when the credibility of security candidate instructions is insufficient, not all user tasks can generate credible candidate instructions. If the candidate instructions generated by multiple large language models have low consistency with each other, or if all candidate instructions are filtered out by security policies, the consensus score of the target cluster will be lower than the set clustering threshold. In this case, instead of forcibly generating an output instruction, a "rejection decision" is output. This decision explicitly informs the upper-layer system that the candidate instructions generated by this user task are not adopted and no operation is performed. The rejection decision mechanism is an important part of the system's security design. It avoids the system "forcing" the execution of a potentially erroneous operation when the quality of candidate instructions is poor and consistency is insufficient. This design principle is particularly important for scenarios with high security requirements.

[0067] In one embodiment, the system further includes an audit log module for: Record user tasks, candidate instruction set summary, aggregation result summary, authentication information, and execution results.

[0068] By recording these five types of information, the audit log module constructs a complete "chain of evidence." When a system anomaly occurs, administrators can trace back through the audit logs: what tasks the user submitted, what instructions the model generated, what rules the system used to make decisions, what operations were ultimately performed, and what the results were.

[0069] The following is a specific embodiment to illustrate the specific application of the method proposed in this invention.

[0070] In this embodiment, a large language model instruction verification system is deployed in the intelligent operation and maintenance system, and the large language model instruction verification device is deployed in a trusted execution environment. Figure 4 This is a schematic diagram illustrating the deployment principle of the instruction verification device for the large language model in this embodiment of the invention. The administrator inputs tasks via natural language, such as: "Check the server logs and clean up temporary log files older than 30 days".

[0071] After receiving the task, the instruction verification system of the large language models first parses the task, generating a task description T. The task scheduling module then sends T to multiple large language models, such as model M1, model M2, and model M3. Each language model outputs candidate instructions, for example: Model M1 output: action1: delete_file path: / var / log / tmp condition: older than 30 days Model M2 output: action2: clean_log directory: / var / log / tmp filter: file_age > 30_days Model M3 output: action3: remove_old_logs target: / var / log / tmp rule: older_than_30_days Subsequently, these candidate instructions are sent to the instruction verification device of the large language model deployed within the Trusted Execution Environment (TEE). The instruction verification of the large language model performs normalization processing within the TEE, converting the different expressions into a unified structured representation, for example: operation = delete_files target = / var / log / tmp condition = age > 30_days Subsequently, the instruction verification of the large language model within TE performs legality checks according to the security policy. For example, the security policy may prohibit the deletion of files in the system core directory, root directory, configuration directory, or certificate directory, and restrict recursive deletion operations that do not explicitly define the path range, as these are actions whose risk exceeds the risk threshold.

[0072] In this embodiment, since the candidate instructions output by the three models are highly consistent in terms of target path and execution conditions, and none of them violate the security policy, they are all considered safe candidate instructions. The instruction verification device of the large language model within the TEE categorizes them into the same cluster and calculates a consensus score. When the consensus score is higher than the clustering threshold, the instruction verification device of the large language model generates the final output instruction a. ,For example: delete_files(path= / var / log / tmp, condition=age>30_days) Then, the instruction verification device of the large language model within the TEE calculates: Output instruction digest action_digest = H(a ); The candidate instruction set summary consensus_commit = H(action 1 || action 2 || action 3); And use the protected private key inside the TEE to generate authentication information: auth_tag= SignTEE(action_digest || consensus_commit); Finally, the decision result R is output: R = (a , auth_tag, consensus_commit); Upon receiving the R, the execution control module first verifies the validity of the authentication information and confirms that the signing public key corresponds to the TEE instance bound through remote authentication. If the verification is successful, the execution control module performs the corresponding operations, cleaning up log files in the specified directory that meet the conditions; if the signature verification fails, the remote authentication fails, or the policy check is not met, execution is refused.

[0073] As can be seen from this embodiment, the TEE serves as a preferred trusted entity in this invention, providing the following additional security capabilities: isolated execution of the instruction verification aggregation logic of the large language model; hardware-level protection of the authentication private key; trusted authentication of the aggregation result; and remote proof to the outside world, demonstrating that the instruction verification of the large language model runs in the expected code and expected environment.

[0074] In one embodiment, the security filtering module is further configured to: Before filtering all candidate instructions based on the security policy set, all candidate instructions are normalized into a structured representation to obtain a normalized candidate instruction set. The structured representation includes, but is not limited to, operation type, target object, operation parameters, execution conditions, risk level, required permissions, source model identifier, timestamp, and session identifier. The normalized candidate instruction set is filtered based on the set of security policies to obtain a set of secure candidate instructions.

[0075] In one embodiment, the decision evaluation module is used to: A consistency calculation is performed on the set of security candidate instructions to obtain the consistency matrix; Cluster the consistency matrix to obtain the output instructions; The candidate instructions generated by all large language models are concatenated and their hashes are calculated to obtain a summary of the candidate instruction set. Calculate the hash of the output instruction, generate an output instruction digest, and generate authentication information based on the output instruction digest; The decision result is constructed based on the output instruction, authentication information, and a summary of the candidate instruction set.

[0076] In one embodiment, the decision evaluation module is used to: Each candidate instruction in the security candidate instruction set is split into multiple layers of feature vectors. The multiple layers of feature vectors include surface feature vectors, middle feature vectors, deep feature vectors, and meta-layer feature vectors. The surface feature vectors are used to represent the operation type and target object. The middle feature vectors are used to represent the operation parameters and execution conditions. The deep feature vectors are used to represent the required permissions and risk level. The meta-layer feature vectors are used to represent the source model identifier, timestamp, and session identifier. Construct a graph structure, wherein each security candidate instruction serves as a graph node, and multi-layer feature vectors serve as node features of the graph nodes; For any two graph nodes, calculate the similarity of the feature vectors at each layer. Based on the similarity, perform weighted fusion of all feature vectors to obtain the fusion similarity, and use the fusion similarity as the edge weight. The graph structure is normalized, and the normalized graph structure is input into the graph neural network for graph convolution to obtain the global consistency score of each graph node. All global consistency scores are combined to form a consistency matrix; Based on the consistency matrix, the average global consistency score of each security candidate instruction with other security candidate instructions is calculated. If the average global consistency score is lower than the consistency threshold, the element corresponding to the security candidate instruction is deleted from the consistency matrix.

[0077] In one embodiment, the decision evaluation module is used to: Candidate instructions are clustered based on the consistency matrix to obtain a set of candidate clusters. Calculate a consensus score for each cluster in the candidate cluster set; The cluster with the highest consensus score is selected as the target cluster. If the consensus score of the target cluster is greater than the clustering threshold, an output instruction is generated based on the target cluster.

[0078] In one embodiment, the decision evaluation module is used to: For each cluster, take the average of the weights of the large language model corresponding to all safe candidate instructions within that cluster; Calculate the ratio of the number of secure candidate instructions within each cluster to the total number of secure candidate instructions in the candidate cluster set; For each cluster, take the average of the global consistency scores between all pairs of security candidate instructions within that cluster; Calculate the success rate of historical candidate instructions for each major language model within each cluster; For each cluster, the consensus score is calculated based on the average and percentage of the weights of the large language model, the average global consistency score and the success rate, as well as the corresponding weights.

[0079] In one embodiment, the decision evaluation module is used to: According to preset rules, select safe candidate instructions from the target cluster, or merge all safe candidate instructions in the target cluster to generate output instructions.

[0080] In one embodiment, the decision evaluation module is used to: After concatenating all clusters, calculate the hash to obtain a summary of the clustering results; The decision chain digest is obtained by concatenating the user task, security policy set, output command, and security policy set version and then calculating the hash. Authentication information is generated based on the output instruction digest, clustering result digest, and decision chain digest.

[0081] In one embodiment, the decision evaluation module is used to: If the consensus score of the target cluster is less than or equal to the clustering threshold, a rejection decision is obtained, which means that the candidate instructions generated by the user task will not be executed.

[0082] This invention also provides a computer device. Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements a method.

[0083] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described instruction verification method for a large language model.

[0084] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the instruction verification method for the large language model described above.

[0085] In this embodiment of the invention, at least one candidate instruction generated by a large language model for a user task is obtained; all candidate instructions are filtered based on a security policy set to obtain a set of secure candidate instructions; the set of secure candidate instructions is evaluated to obtain a decision result, which includes an output instruction and authentication information. The output instruction can only be executed after the authentication information is verified. Compared with existing technical solutions where the reliability of instructions generated by large language models is insufficient, filtering candidate instructions using a security policy set can effectively intercept instructions with security risks, avoiding system risks, data leakage risks, and operational security risks caused by harmful instructions generated by large language models from the source, and ensuring the security and reliability of the instruction execution environment. By evaluating the set of secure candidate instructions, output instructions that meet the user task requirements, are logically consistent, and have compliant content can be selected, and authentication information can be generated. Execution is only possible after authentication, further enhancing the controllability of instruction execution and avoiding mis-execution and illegal execution. Binding authentication information to output instructions ensures that only instructions that have passed security verification can be executed, and also enables traceability management of instructions through authentication information, facilitating subsequent investigation of problems in the instruction execution process and improving the regulatory oversight of the entire instruction processing flow. The methodology is clear and adaptable to the instruction generation and verification needs of large language models in different scenarios. It does not require large-scale modification of the large language model itself, and security control can be achieved solely through the verification of authentication information. It balances the flexibility of the large language model with the security of instruction execution, making it highly practical.

[0086] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for verifying instructions in a large language model, characterized in that, include: Obtain at least one candidate instruction generated by a large language model for the user task; All candidate instructions are filtered based on the set of security policies to obtain a set of secure candidate instructions; A decision evaluation calculation is performed on the set of security candidate instructions to obtain a decision result. The decision result includes an output instruction and corresponding authentication information. The output instruction can only be executed after the authentication information is verified.

2. The method as described in claim 1, characterized in that, Before filtering all candidate instructions based on the set of security policies, the following is also included: All candidate instructions are normalized into a structured representation to obtain a normalized candidate instruction set. The structured representation includes, but is not limited to, operation type, target object, operation parameters, execution conditions, risk level, required permissions, source model identifier, timestamp, and session identifier. All candidate instructions are filtered based on a set of security policies, including: The normalized candidate instruction set is filtered based on the set of security policies to obtain a set of secure candidate instructions.

3. The method as described in claim 1, characterized in that, The set of security policies includes, but is not limited to, candidate instructions that prohibit access to sensitive directories, candidate instructions that prohibit execution of risk levels greater than the level threshold, candidate instructions that restrict actions involving risks greater than the risk threshold, candidate instructions that restrict cross-system calls, candidate instructions that restrict access to preset types of data, candidate instructions that require the principle of least privilege to meet the requirements, candidate instructions that require manual confirmation, candidate instructions that require secondary verification or multiple consensus, and candidate instructions that prohibit the deletion of preset files.

4. The method as described in claim 1, characterized in that, The decision evaluation calculation is performed on the set of security candidate instructions to obtain the decision results, including: A consistency calculation is performed on the set of security candidate instructions to obtain the consistency matrix; Cluster the consistency matrix to obtain the output instructions; The candidate instructions generated by all large language models are concatenated and their hashes are calculated to obtain a summary of the candidate instruction set. Calculate the hash of the output instruction, generate an output instruction digest, and generate authentication information based on the output instruction digest; The decision result is constructed based on the output instruction, authentication information, and a summary of the candidate instruction set.

5. The method as described in claim 4, characterized in that, A consistency calculation is performed on the set of security candidate instructions to obtain a consistency matrix, including: Each candidate instruction in the security candidate instruction set is split into multiple layers of feature vectors. The multiple layers of feature vectors include surface feature vectors, middle feature vectors, deep feature vectors, and meta-layer feature vectors. The surface feature vectors are used to represent the operation type and target object. The middle feature vectors are used to represent the operation parameters and execution conditions. The deep feature vectors are used to represent the required permissions and risk level. The meta-layer feature vectors are used to represent the source model identifier, timestamp, and session identifier. Construct a graph structure, wherein each security candidate instruction serves as a graph node, and multi-layer feature vectors serve as node features of the graph nodes; For any two graph nodes, calculate the similarity of the feature vectors at each layer. Based on the similarity, perform weighted fusion of all feature vectors to obtain the fusion similarity, and use the fusion similarity as the edge weight. The graph structure is normalized, and the normalized graph structure is input into the graph neural network for graph convolution to obtain the global consistency score of each graph node. All global consistency scores are combined to form a consistency matrix; Based on the consistency matrix, the average global consistency score of each security candidate instruction with other security candidate instructions is calculated. If the average global consistency score is lower than the consistency threshold, the element corresponding to the security candidate instruction is deleted from the consistency matrix.

6. The method as described in claim 4, characterized in that, The clustering formula for the consistency matrix yields the output instructions, including: Candidate instructions are clustered based on the consistency matrix to obtain a set of candidate clusters. Calculate a consensus score for each cluster in the candidate cluster set; The cluster with the highest consensus score is selected as the target cluster. If the consensus score of the target cluster is greater than the clustering threshold, an output instruction is generated based on the target cluster.

7. The method as described in claim 6, characterized in that, For each cluster in the candidate cluster set, a consensus score is calculated, including: For each cluster, take the average of the weights of the large language model corresponding to all safe candidate instructions within that cluster; Calculate the ratio of the number of secure candidate instructions within each cluster to the total number of secure candidate instructions in the candidate cluster set; For each cluster, take the average of the global consistency scores between all pairs of security candidate instructions within that cluster; Calculate the success rate of historical candidate instructions for each major language model within each cluster; For each cluster, the consensus score is calculated based on the average and percentage of the weights of the large language model, the average global consistency score and the success rate, as well as the corresponding weights.

8. The method as described in claim 6, characterized in that, Output instructions are generated based on the target cluster, including: According to preset rules, select safe candidate instructions from the target cluster, or merge all safe candidate instructions in the target cluster to generate output instructions.

9. The method as described in claim 6, characterized in that, Also includes: If the consensus score of the target cluster is less than or equal to the clustering threshold, a rejection decision is obtained, which means that the candidate instructions generated by the user task will not be executed.

10. The method as described in claim 4, characterized in that, Also includes: After concatenating all clusters, calculate the hash to obtain a summary of the clustering results; The decision chain digest is obtained by concatenating the user task, security policy set, output command, and security policy set version and then calculating the hash. Authentication information is generated based on the output instruction digest, clustering result digest, and decision chain digest.

11. A large language model instruction verification device, characterized in that, include: The candidate instruction acquisition module is used to obtain at least one candidate instruction generated by a large language model for the user task. The security filtering module is used to filter all candidate instructions based on a set of security policies to obtain a set of secure candidate instructions. The decision evaluation module is used to perform decision evaluation calculations on the set of security candidate instructions to obtain decision results. The decision results include output instructions and corresponding authentication information. The output instructions can only be executed after the authentication information is verified.

12. A large language model instruction verification system, characterized in that, It includes the instruction verification device, intelligent agent module, task scheduling module, multiple large language models and execution control module of the large language model as described in claim 11; The intelligent agent module is used to: receive user tasks and generate task descriptions; The task scheduling module is used to: distribute task descriptions to at least one large language model, and send the received candidate instructions to the instruction verification device of the large language model. The large language model is used to: generate candidate instructions based on the received task description and return them to the task scheduling module; The execution control module is used to: verify the authentication information in the decision result output by the instruction verification device of the large language model, and execute the output instruction after the verification is successful.

13. The system as described in claim 12, characterized in that, The instruction verification device of the large language model is deployed in a trusted entity or a distributed system composed of multiple trusted entities. The trusted entity includes, but is not limited to, a trusted execution environment, a hardware security module, an isolated virtual machine, a trusted coprocessor, and a trusted execution platform with security isolation, key protection, and integrity proof capabilities.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

Images