Runtime shortcut action semantic authorization method, system and terminal equipment for branched large language model program

Abstract

The application is suitable for the technical field of large language model reasoning and optimization, and provides a runtime shortcut action semantic authorization method, system and terminal equipment for a branch type large language model program, which comprises the following steps: receiving a user input original task request; creating a multi-branch reasoning program carrying a preset convergence semantic according to the original task request; wherein the preset convergence semantic represents a fixed rule for pre-set aggregation and output of the running result of the multi-branch reasoning program; generating a shortcut action according to the running state of the multi-branch reasoning program; wherein the shortcut action is an operation instruction for optimizing the running state; and performing authorization verification on the program running result corresponding to the shortcut action based on the preset convergence semantic to obtain a semantic authorization result. The above method can realize the running optimization of the multi-branch reasoning program under the premise of safety without changing the final output result, and takes into account reasoning acceleration and result output accuracy.

Description

Technical Field

[0001] This application belongs to the field of large language model reasoning and optimization technology, and in particular relates to a runtime shortcut action semantic authorization method, system and terminal device for branching large language model programs. Background Technology

[0002] As large model inference gradually evolves into a multi-branch parallel operation mode, in order to reduce latency and resource consumption, the industry generally adopts optimization operations such as early stopping, canceling branches, and releasing cache. However, such operations can easily violate the preset result aggregation rules.

[0003] Currently, most related technologies rely on confidence levels, empirical strategies, or underlying scheduling for optimization. They have not established standardized and verifiable semantic authorization mechanisms for different branch aggregation rules, making it difficult to rigorously determine whether optimization operations affect the final result. They are also difficult to adapt to diverse aggregation rules and the operational optimization requirements of branch-based large language models for security, compliance, and traceability. Summary of the Invention

[0004] This application provides a runtime shortcut action semantic authorization method, system, and terminal device for branching large language model programs. It can perform authorization verification on shortcut actions for optimization based on preset convergence semantics, and can optimize the operation of multi-branch reasoning programs without changing the final output results, thus balancing reasoning acceleration and result output accuracy.

[0005] In a first aspect, embodiments of this application provide a runtime shortcut action semantic authorization method for branching large language model programs, including: Receive the user's original task request; Create a multi-branch reasoning program carrying preset convergence semantics based on the original task request; wherein, the preset convergence semantics represent a fixed rule for summarizing and outputting the running results of the multi-branch reasoning program. Quick actions are generated based on the running state of the multi-branch reasoning program; where quick actions are operation instructions that optimize the running state. Based on the preset convergence semantics, the program execution result corresponding to the shortcut action is authorized and verified to obtain the semantic authorization result.

[0006] In this embodiment, the system receives the user's original task request, generates a multi-branch reasoning program with preset result aggregation rules (convergence semantics), generates branch optimization shortcut actions based on the running status, and verifies the legality of the actions using convergence semantics to obtain the authorization result. This solution can perform authorization verification on the optimization shortcut actions based on preset convergence semantics, and can optimize the operation of the multi-branch reasoning program without changing the final output result, balancing reasoning acceleration and result output accuracy.

[0007] In one possible implementation of the first aspect, a multi-branch reasoning program carrying preset convergence semantics is created based on the original task request, including: Perform scenario identification on the original task request to obtain the corresponding business scenario type; Perform rule matching on business scenario types to obtain task configurations that are bound to the corresponding preset convergence semantics; Parallel inference branches are created based on the task configuration, generating a multi-branch inference program carrying preset convergence semantics.

[0008] In this embodiment, scene recognition and rule matching are used to automatically bind the corresponding convergence semantics, standardize the generation of multi-branch programs, unify and solidify the result summary logic, and the subsequent verification process can directly call the preset rules to complete the invariant verification, which simplifies the deployment process and avoids the problems of verification failure and inference result error caused by inconsistent custom rules.

[0009] In one possible implementation of the first aspect, a shortcut action is generated based on the running state of the multi-branch reasoning program, including: The reasoning state of a multi-branch reasoning program is statistically analyzed. The reasoning state includes the number of branches that have completed the reasoning task, the reasoning results of branches that have completed the reasoning task, the number of branches that have not completed the reasoning task, and the remaining decoding quota of branches. Generate quick actions based on the reasoning state.

[0010] In this embodiment of the application, by statistically analyzing multi-dimensional quantitative indicators such as the number of completed branches, branch results, number of incomplete branches, and remaining decoding quota, a complete picture of the real-time operation of multiple branches is depicted. This serves as an objective basis for generating quick actions, providing data support for resource optimization and scheduling, and making scheduling more accurate and efficient.

[0011] In one possible implementation of the first aspect, authorization verification is performed on the program execution result corresponding to the shortcut action based on preset convergence semantics to obtain a semantic authorization result, including: Obtain the full metadata corresponding to the current multi-branch inference program; the full metadata includes the hash value corresponding to the original task request, the preset convergence semantics, the real-time running snapshot of the multi-branch program, the action category corresponding to the shortcut action, and the estimated savings; Generate a typed certificate based on full metadata and shortcut actions; Based on the typed certificate, the program execution result corresponding to the shortcut action is authorized and verified to obtain the semantic authorization result.

[0012] In this embodiment, by collecting full-dimensional metadata covering tasks, rules, real-time running status, and optimization operations and binding it to the certificate, the certificate can fully record the current inference environment and optimization intent. Subsequent verification can rely on the complete set of information built into the certificate and combine it with the corresponding converged semantics to complete rigorous inference and verification, avoiding the problems of information loss and judgment distortion that exist in single data verification.

[0013] In one possible implementation of the first aspect, a typed certificate is generated based on the full metadata and shortcut actions, including: A custom certificate template is matched based on quick actions and preset semantic matching. Add custom parameters to the custom certificate template based on the full metadata. Generate a typed certificate based on the custom certificate template with added custom parameters.

[0014] In this embodiment, full-dimensional metadata such as task hash, convergence semantics, branch running snapshots, action type and optimization benefits are collected. Combined with business semantics, corresponding special type certificates are generated. The certificate relies on the exclusive invariant rules built into it to complete rigorous mathematical verification, rather than simple experience judgment, thus ensuring the rigor and universality of the verification mechanism.

[0015] In one possible implementation of the first aspect, the authorization verification of the program execution result corresponding to the shortcut action is performed based on the typed certificate to obtain the semantic authorization result, including: Obtain the current running status data of the multi-branch inference program; The runtime status data is matched with the full metadata contained in the typed certificate to obtain the first matching result; If the first matching result indicates a successful match, then the semantic authorization result is obtained by performing a deduction and verification based on the typed certificate.

[0016] In this embodiment, a two-layer verification mechanism of state matching and semantic verification is adopted. First, the certificate is verified to be consistent with the current operating environment to prevent expired or forged certificates from allowing illegal operations. Then, the exclusive invariant is verified to prove from a logical level that the shortcut action does not change the final output. Layered interception of risk-optimized operations enhances the verification's fault tolerance and security.

[0017] In one possible implementation of the first aspect, semantic authorization results are obtained by performing deductive verification based on the typed certificate, including: Based on the full amount of metadata contained in the typed certificate and the preset convergence semantics contained therein, the inference and verification results are obtained. The deduction and verification results are matched with the semantic compliance statements bound in the typed certificate to obtain the second matching result; wherein, the semantic compliance statement represents the statement information added in advance in the typed certificate that does not change the running result of the multi-branch program after executing the shortcut action; If the second matching result indicates a successful match, then the semantic authorization result indicates a successful authorization.

[0018] In this embodiment, the logical deduction and verification are first completed by combining the full metadata of the certificate with the exclusive converged semantics, and then the compliance statement of the result pre-bound to the certificate is checked. The double verification closed-loop proof that the quick action will not tamper with the summary result, and authorization can only be granted if it fully complies with the preset compliance constraints, thus eliminating the result deviation caused by the optimization operation from the logical level.

[0019] In one possible implementation of the first aspect, the method further includes: If the semantic authorization result indicates successful authorization, then execute the shortcut action; If the semantic authorization result indicates that authorization has failed, the shortcut action will not be executed, and the multi-branch reasoning program will continue to run.

[0020] In this embodiment, the process is divided into branches based on the semantic authorization result. If the authorization is successful, it means that the shortcut action will not change the final summary result, and resource optimization can be performed with confidence. If the authorization fails, it means that the operation has the risk of tampering with the reasoning result, so the optimization is abandoned and the original complete reasoning process is maintained.

[0021] In a second aspect, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the runtime shortcut action semantic authorization method for a branching large language model program as described in any of the first aspects above.

[0022] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements a runtime shortcut action semantic authorization method for a branching large language model program as described in any of the first aspects above.

[0023] Fourthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the runtime shortcut action semantic authorization method for branching large language model programs described in any of the first aspects above.

[0024] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the runtime shortcut action semantic authorization method for branching large language model programs provided in this application embodiment; Figure 2 This is a flowchart illustrating the creation of a multi-branch reasoning program according to an embodiment of this application; Figure 3 This is a schematic diagram of the process for generating shortcut actions according to an embodiment of this application; Figure 4 This is a schematic diagram of an authorization verification process provided in an embodiment of this application; Figure 5 This is a schematic diagram of the process for generating a typed certificate according to an embodiment of this application; Figure 6 This is a schematic diagram of the authorization verification process provided in another embodiment of this application; Figure 7 This is a schematic diagram of a semantic authorization verification process provided in another embodiment of this application; Figure 8 This is a schematic diagram of the overall architecture of the runtime shortcut action semantic authorization method for branching large language model programs provided in the embodiments of this application; Figure 9 This is a schematic diagram illustrating different certificate types and inspection conditions corresponding to different convergence semantics provided in the embodiments of this application; Figure 10 This is a schematic diagram of the KV cache shared storage architecture for branched LLM inference provided in an embodiment of this application; Figure 11 This is a schematic diagram of various types of certificates provided in the embodiments of this application; Figure 12 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0033] As large model inference gradually evolves into a multi-branch parallel operation mode, in order to reduce latency and resource consumption, the industry generally adopts optimization operations such as early stopping, canceling branches, and releasing cache. However, such operations can easily violate the preset result aggregation rules.

[0034] Currently, most related technologies rely on confidence levels, empirical strategies, or underlying scheduling for optimization. They have not established standardized and verifiable semantic authorization mechanisms for different branch aggregation rules, making it difficult to rigorously determine whether optimization operations affect the final result. They are also difficult to adapt to diverse aggregation rules and the operational optimization requirements of branch-based large language models for security, compliance, and traceability.

[0035] To address the aforementioned technical problems, this application provides a runtime shortcut action semantic authorization method and system for branching large language model programs. The system includes a branching program controller, a backend executor, a shortcut action proposer, a certificate generator, a trusted certificate verifier, a certificate / action audit log, and an optional backend resource manager. At runtime, the large language model inference is represented as P=(x,C,J,B0), where x is the input, C is the controller, J is the declared convergence semantics, and B0 is the initial branch set. The runtime state includes the current branch set, partial output, remaining budget, verifier state, backend node information, and side effect commit logs.

[0036] During operation, an untrusted action proposer generates various performance optimization shortcut actions based on the running status. The certificate generator matches different convergence semantics such as majority voting, first success, and all required to generate exclusive typed convergence certificates that bind real-time running snapshots, version identifiers, and risk constraints. Then, an independent trusted verifier performs triple verification of certificate type, running status consistency, and convergence semantic invariant. Only when the verification passes is the shortcut action allowed to be executed. At the same time, all kinds of action audit logs are fully recorded. This can securely optimize inference latency, token overhead, and memory usage, and ensure that the output results strictly conform to the preset convergence semantics. It has the advantages of low verification overhead, multi-scenario adaptability, auditability, and easy integration with existing large model inference service backends.

[0037] See Figure 1 This is a flowchart illustrating a runtime shortcut action semantic authorization method for branching large language model programs provided in this application embodiment. It is intended as an example and not a limitation. The method may include the following steps: S101 receives the original task request input by the user.

[0038] In this embodiment, the system's front-end interaction module continuously listens to the user input interface. When the user submits a raw task request in text form, the interface layer first performs basic cleaning on the raw request, removing invalid whitespace characters and illegal special symbols, generating a globally unique task hash identifier to bind the full text of the request, and then forwarding the cleaned raw task request carrying the unique task hash to the scheduling controller. If the user submits text in the input box, such as explaining the molecular principle of water freezing at 0 degrees Celsius, the system directly receives this complete raw text and completes this step.

[0039] S102, Create a multi-branch reasoning program carrying preset convergence semantics according to the original task request; wherein, the preset convergence semantics represent the fixed rules for summarizing and outputting the running results of the multi-branch reasoning program.

[0040] In this embodiment, multiple parallel inference branches are built using the user-submitted original task request as input, and predefined result aggregation rules (merging semantics) are bound to these parallel branches. These rules specify the criteria for integrating and outputting the final answer after the subsequent multiple branches have completed their inference. This is the core initialization step for enabling multi-branch parallel inference and determining the result merging logic.

[0041] Among them, the pre-defined join semantics generally include majority join, first-success join, all-required join, verifier-accept join, and statistical risk join.

[0042] In one embodiment, see Figure 2 This is a flowchart illustrating the creation of a multi-branch reasoning program according to an embodiment of this application, as shown below. Figure 2 As shown, step S102 includes: S201, perform scenario identification on the original task request to obtain the corresponding business scenario type.

[0043] In this embodiment, after the system obtains the cleaned original task request text, it calls the built-in scene recognition and classification model, completes the scene determination through keyword matching and intent feature extraction, and outputs the corresponding business scene type. Different scenes are pre-bound with exclusive convergence semantics and parallel branch parameters. After the recognition is completed, the scene type is synchronously sent to the controller. The controller matches the corresponding preset convergence semantics and branch configuration according to the scene to build the subsequent multi-branch inference program.

[0044] For example, when a user inputs the original request "Calculate a math word problem: If person A travels at a speed of 5 km / h, how far will they travel in 3 hours?", the scene recognition model extracts the features of "calculation and math word problem", determines that the business scenario is a math calculation scenario, and passes the scene recognition result to the scheduling controller.

[0045] S202, perform rule matching on the business scenario type to obtain the task configuration bound to the corresponding preset convergence semantics.

[0046] In this embodiment, the system has a built-in scenario-rule mapping library. Each business scenario type is pre-associated with a unique matching rule. The rule is bound with exclusive convergence semantics, parallel branch number, decoding quota and other parameters. After obtaining the business scenario type, the system retrieves the mapping library to complete the rule matching, extracts the preset convergence semantics and supporting operating parameters bound under the rule, integrates them to generate a complete task configuration file, and sends it to the downstream controller for initializing the inference program.

[0047] For example, if the scene recognition result is a mathematical calculation scene, the system searches the scene mapping rule library and matches the mathematical scene-specific rule. If the rule is bound to majority voting convergence semantics, 8 parallel inference branches, and uniform decoding length, the system integrates and generates a task configuration that includes the above content.

[0048] S203: Create parallel inference branches according to the task configuration and generate a multi-branch inference program carrying preset convergence semantics.

[0049] In this embodiment, the scheduling controller loads task configurations including the number of branches, decoding quotas, and convergence semantic identifiers. Based on the number of branches specified in the configuration, it batch initializes independent inference branches that do not interfere with each other. All branches load the same user's original task request as inference input. At the same time, the corresponding preset convergence semantics and matching judgment parameters in the configuration are attached to the program's main controller, assembling a multi-branch inference program that binds fixed result summarization rules and can execute inference in parallel. Then, each branch is started to synchronously carry out text generation inference.

[0050] For example, the task configuration obtained by rule matching is 8 inference branches, and the preset convergence semantics is majority voting. The controller creates 8 independent mathematical inference branches in batches. Each branch takes in the user's mathematical question. The majority voting vote determination rule is bound to the top-level controller of the program to generate a set of 8 branches that follow the majority voting aggregation rule and start parallel inference.

[0051] In the above method, scene recognition and rule matching are used to automatically bind the corresponding convergence semantics, standardize the generation of multi-branch programs, unify and solidify the result summary logic, and the subsequent verification process can directly call the preset rules to complete the invariant verification. This simplifies the deployment process and avoids verification failures and inference errors caused by inconsistent custom rules.

[0052] S103, Generate shortcut actions based on the running state of the multi-branch reasoning program; wherein, shortcut actions are operation instructions that optimize the running state.

[0053] In this embodiment, the inference progress, output results, memory usage, token consumption, and other running status of each branch of the multi-branch inference program are monitored in real time. An untrusted action proposer generates various optimization operation instructions based on the current running status to reduce latency, reduce token overhead, and release memory resources. These optimization instructions are called shortcut actions. The proposer is only responsible for producing optimization proposals and has no right to decide whether the action can be executed.

[0054] In one embodiment, see Figure 3 This is a flowchart illustrating the generation of shortcut actions according to an embodiment of this application, as shown below. Figure 3 As shown, step S103 includes: S301, Statistics on the reasoning state of the multi-branch reasoning program; wherein, the reasoning state includes the number of branches that have completed the reasoning task, the reasoning results of branches that have completed the reasoning task, the number of branches that have not completed the reasoning task, and the remaining decoding quota of branches.

[0055] In this embodiment, the system's built-in status monitoring module continuously polls all parallel inference branches, and in real time calculates the total number of branches that have finished decoding, the normalized inference answer of each completed branch, and the number of incomplete branches still being decoded. Simultaneously, it reads data such as the remaining token generation quota for each incomplete branch. This multi-type information is integrated into unified inference status data and output externally, providing it to the quick action proposer for analysis to determine if there is room for optimization.

[0056] For example, during the operation of an 8-branch majority voting reasoning program, the monitoring module obtains the reasoning status as follows: 5 branches have been completed, the answer to all 5 branches is A, 3 branches have not been completed, and each of the 3 branches has a remaining decoding quota of 256 tokens. This set of data is integrated as the complete reasoning status.

[0057] In addition, the controller starts all inference / tool ​​call branches according to the task configuration. Each branch independently executes decoding or tool requests. The status acquisition component synchronously captures the intermediate output content generated by each branch, the remaining token decoding budget, the status hash representing the unique identifier of the current branch snapshot, the backend hardware node number where the branch is running, the version number of the currently loaded normalizer and validator, and the side effect records such as file reading and writing and interface calls generated by the branch. All information is bound to the corresponding branch identity and stored and archived.

[0058] S302, Generate a quick action based on the reasoning state.

[0059] In this embodiment, the shortcut action proposer receives the summarized inference state, makes a comprehensive judgment based on the convergence semantic rules bound to the current program, and if it is determined that the current branch operation meets the conditions for early optimization, it generates the corresponding shortcut action instruction. The instruction is marked with the action type, target branch, and expected resource benefits. After fully recording the relevant information of the action, it is submitted to the certificate generator for processing.

[0060] For example, the multi-branch reasoning program counts that 5 out of 8 branches are completed and the answer is A, and the remaining 3 are incomplete. The proposer combines the majority vote and semantic determination to determine that the remaining branches cannot overturn the current result, and then generates a shortcut action of "cancel the 3 incomplete branches and release the corresponding KV cache".

[0061] In addition, based on real-time running data such as the remaining decoding quota of the current branch, the number of unreleased KV cache pages, and the number of GPU queued tasks, the proposer quantifies and calculates the inference time that can be shortened by executing the current shortcut action, the total amount of tokens that can be saved, the number of KV cache pages that can be reclaimed, and the backend queuing congestion and GPU memory usage pressure that can be alleviated. The proposer outputs all the estimated benefit values ​​along with the shortcut action and stores them in metadata for use by the certificate generator and audit module.

[0062] The above method uses multi-dimensional quantitative indicators such as the number of completed branches, branch results, number of incomplete branches, and remaining decoding quotas to fully depict the real-time operation of multiple branches. This serves as an objective basis for generating quick actions, providing data support for resource optimization and scheduling, and making scheduling more accurate and efficient.

[0063] S104, based on the preset convergence semantics, perform authorization verification on the program execution result corresponding to the shortcut action to obtain the semantic authorization result.

[0064] In this embodiment of the application, the convergence semantics pre-bound by the program are used as the judgment criteria to verify whether the shortcut action to be executed will change the final result that the multi-branch program should originally output, thereby determining whether the optimization action is legal and usable, and finally outputting the semantic authorization result of pass or rejection. Only when the authorization is passed can the shortcut action be executed.

[0065] In the above method, authorization verification is carried out on the shortcut action for optimization based on the preset convergence semantics. This can optimize the operation of multi-branch reasoning program without changing the final output result, thus balancing reasoning acceleration and result output accuracy.

[0066] In one embodiment, see Figure 4 This is a schematic diagram of an authorization verification process provided in an embodiment of this application, such as... Figure 4 As shown, step S104 includes: S401, obtain the full metadata corresponding to the current multi-branch inference program; wherein, the full metadata includes the hash value corresponding to the original task request, the preset convergence semantics, the real-time running snapshot of the multi-branch program, the action category corresponding to the shortcut action, and the estimated savings.

[0067] In this embodiment, after receiving the shortcut action to be verified, the certificate generator retrieves the unique hash identifier generated by the original task request from the task scheduling module, reads the preset convergence semantics bound to the current multi-branch inference program, retrieves the branch running snapshots statistically collected in real time by the monitoring module, and extracts the action type to which the shortcut action belongs and the estimated savings in tokens, video memory, latency, etc., of the action. All of the above information is summarized and encapsulated into a complete set of full metadata, which is used to construct a convergence certificate bound to the real-time state.

[0068] For example, for the shortcut action of "canceling 3 incomplete mathematical reasoning branches and releasing KV cache", the system obtains full metadata: the task hash corresponding to the mathematical question, the majority voting convergence semantics, the real-time snapshot of 8-branch reasoning (5 branches completed answer A, 3 branches not completed), the action category is branch cancellation + cache release, and it is estimated to save 1200 tokens and 3GB of video memory usage.

[0069] S402 generates a typed certificate based on full metadata and shortcut actions.

[0070] In this embodiment, based on full metadata including task hash, convergence semantics, branch running snapshot, and shortcut action type, a unique typed convergence certificate is generated by matching the convergence semantics bound to the current program. The real-time running status, action information, and semantic rules are all bound in the certificate, which serves as a machine-verifiable credential for subsequent trusted verifiers to verify whether the shortcut action is legal and executable.

[0071] In one embodiment, see Figure 5 This is a schematic diagram of the process for generating a typed certificate according to an embodiment of this application, such as... Figure 5 As shown, step S402 includes: S501 matches a unique certificate template based on quick actions and preset semantic matching.

[0072] In this embodiment, the system pre-stores multiple certificate templates that are adapted to different convergence semantics and shortcut actions. Each template has a fixed field structure. The certificate generator reads the convergence semantic identifier and shortcut action type of the current program, searches the template mapping table to complete the matching, and selects the unique and suitable exclusive certificate template. Subsequently, all metadata is filled into the corresponding fields of the template to generate a complete certificate.

[0073] For example, the current program uses majority voting convergence semantics. The shortcut action is to cancel incomplete branches and release the KV cache. After the system searches the template library, it matches the majority voting exclusive certificate (MajorityCert) template. This template is pre-loaded with exclusive fields such as joinid, branch universe hash, leading votes, and maximum remaining votes.

[0074] S502 adds exclusive parameters to the exclusive certificate template based on the full metadata.

[0075] In this embodiment, the certificate generator splits the full metadata and extracts the exclusive verification parameters corresponding to the current convergence semantics, such as the leading votes, second leading votes, and maximum residual votes of incomplete branches in the majority voting scenario, the verifier version and witness hash in the first successful scenario, and then fills the exclusive parameters along with general data such as task hash, branch universe hash, action type, and risk threshold into the preset fields of the exclusive certificate template to assemble a complete and verifiable convergence certificate.

[0076] For example, after matching the majority voting certificate template, specific parameters are extracted from the full metadata: 5 votes for completed answer A, 0 votes for second-leading answer B, and 3 maximum remaining votes for incomplete branches. These parameters, along with the task hash, branch running snapshot hash, and action type, are filled into the corresponding positions in the template to generate a complete majority voting certificate.

[0077] S503 generates a typed certificate based on the exclusive certificate template with added exclusive parameters.

[0078] In this embodiment, the certificate generator serializes the template with all general metadata and semantic-specific parameters filled in, adds a unique certificate number, timestamp, and status binding hash, packages and encrypts it according to the standard format of the corresponding certificate type, generates a typed certificate corresponding to the semantic category, fully binds the current task, branch running snapshot, pending shortcut action and semantic verification conditions, and sends it to the certificate verifier for legality verification.

[0079] For example, a majority voting certificate template filled with parameters such as vote count and status hash is serialized and encapsulated to generate a majority voting type certificate. The certificate fully records the task hash, 5 leading votes, 3 remaining votes limit, branch snapshot, and action information such as canceling the branch and releasing the cache, and is handed over to a trusted verifier for verification.

[0080] The above method fully collects metadata from all dimensions, including task hash, convergence semantics, branch execution snapshots, action types, and optimization benefits. It then generates corresponding dedicated type certificates based on business semantics and relies on the exclusive invariant rules built into the certificate to complete rigorous mathematical verification, rather than simple experience-based judgment, thus ensuring the rigor and universality of the verification mechanism.

[0081] S403, based on the typed certificate, performs authorization verification on the program execution result corresponding to the shortcut action to obtain the semantic authorization result.

[0082] In this embodiment, the trusted certificate verifier reads the complete typed certificate and sequentially verifies the certificate type legality, whether the running status bound to the certificate is consistent with the real-time snapshot of the current branch, and whether the invariant conditions of the corresponding convergence semantics are met. In this way, it is determined whether executing the shortcut action will change the predetermined output result of the program, and finally outputs a semantic authorization conclusion on whether the shortcut action is allowed to be executed.

[0083] In the above method, by collecting full-dimensional metadata covering tasks, rules, real-time running status, and optimization operations and binding it to the certificate, the certificate can fully record the current inference environment and optimization intent. Subsequent verification can rely on the complete set of information built into the certificate and combine it with the corresponding converged semantics to complete rigorous inference and verification, avoiding the problems of information loss and judgment distortion that exist in single data verification.

[0084] In another embodiment, see Figure 6 This is a schematic diagram of the authorization verification process provided in another embodiment of this application, such as... Figure 6 As shown, step S403 includes: S601, obtain the current running status data of the multi-branch inference program.

[0085] In this embodiment, the system status monitoring module continuously polls each parallel branch, collects and summarizes all status information in real time, including the number of completed branches, the inference results of each branch, the number of incomplete branches, the remaining decoding token quota, KV cache usage, and the load of backend nodes. This information is then uniformly organized into structured operational status data and synchronously provided to the certificate verifier for consistency comparison with the snapshot recorded in the certificate.

[0086] For example, during the operation of an 8-branch majority voting inference program, the monitoring module collects the following running status data: 5 branches have completed inference and the result is A for all of them; the remaining 3 branches are still decoding, each with 256 tokens remaining; and the current KV cache of 24 pages is recorded. The entire set of data serves as the current running status of the program.

[0087] S602 matches the runtime status data with the full metadata contained in the typed certificate to obtain the first matching result.

[0088] In this embodiment, the certificate verifier extracts real-time collected running status data and fields such as branch snapshots, task hashes, number of branches, and completion status of each branch stored in the full metadata of the typed certificate. It compares the key values ​​of the two types of data with the unique hash identifier item by item. If all fields are indistinguishable, the match is determined to be successful. If any core information is inconsistent, the match is determined to be unsuccessful, and the corresponding first matching result is output.

[0089] For example, the real-time running status is 8 branches, 5 branch results A, and 3 incomplete; the snapshot of the record in the metadata of the majority voting certificate is also 8 branches, 5 votes A, and 3 remaining branches. If all are matched, the first matching result is obtained; if the certificate record is 4 votes A, which does not match the real-time 5 votes, the matching fails.

[0090] It should be noted that before performing branch snapshot data verification based on the typed certificate, it is necessary to verify the certificate's eligibility or check whether the certificate type is supported. After verifying that the certificate type is supported, the aforementioned branch snapshot verification can be performed.

[0091] S603 If the first matching result indicates a successful match, then the semantic authorization result is obtained by performing a deduction and verification based on the typed certificate.

[0092] In this embodiment of the application, after verifying that the branch snapshot recorded in the certificate is completely consistent with the current actual running state, the verifier performs logical deduction and verification based on the preset convergence semantic rules and various verification parameters carried in the certificate, and determines that the final output result of the multi-branch program will not change after executing the shortcut action recorded in the certificate, thereby generating a semantic authorization result that allows or prohibits the execution of the shortcut action.

[0093] The above method employs a two-layer verification mechanism of state matching and semantic verification. First, it verifies that the certificate is consistent with the current operating environment to prevent expired or forged certificates from allowing illegal operations. Then, it verifies the convergence of exclusive invariants to logically prove that the shortcut action does not change the final output. This layered interception of risk-optimized operations enhances the verification's fault tolerance and security.

[0094] In another embodiment, see Figure 7 This is a flowchart illustrating the semantic authorization verification process provided in another embodiment of this application, such as... Figure 7 As shown, step S603 includes: S701, based on the full amount of metadata contained in the typed certificate and the preset convergence semantics contained therein, perform inference and verification to obtain the inference and verification conclusion.

[0095] In this embodiment, the verifier extracts all metadata recorded in the certificate, such as the total number of branches, the number of completed branches, the reasoning results of each branch, the number of incomplete branches, and the type of shortcut action. Then, it retrieves the preset convergence semantic rules bound to the certificate, substitutes them into the corresponding semantic verification formula, simulates the output of all remaining incomplete branches that is opposite to the current leading result, calculates whether there is a possibility of overturning the existing conclusion, and generates a deductive verification conclusion that the verification is passed or failed based on the verification results.

[0096] For example, there are 8 branches in total, 5 of which output answer A, and the remaining 3 are incomplete; the default convergence semantics are majority voting. The deduction and verification assume that all 3 incomplete branches output answer B. B has a maximum of 3 votes, which cannot exceed A's 5 votes. Executing the shortcut action to terminate the remaining branches will not change the final output, thus the deduction and verification conclusion is passed.

[0097] S702, the deduction and verification conclusion is matched with the semantic compliance statement bound in the typed certificate to obtain the second matching result; wherein, the semantic compliance statement represents the statement information that the execution shortcut action added in advance in the typed certificate does not change the running result of the multi-branch program.

[0098] In this embodiment, the verifier extracts the conclusion obtained from the deduction and verification, as well as the semantic compliance declaration text and judgment identifier pre-carried in the typed certificate, and compares whether the two judgment logics on "whether the execution of the shortcut action affects the final output of the multi-branch" are consistent. If the verification proves that the action does not change the result and the certificate declaration also marks that the action is compliant and there is no risk of result tampering, then the second matching result is a successful match. If the two judgments contradict each other, then the matching fails.

[0099] For example, the majority voting certificate has a built-in semantic compliance statement: terminating the remaining 3 branches will not change the final answer A; the deduction and verification conclusion is that even if all the remaining branches output B, they cannot overtake A. The two judgments are consistent, and a second matching result is obtained; if the verification shows that the remaining branches may overtake, but the certificate is still marked as compliant, then the second matching result is a failed match.

[0100] S703, if the second matching result indicates a successful match, then the semantic authorization result indicates a successful authorization.

[0101] In this embodiment of the application, after completing two layers of verification—matching the running status with the certificate snapshot and the semantic inference verification with the certificate compliance statement—and both matching results are successful, the verification process determines that the shortcut action to be executed is legal, generates an authorization success identifier, and issues an instruction to allow the system to execute the optimization operation corresponding to the shortcut action.

[0102] It should be noted that during the verification process, the trusted verifier executes the following function: VerifyCert(P,S_t,a,c) Where P represents the global configuration of the current multi-branch inference task, including convergence semantic rules, validator version, and global constraints; S_t represents the real-time branch running status data collected at time t (current branch completion votes, KV cache, number of incomplete branches, snapshot hash, etc.); a represents the shortcut action to be executed (optimization operations such as early convergence, branch pruning, and releasing KV cache); and c represents the typed certificate generated based on the full metadata (one of five types of certificates, including majority vote convergence certificate and first-success convergence certificate). If the certificate type is unknown, the state binding does not match, or the convergence-specific invariant does not hold, the request is rejected.

[0103] In the above method, the logical deduction and verification are first completed by combining the full metadata of the certificate with the exclusive converged semantics, and then the compliance statement of the result pre-bound by the certificate is checked. The double verification closed loop proves that the quick action will not tamper with the summary result. Only when it fully complies with the preset compliance constraints can it be authorized, thus eliminating the result deviation caused by the optimization operation from the logical level.

[0104] In one embodiment, the method further includes: If authorization is successful, the shortcut action is executed; if authorization fails, the shortcut action is not executed, and the multi-branch reasoning program continues to run.

[0105] In this embodiment, the scheduling module receives the final authorization identifier for traffic splitting. If the authorization is successful, it immediately executes the quick action instruction, such as terminating the incomplete branch, releasing the KV cache, or reducing the reserved Token budget. If the authorization fails, it ignores the current quick action proposal, does not perform any branch interruption or resource reclamation operations, and all inference branches continue to run the original decoding process, continuously collect new inference states, and regenerate quick actions to carry out the next round of verification.

[0106] In the above method, the process is divided into branches based on the semantic authorization result. If the authorization is successful, it means that the shortcut action will not change the final summary result, and resource optimization can be performed with confidence. If the authorization fails, it means that the operation has the risk of tampering with the reasoning result, so the optimization is abandoned and the original complete reasoning process is maintained.

[0107] In addition, this application provides other embodiments besides majority voting, as follows: First successful convergence of deterministic checker certificate Application environments: code generation, program synthesis, constraint solving, or tool execution search. Multiple candidate branches generate programs, commands, or solutions simultaneously. The program declarations are successfully merged first, meaning that as long as one candidate passes the registered deterministic checker, the solution can be returned.

[0108] After each branch completes its candidate, the runtime performs deterministic checks in a sandbox or fixed environment, such as unit testing, arithmetic constraint checks, format validation, or domain rule checks.

[0109] When a candidate passes the verification, the certificate generator successfully merges the certificates, including the candidate hash, input hash, verifier number / version, execution environment hash, and witness hash.

[0110] The validator re-executes or verifies the checker result to confirm that the candidate satisfies the first successful predicate under the current input and version binding.

[0111] Once the certificate is approved, the runtime returns to the candidate and cancels the sibling branch; if the checker overhead is high, this shortcut action can only be enabled when deterministic witnesses can be cached or quickly replayed.

[0112] A safe extension, not deletion, under the condition that all must converge Application environment: In scenarios such as task decomposition, report generation, and multi-step tool invocation, each subtask must be completed to obtain the final output. All program declaration fields are required.

[0113] The controller maintains four types of branch sets: required, completed, deferred, and deleted.

[0114] If the proposer suggests releasing resources or skipping a branch, the certificate generator must construct an AllRequiredCert to prove that the branch has been completed or has only been postponed, checked, offloaded, or rescheduled, rather than logically deleted.

[0115] The validator checks the required branch set ∪ the completed branch set ∪ the deferred branch set, and the required branch set ∩ the deleted branch set = empty set; it also checks whether the deferred branch set can be recovered from the checkpoint or the scheduling state.

[0116] Resource-level migrations or deferrals are only permitted when the above conditions are met; any action that deletes a required branch is rejected.

[0117] The validator accepts convergence and statistical risk certificates. In the validator acceptance semantics, the program declares that it will only return if a registered deterministic validator accepts a proof or witness. The system constructs a verification by using a merge certificate (VerifierAcceptCert), binding the candidate, proof object, validator version, and state hash. After the validator passes, other search branches can be canceled.

[0118] For scenarios where deterministic proof is not possible, a statistical risk boundary convergence certificate (StatisticalCert) can be used. The certificate records the stopping rule, calibration set hash, observation statistics, single-risk budget alpha_t, and global risk budget delta. The validator confirms that the estimated residual risk does not exceed the budget and records the budget consumption for that instance. This embodiment does not claim zero error but rather provides an explicit risk boundary.

[0119] Integration with LLMserving backend The front-end semantic authorization layer can be deployed on SGLang, vLLM, and Orca-style back-ends without requiring modification of the basic model parameters.

[0120] Backend nodes can record parent nodes, logical token spans, physical key-value cache page IDs, refcounts, residency tiers, last access timestamps, and branch summaries.

[0121] After authorization by certificate, the system can securely perform resource actions, including undecoding, releasing or downgrading key-value pages, host offload, checkpoint saving / resume, and rescheduling.

[0122] Backend optimizations do not define validity; validity is determined solely by the declaration confluence semantics and the certificate validator.

[0123] See Figure 8 This is a schematic diagram of the overall architecture of the runtime shortcut action semantic authorization method for branching large language model programs provided in this application embodiment, as shown below. Figure 8 As shown, it includes: (a) Branched Large Model Inference (BranchyLLMinference) Core Process Prompt: Input the question (example: explain why water freezes at 0℃) and send it to the scheduling controller (strategy planner).

[0124] Multi-branch parallel reasoning: The controller initiates N independent reasoning branches, each generating text independently; green squares = generated tokens, gray squares = remaining tokens to be generated. Example: The four branches output four types of reasoning content: molecular motion, kinetic energy, hydrogen bonds, and phase transition dynamics, respectively.

[0125] Two core rule systems Merge semantics (summary rules): Defines how multi-branch results are merged and output. Examples include: majority voting, tool call (successful), and tree search (optimal K selection). Heuristic shortcut actions (optimization suggestions): The system proposes the following acceleration methods: early merging (early termination of branches), pruning and canceling branches, and early verification (completing verification earlier).

[0126] (b) Fast optimization without a certificate: poses security risks. Using a 7-branch majority vote as an example, this demonstrates the fatal flaw of pure heuristic early stopping: Current status: Branches 1 / 2 / 3 output answer A, branch 4 outputs answer B, branches 5 / 6 / 7 are incomplete; Heuristic shortcut operation: If you see that A has 3 votes in the lead, stop all branches in advance and return directly to the answer A; There are two risks of overturning the result: ① If all three remaining branches output B, the final 4 votes B > 3 votes A, and the correct result should be B; ② If all three remaining branches output C, the votes 3A:3B:3C will result in a tie, and the final output will be determined by the tie rule, which conflicts with the early return of A. Conclusion: Relying solely on "answer stability" and "heuristic score" to terminate prematurely will result in semantic violations and ultimately incorrect reasoning results.

[0127] (c) CertiJoin Trusted Verification Gateway (Complete Security Verification Pipeline) Complete data flow chain Quick Action Proposer → Certificate Generator → CertiJoin Verification Gateway → Run Executor 1. Certificate Family (Preset Standardized Certificate Templates) Dedicated certificate types adapted to different convergence semantics: Majority voting certificate: secure early majority convergence; First successful certificate verification: Verifier confirmation is required; All required certificates: All branches must be completed; Verifier receives certificate: Registered verifier is recognized; Statistical Certificate: Risk Boundary Constraints.

[0128] 2. Trusted Closed-Loop Verifier (Fully Enclosed Security Verification with Three-Layer Core Check) Type validation: The certificate type is valid and matches the current pooling semantics; State binding verification: The branch execution snapshot recorded in the certificate is completely matched with the real-time state of the program; Consolidation invariant verification: Through mathematical deduction, it is proven that executing the shortcut action will not change the final aggregated result.

[0129] 3. Verify the branch logic (core decision formula: VerifyCert(P,S_t,a,c)=1) If the validation fails (loop rejection): discard the shortcut action and continue the complete reasoning in the branch; Verification passed (legitimate shortcut action): Authorized to perform optimization operations such as early termination and branch pruning.

[0130] (d) Certified outcome & empirical takeaway 1. Trusted and rapid actions ensure that the semantics of the convergence remain unchanged. After the 7-branch example verification is passed, it is authorized to merge early when the 5th branch is completed: even if the remaining 2 branches are not completed, it can be proven through mathematical calculation that no matter what answer is output by the remaining branches, it cannot surpass the number of votes for A, and the final result must be A; at the same time, it is verified that the output result after running all branches is completely consistent with the early termination result.

[0131] 2. Empirical experimental indicators Semantic violation rate: 0% (compared to a maximum of 10.95% error rate with unvalidated heuristics); Offline acceleration up to 1.75x; Reasoning time saved: up to 1.54 times; The majority voting certificate verification time is only <1.15 milliseconds at the 95th percentile, with extremely low verification overhead.

[0132] 3. Test Benchmark Validation using standard test sets for multiple large-scale models, including GSMMB, MATHGO, MMLU, HumanEval, and MBPP.

[0133] See Figure 9 This is a schematic diagram illustrating different certificate types and inspection conditions corresponding to different convergence semantics provided in the embodiments of this application, such as... Figure 9 As shown, it includes: (a) Majority vote convergence + majority vote certificate 1. Process Logic , , ... This represents all parallel inference branches, with each branch outputting the voting results. The votes were counted in three categories: the number of votes for the leading answer (leader=92), the number of votes for the second-place answer (runner-up=61), and the number of unresolved votes (unresolved=23). Release determination formula: Leading votes The number of votes for the second place is greater than the total number of votes for the unfinished branches. In this example, 92-61=31>23, which meets the condition and allows early-join. Mathematical meaning: Even if all the votes for the unfinished branches are cast for the second place, it is impossible to overtake the leading answer. Early termination will not change the final result.

[0134] 2. Built-in parameters of the majority voting (MajorityCert) certificate The pool number, leading votes, second-place votes, number of unfilled splits, tie rules, and running status hash.

[0135] (b) First successful rendezvous + First success certificate 1. Process Logic Multiple branches generate candidate results in parallel; A deterministic checker exists, which generates an "accepted witness" as long as the output of any branch passes the checker's verification. Siblings step: Other branches that have not been completed can be terminated early without further reasoning.

[0136] 2. First Success Certificate (FirstSuccessCert) Built-in Parameters Merge number, verifier number, voucher hash, candidate result hash, running status hash.

[0137] (c) All tasks must be completed and all required certificates must be filled in (AllRequiredCert) 1. Process Logic These are tasks / branches that the system requires to be fully executed. Only when all branches are checked off will the task be considered complete. Restrictions: Required tasks cannot be deleted or prematurely terminated; only partial branches can be deferred, but cannot be directly terminated.

[0138] 2. Built-in parameters for AllRequiredCert certificates The hashes are: pool number, required task set hash, completed task set hash, delayed task set hash, and running status hash.

[0139] (d) Verifier Acceptance Certificate Verification (VerifierAcceptCert) 1. Process Logic The branch generates candidate results, which are then sent to the registered verifier. After the verifier approves the application, it generates an acceptance witness. If the credentials are valid, the process is initiated directly without waiting for other branches.

[0140] 2. Verify the built-in parameters of the receiving certificate (VerifierAcceptCert). Merge number, verifier number, voucher hash, timestamp, running status hash.

[0141] (e) Statistical probability convergence + Statistical risk certificate (StatisticalCert) 1. Process Logic Statistical results of probability distribution output from multiple branches; The tail risk interval δ is calculated using a probability distribution, with the constraint that total risk ≤ δ. Generates risk-bound certified certificates, allowing early branch termination under controlled error probabilities. Suitable for statistical reasoning tasks that do not require 100% absolute accuracy and allow for extremely low probability errors.

[0142] 2. Built-in parameters of the certificate (Statistical Cert) Merge number, confidence level, stop rule, risk threshold delta, calibration hash, running status hash.

[0143] See Figure 10 This is a schematic diagram of the KV cache shared storage architecture for branched LLM inference provided in an embodiment of this application, as shown below. Figure 10 As shown, it includes: Module 1: Shared Prefix Branch DAG (Underlying Storage Node) Basic concepts DAG: Directed Acyclic Graph, used to model the token prefix shared by multiple inference branches; Nodes n0~n12: Each node corresponds to a continuous token range and is bound to a set of key-value cache pages; rc=refcount: Reference count, indicating how many inference branches are currently reusing this node; Three-tier storage diagram: Deep Blue: Hot Layer (HOT) (High-speed GPU memory, fastest access) Cyan: Temperature layer WARM (ordinary GPU memory) White: Cold layer (host memory / disk, slowest speed) Structural logic in the diagram Prefix shared chain: n0→n1→n2→n3 is the pre-inference prefix shared by all branches. The reference count of these 4 nodes is rc=5, which means that 5 branches reuse this KV cache. The entire branch stores only 1 copy of KV data and does not copy it repeatedly. Branching: After n3, multiple independent reasoning branches split off: Branch 1: n3→n8, n3→n9 Branch 2: n3→n5→n10 Branch 3: The leaf nodes (n8 / n9 / n10 / n11 / n12) after the branching from n3→n6→n7→n11 and n7→n12 are used only for a single branch, rc=1; Bottom Explanation: Shared prefix nodes occupy only one KV cache, relying on reference counting to track reuse, which greatly saves video memory.

[0144] KV Cache Saving Calculation Formula Comparing the two storage solutions: Original scheme (Naiveper-branch): Each branch stores the complete set of key-value pages independently, and repeatedly stores the common prefix; Shared substrate solution: The common prefix is ​​stored only once; Savings rate = 1 - total number of pages in the shared solution / total number of pages in the original branch's independent storage; A shared architecture can significantly reduce GPU memory usage.

[0145] Module 2: Example of underlying node metadata (taking node n3 as an example) Displays complete management fields for a single cache node, divided into five categories: basic association, caching, tiering, inference, and verification statistics. Related Links Node ID: n3; Parent node: n2 (records the hierarchical relationship of the prefix DAG) KV cache stores information Token span: This node covers the token range [128, 256]. Key-value page IDs: Bound physical cache page numbers [1034, 1035, 1036, 1037] refcount: Reference count = 5 Layered residency attributes tier: HOT (Hot Layer GPU Memory) Last access: The most recent access timestamp, used for hot / cold tier scheduling. Inference result information Answer snippet: This branch snippet outputs a text summary. Decode length: The current node is decoding 128 tokens. CertiJoin validation of associated fields verifierstate: Binds the certificate type VerifierAcceptCert confidencestats: Confidence statistics (mean 0.87, minimum 0.62, 95th percentile 0.64), used for statistical risk assessment of convergence certificates. Module 3: Backend Execution Flow + Three-Level Layered Resident Scheduling Upper half: Quick Action Distribution Link (linked with CertiJoin Certificate Gateway) Quick Action Proposer → Certificate Verification Gateway (Semantic Validity Verification) → Backend Resource Executor Two rules: Simple resource operations (cache layer replacement, recycling): handled directly by the backend; Quick actions that alter the convergence semantics (early branch termination): must be verified and authorized by the certificate gateway before execution.

[0146] Second half: Three-tier storage architecture (speed decreases from top to bottom) GPU Hot Tier: Stores shared prefix nodes (n0 / n1 / n2 / n3) for high-frequency multiplexing, with a capacity of... ; GPU Warm Tier: Stores branch nodes (n4 / n8 / n9 / n10) with moderate access frequency, capacity... ; Cold Tier for Host Systems: Low-speed disks / host memory, storing infrequently used leaf nodes (n6 / n7 / n11 / n12), capacity... ; Hierarchical scheduling logic Demote: Node access frequency decreases → moves down from the hot layer to the warm layer, and then down to the cold layer; Upgrade (promote): Nodes are frequently read and reused → moved from the cold layer to the warm layer and then the hot layer; Garbage collect: When the node reference count (refcount) is 0 and there is no branch reuse, the KV cache is released directly, and video memory / disk space is reclaimed.

[0147] See Figure 11 This is a schematic diagram of various types of certificates provided in the embodiments of this application, such as... Figure 11 As shown, it includes: 1. Majority Certificate Applicable shortcuts It is suitable for multi-branch reasoning that uses majority voting aggregation rules and supports three types of optimizations: early branch termination (early merging), pruning and canceling redundant branches, and releasing key-value cache pages.

[0148] Certificate core fields Merge number, branch global snapshot hash, normalizer identifier and version, tie determination rule, leading answer hash, second-place answer hash, completed leading votes m1, second-leading votes m2, and maximum potential votes for uncompleted branches rmax.

[0149] Verification conditions (mathematical safety boundary) Recount the normalized valid votes and verify the inequality: m1 > m2 + rmax. This means that even if all unfinished branches are voted for the second-place answer, the second-place answer's total votes cannot exceed the current leading answer; early termination will not alter the final output.

[0150] 2. First Success Certificate (FirstSuccessCert) Applicable shortcuts For business semantics such as code generation and search tasks where the first valid result is effective, after the first branch passes validation, the remaining incomplete sibling branches from the same source are canceled.

[0151] Certificate core fields Candidate result hash, deterministic validator ID and version, input context hash, inference environment hash, verification pass credential hash, and pass flag.

[0152] Verification conditions Rerun the registered deterministic validator to verify that the candidate results output by the current branch do indeed satisfy the "first success" criterion, confirming the authenticity and validity of the credentials. Simple confidence scores cannot be used as reliable evidence; they must be verified through fixed validation logic.

[0153] 3. All required certificates (AllRequiredCert) Applicable shortcuts This is suitable for strongly constrained scenarios where all branches must be executed completely. It only allows delayed scheduling and task breakpoint saving / offline unloading from video memory, and prohibits the direct deletion of any required branches.

[0154] Certificate core fields Required branch set hash, completed branch set hash, delayed execution branch set hash, deleted set hash, global running status hash.

[0155] Verification conditions The validation process resulted in the direct deletion of any branches that had no required fields. The union of branches and delayed branches has been completed, fully covering all required tasks; Delayed branches can be resumed at any time, and there is no risk of permanent loss.

[0156] 4. Verifier Accept Certificate Applicable shortcuts For inference tasks that use a trusted registration validator to review results, once the validator approves a candidate, it can quickly return the result and cancel the remaining unexecuted branches.

[0157] Certificate core fields Candidate output hash, registered verifier ID and version, verification certificate / credential object, and global state hash.

[0158] Verification conditions Only credentials issued by a deterministic registration verifier are considered valid proof; confidence scores obtained solely through model learning cannot constitute credible proof and cannot be used as a basis for release.

[0159] 5. Statistical Risk Certificate (StatisticalCert) Applicable shortcuts For statistical reasoning that cannot be proven with 100% certainty using mathematical proof, stop early at the risk boundary under a pre-set controllable error probability.

[0160] Certificate core fields Stop rule identifier, calibration dataset hash, statistical distribution computational cost, confidence level, and risk prediction threshold.

[0161] Verification conditions Recalculate the residual error risk from the remaining branches to confirm that the overall risk does not exceed the threshold stated in the certificate and is within the globally allowed risk budget.

[0162] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0163] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0164] Figure 12 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 12 As shown, the terminal device 12 of this embodiment includes: at least one processor 120 ( Figure 12 (Only one is shown in the image) a processor, a memory 121, and a computer program 122 stored in the memory 121 and capable of running on at least one processor 120. When the processor 120 executes the computer program 122, it implements the steps in any of the above embodiments of the runtime shortcut action semantic authorization method for branching large language model programs.

[0165] The terminal device can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 12 This is merely an example of terminal device 12 and does not constitute a limitation on terminal device 12. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0166] The processor 120 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0167] In some embodiments, memory 121 may be an internal storage unit of terminal device 12, such as a hard disk or memory of terminal device 12. In other embodiments, memory 121 may be an external storage device of terminal device 12, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on terminal device 12. Furthermore, memory 121 may include both internal and external storage units of terminal device 12. Memory 121 is used to store operating system, application programs, boot loader, data, and other programs, such as program code of computer programs. Memory 121 may also be used to temporarily store data that has been output or will be output.

[0168] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0169] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0170] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0171] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0172] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0174] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technology of the embodiments of this application.

Claims

1. A runtime shortcut action semantic authorization method for branching large language model programs, characterized in that, The method includes: Receive the user's original task request; A multi-branch reasoning program carrying a preset convergence semantic is created based on the original task request; wherein the preset convergence semantic represents a fixed rule that is pre-set to summarize and output the running results of the multi-branch reasoning program. Quick actions are generated based on the running state of the multi-branch reasoning program; wherein, the quick actions are operation instructions that optimize the running state; Based on the preset convergence semantics, the program execution result corresponding to the shortcut action is authorized and verified to obtain the semantic authorization result; The step of authorizing the program execution result corresponding to the shortcut action based on the preset convergence semantics to obtain the semantic authorization result includes: Obtain the full metadata corresponding to the current multi-branch inference program; wherein, the full metadata includes the hash value corresponding to the original task request, the preset convergence semantics, the real-time running snapshot of the multi-branch program, the action category corresponding to the shortcut action, and the estimated savings; Generate a typed certificate based on the full metadata and the shortcut action; Based on the typed certificate, the program execution result corresponding to the shortcut action is authorized and verified to obtain a semantic authorization result; The step of generating a typed certificate based on the full metadata and the shortcut action includes: Based on the aforementioned shortcut action and preset convergence semantics, a dedicated certificate template is matched; Add exclusive parameters to the exclusive certificate template based on the full metadata. Generate the typed certificate based on the exclusive certificate template with added exclusive parameters; The authorization verification of the program execution result corresponding to the shortcut action based on the typed certificate to obtain the semantic authorization result includes: Obtain the current running status data of the multi-branch inference program; The running status data is matched with the full metadata contained in the typed certificate to obtain the first matching result; If the first matching result indicates a successful match, then the semantic authorization result is obtained by performing a deduction and verification based on the typed certificate.

2. The runtime shortcut action semantic authorization method for branching large language model programs as described in claim 1, characterized in that, The step of creating a multi-branch reasoning program carrying preset convergence semantics based on the original task request includes: The original task request is subjected to scenario identification to obtain the corresponding business scenario type; The business scenario types are matched according to rules to obtain the task configurations that are bound to the corresponding preset convergence semantics; Parallel reasoning branches are created according to the task configuration, generating the multi-branch reasoning program carrying preset convergence semantics.

3. The runtime shortcut action semantic authorization method for branching large language model programs as described in claim 1, characterized in that, The step of generating shortcut actions based on the running state of the multi-branch reasoning program includes: The reasoning state of the multi-branch reasoning program is statistically analyzed; wherein the reasoning state includes the number of branches that have completed the reasoning task, the reasoning results of branches that have completed the reasoning task, the number of branches that have not completed the reasoning task, and the remaining decoding quota of branches. The shortcut action is generated based on the reasoning state.

4. The runtime shortcut action semantic authorization method for branching large language model programs as described in claim 3, characterized in that, The step of performing inference and verification based on the typed certificate to obtain the semantic authorization result includes: Based on the full amount of metadata contained in the typed certificate and the preset convergence semantics contained therein, the inference and verification results are obtained. The deduction and verification conclusion is matched with the semantic compliance statement bound in the typed certificate to obtain a second matching result; wherein, the semantic compliance statement represents the declaration information added in advance in the typed certificate that the execution of the shortcut action does not change the running result of the multi-branch program; If the second matching result indicates a successful match, then the semantic authorization result indicates a successful authorization.

5. The runtime shortcut action semantic authorization method for branching large language model programs as described in any one of claims 1-4, characterized in that, The method further includes: If the semantic authorization result indicates successful authorization, then the shortcut action is executed; If the semantic authorization result indicates that the authorization has failed, the shortcut action will not be executed, and the multi-branch reasoning program will continue to run.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images