Large language model output verification method and device, electronic equipment and computer readable storage medium

Abstract

The present application relates to artificial intelligence technology, and provides a large language model output verification method and device, electronic equipment and computer readable storage medium, the method comprises: in the process that a large language model inquires for a preset generation task and outputs Token stream, the Token segment of current time output is obtained;Token segment is verified using pre-initialized verifier, and the verifier is built based on finite state machine according to the preset specification for constraining the syntax and semantics of Token stream of large language model output;When the verification fails, terminate the Token stream of large language model and continue to output.This application terminates the continuous output of Token stream when the real-time verification of Token stream output by large language model fails, which avoids the serious waste of reasoning computing power and Token quota.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and more specifically, to a method, apparatus, electronic device, and computer-readable storage medium for large language model output verification. Background Technology

[0002] With the widespread application of LLM (Large Language Model), ensuring that the model output strictly conforms to a predefined schema (such as JSON Schema) has become a key step in building a stable system.

[0003] Traditional technical solutions typically employ a "prompt engineering + post-processing verification" model. This involves injecting preset formatting requirements into the prompts, and then performing a full parsing of the generated text after the LLM has completed all text generation. However, in actual high-concurrency engineering practices, this "post-generation verification" strategy suffers from a significant waste of inference computing power and token quotas because the verification logic lags behind the generation process. If the model generates an error that violates the preset formatting requirements in the early stages of response, the exception cannot be detected until the entire sequence has been generated. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, electronic device, and computer-readable storage medium for large language model output verification.

[0005] The embodiments of the present invention can be implemented as follows: In a first aspect, the present invention provides a method for validating the output of a large language model, the method comprising: During the process of the large language model inferring and outputting a token stream for a preset generation task, the token fragment output at the current moment is obtained; The token fragments are validated using a pre-initialized validator, which is constructed based on a finite state machine according to a preset specification for constraining the syntax and semantics of the token stream output by the large language model. If the validation fails, the large language model will stop outputting the Token stream.

[0006] In an optional implementation, before the step of verifying the token fragment using a pre-initialized verifier, the method further includes: Obtain a predefined structured data class, wherein the structured data class includes at least one field; The metadata of each field is parsed to obtain the semantic constraints of each field; All the aforementioned semantic constraints are converted into preset specifications according to preset grammatical constraints; Generate a finite state machine that conforms to the preset specification and bind the finite state machine and the verifier to initialize the verifier.

[0007] In an optional implementation, after the step of converting all the semantic constraints into a preset specification according to preset syntactic constraints, the method further includes: Obtain the configuration strategy of the large language model; If the configuration strategy is a tool invocation configuration, then a preset tool definition object is constructed and the preset specification is embedded into the parameters of the preset tool definition object, so as to inject the preset specification into the input context of the large language model through the preset tool definition object; If the configuration strategy is a text generation configuration, the preset specification is formatted into a string according to a preset format and the string is injected into a prompt word, so as to inject the preset specification into the input context of the large language model through the prompt word.

[0008] In an optional implementation, the verifier is pre-bound to a finite state machine, and the step of verifying the token fragment using the pre-initialized verifier includes: The token fragment is input into the finite state machine as a byte stream. If the currently input byte triggers the finite state machine to jump to a preset illegal state, the verification is determined to have failed. If none of the bytes in the Token fragment trigger the finite state machine to jump to the preset illegal state, then the verification is deemed successful.

[0009] In an optional implementation, the step of terminating the output of the Token stream by the large language model when the verification fails includes: When the verification fails, a cancellation signal is sent to the large language model so that the large language model terminates the output of the token stream based on the cancellation signal; or, If the verification fails, the network connection with the large language model is disconnected, and the large language model is terminated from continuing to output the Token stream.

[0010] In an optional implementation, the method further includes: When the verification fails and the number of retries is less than the preset number, an error object is constructed based on the current verification location, error type, and expected type. Update the context of the large language model based on the error object; The verifier is reinitialized based on the updated context, and the pre-defined generation task is re-inferred using the large language model.

[0011] In an optional implementation, the Token stream is temporarily stored in a target buffer, and the step of obtaining the Token fragment output at the current moment includes: The memory address and fragment length of the Token fragment in the target buffer are obtained using a preset buffer protocol; The token fragment is obtained from the target buffer based on the memory address and fragment length.

[0012] In an optional implementation, the method further includes: When the verification passes, wait to obtain the Token fragment of the next moment and verify it until the Token stream is completely output.

[0013] In an optional implementation, after the token stream is output, the method further includes: Convert the Token stream into an object instance of a preset structure type; Identify each target field in the object instance that has a preset external dependency; Retrieve the active resource instances of the preset external dependencies required for each of the target fields; All active resource instances required by the target fields are treated as active resource instances context-dependent with respect to the object instance. Based on the object instance and the activity resource instance, a target object carrying context information is generated.

[0014] Secondly, the present invention provides a computer program product, which, when executed by a processor, implements the large language model output verification method described in the first aspect.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention, during the process of a large language model inferring and outputting a token stream for a preset generation task, utilizes a pre-initialized validator to verify the token fragment acquired at the current moment. When the verification fails, the large language model terminates its output of the token stream. Since the validator is constructed based on a preset specification and a finite state machine, and this preset specification is a pre-determined specification used to constrain the syntax and semantics of the token stream output by the large language model, real-time verification of the token stream output by the large language model is achieved. This allows for timely termination of the token stream output when verification fails, avoiding the severe waste of inference computing power and token quotas caused by the "post-generation verification" approach. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an example diagram illustrating the architecture of a verification method in the conventional technology provided in this embodiment.

[0018] Figure 2 This is an example architecture diagram of the large language model output verification method provided in this embodiment.

[0019] Figure 3 This is a flowchart illustrating the large language model output verification method provided in this embodiment.

[0020] Figure 4 Example of a block diagram for the large language model output verification device provided in this embodiment. Figure 1 .

[0021] Figure 5 Example of a block diagram for the large language model output verification device provided in this embodiment. Figure 2 .

[0022] Figure 6 This is a block diagram of the electronic device provided in this embodiment.

[0023] Icons: 10-Electronic device; 11-Processor; 12-Memory; 13-Bus; 100-Large language model output verification device; 110-Acquisition module; 120-Verification module; 130-Termination module; 140-Retry module; 150-Generation module. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0027] In the description of this invention, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0028] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0029] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0030] To more clearly illustrate the "post-generation verification" verification method in traditional techniques, this embodiment combines... Figure 1 For further explanation, please refer to [link / reference]. Figure 1 , Figure 1 This is an example diagram illustrating the architecture of a verification method in the conventional technology provided in this embodiment. Figure 1 In this context, the requirements for the output format of the LLM model are injected through prompts. The LLM model performs inference based on the output format requirements in the prompts to obtain the model output results. The model output results are fully validated through the built-in JSON library of the Python interpreter or a third-party validation library. The validated model output results are then sent to downstream businesses.

[0031] This approach suffers from at least the following technical drawbacks: First, because the validation logic lags behind the generation process, if the model generates character or field type errors that violate JSON syntax in the early stages of the response (such as the first few tokens), the system still needs to wait for the entire sequence to be generated before it can capture the exception. This leads to a serious waste of expensive GPU (Graphics Processing Unit) inference computing power and API (Application Programming Interface) token quotas. Second, when processing long text streaming responses, the pure Python-implemented validator is limited by the GIL (Global Interpreter Lock) and the overhead of interpretation execution, making it difficult to complete complex schema recursive validation within the millisecond time window when the token arrives, resulting in backlog and latency jitter in streaming transmission. Finally, traditional error handling mechanisms often only perform simple retries and lack a closed-loop mechanism to feed back the specific validation error stack to the model, causing the model to easily get stuck in an infinite loop of repeated errors during the generation of complex structures.

[0032] In view of this, this embodiment provides a large language model output verification method, apparatus, electronic device, and computer-readable storage medium to overcome at least the above-mentioned problems, which will be described in detail below.

[0033] Please refer to Figure 2 , Figure 2 This is an example architecture diagram of the large language model output verification method provided in this embodiment. Figure 2 In this process, users predefine structured data classes, which are then converted into a schema. One side of the process uses the schema to construct prompts, and based on these prompts, performs context injection and request input to the LLM model. The other side uses the schema byte stream to initialize a validator for validation. These two processes can be performed in parallel. When the LLM model outputs a token stream, each generated token fragment is monitored through asynchronous stream processing and sent to the validator for verification. If the validation fails, the continued output of the token stream is terminated; if the validation passes, the next token fragment is processed until the current token stream is completed, outputting a target object that can be used by downstream businesses.

[0034] It should be noted that, Figure 2 This is just one example of an architecture; in fact, other processing can be added on top of it to meet actual needs. The following is based on... Figure 2 The architecture description of the large language model output verification method provided in this embodiment is as follows: Figure 3 , Figure 3 This is a flowchart illustrating the large language model output verification method provided in this embodiment. The method includes the following steps: Step S101: During the process of the large language model performing inference for the preset generation task and outputting the token stream, obtain the token fragment output at the current moment.

[0035] In this embodiment, the preset generation task can be in the form of information such as text, language, or images input by the user to the large language model. The large language model performs inference on the preset generation task and outputs the inference result as a text stream. The token stream can be a data sequence composed of text fragments segmented according to preset semantic rules arranged in sequence, and the token fragment is the data sequence output by the large language model at the current moment.

[0036] Step S102: The token fragment is validated using a pre-initialized validator, which is constructed based on a finite state machine according to the preset specifications for the syntax and semantics of the token stream output by the large language model.

[0037] In this embodiment, the preset specification can originate from a user-defined structured data class, such as a data model inheriting from Pydantic BaseModel. Its metadata, including field types, enumerated values, numerical ranges, and regular expression constraints, is parsed and converted into a standard specification, such as JSON Schema. The preset specification is compiled into state transition rules for a Finite State Machine (FSM). After the FSM is bound to the validator and initialized, the validator can instantly determine whether the token conforms to the preset specification based solely on the currently input byte fragment and its historical parsing state at any given time.

[0038] In this embodiment, as an implementation method, the preset specification may include JSON syntax structure (such as object start, key-value separation, quotation mark closure, etc.) and schema semantic constraints (such as a field must be an integer, a string must match a specific regular expression pattern, and an enumeration field can only take "Success" or "Failed", etc.).

[0039] Step S103: When the verification fails, the large language model is terminated from continuing to output the Token stream.

[0040] In this embodiment, a failure to verify may be caused by a byte in the token fragment causing the finite state machine to jump to an invalid state or a preset abnormal state. In this case, in order to avoid continuing to consume inference computing power and token quota meaninglessly, the large language model is terminated from continuing to output the token stream.

[0041] The method provided in this embodiment achieves real-time verification of the token stream output by the large language model through a pre-determined preset specification for constraining the syntax and semantics of the token stream output by the large language model and a verifier constructed from a finite state machine. When the verification fails, the continued output of the token stream is terminated in a timely manner, avoiding the serious waste of inference computing power and token quota caused by the "post-generation verification" implementation method.

[0042] In an optional implementation, in order to automatically obtain a verifier capable of performing verification according to a preset specification, this embodiment provides an implementation method for initializing the verifier: First, obtain a predefined structured data class, which includes at least one field; In this embodiment, a structured data class can be a pre-defined data structure defined by the user to constrain the output of a large language model, including requirements for output format and content in business logic. A field is the basic unit in the structured data class used to carry specific information. Each field carries its associated metadata, which can be interpreted as descriptive information attached to the field at the code level, including type declarations, value ranges, regular expression patterns, whether a field is required, nesting relationships, etc. These metadata collectively constitute the semantic constraints that the field must satisfy in practical applications. For example, a structured data class may contain fields such as username, age, and role type, and specify the conditions that each field should satisfy through type annotations (e.g., username is of type str, age is of type int), field constraints (e.g., Field(ge=0, le=120) indicates that the age is between 0 and 120), and enumeration declarations (e.g., role: Literal["admin", "user"]).

[0043] Secondly, the metadata of each field is parsed to obtain the semantic constraints of each field; In this embodiment, for structured data classes that inherit from Python's `pydantic.BaseModel` base class, by parsing metadata, the constraints defined in `const` or `Field` can be converted into validation keywords in the JSON Schema. For example, regular expressions are mapped to the `pattern` field, and numerical ranges are mapped to the `minimum` and `maximum` fields. When nesting exists, nested reference relationships ($defs) are processed, recursively parsing all constraints of the internal fields and organizing the reference relationships into a self-contained structure, ensuring that the generated Schema is a self-contained closure. For example, when a field is labeled as an `int` type with a numerical range of 0-120, 0 and 120 are mapped to the `minimum` and `maximum` fields, respectively; when a field is a nested model (such as `address: AddressModel`), `AddressModel` is defined as: class AddressModel(BaseModel): street:constr(min_length=1, max_length=100, pattern=r'^[a-zA-Z0-9\s\-, / ]+$') city: str = Field(..., min_length=2, max_length=50) The final JSON Schema representation of the address is as follows: “AddressModel”: { "street": { "type": "string", "minLength": 1, "maxLength": 100, "pattern": "^[a-zA-Z0-9\\s\\-, / ]+$" }, "city": { "type": "string", "minLength": 2, "maxLength": 50 } }

[0044] Third, convert all semantic constraints into preset specifications according to preset grammatical constraints; In this embodiment, the preset syntax constraints can be syntax constraints of a pre-specified language, such as JSON syntax constraints.

[0045] Finally, a finite state machine conforming to the preset specifications is generated and the finite state machine and the checker are bound together to initialize the checker.

[0046] In this embodiment, the preset specification can be to recursively serialize Python class attributes and their type annotations into JSON Schema dictionary objects conforming to the Draft 2020-12 standard. Each state of the finite state machine represents a certain context in the token stream parsing process (e.g., reading the key name of a JSON object, reading a string value, expecting an integer), and each state transition rule can be determined by the current state and the next input byte. The state transition table of the finite state machine is constructed and pre-compiled based on the preset specification in byte stream form. Initializing the validator can be done by generating a runnable validator instance and initializing it.

[0047] One implementation approach is to implement the validator within the Rust core module. This validator is passed across language boundaries from Python (the host environment) to the underlying Rust core module via Rust FFI (Foreign Function Interface). After receiving the schema byte stream, the Rust core module constructs and pre-compiles the finite state machine (FSM) transformation table based on the schema, initializes an instance of the validator, and maintains the current parsing cursor, stack depth, and expected token type, among other states and intermediate information generated during runtime.

[0048] In optional implementations, the inference interfaces of different large language models may be different. In order to adapt to large language models with different inference interfaces, this embodiment, after obtaining the preset specification, also provides an implementation method for injecting the preset specification into large language models with different inference interfaces: First, obtain the configuration strategy for the large language model; In this embodiment, the configuration strategy reflects the inference interface types supported by the large language model; the large language model can support both function calling and text generation. To adapt to different configuration strategies, the way preset specifications are injected into the large language model also varies.

[0049] Secondly, if the configuration strategy is tool invocation configuration, a preset tool definition object is constructed and the preset specification is embedded into the parameters of the preset tool definition object, so as to inject the preset specification into the input context of the large language model through the preset tool definition object; if the configuration strategy is text generation configuration, the preset specification is formatted into a string according to the preset format and the string is injected into the prompt word, so as to inject the preset specification into the input context of the large language model through the prompt word.

[0050] In this embodiment, the preset tool definition object typically includes three core fields: name, description, and parameters. The preset specification is fully embedded in the parameter field, allowing the model to simultaneously understand the syntactic and semantic constraints that the output must meet when parsing the tool definition. When the configuration strategy is text generation configuration, the preset specification can be serialized into a compact and syntactically correct JSON string according to the preset format, and this string is injected into the system prompt (i.e., the instructional context received when the model starts). The prompt explicitly instructs the model to strictly follow the structural requirements expressed by the string when outputting. In this way, whether the large language model is deployed in a professional interface environment that supports tool calls or runs in a general text generation environment that only accepts natural language prompts, the preset specification can be integrated into its input context in a way that the model can directly understand and respond to, thereby ensuring that the verification mechanism has feasibility and consistency in different deployment forms.

[0051] As one implementation, after obtaining the Python structured data class, converting the Python structured data class into a context that the large language model can understand and initializing the validator in the Rust core module can be done simultaneously. The processing method is as follows: First, schema extraction and standardization. This step aims to leverage Pydantic's built-in capabilities to map Python-type structured data classes to the Draft 2020-12 JSON Schema. The main processing includes: parsing metadata (such as regex patterns, ge / le ranges) from the fields of the structured data classes and converting them into schema constraints; handling the reference relationships ($defs) of nested models to ensure that the generated schema is a self-contained closure with no external file dependencies.

[0052] Secondly, the Rust FFI core initialization and prompt word injection and payload construction are performed in parallel.

[0053] For Rust FFI core initialization, this step aims to synchronously build and verify the validator before initiating network requests, implementing a "Fail-Fast" mechanism. The main processes include: serializing the JSON object into a UTF-8 byte stream, passing it across the FFI boundary to the Rust module, which immediately parses the schema and builds a transformation table for a Deterministic Finite Automaton Machine (DFA). If the schema contains complex types that cannot be converted to state machine rules (such as dynamic recursion depth), an exception is thrown, terminating subsequent invalid LLM inference requests, and outputting the message "Rust FSM compilation failed; the schema structure may be too complex or ambiguous," displaying the erroneous schema along with the message. If no exception occurs, the Rust-exposed C-ABI interface is called, returning a memory-safe handle (Handle ID), which is used to route subsequent streaming token chunks to the corresponding FSM.

[0054] For prompt injection and payload construction, this step aims to maximize instruction compliance by employing different schema injection strategies based on the varying capabilities of the large language model. For the tool invocation strategy, the schema is encapsulated within the tool definition, leveraging the inherent structured capabilities of the large language model to directly inject the complete schema object into the parameters field. The resulting payload can be represented as follows: request_payload = { "tools": [tool_definition], "tool_choice": {"type": "function", "function": {"name": "generate_structured_data"}}.

[0055] For the text generation strategy, the schema is compressed into a string and embedded in the System Prompt, and a preset template is used to explicitly instruct the model to output a JSON-type string. The resulting payload can be represented as: request_payload = { "messages": [ {"role": "system", "content": formatted_system_prompt} ] }

[0056] Finally, you get the constructed HTTP payload and a handle to the ready Rust module-based validator.

[0057] In an optional implementation, after obtaining the initialized validator, the state transitions of a finite state machine can be used to automatically perform syntax and semantic validation on the token fragments. To promptly detect validation failures, this embodiment provides a byte stream-based validation implementation: First, input the token fragments into the finite state machine as a byte stream; Secondly, if the currently input byte triggers the finite state machine to jump to the preset illegal state, the verification is deemed to have failed; if none of the bytes in the Token fragment trigger the finite state machine to jump to the preset illegal state, the verification is deemed to have passed.

[0058] In this embodiment, the finite state machine reads and advances state transitions byte by byte based on its current parsing state, such as the object start state, the expected field name state, or the internal state of a string value. During this process, the finite state machine not only performs basic syntax checks but also simultaneously performs semantic constraint verification required by the preset specification. For example, when the preset specification limits a field's value to only a few enumeration items, a pre-generated prefix tree structure can be used to compare the character sequence composed of the currently input bytes with the common prefix of all valid enumeration values ​​in real time. Once it is found that the current byte causes the sequence to deviate from all valid paths—for example, expecting "Success" or "Failed" but receiving "Fail"—the finite state machine is triggered to jump to a preset illegal state (such as the Reject state). If the currently input byte triggers the finite state machine to jump to the preset illegal state, the verification is deemed to have failed. If none of the bytes in the token fragment trigger the finite state machine to jump to the preset illegal state, the verification is deemed to have passed. Thus, the entire verification process uses bytes as the smallest processing unit, and the finite state machine ensures both high real-time performance and accurate and efficient constraint judgment of the large language model output.

[0059] In an optional implementation, to ensure timely termination of the large language model's continued output of the token stream when the validation fails, this embodiment also provides at least the following two implementation methods: Method 1: When the validation fails, a cancellation signal is sent to the large language model so that the large language model will stop outputting the token stream based on the cancellation signal.

[0060] Method 2: When the validation fails, disconnect the network connection with the large language model and stop the large language model from continuing to output the token stream.

[0061] In this embodiment, a failure to validate may be due to the validator detecting a syntax error or schema violation (such as generating an illegal JSON structure or an incorrect field type). In this case, the large language model needs to be terminated immediately from continuing to output the token stream.

[0062] In this embodiment, the cancellation request can be a standard control command initiated by the client to notify the remote inference service to immediately suspend the current inference task. The network connection refers to the underlying transport layer connection established between the client and the large language model inference service, carrying the token stream transmission, including persistent connections based on the TCP protocol or multiplexed connections based on HTTP / 2.

[0063] As an engineering implementation, the Python side maintains a streaming response processor, StreamHandler, which can subscribe in real time to the streaming responses output by the large language model inference interface, including server-sent event streams (SSE streams or gRPC streams) conforming to OpenAI compatible protocols. When the large language model generates a new token fragment, the Python side can synchronously call the underlying core validator, RustValidator::process_token, implemented in Rust, via the external function interface FFI. The finite state machine maintained internally by the validator in the Rust module can use the JSONSchema bytecode pre-generated by the Pydantic model. If the finite state machine is in a state transition process... The system determines that the currently input token fragment is syntactically invalid, for example, due to a missing key-value pair separator in a JSON object or a schema violation. Specifically, this could be due to a quotation mark character detected in a field defined as an integer type. The validator in the Rust module can immediately return a ValidationResult::Err structure containing the error type, error path, and expected character set, without waiting for the entire token stream to end or the JSON structure to close. Upon receiving this error signal, the Python side can immediately perform a hard interrupt operation, with two options: First, trigger the CancelToken of the asynchronous task, sending a cancellation signal (Cancel Request) to the large language model inference service, causing the large language model to terminate the token stream based on the cancellation signal; second, directly close the underlying TCP or HTTP connection handle, disconnecting the network connection with the large language model, and forcibly terminating the large language model's token stream output at the communication link level. Either of these methods can be completed within milliseconds, physically cutting off the generation path of invalid tokens and preventing wasted computing power.

[0064] It should be noted that the above-mentioned method one and method two can be parallel and mutually substitutable technical solutions.

[0065] In an optional implementation, in the event of a failed validation, in addition to promptly terminating the output of the token stream by the large language model, this embodiment also provides a corresponding processing method to improve the robustness of the large language model's inference: First, when the validation fails and the number of retries is less than the preset number, an error object is constructed based on the current validation location, error type, and expected type. In this embodiment, the preset number of retries can be configured according to actual needs, or a default value can be preset, for example, the preset number of retries can be set to three. As one implementation method, a counter retry_count can be set to count the number of retries, and the counter is incremented once for each additional retries.

[0066] In this embodiment, the error object includes, but is not limited to, the specific field path when the verification fails, the content of the token fragment determined to be incorrect, and the character set or data type that the position should conform to. For example, the error object may also include information such as the field hierarchy relationship corresponding to the current verification position.

[0067] Secondly, update the context of the large language model based on the error object; In this embodiment, as one implementation method, the error object is parsed and converted into a prompt word in natural language format. The prompt word can be encapsulated into a new Message object of User or System role and appended to the end of the current dialogue history list (History Context), thereby completing the update of the large language model context.

[0068] Finally, the validator is reinitialized based on the updated context and the pre-defined generation task is re-inferred using a large language model.

[0069] In this embodiment, based on the updated context, the validator initialization process is called again to generate a finite state machine adapted to the new context, and an inference request for the same preset generation task is sent to the large language model again. In the new round of inference, the large language model can use its own attention mechanism to focus on the specific fields and constraints pointed to by the aforementioned error message. When regenerating the token stream, it actively adjusts the output strategy to avoid repeating the same type of illegal syntax or schema violation. The entire verification and generation process is based on a closed-loop mechanism, with each iteration using the previous verification conclusion as input, thereby ensuring that the output result gradually approaches the correct structure defined by the preset specification.

[0070] It should be noted that if the verification still fails after several retries, the retry process will be terminated and a final preset exception will be thrown to the client.

[0071] In an optional implementation, if the monitoring of the token stream is done by Python, and if the verification is also handled by Python, then Python's processing efficiency cannot adequately meet the real-time requirements. To fully utilize the advantages of other languages, such as Rust, which have higher processing efficiency than Python, while avoiding the space and time overhead caused by introducing intermediate data construction and cross-language memory copying that could affect verification efficiency, this embodiment provides a processing method: First, the memory address and length of the token fragment in the target buffer are obtained using a preset buffer protocol; In this embodiment, the token stream output by the large language model is temporarily stored in the target buffer. The target buffer refers to a contiguous memory region temporarily allocated by the Python runtime environment for the output data of the large language model during token generation. The default buffer protocol refers to the widely supported standard buffer protocol defined in the Python language, which enables high-level language objects to safely expose their contiguous memory blocks to the underlying system or external compiled languages.

[0072] Secondly, the token fragment is retrieved from the target buffer based on the memory address and fragment length.

[0073] In this embodiment, the token fragment is obtained from the target buffer according to the memory address and fragment length. There is no need to construct a new string copy or intermediate data structure, which realizes zero-copy of data and avoids the time and space overhead caused by additional space allocation and data copying.

[0074] In an optional implementation, when the verification passes, in order to ensure that subsequent token fragments in the token stream are verified one by one in real time, the processing method provided in this embodiment is as follows: When the verification passes, wait to obtain the token fragment of the next moment and verify it until the token stream is completely output.

[0075] In this embodiment, if Python receives a preset normal state returned by the validator, such as a valid state or an incomplete state (e.g., Valid or Incomplete), it will suspend and wait for the generation of the token fragment at the next moment until the token stream is completely output. The token stream can be completed by the validator returning a preset completion state (e.g., Complete) to Python. Since the validator returns a status code instead of a complete data object, the serialization overhead is greatly reduced.

[0076] It should also be noted that if a network error occurs during the token fragment verification process instead of a verification error, a retry is not necessary. Instead, an error message indicating a network error is returned directly to the client.

[0077] In an optional implementation, after the Token stream is output, in order to ensure that the data in the output Token stream conforms to the preset specifications in terms of syntax and semantics, and further enable the data to be naturally integrated into the actual business execution environment, thereby directly supporting downstream program calls and avoiding the risks of type errors and missing context caused by manual intervention or secondary parsing, this embodiment also provides an implementation method: First, the token stream is converted into an object instance of a preset structure type; In this embodiment, as an implementation method, Rust utilizes the zero-copy mechanism of the FFI interface to efficiently transmit the accumulated and cleaned JSON data stream in the target buffer to Python. Based on the class definition generated during the Schema injection phase, Python calls the `model_validate` method of Pydantic Models or an equivalent deserialization method to convert the JSON data stream into a concrete Python object instance. That is, it constructs the Schema-compliant JSON data into a Python object with clearly defined field definitions, type constraints, and validation logic. It should be noted that during the conversion, a field-level deep constraint check is also performed to ensure that all enumeration values, numerical ranges, and regular expression constraints are satisfied. If the deep constraint check fails, an exception message can be returned to the client, or a retry can be attempted, updating the large language model context, re-initializing the validator based on the updated context, and re-inferring the preset generation task using the large language model.

[0078] Secondly, obtain the active resource instance associated with the context of the object instance; In this embodiment, active resource instances include, but are not limited to, active ORM database session handles, authenticated API keys, or distributed trace IDs, and other resource instances that are context-dependent on the object instance.

[0079] Third, generate a target object carrying context information based on the object instance and the active resource instance.

[0080] In this embodiment, active resource instances are dynamically injected into reserved slots of model instances, transforming DTOs (Data Transfer Objects) that were originally just data carriers into rich model objects with context-aware capabilities. For example, a user object injected with a database connection can directly call its own ".save()" method to achieve persistence without requiring the business layer code to explicitly pass connection parameters. If a critical dependency is detected as missing or type mismatched during the injection process, a preset error (such as DependencyResolutionError) can be immediately thrown and output blocked to prevent invalid objects from entering downstream.

[0081] In this embodiment, the target object is a strongly typed Python object that has undergone rigorous validation and completed full dependency assembly. Downstream code can directly access attributes or call methods via dot notation, eliminating the need for manual type assertions or context binding. This achieves an end-to-end type safety loop from prompt definition to business code execution. For example, if the object instance is {"username": "alice", "role": "admin"}, following the above steps, db_session can be directly injected into the target object, and downstream business logic can directly call the db_session's ".save()" method.

[0082] In an optional implementation, in order to clearly identify the service resources that need to be accessed in the real runtime environment from multiple fields, and to accurately obtain and integrate these resources, this embodiment provides an implementation method for obtaining active resource instances related to the context of an object instance: First, identify each target field in the object instance that has a pre-defined external dependency; In this embodiment, the generated Pydantic object instance is first scanned based on predefined type hints and decorator metadata to identify special fields marked as "runtime dependent" or "exclude serialization". The target field is the special field marked as "runtime dependent" or "exclude serialization".

[0083] Secondly, retrieve the active resource instances of the preset external dependencies required for each target field; In this embodiment, for each target field, the context of the preset generation task and the global service registry are obtained. The context of the preset generation task includes, but is not limited to, the distributed trace ID (Trace ID) or the user ID (UserID). The global service registry includes, but is not limited to, database connection pools, API clients, etc. The search is first performed in the context of the preset generation task (e.g., the Trace ID or User Session of the current request). If it is not found in the context, the search is performed in the global service registry (e.g., the database connection handle ORM Session). If a resource is marked as essential but no corresponding active resource instance is found, the process is terminated immediately, and an exception is returned to prevent the exception from spreading to downstream services.

[0084] Finally, all active resource instances required by the target fields are set as active resource instances context-dependent with respect to the object instance.

[0085] To perform the corresponding steps in the above embodiments and various possible implementations, an implementation method of the large language model output verification device 100 is given below. Please refer to... Figure 4 , Figure 4 Example of a block diagram for the large language model output verification device provided in this embodiment. Figure 1 It should be noted that the basic principle and technical effect of the large language model output verification device 100 provided by the present invention are the same as those of the corresponding embodiments described above. For the sake of brevity, this embodiment does not mention or point out these aspects.

[0086] The large language model output verification device 100 includes an acquisition module 110, a verification module 120, and a termination module 130.

[0087] The acquisition module 110 is used to acquire the token fragments output at the current moment during the process of the large language model performing inference for a preset generation task and outputting a token stream.

[0088] The verification module 120 is used to verify the token fragments using a pre-initialized verifier, which is constructed based on a finite state machine according to the preset specifications for constraining the syntax and semantics of the token stream output by the large language model.

[0089] Termination module 130 is used to terminate the large language model from continuing to output the token stream when the verification fails.

[0090] In an optional implementation, the verification module 120 is further configured to: Retrieve a predefined structured data class, which includes at least one field; The metadata of each field is parsed to obtain the semantic constraints of each field; Convert all semantic constraints into preset specifications according to preset grammatical constraints; Generate a finite state machine that conforms to the preset specifications and bind the finite state machine and the checker to initialize the checker.

[0091] In an optional implementation, the verification module 120 is further configured to: Obtain the configuration strategy for the large language model; If the configuration strategy is tool invocation configuration, then a preset tool definition object is constructed and the preset specification is embedded into the parameters of the preset tool definition object, so as to inject the preset specification into the input context of the large language model through the preset tool definition object; If the configuration strategy is text generation configuration, the preset specification will be formatted into a string according to the preset format and the string will be injected into the prompt words so that the preset specification can be injected into the input context of the large language model through the prompt words.

[0092] In an optional implementation, the verifier is pre-bound to a finite state machine, and the verification module 120, when specifically used to verify the token fragment using the pre-initialized verifier, is further used to: Input the token fragments into the finite state machine as a byte stream; If the currently input byte triggers the finite state machine to jump to a preset illegal state, the verification is deemed to have failed. If none of the bytes in the token fragment trigger the finite state machine to jump to the preset illegal state, the verification is considered successful.

[0093] In an optional implementation, the termination module 130 is specifically used for: When the validation fails, a cancellation signal is sent to the large language model so that the large language model will stop outputting the token stream based on the cancellation signal; or, If the validation fails, disconnect the network connection with the large language model and terminate the large language model from continuing to output the token stream.

[0094] In an optional implementation, the large language model output verification device 100 further includes a retry module 140, which is used for: When the verification fails and the number of retries is less than the preset number, an error object is constructed based on the current verification location, error type, and expected type. Update the context of the large language model based on the error object; The validator is reinitialized based on the updated context, and the pre-defined generation task is re-inferred using a large language model.

[0095] In an optional implementation, the acquisition module 110 is specifically used for: Use a preset buffer protocol to obtain the memory address and length of the token fragment in the target buffer; Retrieve the token fragment from the target buffer based on the memory address and fragment length.

[0096] In an optional implementation, the verification module 120 is further configured to: When the verification passes, wait to obtain the token fragment of the next moment and verify it until the token stream is completely output.

[0097] In an optional implementation, the large language model output verification device 100 further includes a generation module 150, which is used for: Convert the token stream into an object instance of a preset structure type; Identify each target field in an object instance that has a pre-defined external dependency; Retrieve the active resource instances of the preset external dependencies required for each target field; Treat all active resource instances required by the target fields as active resource instances context-dependent to the object instance; Generate a target object carrying context information based on the object instance and the active resource instance.

[0098] It should be noted that, during implementation, the large language model output verification method provided in this embodiment can be adapted to different functions involved in the corresponding large language model output verification device according to processing needs. Figure 4 This embodiment also provides an example of another way to divide functional modules, please refer to... Figure 5 , Figure 5 Example of a block diagram for the large language model output verification device provided in this embodiment. Figure 2 . Figure 5 In this framework, the large language model output validation device can include a schema injection module, a Rust FFI-based streaming validation core module, an error feedback and autocorrect module, and a dependency injection container. The schema injection module includes a schema generation submodule and a prompt word construction submodule. The Rust FFI streaming validation core module includes a validator and a finite state machine maintained by the validator. Furthermore, an asynchronous stream processor is implemented in Python.

[0099] The Schema Injection module automatically converts user-defined Python Pydantic models (i.e., the structured data classes mentioned earlier) into standard JSON Schema descriptions. This schema is formatted as system prompts or tool definitions and injected into the input context of the large language model, thereby constraining the model to generate JSON data that conforms to specific structure and type definitions.

[0100] The schema generation submodule receives user-defined structured data classes, typically inheriting from Python's `pydantic.BaseModel` base class. The schema injection module first calls Pydantic's built-in `model_json_schema` method to recursively serialize Python class attributes and their type annotations into JSONSchema dictionary objects conforming to the Draft 2020-12 standard. During this process, the module parses the field metadata, converting constraints defined in `constr` or `Field` into validation keywords in the JSON Schema. For example, it maps regular expressions to the `pattern` field, numerical ranges to `minimum` and `maximum` fields, and handles nested model references ($defs) to ensure the generated schema is a self-contained closure.

[0101] The prompt word construction submodule, to adapt to different LLM inference interfaces, formats the schema according to the configuration strategy. For models that support Function Calling, the module constructs a Tool Definition object containing name, description, and parameters fields, and fully embeds the generated JSON schema into the parameters. For general models that only support text generation, the module serializes the schema into a compact JSON string and injects it into the System Prompt's preset template, explicitly instructing the model to strictly follow this schema when outputting JSON data.

[0102] In Python, the Async Stream Processor directly uses the Python Buffer Protocol to obtain the underlying memory address pointer and byte length of the current Token object when the large language model starts generating the Token stream. It then uses the External Function Interface (FFI) to pass the received Token fragments to the underlying core module verifier written in Rust in real time with zero copy.

[0103] The validator and finite state machine: The schema injection module serializes the standardized JSON schema into a UTF-8 encoded byte stream, which is then passed to the validator in the underlying Rust core module via Rust FFI across language boundaries. Upon receiving the schema byte stream, the validator immediately builds and pre-compiles the finite state machine's transformation table based on the schema, initializing the streaming validator instance. If the schema contains complex types that cannot be converted to deterministic state machine rules, a pre-defined exception `SchemaCompilationError` will be thrown at this stage, terminating the process. Finally, after initialization, the validator receives an HTTP request payload containing the injected schema information and a ready Rust validator handle, preparing for subsequent streaming inference and incremental validation.

[0104] After receiving the token fragment from the asynchronous stream processor, the validator uses a finite state machine maintained for the target JSON syntax to update the parsing state within the millisecond intervals between token fragment generation. It checks in real time whether the current character sequence conforms to the JSON syntax specification and whether it meets the constraints such as field types and enumeration value ranges defined in the schema.

[0105] As an implementation method, the process of initializing the verifier and using the verifier for verification mainly includes four core stages: initial binding, zero-copy data transmission, state machine incremental stepping, and immediate exception blocking.

[0106] First, during the system initialization phase, Python's dependency injection container loads the schema definition of the target data structure, compiles it into a compact binary validation graph, and passes it to the Rust-written Core module through external function interfaces such as PyO3 or CFFI. Based on this, the module initializes a resident Validator instance, which internally maintains the current parsing cursor, stack depth, and expected token type.

[0107] Next, when LLM inference begins, the asynchronous stream processor on the Python side listens for each token string (i.e., token fragment) returned by the generator. To achieve zero-copy transfer, the asynchronous stream processor does not perform conventional string concatenation or encoding conversion. Instead, it directly uses the Python Buffer Protocol to obtain the underlying memory address pointer and byte length of the current token object. This memory view is immediately passed to the `validate_chunk` interface exposed by the Rust core module, ensuring that no heap memory allocation or data copying occurs during cross-language calls, keeping the overhead of FFI calls to the microsecond level.

[0108] After entering the Rust core module, the finite state machine (FSM) maintained by the validator takes over control. The FSM reads the incoming byte slices based on the current parsing state (e.g., InObject, ExpectKey, InString). The validator uses a byte-stream driven approach, advancing the state machine's transitions byte by byte. While checking JSON syntax compliance, the validator performs schema constraint checks in parallel. For example, if the schema defines a field as an enumeration type, the internally maintained trie compares the accumulated character sequence in real time; once it detects that the character sequence caused by the current token deviates from the common prefix of all valid enumeration values, the FSM immediately enters the Reject state.

[0109] Finally, there's the handling of immediate exception blocking. The validator returns a lightweight status enumeration value to the asynchronous stream processor. If the returned status is Valid or Incomplete, the asynchronous stream processor will continue to suspend and wait for the next token; if it returns Invalid, the validator will simultaneously construct a structured error object containing the specific error path, the expected character set, and the fragment of the invalid token. Upon capturing this error, the asynchronous stream processor immediately sends a CancelRequest signal to the LLM inference service to interrupt the subsequent token generation stream, preventing wasted computing power, and passes the structured error object to the error feedback module.

[0110] The error feedback and automatic correction module is used to retry according to a preset number of times and build a correction prompt, thereby completing closed-loop control from flow monitoring to immediate blocking.

[0111] The Dependency Injection Container is used to perform the final data delivery and object assembly process. That is, it dynamically injects the required external dependencies (such as database connections, API keys, etc.) according to the runtime context, and finally outputs Python objects that have undergone strict type validation and structured parsing, which can be directly used by downstream business logic, ensuring type safety throughout the entire chain.

[0112] The data delivery and object assembly process includes the following steps: First, to obtain the data to be assembled, the underlying Rust uses the zero-copy mechanism of the FFI interface to efficiently transfer the cleaned JSON structured data accumulated in the memory buffer to Python. Based on the class definition generated during the Schema injection phase, Python calls the `model_validate` method of Pydantic Models or an equivalent deserialization method to convert the raw JSON data stream into concrete Python object instances. During this process, not only is the data format mapping completed, but a final field-level deep constraint check is also performed to ensure that all enumeration values, numerical ranges, and regular expression constraints are satisfied.

[0113] Building upon this foundation, the dependency injection container addresses the disconnect between the static data generated by the LLM and the system's runtime environment. Based on predefined type hints and decorator metadata, the container scans the generated Pydantic object instances, identifying special fields marked as "runtime dependencies" or "excluded from serialization." It then retrieves corresponding active resource instances from the current Async Task Context or the global service registry and dynamically injects these entities into reserved slots in the model instance. This mechanism transforms the DTO (Data Transfer Object), which was originally just a data carrier, into a rich model object with context awareness. If a critical dependency is missing or a type mismatch is detected during injection, the container immediately throws a predefined exception, `DependencyResolutionError`, and blocks output to prevent invalid objects from entering downstream. Finally, the dependency injection container returns a strongly typed Python object to the business logic layer of the calling end, which has undergone rigorous validation and full dependency assembly. Downstream code can directly access attributes or call methods using dot notation without manual type assertions or context binding, thus achieving an end-to-end type-safe closed loop from Prompt definition to business code execution.

[0114] This invention also provides a block diagram of an electronic device 10, which implements the large language model output verification method described in the foregoing embodiments. Please refer to... Figure 6 , Figure 6 This is a block diagram of an electronic device 10 provided in this embodiment. The electronic device 10 includes a processor 11, a memory 12, and a bus 13. The processor 11 and the memory 12 are connected through the bus 13.

[0115] Processor 11 can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the large language model output verification method described above can be completed by the integrated logic circuits in the hardware of processor 11 or by software instructions. Processor 11 can be a general-purpose processor, including CPU (Central Processing Unit), NP (Network Processor), GPU (Graphics Processing Unit), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Logic Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0116] The memory 12 is used to store the program that implements the large language model output verification method. The program can be a software function module stored in the memory 12 in the form of software or firmware or embedded in the OS (Operating System) of the electronic device 10.

[0117] After receiving the execution instruction, the processor 11 executes the program to implement the large language model output verification method of the aforementioned embodiment.

[0118] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the large language model output verification method as described in this embodiment.

[0119] This embodiment provides a computer program product that, when executed by a processor, implements the large language model output verification method as described in this embodiment.

[0120] In summary, embodiments of the present invention provide a method, apparatus, electronic device, and computer-readable storage medium for validating the output of a large language model. The method includes: acquiring a token fragment currently being output during the process of a large language model performing inference for a preset generation task and outputting a token stream; validating the token fragment using a pre-initialized validator, which is constructed based on a finite state machine according to preset specifications for constraining the syntax and semantics of the token stream output by the large language model; and terminating the large language model from continuing to output the token stream when the validation fails. Compared with the prior art, this embodiment has at least the following advantages: (1) It realizes real-time verification of the token stream output by the large language model and terminates the continued output of the token stream in time when the verification fails, avoiding the serious waste of inference computing power and token quota caused by the "post-generation verification" implementation method; (2) It uses the Python buffer protocol to obtain the underlying memory address pointer and byte length of the current token fragment, realizes the transfer of the token fragment to the verifier in a zero-copy manner, ensures that no heap memory allocation or data copying occurs during cross-language calls, and controls the additional overhead to the microsecond level; (3) When the verification fails, it also constructs a structured error object containing the specific error path, expected character set and illegal token fragment, and constructs a correction Prompt based on the structured error object to complete the transfer from the stream. The closed-loop control that detects and blocks immediately can force the model to use its own attention mechanism to pay attention to the error information and improve the probability of successful retry; (4) The high performance characteristics of the Rust language are used to embed a finite state machine in Python, which further improves the efficiency of cross-language calls; (5) Different configuration strategies are used to format the schema for different LLM inference interfaces; (6) The number of retries is controlled to avoid an infinite loop of retries when an unrepairable error occurs; (7) When all Token streams are verified, the association between the static data in the Token and its runtime environment is automatically injected, so that the final object can be used directly by the downstream code without manual type assertion or context binding, thus realizing an end-to-end type safety closed loop from Prompt definition to business code execution.

[0121] The above descriptions are merely various embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for validating the output of a large language model, characterized in that, The method includes: During the process of the large language model inferring and outputting a token stream for a preset generation task, the token fragment output at the current moment is obtained; The token fragments are validated using a pre-initialized validator, which is constructed based on a finite state machine according to a preset specification for constraining the syntax and semantics of the token stream output by the large language model. If the validation fails, the large language model will stop outputting the Token stream.

2. The method according to claim 1, characterized in that, Before the step of verifying the token fragment using a pre-initialized verifier, the method further includes: Obtain a predefined structured data class, wherein the structured data class includes at least one field; The metadata of each field is parsed to obtain the semantic constraints of each field; All the aforementioned semantic constraints are converted into preset specifications according to preset grammatical constraints; Generate a finite state machine that conforms to the preset specification and bind the finite state machine and the verifier to initialize the verifier.

3. The method according to claim 2, characterized in that, After the step of converting all the semantic constraints into a preset specification according to the preset syntax constraints, the method further includes: Obtain the configuration strategy of the large language model; If the configuration strategy is a tool invocation configuration, then a preset tool definition object is constructed and the preset specification is embedded into the parameters of the preset tool definition object, so as to inject the preset specification into the input context of the large language model through the preset tool definition object; If the configuration strategy is a text generation configuration, the preset specification is formatted into a string according to a preset format and the string is injected into a prompt word, so as to inject the preset specification into the input context of the large language model through the prompt word.

4. The method according to claim 1, characterized in that, The verifier is pre-bound to a finite state machine, and the step of verifying the token fragment using the pre-initialized verifier includes: The token fragment is input into the finite state machine as a byte stream. If the currently input byte triggers the finite state machine to jump to a preset illegal state, the verification is determined to have failed. If none of the bytes in the Token fragment trigger the finite state machine to jump to the preset illegal state, then the verification is deemed successful.

5. The method according to claim 1, characterized in that, The step of terminating the output of the Token stream by the large language model when the verification fails includes: When the verification fails, a cancellation signal is sent to the large language model so that the large language model terminates the output of the token stream based on the cancellation signal; or, If the verification fails, the network connection with the large language model is disconnected, and the large language model is terminated from continuing to output the Token stream.

6. The method according to claim 1 or 5, characterized in that, The method further includes: When the verification fails and the number of retries is less than the preset number, an error object is constructed based on the current verification location, error type, and expected type. Update the context of the large language model based on the error object; The verifier is reinitialized based on the updated context, and the pre-defined generation task is re-inferred using the large language model.

7. The method according to claim 1, characterized in that, The token stream is temporarily stored in the target buffer, and the step of obtaining the token fragment output at the current moment includes: The memory address and fragment length of the Token fragment in the target buffer are obtained using a preset buffer protocol; The token fragment is obtained from the target buffer based on the memory address and fragment length.

8. The method according to claim 1, characterized in that, The method further includes: When the verification passes, wait to obtain the Token fragment of the next moment and verify it until the Token stream is completely output.

9. The method according to claim 8, characterized in that, After the token stream is output, it also includes: Convert the Token stream into an object instance of a preset structure type; Identify each target field in the object instance that has a preset external dependency; Retrieve the active resource instances of the preset external dependencies required for each of the target fields; All active resource instances required by the target fields are treated as active resource instances context-dependent with respect to the object instance. Based on the object instance and the activity resource instance, a target object carrying context information is generated.

10. A computer program product, characterized in that, When the computer program product is executed by the processor, it implements the large language model output verification method according to any one of claims 1-9.

Images