Text reasoning system and method, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a text reasoning system and method, electronic equipment, a storage medium and a computer program product, and the text reasoning system comprises an AI gateway layer which is provided with a token statistical plug-in; the AI gateway layer is used for receiving the reasoning request and calling back the token statistical plug-in in each reasoning stage responding to the reasoning request; and the token statistics plug-in is used for performing token statistics in each reasoning stage in response to the reasoning request to obtain a token statistics result of the reasoning request. According to the embodiment of the invention, the decoupling between the token statistics and the inference engine at the lower layer can be effectively realized, a multi-inference engine scene can be adapted, and a plurality of heterogeneous inference engines can share the unified token statistics logic.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a text reasoning system and method, electronic device, storage medium and computer program product. Background Technology

[0002] In text inference services, such as Large Language Model (LLM) inference services, it is typically necessary to perform token statistics for each inference request to support subsequent billing, monitoring, and optimization. In existing text inference systems, token statistics logic is usually implemented separately within each inference engine. The token statistics process is highly coupled with the inference engine, making it difficult to uniformly adapt to multiple heterogeneous inference engines. Summary of the Invention

[0003] In view of this, this disclosure presents a text reasoning system and method, electronic device, storage medium and computer program product.

[0004] According to one aspect of this disclosure, a text inference system is provided, including an AI gateway layer, wherein the AI ​​gateway layer is deployed with a token statistics plugin; the AI ​​gateway layer is configured to receive inference requests and call back the token statistics plugin at each inference stage in response to the inference request; the token statistics plugin is configured to perform token statistics at each inference stage in response to the inference request to obtain the token statistics result of the inference request.

[0005] In one possible implementation, the token statistics of the inference request include at least one of the following: request identifier, user identifier, model identifier, number of input tokens, number of output tokens, total number of tokens, first token delay, total request delay, and response type, wherein the response type is a streaming response or a non-streaming response.

[0006] In one possible implementation, the system further includes: a client; the client being configured to send the inference request to the AI ​​gateway layer; the AI ​​gateway layer being configured to call back the token statistics plugin during the request phase of the inference request; and the token statistics plugin being configured to parse the inference request, determine and store the request identifier, the user identifier, the model identifier, and the request start time of the inference request.

[0007] In one possible implementation, the system further includes: an inference engine; the AI ​​gateway layer, configured to send the inference request to the inference engine and, during the response phase of the inference request, call back the token statistics plugin; the token statistics plugin, configured to determine and mark the response type.

[0008] In one possible implementation, the AI ​​gateway layer is configured to, when the response type is a streaming response, call back the token statistics plugin after receiving the first response data block returned by the inference engine; the token statistics plugin is configured to parse the first response data block returned by the inference engine, determine the first token time, and determine the first token delay based on the first token time and the request start time.

[0009] In one possible implementation, the AI ​​gateway layer is configured to, when the response type is a streaming response, call back the token statistics plugin after receiving each response data block returned by the inference engine; the token statistics plugin is configured to parse and accumulate the number of tokens included in each response data block returned by the inference engine to determine the number of output tokens.

[0010] In one possible implementation, the token statistics plugin is used to: parse the metadata information included in the last response data block returned by the inference engine, determine the number of output tokens counted by the inference engine; and, based on the number of output tokens counted by the inference engine, correct the number of output tokens obtained by summing all response data blocks returned by the inference engine to obtain the corrected number of output tokens.

[0011] In one possible implementation, the token statistics plugin is used to: parse the prompt text included in the inference request to determine the number of input tokens; or, parse the metadata information included in the last response data block returned by the inference engine to determine the number of input tokens counted by the inference engine.

[0012] In one possible implementation, the token statistics plugin is used to: sum the input token count and the output token count to determine the total token count.

[0013] In one possible implementation, the system further includes: a metering system; the token statistics plugin is used to generate a complete inference event corresponding to the inference request by taking the token statistics results of the inference request according to a preset event format after the inference request ends; the token statistics plugin is used to send the complete inference event to the metering system.

[0014] In one possible implementation, the system further includes: a metering system; the token statistics plugin is used to generate a partial inference event corresponding to the inference request by taking the token statistics result of the inference request at the preset reporting time according to a preset event format at any preset reporting time before the inference request ends; the token statistics plugin is used to send the partial inference event to the metering system.

[0015] In one possible implementation, the preset event format is the Cloud Events format.

[0016] In one possible implementation, the token statistics plugin is a WASM plugin.

[0017] According to another aspect of this disclosure, a text reasoning method is provided, the method being applied to a text reasoning system, wherein the system includes: an AI gateway layer, the AI ​​gateway layer having a token statistics plugin deployed thereon; the AI ​​gateway layer receiving reasoning requests and calling back the token statistics plugin at each reasoning stage in response to the reasoning requests; performing token statistics at each reasoning stage in response to the reasoning requests, and using the token statistics plugin to obtain the token statistics result of the reasoning requests.

[0018] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0019] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0020] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0021] The text inference system of this disclosure deploys a token statistics plugin at the AI ​​gateway layer. This allows the AI ​​gateway layer to call back the token statistics plugin at each inference stage in response to a received inference request. The plugin then performs independent token statistics at each inference stage, yielding the token statistics result for the inference request. By deploying the token statistics plugin at the AI ​​gateway layer, the decoupling between token statistics and the underlying inference engine is effectively achieved. This enables adaptation to multi-inference engine scenarios, allowing multiple heterogeneous inference engines to share a unified token statistics logic.

[0022] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0023] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0024] Figure 1 A block diagram of a text reasoning system according to an embodiment of the present disclosure is shown.

[0025] Figure 2 This illustrates an embodiment according to the present disclosure. Figure 1 The flowchart shown illustrates the token counting process in the text reasoning system.

[0026] Figure 3 A flowchart illustrating a text reasoning method according to an embodiment of the present disclosure is shown.

[0027] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0029] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0030] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0031] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0032] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0033] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0034] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0035] In text inference services, such as LLM inference services, it is usually necessary to perform token statistics for each inference request. For example, it is necessary to perform statistics on key indicators such as the number of input tokens, the number of output tokens, the total number of tokens, and the first token latency (TTFT) to support subsequent billing, monitoring and optimization work.

[0036] In existing text reasoning systems, token statistics logic is typically implemented separately within each inference engine. This high coupling between the token statistics process and the inference engine leads to the following problems: 1. High coupling: Each inference engine needs to implement its own token statistics logic, resulting in redundant development and high maintenance costs. 2. Difficulty in unified management: In multi-inference engine scenarios, unified token statistics and measurement cannot be achieved. 3. Limited scalability: Adding new statistical metrics requires modifying the inference engine code, posing a risk.

[0037] Furthermore, most current inference engines can only count the number of input tokens, output tokens, and total tokens, lacking the ability to uniformly collect and report data such as first token latency (TTFT) and streaming inference performance metrics. The statistical information reporting methods of inference engines often rely on custom APIs or log collection, which is detrimental to system decoupling and scalability.

[0038] To address the aforementioned technical problems, this disclosure provides a text reasoning system that decouples token statistics from the reasoning engine, effectively adapts to multi-inference engine scenarios, and enables multiple reasoning engines to share the same token statistics logic. The text reasoning system of this disclosure is described in detail below.

[0039] Figure 1 A block diagram of a text reasoning system according to an embodiment of the present disclosure is shown. Figure 1 As shown, the system includes: an AI gateway layer, wherein the AI ​​gateway layer is equipped with a token statistics plugin; the AI ​​gateway layer is used to receive inference requests and to call back the token statistics plugin at each inference stage in response to the inference request; the token statistics plugin is used to perform token statistics at each inference stage in response to the inference request to obtain the token statistics result of the inference request.

[0040] The text inference system of this disclosure deploys a token statistics plugin at the AI ​​gateway layer. This allows the AI ​​gateway layer to call back the token statistics plugin at each inference stage in response to a received inference request. The plugin then performs independent token statistics at each inference stage, yielding the token statistics result for the inference request. By deploying the token statistics plugin at the AI ​​gateway layer, the decoupling between token statistics and the underlying inference engine is effectively achieved. This enables adaptation to multi-inference engine scenarios, allowing multiple heterogeneous inference engines to share a unified token statistics logic.

[0041] Figure 1 The components of the text reasoning system shown can be deployed on the same or different physical devices according to the actual implementation environment and needs, and this disclosure does not make any specific limitations in this regard.

[0042] In one example, the text reasoning system can be an LLM reasoning system, or it can be applied to other text reasoning systems that require token statistics, depending on actual needs. This disclosure does not make any specific limitations on this.

[0043] In one example, the reasoning phase of a reasoning request includes a request phase and a response phase. The request phase includes a request header phase and a request body phase; the response phase includes a response header phase and a response body phase. The specific form of each phase can be found in related technologies, and this disclosure does not impose specific limitations on it.

[0044] The callback token statistics plugin in this disclosure refers to the call to the token statistics plugin that is automatically triggered in response to events or time nodes in the inference phase. This will be described in more detail in the following embodiments.

[0045] Because the token statistics plugin is deployed at the AI ​​gateway layer, it does not intrude on the inference engine. The inference engine only needs to focus on inference computation and does not need to worry about the token statistics logic or modify the inference engine source code. In addition, using a separate token statistics plugin for token statistics offers strong scalability and facilitates the addition of new statistical dimensions and reporting methods. The token statistics plugin can be built and updated independently, offering flexible deployment.

[0046] In one possible implementation, the token statistics plugin is a WASM plugin.

[0047] A WASM (WebAssembly) plugin is a module that uses WebAssembly technology to extend the functionality of the main program. Leveraging the sandbox mechanism of WASM plugins, a token statistics plugin in the form of a WASM plugin is implemented. Running in a sandbox environment, it ensures that the stable operation of the AI ​​gateway layer is not affected even if the token statistics plugin fails, effectively achieving security isolation.

[0048] In one example, an agent that supports WASM plugins (e.g., Envoy) is deployed at the AI ​​gateway layer. The agent supports token statistics by extending functionality through WASM plugins.

[0049] In addition to the WASM plugin, the token statistics plugin can also use other plugins that can achieve secure isolation. This disclosure does not make any specific restrictions on this.

[0050] In one possible implementation, the token statistics of the inference request include at least one of the following: request identifier, user identifier, model identifier, number of input tokens, number of output tokens, total number of tokens, first token delay, total request delay, and response type, wherein the response type is a streaming response or a non-streaming response.

[0051] To support subsequent billing, monitoring, and optimization efforts, the token statistics for inference requests can include: request ID (request_id), user ID (user_id), model ID (model), number of input tokens (input_tokens), number of output tokens (output_tokens), total number of tokens (total_tokens), first token latency (first_token_latency), total request latency (total_latency), and response type (streaming).

[0052] The request identifier (request_id) is a unique identifier used to indicate an inference request, and its data type is string. The request identifier (request_id) includes at least one of the following: a user-level business trace identifier (trace_ID) or a single inference request-level correlation identifier (correlation_ID). The business trace identifier (trace_ID) is used to identify the entire user session, and the correlation identifier (correlation_ID) is used to identify any inference request included in the complete user session. Based on the request identifier (request_id), the corresponding inference request can be traced.

[0053] The user identifier (user_id) is used to indicate the user who triggered the inference request; its data type is string. In a multi-tenant environment of a text inference system, the user identifier (user_id) is used to distinguish different users and provide a unique identifier for subsequent user billing.

[0054] The model identifier (model) indicates the name and version of the model that the inference request needs to invoke; its data type is string. The model identifier (model) can be parsed from one of the following: the request path of the inference request, the request headers of the inference request, or the request body of the inference request. Based on the model identifier (model), the name and version of the model that the inference request needs to invoke can be determined.

[0055] The number of input tokens (input_tokens) indicates the number of tokens that need to be input to the model for an inference request. The data type is integer.

[0056] The number of output tokens (output_tokens) indicates the number of tokens actually generated and output by the model after responding to the inference request. The data type is integer.

[0057] The total number of tokens (total_tokens) indicates the sum of the number of input tokens (input_tokens) and the number of output tokens (output_tokens), and is an integer.

[0058] The text reasoning system of this disclosure supports both streaming and non-streaming responses during the response phase of a reasoning request.

[0059] When the response type corresponding to the inference request is a streaming response (streaming=true), that is, the model running on the inference engine returns multiple response data blocks that constitute the inference result while performing inference.

[0060] The first token latency (first_token_latency) indicates the time elapsed from the start of the inference request to the generation of the first output token by the model, assuming the response type for the inference request is streaming. The first token latency (first_token_latency) measures response latency, i.e., the end-to-end waiting time from the issuance of the inference request to the output of the first perceptible token. In one example, the unit of first token latency (first_token_latency) can be milliseconds, but it can also be set to other time units as needed; this disclosure does not impose specific limitations on this.

[0061] The total latency (total_latency) indicates the time from the start of the inference request to the end of the complete response when the response type corresponding to the inference request is a streaming response; that is, the total time of the entire inference request. The data type is integer. The unit of total latency (total_latency) can be milliseconds, or it can be set to other time units according to actual needs; this disclosure does not specifically limit this.

[0062] When the response type corresponding to the inference request is a non-streaming response (streaming=false), that is, when the inference engine returns a complete inference result after completing the inference, the first token latency (first_token_latency) and the total request latency (total_latency) are the same.

[0063] In one example, in addition to the above-mentioned statistical indicators, the token statistics of the inference request may also include other statistical indicators according to the actual application requirements, such as streaming inference performance indicators (time between tokens (TBT), token generation rate (Tokens Per Second, TPS), etc.), which are not specifically limited in this disclosure.

[0064] In one possible implementation, the text inference system further includes: a client; the client for sending inference requests to the AI ​​gateway layer; the AI ​​gateway layer for calling back the token statistics plugin during the request phase of the inference request; and the token statistics plugin for parsing the inference request, determining and storing the request identifier, user identifier, model identifier, and the request start time of the inference request.

[0065] like Figure 1 As shown, the text inference system also includes a client. The client sends an inference request to the AI ​​gateway layer, whereby the inference request is used to initiate an inference session. For example, the client sends the inference request to the AI ​​gateway layer via HTTP or HTTPS protocols.

[0066] An inference request consists of request headers and a request body. The request headers contain the metadata of the inference request, while the request body contains the specific request content. The request phases for an inference request include: the request header phase (on Request Headers) and the request body phase (on Request Body). The AI ​​gateway layer calls back the token statistics plugin during the request header phase (on Request Headers), i.e., after receiving the complete request headers corresponding to the inference request.

[0067] Figure 2 This illustrates an embodiment according to the present disclosure. Figure 1 The flowchart shown illustrates the token counting process in the text reasoning system. Figure 2 As shown, the text reasoning system includes: a client, an AI gateway layer, a token statistics plugin deployed on the AI ​​gateway layer, an inference engine, and a measurement system.

[0068] like Figure 2 As shown, the client sends an inference request to the AI ​​gateway layer, and the token statistics plugin callback is triggered in the request header stage (onRequest Headers).

[0069] The token statistics plugin parses the inference request and determines the corresponding request identifier (request_id), user identifier (user_id), and model identifier (model).

[0070] The token statistics plugin allocates memory space for inference requests and creates a context object (ctx) for the inference request in the memory space, which is used to record the token statistics results of the inference request later.

[0071] In addition, the token statistics plugin initializes timers and counters in the context object corresponding to the inference request. The initialized timer records the start time (t_start) of the inference request. The initialized counter records the number of output tokens (output_tokens), at which point output_tokens = 0.

[0072] like Figure 2 As shown, the token statistics plugin stores the request identifier (request_id), user identifier (user_id), model identifier (model), and request start time (t_start) in the context object corresponding to the inference request.

[0073] In one possible implementation, a token counting plugin is used to: parse the prompt text included in the inference request and determine the number of input tokens.

[0074] During the request body phase of the inference request, that is, after the AI ​​gateway layer receives the complete request body corresponding to the inference request, the token statistics plugin uses a built-in or external tokenizer to parse the complete prompt text included in the request body, determine the number of input tokens corresponding to the inference request, and store the number of input tokens in the context object corresponding to the inference request.

[0075] In one possible implementation, the text reasoning system also includes: an inference engine; and an AI gateway layer for sending inference requests to the inference engine.

[0076] The AI ​​gateway layer sends inference requests to the corresponding inference engine according to a preset routing strategy.

[0077] In one example, a model may run on multiple inference engines, meaning one model corresponds to multiple inference engines. If the model identifier (model) corresponding to an inference request has multiple corresponding inference engines, the inference request can be sent to one inference engine based on a load balancing strategy.

[0078] like Figure 2 As shown, the AI ​​gateway layer sends inference requests to the inference engine.

[0079] The inference engine responds to inference requests, executes the inference process, and enters the response phase of the inference request. The token statistics plugin does not participate in the internal inference logic of the inference engine.

[0080] In one possible implementation, the AI ​​gateway layer is used to call back the token statistics plugin during the response phase of the inference request; the token statistics plugin is used to determine and mark the response type.

[0081] The text reasoning system of this disclosure supports both streaming and non-streaming responses.

[0082] The response phase for an inference request includes: the response header phase (on Response Headers) and the response body phase (on Response Body). The AI ​​gateway layer calls back the token statistics plugin during the response header phase (on Response Headers), which is after the AI ​​gateway layer receives the complete response headers corresponding to the inference request from the inference engine.

[0083] The token statistics plugin determines whether the response type of the inference request is a streaming response or a non-streaming response based on the complete response headers corresponding to the inference request, and marks the response type of the inference request in the context object corresponding to the inference request (streaming=true / false).

[0084] In one example, the response header includes a field indicating the response type: Content-Type. The Token statistics plugin uses the Content-Type field to determine the response type of the inference request. When the Content-Type field is text / event-stream, it indicates a streaming response; when the Content-Type field is application / json, it indicates a non-streaming response.

[0085] In one possible implementation, the AI ​​gateway layer is used to call back the token statistics plugin after receiving the first response data block returned by the inference engine when the response type is streaming response; the token statistics plugin is used to parse the first response data block returned by the inference engine, determine the first token time, and determine the first token delay based on the first token time and the request start time.

[0086] When the response type corresponding to the inference request is a streaming response, during the response body phase, the inference engine streams multiple response data blocks back to the AI ​​gateway layer via the Server-Sent Events (SSE) protocol.

[0087] After receiving the first response data chunk from the inference engine, the AI ​​gateway layer calls back the token statistics plugin. For example... Figure 2 As shown, during the response body phase, after the AI ​​gateway layer receives the first response data block returned by the inference engine, it calls back the token statistics plugin. The token statistics plugin parses the first token arrival event from the first response data block and uses the timer in the context object corresponding to the inference request to record the first token time (t_first token), and calculates the first token latency = first token time - request start time, i.e., first_token_latency = t_firsttoken - t_start. Then, it stores the first token latency (first_token_latency) in the context object corresponding to the inference request.

[0088] In one possible implementation, the AI ​​gateway layer, when the response type is streaming, calls back the token statistics plugin after receiving each response data chunk returned by the inference engine; the token statistics plugin is used to parse and accumulate the number of tokens included in each response data chunk returned by the inference engine to determine the number of output tokens.

[0089] The inference engine continuously generates and returns response data blocks to the AI ​​gateway layer until inference is complete or interrupted (due to the real-time output token count reaching the maximum output threshold max_tokens, inference timeout, user cancellation, etc.). The maximum output threshold max_tokens can be flexibly set according to actual needs, and this disclosure does not impose specific limitations on it.

[0090] After receiving each response data block, the AI ​​gateway layer calls back the token statistics plugin. The token statistics plugin parses each response data block and uses the counter in the context object corresponding to the inference request to count and accumulate the number of tokens included in each response data block in real time, which is then used as the real-time output token count (output_tokens) and stored in the context object corresponding to the inference request.

[0091] like Figure 2As shown, for any response data block, the token statistics plugin parses the response data block and uses the counter in the context object corresponding to the inference request to count the number of newly added tokens (delta_tokens) in the response data block in real time. Then, it accumulates to determine the number of output tokens in real time, output_tokens+=delta_tokens, and updates the number of output tokens (output_tokens) stored in the context object corresponding to the inference request.

[0092] In addition, the token statistics plugin can also use the timer in the context object corresponding to the inference request to record the time of the latest received response data chunk (last_chunk_time) to calculate the token generation rate (TPS) for throughput evaluation of the inference engine.

[0093] In one possible implementation, a token counting plugin is used to: sum the input token count and the output token count to determine the total token count.

[0094] The token statistics plugin can sum the real-time output token count (output_tokens) with the input token count (input_tokens) before the inference request ends, determine the real-time total token count (total_tokens), and store it in the context object corresponding to the inference request.

[0095] In one possible implementation, the text reasoning system further includes: a measurement system; a token statistics plugin, used to generate partial reasoning events corresponding to the reasoning request by taking the token statistics results of the reasoning request at any preset reporting time before the end of the reasoning request according to a preset event format; and a token statistics plugin, used to send partial reasoning events to the measurement system.

[0096] The preset event format is pre-configured and is the format for reporting token statistical results to the metering system in the form of events. The specific content can be flexibly set according to actual needs, and this disclosure does not impose specific limitations on it.

[0097] The text reasoning system disclosed herein supports rolling reporting. That is, at any preset reporting time before the end of the reasoning request, the token statistics plugin generates a partial reasoning event corresponding to the reasoning request according to a preset event format based on the token statistics results of the reasoning request at that preset reporting time, and reports the partial reasoning event to the metering system to support real-time billing or real-time visualization.

[0098] Since the inference request has not yet ended, some inference events include: request ID, user ID, model ID, number of input tokens, number of real-time output tokens, total number of real-time tokens, first token latency, total real-time request latency (the time elapsed between the preset reporting time and the request start time), response type (streaming), number of newly added tokens (the number of tokens added since the last report), and time interval (the time since the last report).

[0099] In one possible implementation, the default event format is Cloud Events.

[0100] In existing technologies, token statistics are mostly reported through custom APIs or log collection, which is not conducive to decoupling and expansion between systems.

[0101] The Cloud Events standard protocol has cross-system interoperability, enabling easy access to cloud-native event buses or metering systems and supporting standardized extensions. Therefore, the text inference system of this disclosure embodiment can report the token statistics results obtained by the token statistics plugin to the metering system in the form of events, based on the Cloud Events standard protocol format.

[0102] In addition to Cloud Events format, the preset event format can also be flexibly set to other pre-defined event formats according to actual needs. This disclosure does not impose any specific limitations on this.

[0103] After the inference engine completes the inference request, it returns the last response data block to the AI ​​gateway layer. The last response data block includes an end marker (SSE [DONE] EOS) to indicate the end of inference or abnormal termination.

[0104] The token statistics plugin parses the last response data block and uses the counter in the context object corresponding to the inference request to count the number of newly added tokens (delta_tokens) in the last response data block in real time. Then, it accumulates to determine the final number of output tokens: output_tokens += delta_tokens.

[0105] In one possible implementation, the token statistics plugin is used to: parse the metadata information included in the last response data block returned by the inference engine to determine the number of output tokens counted by the inference engine; and based on the number of output tokens counted by the inference engine, correct the number of output tokens obtained by accumulating all response data blocks returned by the inference engine to obtain the corrected number of output tokens.

[0106] Since the inference engine itself has the ability to count the number of output tokens (output_tokens), the token counting plugin parses the metadata information included in the last response data block returned by the inference engine to determine the number of output tokens counted by the inference engine.

[0107] Based on the number of output tokens counted by the inference engine, the number of output tokens obtained by the token statistics plugin by accumulating all response data blocks is corrected to obtain the corrected number of output tokens, and the number of output tokens (output_tokens) stored in the context object corresponding to the inference request is updated.

[0108] In one example, if the difference between the number of output tokens counted by the inference engine and the number of output tokens obtained by the token statistics plugin by summing all response data blocks is less than a preset threshold, the number of output tokens counted by the inference engine can be determined as the modified number of output tokens.

[0109] In one example, if the difference between the number of output tokens counted based on the inference engine and the number of output tokens obtained by the token statistics plugin by summing all response data blocks is greater than or equal to a preset threshold, both values ​​can be recorded in the number of output tokens and an anomaly can be marked.

[0110] The specific value of the preset threshold can be flexibly set according to actual needs, and this disclosure does not impose specific limitations on it.

[0111] In one possible implementation, a token statistics plugin is used to parse the metadata information included in the last response data block returned by the inference engine to determine the number of input tokens counted by the inference engine.

[0112] Since the inference engine itself has the ability to count the number of input tokens (input_tokens), if the token counting plugin does not count and determine the number of input tokens (input_tokens) in the request body stage of the inference request, the number of input tokens counted by the inference engine can be determined by parsing the metadata information included in the last response data block returned by the inference engine, and stored in the context object corresponding to the inference request.

[0113] The token statistics plugin can sum the number of output tokens (output_tokens) and the number of input tokens (input_tokens) after the inference request ends to determine the final total number of tokens (total_tokens) and store it in the context object corresponding to the inference request.

[0114] like Figure 2 As shown, the token statistics plugin further parses the last token arrival event from the last response data block and uses a timer in the context object corresponding to the inference request to record the last token time (t_last token). It then calculates the total request latency as: total_latency = t_last token - t_start. Finally, it stores the total request latency (total_latency) in the context object corresponding to the inference request.

[0115] In one possible implementation, the system further includes: a metering system; a token statistics plugin, used to generate a complete inference event corresponding to the inference request by using the token statistics results of the inference request in a preset event format after the inference request ends; and a token statistics plugin, used to send the complete inference event to the metering system.

[0116] At the end of the inference request, the token statistics plugin generates a complete inference event corresponding to the inference request according to the preset event format, and reports the complete inference event to the metering system.

[0117] like Figure 2 As shown, the token statistics plugin reports the complete inference event to the metering system. Additionally, the AI ​​gateway layer returns the complete inference result to the client.

[0118] The complete inference event includes: request ID, user ID, model ID, number of input tokens, number of output tokens, total number of tokens, first token latency, total latency, and response type (streaming).

[0119] The text inference system of this disclosure deploys a token statistics plugin at the AI ​​gateway layer. This allows the AI ​​gateway layer to call back the token statistics plugin at each inference stage in response to a received inference request. The plugin then performs independent token statistics at each inference stage, yielding the token statistics result for the inference request. By deploying the token statistics plugin at the AI ​​gateway layer, the decoupling between token statistics and the underlying inference engine is effectively achieved. This enables adaptation to multi-inference engine scenarios, allowing multiple heterogeneous inference engines to share a unified token statistics logic.

[0120] Figure 3 A flowchart illustrating a text reasoning method according to an embodiment of this disclosure is shown. The method is applied to... Figure 1 , Figure 2 The text reasoning system shown includes an AI gateway layer, which deploys a token statistics plugin. (Example:) Figure 3 As shown, the method may include:

[0121] In step S31, the AI ​​gateway layer is used to receive inference requests and to call back the token statistics plugin at each inference stage in response to the inference request.

[0122] In step S32, token statistics are performed at each inference stage in response to the inference request, and the token statistics plugin is used to obtain the token statistics results of the inference request.

[0123] In one possible implementation, the token statistics of the inference request include at least one of the following: request identifier, user identifier, model identifier, number of input tokens, number of output tokens, total number of tokens, first token delay, total request delay, and response type, wherein the response type is a streaming response or a non-streaming response.

[0124] In one possible implementation, the text reasoning system also includes a client;

[0125] The AI ​​gateway layer is used to receive inference requests, including: receiving inference requests sent by clients using the AI ​​gateway layer; and calling back the token statistics plugin at each inference stage in response to the inference request, including: calling back the token statistics plugin using the AI ​​gateway layer during the request stage of the inference request; and obtaining the token statistics results of the inference request using the token statistics plugin, including: parsing the inference request using the token statistics plugin during the request stage of the inference request, determining and storing the request identifier, user identifier, model identifier, and the request start time of the inference request.

[0126] In one possible implementation, the text inference system further includes: an inference engine; a token statistics plugin that is called back at each inference stage in response to an inference request, including: using the AI ​​gateway layer to send the inference request to the inference engine, and calling back the token statistics plugin at the response stage of the inference request; and using the token statistics plugin to obtain the token statistics results of the inference request, including: using the token statistics plugin to determine and mark the response type at the response stage of the inference request.

[0127] In one possible implementation, the token statistics plugin is called back at each inference stage in response to the inference request, including: in the case of a streaming response, the AI ​​gateway layer calls back the token statistics plugin after receiving the first response data block returned by the inference engine; the token statistics plugin is used to obtain the token statistics results of the inference request, including: using the token statistics plugin to parse the first response data block returned by the inference engine, determine the first token time, and determine the first token delay based on the first token time and the request start time.

[0128] In one possible implementation, the token statistics plugin is called back at each inference stage in response to the inference request, including: when the response type is a streaming response, the AI ​​gateway layer calls back the token statistics plugin after receiving each response data block returned by the inference engine; the token statistics plugin is used to obtain the token statistics result of the inference request, including: using the token statistics plugin to parse and accumulate the number of tokens included in each response data block returned by the inference engine to determine the number of output tokens.

[0129] In one possible implementation, a token statistics plugin is used to obtain the token statistics result of the inference request, including: using the token statistics plugin to parse the metadata information included in the last response data block returned by the inference engine to determine the number of output tokens counted by the inference engine; using the token statistics plugin to correct the number of output tokens obtained by accumulating all response data blocks returned by the inference engine based on the number of output tokens counted by the inference engine, to obtain the corrected number of output tokens.

[0130] In one possible implementation, a token statistics plugin is used to obtain the token statistics results of the inference request, including: using the token statistics plugin to parse the prompt text included in the inference request to determine the number of input tokens; or, using the token statistics plugin to parse the metadata information included in the last response data block returned by the inference engine to determine the number of input tokens counted by the inference engine.

[0131] In one possible implementation, a token statistics plugin is used to obtain the token statistics results of the inference request, including: using the token statistics plugin to sum the number of input tokens and the number of output tokens to determine the total number of tokens.

[0132] In one possible implementation, the text reasoning system further includes a measurement system; the method further includes: after the reasoning request ends, using a token statistics plugin to generate a complete reasoning event corresponding to the reasoning request according to a preset event format by using the token statistics plugin; and using the token statistics plugin to send the complete reasoning event to the measurement system.

[0133] In one possible implementation, the method further includes: at any preset reporting time before the end of the inference request, using a token statistics plugin, generating partial inference events corresponding to the inference request according to a preset event format based on the token statistics results of the inference request at the preset reporting time; and using the token statistics plugin, sending the partial inference events to the metering system.

[0134] In one possible implementation, the default event format is Cloud Events.

[0135] In one possible implementation, the token statistics plugin is a WASM plugin.

[0136] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0137] In addition, this disclosure also provides electronic devices, computer-readable storage media, and programs, all of which can be used to implement any of the text reasoning methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the relevant section of the method and will not be repeated here.

[0138] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0139] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0140] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0141] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0142] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. For example, electronic device 1900 may be provided as a server or terminal device. (Refer to...) Figure 4 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0143] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0144] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0145] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0146] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0147] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0148] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0149] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0150] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0152] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A text-based reasoning system, characterized in that, include: AI gateway layer, wherein the AI ​​gateway layer is equipped with a token statistics plugin; The AI ​​gateway layer is used to receive inference requests and to call back the token statistics plugin at each inference stage in response to the inference request; The token statistics plugin is used to perform token statistics at each inference stage in response to the inference request, and obtain the token statistics results of the inference request.

2. The system according to claim 1, characterized in that, The token statistics of the inference request include at least one of the following: request identifier, user identifier, model identifier, number of input tokens, number of output tokens, total number of tokens, first token delay, total request delay, and response type, wherein the response type is a streaming response or a non-streaming response.

3. The system according to claim 2, characterized in that, The system also includes: a client; The client is used to send the inference request to the AI ​​gateway layer; The AI ​​gateway layer is used to call back the token statistics plugin during the request phase of the inference request. The token statistics plugin is used to parse the inference request, determine and store the request identifier, the user identifier, the model identifier, and the request start time of the inference request.

4. The system according to claim 3, characterized in that, The system also includes: an inference engine; The AI ​​gateway layer is used to send the inference request to the inference engine, and in the response phase of the inference request, to call back the token statistics plugin; The token statistics plugin is used to determine and mark the response type.

5. The system according to claim 3, characterized in that, The AI ​​gateway layer is used to call back the token statistics plugin after receiving the first response data block returned by the inference engine when the response type is a streaming response. The token statistics plugin is used to parse the first response data block returned by the inference engine, determine the first token time, and determine the first token delay based on the first token time and the request start time.

6. The system according to claim 2, characterized in that, The AI ​​gateway layer is used to call back the token statistics plugin after receiving each response data block returned by the inference engine when the response type is a streaming response. The token statistics plugin is used to parse and accumulate the number of tokens included in each response data block returned by the inference engine to determine the number of output tokens.

7. The system according to claim 6, characterized in that, The token statistics plugin is used for: Parse the metadata information included in the last response data block returned by the inference engine to determine the number of output tokens counted by the inference engine; Based on the number of output tokens counted by the inference engine, the number of output tokens obtained by summing all the response data blocks returned by the inference engine is corrected to obtain the corrected number of output tokens.

8. The system according to claim 6 or 7, characterized in that, The token statistics plugin is used for: Parse the prompt text included in the inference request to determine the number of input tokens; or, The metadata information included in the last response data block returned by the inference engine is parsed to determine the number of input tokens counted by the inference engine.

9. The system according to claim 8, characterized in that, The token statistics plugin is used for: The total number of tokens is determined by summing the number of input tokens and the number of output tokens.

10. The system according to claim 1, characterized in that, The system also includes: a metering system; The token statistics plugin is used to generate a complete inference event corresponding to the inference request by taking the token statistics results of the inference request according to a preset event format after the inference request ends. The token statistics plugin is used to send the complete inference event to the metering system.

11. The system according to claim 1, characterized in that, The system also includes: a metering system; The token statistics plugin is used to generate a partial inference event corresponding to the inference request by taking the token statistics results of the inference request at the preset reporting time according to a preset event format at any preset reporting time before the inference request ends. The token statistics plugin is used to send the aforementioned inference events to the metering system.

12. The system according to claim 10 or 11, characterized in that, The preset event format is Cloud Events format.

13. The system according to claim 1, characterized in that, The token statistics plugin is a WASM plugin.

14. A text-based reasoning method, characterized in that, The method is applied to a text reasoning system, wherein the system includes an AI gateway layer, and the AI ​​gateway layer is equipped with a token statistics plugin; The AI ​​gateway layer is used to receive inference requests and to call back the token statistics plugin at each inference stage in response to the inference request. Token statistics are performed at each inference stage in response to the inference request, and the token statistics plugin is used to obtain the token statistics results of the inference request.

15. An electronic device, characterized in that, The text reasoning system includes any one of claims 1 to 13.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 14.

17. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 14.

18. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 14.

Images