Wordpiece metering method, system, device and artificial intelligence gateway for large models

Abstract

This application discloses a lexical measurement method, system, device, and artificial intelligence gateway for large models, belonging to the field of artificial intelligence. The method is applied to an artificial intelligence gateway and includes: receiving a response data stream corresponding to a model processing request, wherein the response data stream is generated by the model server calling a large model processing client's model processing request; and, when the target response fragment in the response data stream indicates the end of the response data stream, parsing the lexical quantity field in the target response fragment to obtain a first lexical quantity; the first lexical quantity is the number of lexical units consumed by the model processing request. This application, by deploying lexical measurement logic on the artificial intelligence gateway, can be applied to various model servers without requiring modifications to the client and model server.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of artificial intelligence, and particularly to a lexical measurement method, system, device, and artificial intelligence gateway for large models. Background Technology

[0002] In current mainstream artificial intelligence models (such as large models), modeling and inference calculations are all based on tokens. The number of tokens is linearly related to the computation cost, so the number of tokens can be used as the basis for billing.

[0003] The relevant technology provides a method for counting tokens within the model. After receiving input content, the model calls its internal tokenizer to calculate the number of tokens in the input content, and during the generation of response data, it accumulates the number of tokens in the response data in real time. After inference is completed, the model returns the number of tokens in the input content and the number of tokens in the response data to the model caller or the billing system. The caller or the billing system then uses these two numbers for billing.

[0004] However, the related technologies embed the measurement logic inside the model, resulting in strong coupling with the model, poor scalability, and difficulty in applying a unified technical solution to different models. Summary of the Invention

[0005] This application provides a lexical measurement method, system, device, and artificial intelligence gateway for large models. The technical solution of this application includes the following contents.

[0006] According to one aspect of this application, a lexical measurement method for large models is provided, applied to an artificial intelligence gateway, the method comprising the following steps.

[0007] Receive the response data stream corresponding to the model processing request, wherein the response data stream is generated by the model server calling the model processing client to request the model processing; When the target response fragment in the response data stream indicates the end of the response data stream, the word count field in the target response fragment is parsed to obtain the first word count; the first word count is the number of words consumed by the model in processing the request.

[0008] According to another aspect of this application, a lexical measurement system for large models is provided, the system comprising a client, an artificial intelligence gateway, and a model server; The client is used to send a model processing request to the artificial intelligence gateway; The artificial intelligence gateway is used to forward the model processing request to the model server; The model server is used to call the large model to process the model processing request and generate a response data stream, and send the response data stream to the artificial intelligence gateway; The AI ​​gateway is configured to forward the response data stream to the client, and, when the target response segment in the response data stream indicates the end of the response data stream, parse the token count field in the target response segment to obtain a first token count; the first token count is the number of tokens consumed by the model in processing the request.

[0009] According to another aspect of this application, a lexical metering device for large models is provided, which is applied in an artificial intelligence gateway, and the device includes the following modules.

[0010] The receiving module is used to receive the response data stream corresponding to the model processing request. The response data stream is generated by the model server calling the model processing client to request the model processing. The parsing module is used to parse the word count field in the target response segment to obtain a first word count when the target response segment in the response data stream represents the end of the response data stream; the first word count is the number of words consumed by the model in processing the request.

[0011] According to one aspect of this application, an artificial intelligence gateway is provided for performing the lexical measurement method for large models as described above.

[0012] According to one aspect of this application, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the lexical measurement method for large models as described above.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, which stores a computer program that is loaded and executed by a processor to implement the lexical measurement method for large models as described above.

[0014] According to another aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium, the computer program being loaded and executed by a processor to implement the lexical measurement method for large models as described above.

[0015] The beneficial effects of the technical solutions provided in this application include at least the following: In this embodiment, by deploying the lexical measurement logic on the artificial intelligence gateway, it can be applied to various model servers without requiring modifications to the client or model server.

[0016] Furthermore, instead of incrementally parsing the response data stream during the streaming process, the AI ​​gateway utilizes the lexical count field naturally carried in the response data stream to parse the number of lexical units consumed by the model in processing the request. This saves the AI ​​gateway's parsing and storage resources, enabling it to operate stably in high-concurrency scenarios. In addition, directly parsing the lexical count field to obtain the number of lexical units consumed by the model in processing the request also improves measurement efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an architecture diagram of a lexical measurement method for large models provided in one embodiment of this application.

[0019] Figure 2 This is a flowchart of a lexical measurement method for large models provided in one embodiment of this application.

[0020] Figure 3 This is a flowchart of a lexical measurement method for large models provided in another embodiment of this application.

[0021] Figure 4 This is a flowchart of a lexical measurement method for large models provided in another embodiment of this application.

[0022] Figure 5 This is a schematic diagram of a lexical measurement system for large models provided in one embodiment of this application.

[0023] Figure 6 This is a structural block diagram of a lexical metering device for large models provided in one embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0025] Currently, most mainstream large-scale models (such as large language models) are based on lexical units for text modeling and inference computation. The number of lexical units is usually linearly related to the computational cost; therefore, in model inference services, the number of lexical units is the most commonly used basis for billing. The following three methods for lexical unit measurement for large-scale models are provided in related technologies.

[0026] I. Internal Model Counting During inference, the model uses a word parser to parse the input text and counts the number of input and output words at the end of inference. In this type of approach, the model itself performs word counting during the inference process. Specifically, after receiving input, the internal word parser calculates the number of input words; when generating output, it accumulates the output words; after inference is complete, both are returned as metadata to the caller; the caller or billing system then uses this metadata for billing.

[0027] Understandably, this solution embeds the metering logic within the inference engine, resulting in strong coupling with the model structure and lexical parser, making it impossible to achieve unified metering at the proxy layer. This solution makes it difficult for the proxy layer to manage billing behavior uniformly, and it cannot form a unified lexical metering system in multi-model, multi-framework environments. It also cannot be extended through plugins or standard interfaces, exhibiting high structural coupling, poor scalability, and difficulty in forming a unified technical solution.

[0028] II. Application Layer SDK (Software Development Kit) Count Before and after sending a request, the client or API (Application Programming Interface) SDK performs lexicalization on the model input and output, respectively, and then performs statistics based on the results.

[0029] Understandably, the metering logic of this solution is executed by the client. However, in large systems, the behavior of SDKs in different languages / versions is inconsistent. The client may skip the metering code, and the consistency of metering results cannot be guaranteed on the server side. In other words, the metering logic of this solution is scattered and lacks a unified and reliable execution environment.

[0030] Furthermore, this solution requires embedding code in every business system, leading to problems such as multiple implementation schemes for multi-language environments, version inconsistencies causing errors, increased system architecture complexity, and difficulty in standardization within the enterprise. In other words, the solution's measurement logic cannot be unified and has poor reusability.

[0031] III. Server-side business logic counting The measurement logic is integrated within the large model inference server, and the number of terms is counted based on the callback of the intermediate hook function or the completion function.

[0032] This application provides a lexical measurement method executed by the proxy layer, in which the artificial intelligence gateway uses the lexical quantity field naturally carried in the response data stream to parse the number of lexical units consumed by the model in processing the request, so as to solve the problems of inaccurate lexical statistics, poor security, insufficient scalability and low real-time performance in related technologies.

[0033] Figure 1 The diagram illustrates a lexical measurement system for large models provided in an exemplary embodiment of this application. The system includes at least a client 110, an artificial intelligence gateway 120, and N model servers 130; wherein the artificial intelligence gateway 120 is equipped with a quantity reporting plugin 121.

[0034] For example, client 110 is an application that calls a large model during runtime. During one call, client 110 will send a model processing request to AI gateway 120; the model processing request is a request to call the large model. Optionally, the model processing request carries a prompt, which serves as input to the large model, and the large model is used to generate a response data stream; optionally, the model processing request also carries the identifier of the large model being called, the address of the target model server 130, etc., for routing and forwarding.

[0035] The AI ​​gateway 120 forwards the model processing request to the target model server 130; such as Figure 1 As shown, one AI gateway 120 corresponds to multiple model servers 130, meaning that one AI gateway 120 can provide data routing services for multiple model servers 130. Optionally, the target model server 130 is selected by the AI ​​gateway 120 itself, or it can be determined by the AI ​​gateway 120 based on the identifier of the large model being invoked and / or the address of the target model server 130 carried in the model processing request.

[0036] Figure 1 The illustration shows an AI gateway 120 forwarding a model processing request to a model server 1. The model server 1 then runs a large model based on the model processing request to generate a response data stream, which is returned to the AI ​​gateway 120. Optionally, one or more large models can be deployed on the model server 1. If multiple large models are deployed on the model server 1, the model server 1 can determine the target large model from among the multiple large models based on the identifier of the large model being called carried in the model processing request, or it can choose a target large model that matches the model processing request, and then generate a response data stream by calling the target large model.

[0037] Each time the AI ​​gateway 120 receives a response fragment from the response data stream, it transmits that response fragment to the internally installed quantity reporting plugin 121. Correspondingly, the quantity reporting plugin 121 continuously receives response fragments transmitted by the AI ​​gateway 120. The quantity reporting plugin 121 identifies each response fragment, and when it identifies a target response fragment indicating the end of the response data stream, it parses the word count field in the target response fragment to obtain the first word count. Optionally, the target response fragment is the last response fragment in the response data stream.

[0038] Optionally, the metering system also includes a billing system 140, and the AI ​​gateway 120 can also send the first word token count to the billing system 140 for billing.

[0039] It is understood that this application, by deploying measurement logic on the AI ​​gateway 120, can be applied to various model servers 130, and the measurement method of this application does not require modification to the client 110 or the model server 130. Furthermore, the quantity reporting plugin 121 will parse the first word quantity using the word quantity field naturally carried in the response data stream, eliminating the need for complex incremental word parsing of the response data stream. This saves the AI ​​gateway's parsing and storage resources and improves word measurement efficiency.

[0040] In one embodiment, client 110 is an artificial intelligence application. Users can send model processing requests to the AI ​​application through dialogue, file uploads, etc. These requests can include information search, document summary and analysis, multimodal content generation, etc. For example, client 110 can be creative software, where users can utilize AI (Artificial Intelligence) capabilities to generate content during the creation process; for example, client 110 could be photo editing software, karaoke software, animation generation software, etc. For example, client 110 could be office software, such as code generation software, where users can generate code using AI capabilities. It should be noted that this application does not limit the specific type of client 110.

[0041] Optionally, client 110 will run on a terminal device, which may be at least one of a smartphone, smartwatch, in-vehicle terminal, wearable device, smart TV, tablet computer, e-book reader, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 player, laptop computer, and desktop computer.

[0042] Optionally, model server 130 is a server that deploys large models. Model server 130 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0043] Figure 2 An exemplary embodiment of this application provides a lexical measurement method for large models, illustrated by way of example, which is executed by an artificial intelligence gateway. The method includes steps 220 and 240.

[0044] Step 220: Receive the response data stream corresponding to the model processing request. The response data stream is generated by the model server calling the large model processing client to request the model processing. An AI gateway sits between the client and one or more model servers, handling data routing between them; essentially, it operates at the proxy layer. For example, the AI ​​gateway forwards model processing requests initiated by the client to the model server, and forwards the response data stream generated by the model server to the client. The response data stream is the result of the model server calling a large model to process the model processing request. The model processing request is a request to the large model to generate the response content. The model server sends the response data to the AI ​​gateway via a streaming process, and the AI ​​gateway also sends the response data to the client via a streaming process.

[0045] Optionally, the AI ​​gateway can be used to provide routing functionality for multiple model servers, each with different large models. Alternatively, the AI ​​gateway can also provide routing functionality for only one model server. Optionally, a single model server can deploy multiple different large models or only one large model; this application does not impose any restrictions on this.

[0046] In some scenarios, AI gateways, in addition to performing routing operations, can also be used for protocol adaptation, security governance, and cost control. For example, an AI gateway can shield API differences between different vendors, providing a standard interface. Optionally, an AI gateway can dynamically distribute model processing requests to the most suitable model based on model performance, response speed, lexical price, or user level. Optionally, an AI gateway can also perform sensitive word filtering and prompt word injection detection on input and output content to prevent models from outputting unsafe content. Optionally, an AI gateway can also implement quota management and cache duplicate requests to significantly reduce the cost of calling large models. Optionally, an AI gateway can also uniformly record request logs, latency events, and error rates for easy debugging and optimization. In some cases, an AI gateway can also combine with a vector database to implement semantic caching (e.g., directly returning historical answers to similar questions) or constrain the model output format through guardrail rules.

[0047] For example, the response data stream received by the AI ​​gateway includes several consecutive response fragments; each response fragment contains at least one of the following: 1. Response Content. Response content refers to the valid response data generated by the model server, rather than descriptive data such as the length or generation time of the response data stream. Optionally, response content can be text, images, audio, etc. For example, the response content {"id": "...", "choices": [{"delta": {"content": "hello"}}]} contains the text "hello". For instance, some or all response fragments preceding the last response fragment in the response data stream may carry response content.

[0048] 2. Usage Key-Value Pairs. Usage key-value pairs are key-value pairs that carry a term quantity field. The usage key-value pair contains a term quantity field; the term quantity field is the value field in the usage key-value pair; the usage key-value pair also includes a "usage" field, which is the key field in the usage key-value pair. For example, the usage key-value pair is {"usage": {"prompt_tokens": 42, "completion_tokens": 118, "total_tokens": 160}}, where the token count field includes the "prompt_tokens", "completion_tokens", and "total_tokens" fields; "prompt_tokens" is the input token field, where the prompt word contains 42 tokens; "completion_tokens" is the output token field, where the answer content contains 118 tokens; and "total_tokens" is the total token count field, where the model consumes a total of 160 tokens. For example, the last response slice in the response data stream carries the usage key-value pair.

[0049] 3. End Marker. The end marker is located in the last response fragment and is used to separate the response data stream. For example, the end marker is represented as data: [DONE]. For example, the last response fragment in the response data stream carries an end marker.

[0050] Step 240: When the target response fragment in the response data stream represents the end of the response data stream, parse the token count field in the target response fragment to obtain the first token count.

[0051] For example, the AI ​​gateway receives response fragments in a streaming manner and performs a data stream end recognition operation on each response fragment. The data stream end recognition operation is used to determine whether the response data stream has ended. Optionally, the AI ​​gateway performs the data stream end recognition operation as soon as it receives a response fragment, without paying attention to whether the previous response fragment has been recognized. Optionally, the AI ​​gateway may also perform the recognition operation for the next response fragment only after the previous response fragment has been recognized, thereby avoiding wasting resources to recognize the next response fragment when the response data stream has already been identified as ended in the previous response fragment.

[0052] Once the AI ​​gateway identifies the end of the response data stream from the target response fragment, it will parse the token count field in the target response fragment to obtain the first token count. Typically, the target response fragment is the last response fragment in the response data stream, and the last response fragment usually has a token count field, which carries token usage information.

[0053] The first word unit count refers to the number of words consumed by the model in processing the request. Alternatively, the first word unit count refers to the number of words consumed by the model server in processing the model's request. Optionally, the first word unit count includes at least one of the following: the number of input words, the number of output words, and the total number of words. The number of input words is the number of words in the prompt words in the model's request; the number of output words is the number of words in the response content generated by the model server based on the model's request; and the total number of words is the sum of the number of input words and the number of output words.

[0054] It is understandable that the response data streams output by different model servers naturally carry a word count field, which contains the word counts already counted by the model server. This application will reuse the word count field in the response data stream to avoid incremental counting of words in the answer content.

[0055] In one embodiment, the AI ​​gateway sends the number of first tokens and / or the billing result corresponding to the number of first tokens to the client.

[0056] For example, the AI ​​gateway sends the first token count as supplementary header data to the client.

[0057] For example, an AI gateway may send the first token count as supplementary header data to the client in the response data stream.

[0058] For example, the AI ​​gateway sends the billing result corresponding to the number of the first word tokens to the client, and the billing result will be displayed on the client's billing interface.

[0059] This embodiment will also send the first word unit count and / or billing result to the client to ensure that the client can know in a timely manner the word unit count consumed by this model call and / or word unit billing status.

[0060] In one embodiment, the execution process of the lexical measurement method for large models provided in this application is parallel to the process of the AI ​​gateway forwarding the response data stream to the client. Optionally, the AI ​​gateway performs the lexical measurement process in a bypass manner, so that the lexical measurement process does not affect the original process of forwarding the response data stream to the client, and the client can still receive the response data stream at the original receiving rate.

[0061] In summary, in this embodiment of the application, by deploying the lexical measurement logic on the artificial intelligence gateway, it can be applied to various model servers without requiring modifications to the client or model server.

[0062] Furthermore, instead of incrementally parsing the response data stream during the streaming process, the AI ​​gateway uses the inherent token count field in the response data stream to parse and obtain the first token count. This saves the AI ​​gateway's parsing and storage resources, enabling it to operate stably in high-concurrency scenarios. In addition, directly parsing the token count field after identifying it yields the number of tokens consumed by the model in processing the request, which improves measurement efficiency compared to incremental parsing.

[0063] In one embodiment, Figure 2 The lexical measurement method for large models shown is executed by a quantity reporting plugin set up on the AI ​​gateway. Optionally, the quantity reporting plugin is attached to a callback function (such as OnHttpStreamingResponseBody) in the AI ​​gateway's response path. Each time the AI ​​gateway receives a response fragment of the response data stream, it passes the response fragment to the callback function, thereby passing the response fragment to the quantity reporting plugin.

[0064] Optionally, the AI ​​gateway can transmit response fragments to the quantity reporting plugin one by one, or it can transmit the accumulated response fragments to the quantity reporting plugin once every preset number of fragments. It is understandable that since the AI ​​gateway receives response fragments sequentially, it will also transmit response fragments sequentially, and the quantity reporting plugin will also receive response fragments sequentially.

[0065] Optionally, the quantity reporting plugin is a WebAssembly (WASM) plugin located in a WebAssembly sandbox environment. The WebAssembly plugin can securely access the response headers and body data of the response data stream, but its operational capabilities are limited by the sandbox. This embodiment will leverage the WebAssembly plugin's suitability for traffic tagging and lightweight computation to execute the terminology measurement logic.

[0066] The web-based assembly sandbox environment supports memory-safe isolation, explicit permission control, and millisecond-level startup of web-based assembly plugins. This embodiment encapsulates the metering logic within a quantity reporting plugin (web-based assembly plugin) and runs it within the web-based assembly sandbox environment. This securely isolates the metering logic from other processing logic of the AI ​​gateway, ensuring the pluggability and security of the metering logic. It allows for the deployment and upgrading of the metering logic without affecting the stability of the AI ​​gateway, facilitating project implementation.

[0067] based on Figure 2 In the optional embodiment shown, step 240 includes... Figure 3 Step 320 is shown.

[0068] Step 320: In response to identifying at least one of the end marker and word count fields from the target response fragment, determine the end of the response data stream.

[0069] The end marker is used to separate different response data streams. In other words, the end marker indicates that the AI ​​gateway will not subsequently receive response fragments from the same response data stream; that is, the end marker indicates that subsequent response fragments received by the AI ​​gateway do not belong to the same response data stream.

[0070] This embodiment transforms the task of detecting the end of the response data stream into the task of recognizing the end marker and / or word count field. This can effectively detect the end of the response data stream, and by recognizing the end marker and / or word count field, the difficulty of detecting the end state of the response data stream can be reduced.

[0071] In one embodiment, after receiving a response fragment, the AI ​​gateway will perform an end marker recognition operation and / or a token count field recognition operation on the response fragment in parallel. Once the end marker is identified from the response fragment through the end marker recognition operation and / or the token count field is identified from the response fragment through the token count field recognition operation, the end of the response data stream is determined.

[0072] At this point, executing the two identification operations in parallel can effectively improve the efficiency of identifying the end of the response data stream from the response fragments.

[0073] In another embodiment, step 320 includes as follows Figure 4 Steps 420 to 440 are shown.

[0074] Step 420: If the target response fragment is identified and an end marker is found, determine that the response data stream has ended; For example, each time the AI ​​gateway receives a response fragment, it will perform an end marker recognition operation on the response fragment. If an end marker is recognized from a response fragment, it is determined that the response data stream has ended.

[0075] When the AI ​​gateway identifies each response fragment in sequence, if the AI ​​gateway successfully identifies the end marker from the target response fragment, it means that the response data stream has ended, that is, the currently identified target response fragment is the last response fragment.

[0076] Step 440: If no end marker is detected from the target response fragment, and a token count field is detected from the target response fragment, the response data stream is determined to have ended.

[0077] For example, each time the AI ​​gateway receives a response fragment, it will perform an end marker recognition operation on the response fragment. If an end marker cannot be recognized from a response fragment, the AI ​​gateway will also perform a token count field recognition operation on the response fragment. If the token count field is recognized, the response data stream is determined to have ended; if the token count field is not recognized, the end marker recognition operation will be performed on the next response fragment.

[0078] When the AI ​​gateway identifies each response fragment in sequence, it first attempts to identify the end marker from the target response fragment. If the AI ​​gateway cannot identify the end marker, it then attempts to identify the token count field from the target response fragment. If the token count field is successfully identified, the response data stream is determined to have ended.

[0079] In this embodiment, in addition to utilizing the "end marker," a "lexicon count field" is also used as a stream end signal. When the lexicon count field is detected in a response segment, the end of the response data stream is confirmed. By identifying the lexicon count field, the end of the response data stream can be accurately and promptly identified in streaming response scenarios, avoiding the inability to identify the stream end status due to the loss of the "end marker." Furthermore, determining the end of the response data stream by detecting the lexicon count field directly correlates the identification of the stream end status with the semantics of the data returned by the model, improving the universality and reliability of the stream end status identification method in multi-model and multi-implementation environments, and ensuring the scalability of the solution.

[0080] Furthermore, this embodiment first determines whether the target response fragment contains an end marker. If it does, the response data stream is determined to have ended. If not, the token count field is then read. In most cases, the end marker can be identified from the target response fragment to determine the end of the response data stream without needing to identify the token count field, thus improving the efficiency of determining the end of the stream.

[0081] In one embodiment, before step 440, the method further includes: determining that a lexical quantity field has been identified if a usage key-value pair is identified from the target response fragment and a lexical quantity field is parsed from the usage key-value pair; wherein the usage key-value pair is a key-value pair carrying a lexical quantity field.

[0082] For example, the AI ​​gateway reads data within the target response fragment to attempt to match usage key-value pairs; if a usage key-value pair is successfully matched and the token count field can be parsed from the usage key-value pair, it determines that the token count field has been identified.

[0083] Usage key-value pairs are key-value pairs that carry a term count field. For example, the term count field is the value field in the usage key-value pair, and the key field is the usage field. Optionally, the usage key-value pairs can be in JSON (JavaScript Object Notation).

[0084] Optionally, the AI ​​gateway uses a streaming JSON parser to read data within the target response fragments.

[0085] For example, the AI ​​gateway reads data within the target response fragment, and after reading the key field (usage field) of the usage key-value pair, it continues reading until a complete usage key-value pair is matched.

[0086] For example, when attempting to identify the token count field from a response fragment, the AI ​​gateway will attempt to identify usage key-value pairs from the response fragment. If usage key-value pairs cannot be identified, it is determined that the response fragment does not contain the token count field. If usage key-value pairs are successfully identified, the AI ​​gateway will further attempt to parse the token count field from the usage key-value pairs. If the token count field is successfully parsed, it is determined that the token count field has been identified. If the token count field cannot be successfully parsed, it is determined that the token count field has not been identified.

[0087] In this embodiment, by first matching usage key-value pairs within the target response segment and then identifying the term quantity field from the usage key-value pairs, a process of coarse-grained matching followed by fine-grained identification is achieved, improving the identification efficiency and accuracy of the term quantity field. In this embodiment, when a usage key-value pair cannot be matched, it can be directly determined that the term quantity field does not exist, thereby avoiding fine-grained identification of the data within the response segment and improving the efficiency of the AI ​​gateway in identifying whether the term quantity field exists within the response segment.

[0088] based on Figure 2 In an alternative embodiment shown, the method further includes: creating a metering session for the response data stream upon receiving the first response fragment of the response data stream; A metering session is a session dedicated to a single response data stream, or in other words, a metering session corresponds to a model processing request. For example, the AI ​​gateway will maintain state information for the metering session, indicating its session status. Optionally, the AI ​​gateway will record the reception status of the response data stream in real time within the metering session's state information; for example, it will record the most recently received response fragment, the transmission progress of the response data stream, and the elapsed transmission time of the response data stream, etc.

[0089] Optionally, when the AI ​​gateway receives the first response fragment, it records the identifier of the model processing request (request_id) and / or the client identifier (tenant_id) in the state information of the metering session.

[0090] In this embodiment, a metering session dedicated to a single response data stream is created to facilitate the maintenance of metering logic for each response data stream and to ensure that the metering logic for each response data stream is not confused.

[0091] based on Figure 2 In an alternative embodiment shown, the method further includes: in response to identifying a token count field or an end marker from a response fragment in the response data stream, updating at least one of a stream end marker and a stream end load field in the state information of the metering session; the stream end marker is used to indicate whether the response data stream has ended, and the stream end load field is used to record the first token count parsed from the response data stream.

[0092] For example, if the AI ​​gateway identifies a token count field or an end marker from a response fragment in the response data stream, it confirms the end of the response data stream and updates the stream end marker in the state information of the metering session. For example, the AI ​​gateway sets the stream end marker in the state information to True.

[0093] In this embodiment, by recording the flow end marker in the status information of the metering session, the flow end status can be recorded in a timely manner, making it easy for the outside world to know that the response data flow has ended.

[0094] For example, if the AI ​​gateway identifies a token count field or an end marker from a response fragment in the response data stream, it confirms the end of the response data stream. The AI ​​gateway will then parse the first token count from the target response fragment and update the stream end payload field in the state information of the metering session. The stream end payload field is used to carry the first token count.

[0095] In this embodiment, by recording the flow end load field in the status information of the metering session, the number of the first term can be recorded in a timely manner, which facilitates the subsequent processing of the number of the first term from the flow end load field, such as reporting to the billing system.

[0096] based on Figure 2In an optional embodiment shown, the method further includes: in response to the AI ​​gateway identifying the end of the response data stream from the target response fragment, the AI ​​gateway parses the target response fragment. If the target response fragment contains an identifier and / or a call timestamp of a called model, the AI ​​gateway parses the identifier and / or call timestamp of the called model from the target response fragment and records it in the status information of the metering session. Optionally, the AI ​​gateway records the identifier and / or call timestamp of the called model together with the first token count in the stream end payload field. For example, the stream end payload field is represented as {prompt_tokens, completion_tokens, total_tokens, model, ts}, where model is the identifier of the called model and ts is the call timestamp of the model.

[0097] The content carried in the end-of-stream payload field will be in a pending reporting state. When the reporting operation is performed, the AI ​​gateway will read information such as the number of first tokens from the end-of-stream payload field and report it to the billing system.

[0098] In this embodiment, a metering session will be maintained for a response data stream to facilitate the management of the word count parsing and reporting process within the response data stream, thereby improving the reliability of the metered word count.

[0099] based on Figure 2 In an optional embodiment shown, the method further includes: terminating the metering session when a session termination condition is met; the session termination condition includes at least one of the following: 1. Upon receiving the response information from the billing system, the billing system will send a response to the AI ​​gateway, informing it that it has received the first token count. Once the AI ​​gateway receives the response information, it can terminate the metering session.

[0100] 2. The first word count has been recorded. After the AI ​​gateway parses and obtains the first word count, it will record the first word count locally (such as in a persistent queue). The metering session can end after the AI ​​gateway successfully records the first word count. When the first word count is needed, the AI ​​gateway can recover the first word count from the local storage.

[0101] 3. Client Closed; Client closed means the connection between the client and the AI ​​gateway is broken; Client closed indicates an abnormal interruption event has occurred, at which point the AI ​​gateway can directly terminate the metering session. Optionally, after the client is closed, the AI ​​gateway will record a flow error end marker in the metering session status information; optionally, the AI ​​gateway will also generate an abnormal event log, which may include information such as the number of reconnections after the abnormal interruption.

[0102] 4. The connection between the AI ​​gateway and the model server is lost; the loss of connection between the AI ​​gateway and the model server indicates an abnormal interruption event. In this case, the AI ​​gateway can directly terminate the metering session. Optionally, after the connection between the AI ​​gateway and the model server is lost, the AI ​​gateway will record a flow error end marker in the status information of the metering session; optionally, the AI ​​gateway will also generate an abnormal event log, which may include information such as the number of reconnections after the abnormal interruption.

[0103] 5. The lifespan of a metering session exceeds the preset lifespan threshold; when the lifespan of a metering session exceeds the preset lifespan threshold, the metering session will be terminated. Optionally, the AI ​​gateway also generates an anomaly audit log, which may contain information such as the timeout duration and reason for the timeout of the metering session.

[0104] In one embodiment, after a metering session ends, the AI ​​gateway will clear the storage resources used to store the state information of the metering session, and reclaim the instance resources of the metering session or reuse them for other metering sessions, so as to achieve rapid resource reclamation and use and avoid resource waste.

[0105] In this embodiment, multiple termination conditions for metering sessions are designed to improve the reliability of metering sessions, avoid session freezes and infinite operation, and enhance the security and risk control capabilities of the entire metering system.

[0106] based on Figure 2 In an optional embodiment shown, the method further includes: generating an audit record if the second token count does not match the first token count; wherein the audit record is used to analyze at least one of tokenization accuracy and anomaly tracking, the second token count is the sum of the token counts of each response segment in the response data stream, and the token count is the number of tokens in the answer content within the corresponding response segment.

[0107] For example, each response fragment carrying the answer content may also carry the fragment word count, which is the number of words in the answer content within that response fragment. In this embodiment, the sum of the fragment word counts in each response fragment carrying the answer content will be compared with the first word count. If they do not match, an audit record will be generated.

[0108] For example, the first word token count includes the output word token count. If the second word token count does not match the output word token count, the AI ​​gateway will generate an audit record. Optionally, if the difference between the second word token count and the output word token count is greater than a preset alarm threshold, the AI ​​gateway will generate a mismatch warning message and store the mismatch warning message in the audit record.

[0109] For example, audit logs will be sent to an auditing system, which may optionally analyze the accuracy of the tokenizer in the model server. Alternatively, the auditing system may trace the root cause of the anomaly.

[0110] In this embodiment, the AI ​​gateway also performs incremental parsing of the response data stream. If the number of tokens obtained by incremental parsing does not match the number of tokens in the first token count, an audit record is generated, which will be audited by the audit system. In this embodiment, the number of tokens obtained by incremental parsing can be used to verify the number of tokens parsed from the token count field. If the two do not match, an audit is performed, so that the audit system can identify the reliability risk of the number of tokens in the first token count. If the audit system believes that the number of tokens in the first token count is unreliable, it can remind the administrator to optimize the token parser in the model server to improve the reliability of the number of tokens carried by the subsequent token count field.

[0111] In one embodiment, the audit log includes at least one of the following: 1. Meter session identifier; Optionally, the meter session identifier is the identifier of the model processing request. For example, the meter session identifier (session_id) is req-abc.

[0112] 2. First word count; Optionally, the first word count may include only the output word count, or it may include the input word count and the total word count in addition to the output word count.

[0113] 3. Number of second word elements.

[0114] 4. The difference between the first word count and the second word count; for example, if the first word count is 118 and the second word count is 120, then the difference between the two is recorded as 2.

[0115] 5. Reasons for the mismatch between the number of second lexical units and the number of first lexical units; for example, the mismatch reason is "lexicalization mismatch", which means that the number of lexical units at the response fragment level parsed by the model server does not match the number of lexical units at the response data stream level parsed.

[0116] 6. Timestamps for the generation of audit records.

[0117] In this embodiment, an audit record format is provided, which carries the necessary information for the audit system to perform an audit, thus ensuring the feasibility of the audit system to perform the audit.

[0118] In one embodiment, each time the AI ​​gateway receives a response fragment carrying answer content from the response data stream, it extracts the fragment word count from the response fragment carrying answer content and adds the fragment word count to the cumulative word count field until the response data stream ends. Extract the value of the cumulative word count field as the second word count; Among them, the cumulative number of tokens field is a field in the status information of the metering session in the response data stream.

[0119] Optionally, the operation of parsing the number of segmented words and the operation of adding the number of segmented words to the cumulative word count field are asynchronous operations. In this case, after the AI ​​gateway parses the number of segmented words, it will continue to parse the number of the next segmented words without waiting to add the number of segmented words to the cumulative word count field.

[0120] Optionally, the AI ​​gateway processes response fragments carrying answer content one by one. Only after parsing the fragment token count from a response fragment carrying answer content and adding the fragment token count to the cumulative token count field will it continue to process the next response fragment carrying answer content.

[0121] For example, the initial value of the cumulative word count field is zero.

[0122] In this embodiment, the number of segmented words is accumulated in the state information of the metering session in a timely manner, avoiding the need to obtain all segmented word counts before summing. This ensures the accuracy of the summation operation and reduces the storage space required for the summation operation. This embodiment provides an alternative path for obtaining the total number of words in the response data stream by incrementally summing the number of segmented words in the response segment carrying the answer content. The summation result obtained in this way can be used to verify the first word count, avoiding the blind assumption that the first word count is always correct and the blind use of the first word count for billing, thus ensuring the accuracy of billing.

[0123] based on Figure 2 In the optional embodiment shown, step 240 is followed by the following steps.

[0124] Construct a reporting event, which includes at least one of the following: the number of first tokens, the identifier of the metering session to which the response data stream belongs, the identifier of the client, the identifier of the model called by the model server, the generation timestamp of the end marker of the response data stream, the identifier of the AI ​​gateway, the sequence number of the reporting event, and the unique signature of the reporting event. An asynchronous, non-blocking method is used to report events to the billing system.

[0125] For example, the identifier of the metering session is "req-abc-123"; optionally, the identifier of the metering session is the identifier of the model processing request assigned by the AI ​​gateway or obtained by the AI ​​gateway from the upstream header information. For example, the identifier of the client is "tenant-A"; optionally, the identifier of the client is obtained by the AI ​​gateway when performing routing operations or obtained from the upstream header information.

[0126] Optionally, the reported event may also include the identifier of the model server; Optionally, the timestamp for generating the stream end marker is in milliseconds; Optionally, the reported event sequence number refers to the sequence number corresponding to this reported event. After a reporting failure, the reporting will be retried. Each reporting will correspond to a reported event sequence number. The reported event sequence number is monotonically increasing within the scope of the metering session, with the metering session as the boundary. Optionally, the AI ​​gateway can use a hash algorithm or generate a web token to sign the reported event, obtaining a unique signature to prevent forged reported events.

[0127] For example, the asynchronous non-blocking mode represents a way to not block the response forwarding path between the client and the model server; alternatively, the AI ​​gateway uses the outbound call interface (call_outbound) or the asynchronous HTTP (Hypertext Transfer Protocol) call interface (async_http_call interface) provided by the runtime to report events without blocking the response forwarding path between the client and the model server.

[0128] The billing system is used to calculate and generate billing results based on the number of first tokens. For example, the AI ​​gateway sends the number of first tokens to the billing system using a standard CloudEvents protocol or Hypertext Transfer Protocol (HTTP). Optionally, the billing system is a billing server.

[0129] For example, the billing system also returns the billing results to the AI ​​gateway, which then sends the billing results to the client.

[0130] In this embodiment, a format for reporting events is provided. In addition to carrying the number of key first tokens, the reported events also carry other event description information, signature information, etc., to help the billing system fully understand the event content and bill the tokens securely and objectively.

[0131] Furthermore, this embodiment will use an asynchronous non-blocking reporting method, which can avoid the metering logic blocking the original response forwarding path between the client and the model server, and avoid reducing the efficiency of the client obtaining the response data stream.

[0132] In one embodiment, the quantity reporting plugin reports the reporting event to the billing system in an asynchronous, non-blocking manner through the target reporting mechanism; The target reporting mechanism includes at least one of the following sub-mechanisms: The reporting target of the reported events is the application programming interface of the billing system; The reporting protocol used for reporting events is Hypertext Transfer Protocol, Secure Hypertext Transfer Protocol, or Cloud Event Protocol; optionally, the Secure Hypertext Transfer Protocol can be MTLS (Mutual TLS, Bidirectional Secure Transport Layer) or a signed Secure Hypertext Transfer Protocol.

[0133] After reporting, the response information from the billing system must be recorded. For example, after reporting, the billing system should return a 2xx response to indicate that the reporting event was successfully received. The billing system will also return response information (such as ack_seq or received_seq), and the AI ​​gateway will record the response information.

[0134] In this embodiment, a comprehensive and systematic reporting mechanism is designed to achieve secure and reliable reporting.

[0135] In one embodiment, the method further includes at least one of the following: if it is determined that the reporting of the event has failed, re-reporting the event to the billing system; if it is determined that the number of times the event has been reported is not less than a preset number threshold, writing the event locally and issuing an alarm.

[0136] For example, if the reporting fails, the AI ​​gateway will re-report; if the reporting operation still fails after performing a preset threshold number of reporting operations, the AI ​​gateway will write the reporting event to a local persistent queue and issue an alarm.

[0137] For example, after a reporting failure occurs (such as a network error or a 5xx error), the AI ​​gateway will retry R times exponentially (R is the preset number of re-reporting attempts, such as R=3). If the reporting still fails, the reporting event will be written to a persistent queue and an alarm will be triggered.

[0138] Optionally, during the re-reporting process, the AI ​​gateway will retain the reported event until the report is successful or the administrator intervenes.

[0139] In this embodiment, the design of retrying reporting after a failure increases the probability of successful reporting and avoids abandoning reporting due to occasional network fluctuations. Furthermore, writing the reporting event locally and issuing an alarm allows maintenance personnel to be promptly notified of reporting failures and to handle the locally stored reporting events.

[0140] In one embodiment, the reported event includes the identifier of the metering session and the sequence number of the reported event. The identifier of the metering session and the sequence number of the reported event are used by the billing system to determine whether the reported event has been received. The billing system uses the metering session identifier and the sequence number of the reported event as the idempotent key. If a reported event with the same idempotent key is received, it indicates that the billing system has already received the current reported event. The billing system will directly return a 200 error indicating reporting failure and will not charge twice. Optionally, the reported event also includes the identifier and signature information of the AI ​​gateway for double protection. After confirming that the same metering session identifier and reported event sequence number have not been received, the billing system further checks whether the same AI gateway identifier and signature information have been received before. This double check strictly ensures that duplicate billing is not performed.

[0141] In one embodiment, in response to receiving a response message from the billing system and the response message indicating successful reception, the target response fragment is recorded as the last response fragment in the status information of the metering session, and the metering session is cleared. After receiving the response information from the billing system and confirming successful reception, the AI ​​gateway will ultimately confirm the target response fragment as the last response fragment, and then it can clean up the metering session. At this point, the metering session is not cleaned up immediately upon confirming the end of the response data stream, but only after the billing system sends back the response information. This avoids cleaning up the metering session before the actual metering is completed, ensuring accurate cleaning of the metering session and accurately defining the lifecycle of the metering session.

[0142] For example, in response to receiving the response information from the billing system, the AI ​​gateway marks SessionState.last_ack_seq = seq (seq is the fragment identifier of the target response fragment) in the status information and triggers the session cleanup timer. The session cleanup timer will reclaim the storage space of the status information after a short delay.

[0143] The technical solution of this application is described in detail below.

[0144] The purpose of this application is to provide a proxy-layer large-model lexical counting technology based on a web page assembly sandbox, to solve the problems of inaccurate lexical statistics, poor security, insufficient scalability, and low real-time performance in existing technologies. This application intercepts the streaming response of the large language model at the proxy layer and intercepts the response at the web page assembly plugin without modifying the model service or the client, and asynchronously reports the lexical count to the billing system, while ensuring performance, idempotency, consistency, and security isolation under high concurrency.

[0145] The metering system of this application includes the following core components.

[0146] 1. AI Gateway: Intercepts all requests and responses from clients to the model server, and is responsible for traffic routing and forwarding; supports the construction of web page assembly sandbox environments and serves as the mounting point for web page assembly plugins. It also handles authentication and authorization, rate limiting and degradation, and metric statistics.

[0147] 2. Metering Plugin (Quantity Reporting Plugin): This plugin enables the reporting of token statistics (number of first-order tokens) to the MeteringService (Billing System) via a proxy.

[0148] 3. Metering Service: Used to provide unified services to the outside world using the standard CloudEvents protocol or HTTP protocol, facilitating the reporting of event metrics by the AI ​​Gateway.

[0149] 4. LLM Provider (Model Server): The ability to provide inference services to external parties.

[0150] The lexical measurement method of this application includes the following specific steps.

[0151] 1. The client initiates a model processing request. The request passes through the AI ​​gateway. The AI ​​gateway matches the corresponding number of reporting plugins based on the route, intercepts the model processing request, records the model-related information, and writes it into the cache. After writing, the AI ​​gateway forwards the model processing request to the model server.

[0152] 2. After the model server processes the model processing request normally, it returns a response data stream. The quantity reporting plugin intercepts the response data stream, calculates the word count, and requests the billing system interface to asynchronously report the statistical data.

[0153] 3. The billing system receives the data and updates the statistical data according to the idempotent ID.

[0154] The entire process will not block normal requests and responses; the quantity reporting plugin should be processed in a non-blocking path.

[0155] The following details the specific implementation process of the lexical measurement method of this application.

[0156] I. Receiving and Intercepting (Proxy Quantity Reporting Plugin Layer) The source data (response data stream) in this application comes from a backend large language model service. This large language model service can be any open-source or closed-source service. Response chunks in the response data stream are byte sequences, and their content may be partial text fragments, JSON fragments, or trailer metadata. For example, common styles of response chunks are as follows: Text slicing (generated content): {"id": "...", "choices": [{"delta": {"content": "hello"}}]}; Usage information: {"usage": {"prompt_tokens": 42, "completion_tokens": 118, "total_tokens": 160}}; End marker: data: [DONE] (sometimes accompanied by JSON at the end).

[0157] In one embodiment, when the Wasm environment in the proxy layer is running, the quantity reporting plugin is attached to the callback function (OnHttpStreamingResponseBody) of the response path. Each time the proxy layer receives a response fragment sent by the backend, it passes the raw bytes of the response fragment to the callback function.

[0158] In one embodiment, the quantity reporting plugin maintains session state for each streaming session. Specifically, when the quantity reporting plugin receives a response fragment for the first time, it creates a session and records the request ID and tenant ID. Understandably, after step one, the callback function of the quantity reporting plugin will receive the response fragment, the quantity reporting plugin will generate the session state, and receive the original response fragment to wait for the next step of processing.

[0159] II. EOS (End of Stream) Determination and Field Extraction The quantity reporting plugin performs the following judgment order for each incoming response fragment. First, if the byte sequence of the response fragment is exactly equal to a common end marker (such as data: [DONE]), it is considered EOS. Otherwise, a JSON match is performed within the response fragment to find the complete JSON information of the usage or input token (prompt_tokens) field / output token (completion_tokens) field.

[0160] Specifically, the quantity reporting plugin uses a streaming JSON parser. After reading "usage", it continues reading until it matches complete key-value information. If the quantity reporting plugin finds a complete usage object and can parse the input token (prompt_tokens) count (number of input tokens) / output token (completion_tokens) count (number of output tokens), it determines that it is EOS.

[0161] Once identified as EOS, the quantity reporting plugin parses and extracts the numerical field from the response shard: prompt_tokens(integer) completion_tokens(integer) total_tokens (optional) model, request_id (if returned in EOS) In one embodiment, if the quantity reporting plugin performs incremental token parsing during the processing of the stream response, it records agent_count (the accumulated tokens of the quantity reporting plugin under the specified SessionState) (accumulated token count field) and eos_count (completion_tokens in EOS). If abs(agent_count - eos_count) > the preset alarm threshold, a mismatch warning is recorded and this information is included in the report (for subsequent optimization and auditing of the token parser). If the quantity reporting plugin does not perform incremental parsing, eos_count is directly used as the final measurement.

[0162] In one embodiment, the quantity reporting plugin sets SessionState.seen_eos (stream end flag) to true and records eos_payload (stream end payload field) = {prompt_tokens, completion_tokens, total_tokens, model, ts} in the state. It also marks the session as "pending reporting" or directly enters the reporting process.

[0163] Understandably, after step two, the session state information (SessionState) will contain eos_payload and seen_eos=true, and will produce parsed metering values ​​(integers), which will be encapsulated as reported events.

[0164] III. Event Construction and Reporting (Quantity Reporting Plugin Sends to Billing System) In one embodiment, the quantity reporting plugin will construct a reporting event, and an example of the JSON for the reporting event is as follows: { "session_id": "req-abc-123", "tenant_id": "tenant-A", "model": "gpt-4-like", "prompt_tokens": 42 "completion_tokens": 118, "eos_timestamp_ms": 1700000000000, "origin_proxy_id":"proxy-01", “seq”: 1, "signature": "hmac(...)" } Here, `session_id` is a unique request ID assigned by the proxy layer or obtained from the upstream header information. `tenant_id` is the tenant / customer identifier (obtained through routing / header). `Seq` is the reporting sequence number, which monotonically increases within the metering session of the quantity reporting plugin (for idempotency processing). `Signature` is the signature information; optional, the quantity reporting plugin can use HMAC / JWT signatures to prevent forgery.

[0165] In one embodiment, the reporting method of the quantity reporting plugin includes the following: The reporting target is the billing system's API, for example, POST https: / / xx.com / api / v1 / usage. The reporting protocol is HTTP / HTTPS (which can use mTLS or signed HTTPS) or CloudEvents. The sending method is asynchronous and non-blocking (does not block the original response forwarding path), for example, the quantity reporting plugin uses the call_outbound / async_http_call interface provided by the runtime to send the reporting event. An ACK (Acknowledge) mechanism is used; after reporting, the billing system returns a 2xx response indicating success and carrying ack_seq or received_seq, which the quantity reporting plugin records.

[0166] In one embodiment, a retry strategy is used for reporting. Specifically, if the reporting fails (network error or 5xx), the quantity reporting plugin retryes exponentially R times (e.g., 3 times), writes the result to a local persistent queue (or a persistent queue in the proxy layer), and issues an alarm.

[0167] In one embodiment, the billing system employs idempotency to ensure no duplicate billing. For example, the billing system uses (session_id, seq) as the idempotency key; if a duplicate report with the same key is received, it directly returns a 200 error and does not charge repeatedly. The quantity reporting plugin includes seq, origin_proxy_id, and signature in the reported event for double protection.

[0168] In one embodiment, if the quantity reporting plugin receives an ACK, the quantity reporting plugin will mark SessionState.last_ack_seq=seq and trigger the session cleanup timer (reclaim the state after a short delay).

[0169] In one embodiment, if a report fails and retry is not possible, the quantity reporting plugin will write the event to a persistent queue / log and trigger an alarm; administrators can configure the Control Plane to view the alarm and intervene manually.

[0170] Understandably, after step three, the reported event will be sent to the billing system, which will return an ACK and update the backend bill; the quantity reporting plugin will update the local session status (ack / failed).

[0171] IV. Session End, Audit and Cleanup (Detailed processing within the plugin) In one embodiment, a metering session can be terminated and reclaimed when any of the following conditions are met: 1. An EOS has been received and the corresponding report has received a response (or has been recorded in the persistent queue and is recoverable); 2. The connection is closed (client disconnect or upstream connection closed), triggering a flush in the close event (if seen_eos=false, the best-effort estimation or an anomaly can be reported); 3. Timeout (session idle exceeds the configured threshold), triggering reclamation and writing an anomaly audit record.

[0172] In one embodiment, if the quantity reporting plugin performs incremental parsing during stream response processing (optionally) and a difference occurs with the EOS value, the quantity reporting plugin will generate an audit log. An example audit log would look like this: { "session_id": "req-abc", "agent_tokens": 120 “eos_tokens”: 118, “diff”: 2, "reason": "tokenizer mismatch", "timestamp": ... } Optionally, audit logs can be sent to an audit system / Kafka (an open-source distributed event streaming platform) for offline analysis of lexical accuracy and anomaly tracking.

[0173] In one embodiment, the quantity reporting plugin will provide feedback to the user (or business system). The goal of this feedback is to ensure that the response data stream is transmitted transparently without affecting the data flow. Therefore, the user (client) still receives the original response data stream from the model, and the interception and reporting by the quantity reporting plugin are bypassed behaviors.

[0174] Optionally, if metering information needs to be populated in the response header, the proxy layer can return a supplementary header (such as X-Usage-Total: 160) to the client after receiving the EOS, or inform the client of the billing result via a webhook / callback after the response is completed (this depends on business needs and privacy policies). Optionally, the billing system can present usage data to the user on the bill / console.

[0175] In one embodiment, the quantity reporting plugin clears the SessionState's memory (carry buffers, pending events) and reclaims WASM instance resources (or reuses the instance for a new session). Optionally, if a re-upload occurs, the quantity reporting plugin will retain persistent items until successful or administrator intervention.

[0176] Understandably, after step four, the metering data is successfully archived into the billing system and becomes available for billing / monitoring; the client response is not delayed or tampered with (unless explicitly requested to return metering information).

[0177] Compared with related technologies, this application introduces a WebAssembly-based plugin structure at the proxy layer to intercept, parse, and measure the streaming response data of a large language model. It then determines the end state (EOS) of the streaming response based on the input and / or output lexical fields contained in the response data. This achieves accurate acquisition and reporting of model lexical usage, and has at least the following beneficial technical effects: 1. Improve the accuracy and consistency of word unit measurement; 2. Reduce the computational complexity and resource consumption of the proxy layer; 3. Achieve decoupling and security isolation between measurement logic and model services; 4. Supports general econometric capabilities across multiple models; 5. Provides low-latency, non-blocking real-time metering data reporting capabilities; 6. Improve the auditability and stability of the metering system.

[0178] The differences between this application and related technologies include the following: Difference 1: This application directly intercepts the model streaming response data through a web page assembly plugin at the proxy layer, and determines the end state (EOS) of the streaming response based on the number of tokens contained in the response data. Related technologies typically perform lexical measurement in the following locations: (1) Inside the model server, word statistics are generated after inference is completed; (2) The application server performs statistics or estimations after receiving the complete response; (3) On the client side, the number of tokens is parsed or inferred automatically after the request is completed.

[0179] None of these solutions intercept the model's streaming response at the proxy layer, and they use the presence or absence of input and / or output lexical fields in the response data as the basis for EOS determination.

[0180] This application moves the lexical measurement logic forward to the proxy layer, takes over the model response stream in real time through a web page plugin, and uses the number of lexical items appearing in the response data as the technical condition for determining EOS.

[0181] This difference directly solves the problem of not being able to accurately and timely identify the end of the output and obtain the final word unit measurement results in streaming response scenarios, and is the fundamental prerequisite for achieving accurate measurement and real-time reporting.

[0182] Point of difference 2: This application uses whether the response data contains input token fields and / or output token fields as the EOS determination condition, rather than relying on a fixed end marker or complete response reception; Related technologies typically determine the end of a response by using protocol layer signals (connection closed, EOF) or fixed identifiers (such as [DONE]); or by performing lexical statistics after receiving the complete response content. This approach has limited adaptability to different model implementations and streaming protocols.

[0183] This application determines EOS by parsing the field content in each response chunk and detecting whether it contains a token count field, so that the recognition of EOS is directly related to the semantics of the data returned by the model.

[0184] This distinction improves the versatility and reliability of EOS recognition in multi-model and multi-implementation environments, enabling the measurement logic to adapt to the differences in measurement information returned by different models at the end stage. It is a key technical point to ensure the scalability of the solution.

[0185] Point 3: This application only parses and obtains lexical measurement data when it is determined to be EOS, rather than performing incremental lexicalization processing throughout the entire streaming process; The relevant techniques require lexicalization of each response fragment; or secondary lexicalization of the complete text after the stream ends. These techniques are computationally complex and heavily dependent on the lexicalization version.

[0186] This application does not perform lexicalization calculation during the streaming output process. It only reads the final lexical count field returned by the model when EOS is detected, thereby completing the measurement.

[0187] This difference significantly reduces the computational load and resource consumption of the proxy layer, enabling the solution to run stably in high-concurrency proxy scenarios, while avoiding errors caused by inconsistent lexicalization.

[0188] Difference 4: The web page is used to assemble a sandbox to execute the metering logic, thereby achieving secure isolation between the metering function and the agent's core processing logic; In related technologies, metering logic typically exists as embedded code on the proxy or server side; or it requires customized modifications to the proxy program. This approach presents security and maintainability issues.

[0189] This application encapsulates the EOS determination, lexical extraction, and metering reporting logic into a web page assembly plugin, which runs in a sandbox environment provided by the agent.

[0190] This distinction ensures the pluggability and security of the metering logic, enabling the deployment and upgrade of this application solution without affecting the stability of the agent, which is conducive to the implementation of the project.

[0191] This application has the following advantages: (1) This application uses the detection of the token count field in the streaming response fragments as the criterion for determining whether the streaming response enters the End State (EOS). This determination method is related to the protocol format of the model's returned data, which improves the robustness and universality of EOS determination in environments with multiple models and the same protocol format.

[0192] (2) By sandboxing execution, the metering logic and the agent core processing logic are isolated, reducing the impact of metering module anomalies on system stability and improving system security and maintainability.

[0193] (3) This application can support different models and different inference frameworks without modifying the model service, thereby improving the universality and scalability of the metrology system.

[0194] (4) While ensuring the real-time response of the model, this application realizes the timely collection of word usage data and non-blocking reporting, thereby improving the real-time performance of the measurement system.

[0195] This application maintains an independent session state for each streaming request at the proxy layer and generates metering data associated with a unique request identifier during the EOS phase. This approach ensures good traceability of the lexical metering results, facilitating subsequent technical audits and anomaly analysis.

[0196] Figure 5 The diagram illustrates a structural block diagram of a lexical measurement system for large models provided in an exemplary embodiment of this application. The system includes a client 110, an artificial intelligence gateway 120, and a model server 130. like Figure 5As shown, client 110 is used to send a model processing request to AI gateway 120; AI gateway 120 is used to forward the model processing request to model server 130; model server 130 is used to call the large model to process the model processing request, generate a response data stream, and send the response data stream to AI gateway 120; AI gateway 120 is used to forward the response data stream to client 110, and, when the target response fragment in the response data stream represents the end of the response data stream, parse the word count field in the target response fragment to obtain the first word count; wherein, the first word count is the number of words consumed by the model processing request.

[0197] In one embodiment, the system further includes a billing system (not shown in the figure); the AI ​​gateway 120 sends the first word token count to the billing system, which then bills the model processing request based on the first word token count and generates a billing result.

[0198] Optionally, the billing system may also send the billing results to the AI ​​gateway 120, which in turn sends the billing results to the client 110.

[0199] In one embodiment, the AI ​​gateway 120 is equipped with a quantity reporting plugin (not shown in the figure). The AI ​​gateway 120 transmits the response data stream to the quantity reporting plugin. When the target response segment in the response data stream represents the end of the response data stream, the quantity reporting plugin parses the word count field in the target response segment to obtain the first word count.

[0200] Optionally, the AI ​​gateway 120 may also send the first token count to the client 110.

[0201] It should be noted that for further information on lexical measurement systems for large models, please refer to the above introduction on "Lexical Measurement Methods for Large Models," which will not be repeated here.

[0202] Figure 6 This embodiment of the present application shows a structural block diagram of a lexical metering device for large models, the device 600 being located on an artificial intelligence gateway, the device comprising: The receiving module 601 is used to receive the response data stream corresponding to the model processing request. The response data stream is generated by the model server calling the model processing client to process the model processing request. The parsing module 602 is used to parse the word count field in the target response fragment to obtain the first word count when the target response fragment in the response data stream represents the end of the response data stream; the first word count is the number of words consumed by the model in processing the request.

[0203] In an optional embodiment, the parsing module 602 is further configured to determine the end of the response data stream in response to identifying at least one of an end marker and a token count field from the target response fragment.

[0204] In an optional embodiment, the parsing module 602 is further configured to determine the end of the response data stream if an end marker is identified from the target response segment; and to determine the end of the response data stream if no end marker is identified from the target response segment but a token count field is identified from the target response segment.

[0205] In an optional embodiment, the parsing module 602 is further configured to determine that a lexical quantity field has been identified when a usage key-value pair is identified from the target response fragment and a lexical quantity field is parsed from the usage key-value pair; wherein the usage key-value pair is a key-value pair carrying a lexical quantity field.

[0206] In an optional embodiment, the apparatus further includes a session maintenance module 603. The session maintenance module 603 is configured to perform at least one of the following: upon receiving the first response fragment of the response data stream, create a metering session for the response data stream; in response to identifying a token count field or an end marker from a response fragment in the response data stream, update at least one of a stream end marker and a stream end payload field in the status information of the metering session, wherein the stream end marker indicates whether the response data stream has ended, and the stream end payload field records the first token count parsed from the response data stream; terminate the metering session when session termination conditions are met; the session termination conditions include at least one of the following: receiving response information from the billing system; recording the first token count; the client being closed; the connection between the AI ​​gateway and the model server being disconnected; or the metering session's lifetime exceeding a preset lifetime threshold.

[0207] In an optional embodiment, the apparatus further includes an audit log generation module 604. The audit log generation module 604 is used to generate an audit log when the second token count does not match the first token count; wherein the audit log is used to analyze at least one of tokenization accuracy and anomaly tracking, the second token count being the sum of the token counts of each response segment in the response data stream, and the token count being the number of tokens in the answer content within the corresponding response segment.

[0208] In an optional embodiment, the audit log generation module 604 is used to extract the number of fragment words from each response fragment carrying the answer content received in the response data stream, and add the number of fragment words to the cumulative word count field until the response data stream ends; the value of the cumulative word count field is extracted as the second word count; wherein, the cumulative word count field is a field in the status information of the metering session of the response data stream.

[0209] In an optional embodiment, the device further includes a reporting module 605. The reporting module 605 is used to construct a reporting event; the reporting event includes at least one of the following: the number of first tokens, the identifier of the metering session to which the response data stream belongs, the identifier of the client, the identifier of the model called by the model server, the generation timestamp of the stream end marker of the response data stream, the identifier of the AI ​​gateway, the sequence number of the reporting event, and the unique signature of the reporting event; the reporting event is reported to the billing system in an asynchronous non-blocking manner; wherein, the asynchronous non-blocking manner indicates that communication between the client and the model server is not blocked; the billing system is used to bill the model processing request based on the reported event.

[0210] In an optional embodiment, the reporting module 605 is further configured to re-report the reporting event to the billing system if it is determined that the reporting event has failed; and to write the reporting event locally and issue an alarm if it is determined that the number of times the reporting event has been reported is not less than a preset number threshold.

[0211] This application also provides an artificial intelligence gateway, which is used to execute the lexical measurement method for large models provided in the above method embodiments.

[0212] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the lexical measurement method for large models provided in the above method embodiments.

[0213] This application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the lexical analysis method for large models provided in the above-described method embodiments.

[0214] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0215] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0216] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A lexical measurement method for large-scale models, characterized in that, Applied to an artificial intelligence gateway, the method includes: Receive the response data stream corresponding to the model processing request, wherein the response data stream is generated by the model server calling the model processing client to request the model processing; When the target response fragment in the response data stream indicates the end of the response data stream, the word count field in the target response fragment is parsed to obtain the first word count; the first word count is the number of words consumed by the model in processing the request.

2. The method according to claim 1, characterized in that, The method further includes: The end of the response data stream is determined in response to the identification of at least one of the end marker and the word count field from the target response fragment.

3. The method according to claim 2, characterized in that, The response determines the end of the response data stream by identifying at least one of the end marker and the word count field from the target response fragment, including: If the end marker is identified from the target response fragment, the end of the response data stream is determined; If the end marker is not detected from the target response segment, but the token count field is detected from the target response segment, the response data stream is determined to have ended.

4. The method according to claim 3, characterized in that, The method further includes: If a usage key-value pair is identified from the target response fragment, and the term quantity field is parsed from the usage key-value pair, then the term quantity field is determined to be identified. The usage key-value pair is a key-value pair that carries the term quantity field.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes at least one of the following: Upon receiving the first response fragment of the response data stream, a metering session for the response data stream is created; In response to the identification of the token count field or the end marker from a response fragment in the response data stream, at least one of the stream end marker and the stream end load field is updated in the status information of the metering session. The stream end marker is used to indicate whether the response data stream has ended, and the stream end load field is used to record the first token count parsed from the response data stream. The metering session ends when the session termination conditions are met; the session termination conditions include at least one of the following: receiving a response message from the billing system; the first term number has been recorded; the client is closed; the connection between the AI ​​gateway and the model server is disconnected; the lifespan of the metering session exceeds a preset lifespan threshold.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: If the number of second lexical terms does not match the number of first lexical terms, an audit record is generated. The audit log is used to analyze at least one of lexicalization accuracy and anomaly tracking. The second lexical count is the sum of the lexical counts of each response segment in the response data stream, where the lexical count is the number of lexical counts of the answer content within the corresponding response segment.

7. The method according to claim 6, characterized in that, The method further includes: For each response fragment carrying answer content received in the response data stream, the number of fragment tokens is extracted from the response fragment carrying answer content, and the number of fragment tokens is added to the cumulative token count field until the response data stream ends; Extract the value of the cumulative word count field as the second word count; The cumulative token count field is a field in the status information of the metering session of the response data stream.

8. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Construct a reporting event; the reporting event includes at least one of the following: the number of the first lexical token, the identifier of the metering session to which the response data stream belongs, the identifier of the client, the identifier of the model called by the model server, the generation timestamp of the stream end marker of the response data stream, the identifier of the artificial intelligence gateway, the sequence number of the reporting event, and the unique signature of the reporting event; The reported event is reported to the billing system in an asynchronous, non-blocking manner; The asynchronous non-blocking mode indicates that communication between the client and the model server is not blocked; the billing system is used to bill the model processing request based on the reported event.

9. The method according to claim 8, characterized in that, The method further includes at least one of the following: If it is determined that the reporting of the event has failed, the event should be re-reported to the billing system. If the number of times the reported event is reported is not less than a preset threshold, the reported event is written to the local machine and an alarm is triggered.

10. The method according to any one of claims 1 to 4, characterized in that, The method is executed by a quantity reporting plugin set on the artificial intelligence gateway. The quantity reporting plugin is a web page assembly plugin and is located in the web page assembly sandbox environment provided by the artificial intelligence gateway.

11. A lexical measurement system for large models, characterized in that, The system includes a client, an artificial intelligence gateway, and a model server; The client is used to send a model processing request to the artificial intelligence gateway; The artificial intelligence gateway is used to forward the model processing request to the model server; The model server is used to call the large model to process the model processing request and generate a response data stream, and send the response data stream to the artificial intelligence gateway; The AI ​​gateway is configured to forward the response data stream to the client, and, when the target response segment in the response data stream indicates the end of the response data stream, parse the token count field in the target response segment to obtain a first token count; the first token count is the number of tokens consumed by the model in processing the request.

12. A lexical unit counting device for large models, characterized in that, The device, used in an artificial intelligence gateway, includes: The receiving module is used to receive the response data stream corresponding to the model processing request. The response data stream is generated by the model server calling the model processing client to request the model processing. The parsing module is used to parse the word count field in the target response segment to obtain a first word count when the target response segment in the response data stream represents the end of the response data stream; the first word count is the number of words consumed by the model in processing the request.

13. An artificial intelligence gateway, characterized in that, The artificial intelligence gateway is used to perform the method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the method as described in any one of claims 1 to 10.

15. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, which is loaded and executed by a processor to implement the method as described in any one of claims 1 to 10.

Images