A distributed large-scale model token counting method based on link tracing

Abstract

This invention discloses a distributed large model token counting method based on trace linking, relating to the field of large model management technology. The method involves collecting token consumption data from the previous day's data via a distributed cloud large model server, uploading this data to Kafka, and receiving and parsing the token consumption data from Kafka via a large model management center. Data from different nodes is integrated based on the trace ID to form complete request chain data. Following a hierarchical and domain-based approach, the large model management center stores user token consumption data in object storage. Finally, based on business needs, the large model management center displays relevant statistical data of message chains within a filtered region.

Description

Technical Field

[0001] This invention discloses a distributed large model token counting method based on link tracing, which relates to the field of large model management technology. Background Technology

[0002] With the widespread application of large-scale models and the increasing development of intelligent agents, accurate and efficient token statistics for different large-scale models are becoming increasingly important for the billing of large-scale model service management platforms. Currently, token calculation is based on the last response from the large-scale model, which misses the tokens consumed in the interaction between the intelligent agent and the large-scale model. While directly recording the consumed token data at the large-scale model interface can record all token consumption information, it cannot perform accurate statistics for each user due to the lack of effective user information. Summary of the Invention

[0003] This invention addresses the problems of existing technologies by providing a distributed large-scale model token counting method based on link tracing. It utilizes the link tracing principle to accurately record the token data consumed by each user request, achieving multi-dimensional token statistics by user, model type, time period, etc., thus laying a solid foundation for accurate billing.

[0004] The specific solution proposed in this invention is as follows: This invention provides a distributed large-scale model token counting method based on link tracing. It collects token consumption data from the previous day's data using a distributed cloud large-scale model server and uploads this data to Kafka. The large model management center receives and parses token consumption data from Kafka, integrates data from different nodes based on the Trace ID to form complete request chain data, and then places the user's token consumption data into object storage according to the requirements of hierarchical and domain-based management. Based on business needs, the large model management center displays relevant statistical data of message links within the filtered area. The relevant statistical data includes message count, Token P50, Token P99, error rate, overall time consumption, and first response time. Token P50 is the median of token consumption within the filtered area, and Token P99 is the 99th percentile of token consumption within the filtered area.

[0005] Furthermore, in the distributed large model token counting method based on link tracing, when collecting token consumption data through the distributed cloud large model server, the user first initiates a large model service request by configuring an API key in the request header. The distributed cloud large model server verifies the validity of the API key. If valid, it performs content security verification on the user request. After successful verification, the distributed cloud large model server generates a unique trace identifier (TraceID) for the user's request. The Trace ID is used to associate operations and data related to the user request on different nodes. The distributed cloud big model server calls the big model service interface. If the big model interface successfully returns information, content security detection is performed. If the content security verification is successful, the big model server stores the user request arrival time, processing time, input text, and output text into object storage in a standard format.

[0006] Furthermore, in the aforementioned distributed large model token counting method based on link tracing, if the distributed cloud large model server verifies the API key as invalid, the process ends and an error message is returned to the user. If the distributed cloud big model server fails to perform content security verification on the user request, the distributed cloud big model server assembles a message containing the request time, request content, and requesting user, and pushes it to the big model management center. If the large model interface fails to return information, the large model assembles a message containing the request time, request content, request user message, and large model return content, and sends it to the large model management center. If the large model interface successfully returns information but content security detection fails, the large model server assembles a message containing the request time, request content, request user message, and security content detection result, and pushes it to the large model management center.

[0007] Furthermore, in the aforementioned distributed large model token counting method based on link tracing, the large model management center and the distributed large model server use a communication protocol to transmit request and response data. The distributed large model server assembles link detection information and, through the SDWAN network, calls the RESTful link detection interface to push large model service status detection information, using the large model service running status information as a heartbeat message. The large model management center receives the link detection information, parses the link information, and stores it in the database. If the large model management center does not receive a heartbeat from the distributed large model server within 3 minutes, it determines that the distributed large model server has failed and notifies the administrators via an alarm ticket. After receiving a user request, the distributed large model server reports the node task execution status in real time when a task fails, including the request task ID, request time, request content, and the large model's response. The large model management center receives data on the execution status of large model service requests and saves it to object storage. If the number of failed large model service requests exceeds the threshold, or if there is no response for an extended period of time, the large model service on that node is deemed to be malfunctioning, and administrators are notified.

[0008] This invention provides a distributed large model token counting system based on link tracing, including a distributed cloud large model server management module and a large model management center management module. The distributed cloud large model server management module collects token consumption data from the previous day's data collected by the large model service and uploads the token consumption data to Kafka. The management module at the large model management center receives and parses token consumption data from Kafka. Based on the TraceID, it integrates data from different nodes to form complete request chain data. Following the requirements of hierarchical and domain-based management, the module then stores the user's token consumption data in object storage. Based on business needs, the management module of the large model management center displays relevant statistical data of the message chain within the filtered area. The relevant statistical data includes the number of messages, Token P50, Token P99, error rate, overall time consumption, and first response time. Token P50 is the median of Token consumption within the filtered area, and Token P99 is the 99th percentile of Token consumption within the filtered area.

[0009] Furthermore, in the distributed cloud large model token counting system based on link tracing, when the distributed cloud large model server management module collects token consumption data, the user first initiates a large model service request by configuring an API key in the request header. The distributed cloud large model server management module verifies whether the API key is valid. If it is, it performs content security verification on the user request. After the verification passes, the distributed cloud large model server management module generates a unique trace identifier (Trace ID) for the user's request. The trace identifier (Trace ID) is used to associate operations and data related to the user request on different nodes. The distributed cloud big model server management module calls the big model service interface. If the big model interface successfully returns information, it performs content security checks. If the content security check is successful, the distributed cloud big model server management module stores the user request arrival time, processing time, input text, and output text in the object storage according to the standard format.

[0010] Furthermore, in the distributed large model token counting system based on link tracing, if the distributed cloud large model server management module verifies the API key as invalid, the process ends and an error message is returned to the user. If the distributed cloud big model server management module fails to perform content security verification on the user request, the distributed cloud big model server management module assembles a message containing the request time, request content, and requesting user, and pushes it to the big model management center management module. If the large model interface fails to return information, the large model assembles a message containing the request time, request content, request user message, and large model return content, and sends it to the large model management center. If the large model interface successfully returns information but content security detection fails, the large model server assembles a message containing the request time, request content, request user message, and security content detection result, and pushes it to the large model management center.

[0011] Furthermore, in the aforementioned distributed large model token counting system based on link tracing, the large model management center module and the distributed large model server management module use a communication protocol to transmit request and response data. The distributed large model server management module assembles link detection information and, through the SDWAN network, calls the RESTful link detection interface to push large model service status detection information, using the large model service running status information as a heartbeat message. The management module at the large model management center receives the link detection information, parses the link information, and stores it in the database. If the management module at the large model management center does not receive a heartbeat from the distributed large model server management module within 3 minutes, it determines that the distributed large model server management module has malfunctioned and notifies the administrators via an alarm work order. After receiving a user request, the distributed large model server management module reports the node task execution status in real time when a task fails, including the request task ID, request time, request content, and the large model's response. The management module at the large model management center receives data on the execution status of large model service requests and saves it to object storage. If the number of failed large model service requests exceeds the threshold, or if there is no response for an extended period of time, the large model service on that node is deemed to be malfunctioning, and administrators are notified.

[0012] The advantages of this invention are: 1. Accurate Billing: Through link tracing technology, the number of input and output tokens for each request on each node in the distributed large model service can be accurately counted and the data is aggregated to the central management terminal. This provides an accurate basis for billing users of the large model service, avoids the problem of inaccurate billing, improves the fairness and reasonableness of billing, and protects the interests of service providers and users.

[0013] 2. Highly efficient architecture design: The system adopts a two-layer architecture with a central server and branch servers, which can accurately count tokens according to different types of large models. This effectively improves the system's processing efficiency and flexibility, enabling the system to better adapt to the complex needs of different types of large models and enhance the system's scalability and adaptability.

[0014] 3. Reliable Operation Guarantee: The inspection mechanism ensures the effectiveness of the connection between the central end and the branch center through timed reporting and real-time detection. The branch center reports the running status of the large model every minute. If the large model management center does not receive a heartbeat within 3 minutes, it is determined that the large model server of the branch center has failed. This mechanism can detect and handle faults in a timely manner, ensure stable operation, reduce service interruption time caused by faults, and improve reliability and availability.

[0015] 4. Facilitates Performance Analysis: When conditions permit, the requests and responses from the large model can be recorded simultaneously, facilitating subsequent analysis of the model's response performance. This helps to further improve the performance of the large model's responses, providing strong support for its optimization and improvement, and contributing to enhancing the overall quality and user experience of the large model service. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0017] Figure 2 This is a flowchart of the large model's service side.

[0018] Figure 3 This is a flowchart of the timed reporting process for monitoring data.

[0019] Figure 4 It is a process that actively uploads a data flow diagram when a task fails. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention. Example

[0021] This invention provides a distributed large-scale model token counting method based on link tracing: Step 1: Collect the token consumption data collected by the large model service the previous day at 1:00 AM every day through the distributed cloud large model server, and upload the token consumption data to Kafka.

[0022] When the distributed cloud big model server collects token consumption data, the user first initiates a big model service request by configuring an API key in the request header. The distributed cloud big model server verifies the validity of the API key. If the API key is invalid, the process ends and an error message is returned to the user. If it is valid, the user's request is checked for content security. If the verification is successful, the distributed cloud big model server generates a unique trace ID for the user's request. The trace ID is used to associate operations and data related to the user's request on different nodes. If the distributed cloud big model server fails to check the content security of the user's request, the distributed cloud big model server assembles a message containing the request time, request content, and request user information and pushes it to the big model management center. The distributed cloud big model server calls the big model service interface. If the big model interface successfully returns information, content security detection is performed. If the big model interface fails to return information, the big model assembles a message containing the request time, request content, request user message, and big model return content, and sends it to the big model management center. If the content security verification is successful, the large model server stores the user request arrival time, processing time, input text, and output text in object storage according to the standard format. If the large model interface successfully returns information indicating that the content security check has failed, the large model server assembles a message containing the request time, request content, request user message, and security content check result, and pushes it to the large model management center.

[0023] Step 2: Receive and parse token consumption data from Kafka through the large model management center. Integrate data from different nodes based on TraceID to form complete request chain data. Then, according to the requirements of hierarchical and domain-based management, put the user's token consumption data into object storage through the large model management center.

[0024] Based on business needs, the large model management center displays relevant statistical data of message links within the filtered area. The relevant statistical data includes message count, Token P50, Token P99, error rate, overall time consumption, and first response time. Token P50 is the median of token consumption within the filtered area, and Token P99 is the 99th percentile of token consumption within the filtered area. Example

[0025] Based on Example 1, the large model management center and the distributed large model server use a communication protocol to transmit request and response data. The distributed large model server assembles link detection information and, through the SDWAN network, calls the RESTful link detection interface to push large model service status detection information, using the large model service running status information as a heartbeat message. The large model management center receives the link detection information, parses the link information, and stores it in the database. If the large model management center does not receive a heartbeat from the distributed large model server within 3 minutes, it determines that the distributed large model server has failed and notifies the administrators via an alarm ticket. After receiving a user request, the distributed large model server reports the node task execution status in real time when a task fails, including the request task ID, request time, request content, and the large model's response. The large model management center receives data on the execution status of large model service requests and saves it to object storage. If the number of failed large model service requests exceeds the threshold, or if there is no response for an extended period of time, the large model service on that node is deemed to be malfunctioning, and administrators are notified. Example

[0026] This invention provides a distributed large model token counting system based on link tracing, including a distributed cloud large model server management module and a large model management center management module. The distributed cloud large model server management module collects token consumption data from the previous day's data collected by the large model service and uploads the token consumption data to Kafka. The management module at the large model management center receives and parses token consumption data from Kafka. Based on the TraceID, it integrates data from different nodes to form complete request chain data. Following the requirements of hierarchical and domain-based management, the module then stores the user's token consumption data in object storage. Based on business needs, the management module of the large model management center displays relevant statistical data of the message chain within the filtered area. The relevant statistical data includes the number of messages, Token P50, Token P99, error rate, overall time consumption, and first response time. Token P50 is the median of Token consumption within the filtered area, and Token P99 is the 99th percentile of Token consumption within the filtered area.

[0027] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0028] Similarly, the advantages of the system of the present invention are: 1. Accurate Billing: Through link tracing technology, the number of input and output tokens for each request on each node in the distributed large model service can be accurately counted and the data is aggregated to the central management terminal. This provides an accurate basis for billing users of the large model service, avoids the problem of inaccurate billing, improves the fairness and reasonableness of billing, and protects the interests of service providers and users.

[0029] 2. Highly efficient architecture design: The system adopts a two-layer architecture with a central server and branch servers, which can accurately count tokens according to different types of large models. This effectively improves the system's processing efficiency and flexibility, enabling the system to better adapt to the complex needs of different types of large models and enhance the system's scalability and adaptability.

[0030] 3. Reliable Operation Guarantee: The inspection mechanism ensures the effectiveness of the connection between the central end and the branch center through timed reporting and real-time detection. The branch center reports the running status of the large model every minute. If the large model management center does not receive a heartbeat within 3 minutes, it is determined that the large model server of the branch center has failed. This mechanism can detect and handle faults in a timely manner, ensure stable operation, reduce service interruption time caused by faults, and improve reliability and availability.

[0031] 4. Facilitates Performance Analysis: When conditions permit, the requests and responses from the large model can be recorded simultaneously, facilitating subsequent analysis of the model's response performance. This helps to further improve the performance of the large model's responses, providing strong support for its optimization and improvement, and contributing to enhancing the overall quality and user experience of the large model service.

[0032] It should be noted that not all steps and modules in the above processes and system structures are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments can be physical or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0033] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A distributed large-scale model token counting method based on link tracing, characterized by: The distributed cloud big model server collects token consumption data from the previous day's data and uploads it to Kafka. The large model management center receives and parses token consumption data from Kafka, integrates data from different nodes based on the Trace ID to form complete request chain data, and then places the user's token consumption data into object storage according to the requirements of hierarchical and domain-based management. Based on business needs, the large model management center displays relevant statistical data of message links within the filtered area. The relevant statistical data includes message count, Token P50, Token P99, error rate, overall time consumption, and first response time. Token P50 is the median of token consumption within the filtered area, and Token P99 is the 99th percentile of token consumption within the filtered area.

2. The distributed large-scale model token counting method based on link tracing as described in claim 1, characterized in that: When collecting token consumption data through the distributed cloud big model server, the user first initiates a big model service request by configuring an API key in the request header. The distributed cloud big model server verifies the validity of the API key. If valid, it performs content security verification on the user's request. After successful verification, the distributed cloud big model server generates a unique trace ID for the user's request. The trace ID is used to associate operations and data related to the user's request on different nodes. The distributed cloud big model server calls the big model service interface. If the big model interface successfully returns information, content security detection is performed. If the content security verification is successful, the big model server stores the user request arrival time, processing time, input text, and output text into object storage in a standard format.

3. The distributed large-scale model token counting method based on link tracing as described in claim 1, characterized in that: If the distributed cloud big data model server verifies the API key as invalid, the process ends and an error message is returned to the user. If the distributed cloud big model server fails to perform content security verification on the user request, the distributed cloud big model server assembles a message containing the request time, request content, and requesting user, and pushes it to the big model management center. If the large model interface fails to return information, the large model assembles a message containing the request time, request content, request user message, and large model return content, and sends it to the large model management center. If the large model interface successfully returns information but content security detection fails, the large model server assembles a message containing the request time, request content, request user message, and security content detection result, and pushes it to the large model management center.

4. The distributed large-scale model token counting method based on link tracing as described in claim 1, characterized in that: The large model management center and the distributed large model server use a communication protocol to transmit request and response data. The distributed large model server assembles link detection information and, through the SDWAN network, calls the RESTful link detection interface to push large model service status detection information, using the large model service running status information as a heartbeat message. The large model management center receives the link detection information, parses the link information, and stores it in the database. If the large model management center does not receive a heartbeat from the distributed large model server within 3 minutes, it determines that the distributed large model server has failed and notifies the administrators via an alarm ticket. After receiving a user request, the distributed large model server reports the node task execution status in real time when a task fails, including the request task ID, request time, request content, and the large model's response. The large model management center receives data on the execution status of large model service requests and saves it to object storage. If the number of failed large model service requests exceeds the threshold, or if there is no response for an extended period of time, the large model service on that node is deemed to be malfunctioning, and administrators are notified.

5. A distributed large-scale model token counting system based on link tracing, characterized by: This includes a distributed cloud large model server-side management module and a large model management center-side management module. The distributed cloud large model server management module collects token consumption data from the previous day's data collected by the large model service and uploads the token consumption data to Kafka. The large model management center module receives and parses token consumption data from Kafka, integrates data from different nodes based on the Trace ID to form complete request chain data, and places user token consumption data into object storage according to the requirements of hierarchical and domain-based management. Based on business needs, the management module of the large model management center displays relevant statistical data of message links within the filtered area. The relevant statistical data includes message count, Token P50, Token P99, error rate, overall time consumption, and first response time. Token P50 is the median of Token consumption within the filtered area, and Token P99 is the 99th percentile of Token consumption within the filtered area.

6. A distributed large-scale model token counting system based on link tracing as described in claim 5, characterized in that: When the distributed cloud big model server management module collects token consumption data, the user first initiates a big model service request by configuring an API key in the request header. The distributed cloud big model server management module verifies the validity of the API key. If valid, it performs content security verification on the user request. After successful verification, the distributed cloud big model server management module generates a unique trace ID for the user's request. The trace ID is used to associate operations and data related to the user request on different nodes. The distributed cloud big model server management module calls the big model service interface. If the big model interface successfully returns information, it performs content security checks. If the content security check is successful, the distributed cloud big model server management module stores the user request arrival time, processing time, input text, and output text in the object storage according to the standard format.

7. A distributed large-scale model token counting system based on link tracing as described in claim 5, characterized in that: If the distributed cloud big data model server management module fails to verify the API key, the process ends and an error message is returned to the user. If the distributed cloud big model server management module fails to perform content security verification on the user request, the distributed cloud big model server management module assembles a message containing the request time, request content, and requesting user, and pushes it to the big model management center management module. If the large model interface fails to return information, the large model assembles a message containing the request time, request content, request user message, and large model return content, and sends it to the large model management center. If the large model interface successfully returns information but content security detection fails, the large model server assembles a message containing the request time, request content, request user message, and security content detection result, and pushes it to the large model management center.

8. A distributed large-scale model token counting system based on link tracing as described in claim 5, characterized in that: The large model management center module and the distributed large model server management module use a communication protocol to transmit request and response data. The distributed large model server management module assembles link detection information and, through the SDWAN network, calls the RESTful link detection interface to push large model service status detection information, using the large model service running status information as a heartbeat message. The management module at the large model management center receives the link detection information, parses the link information, and stores it in the database. If the management module at the large model management center does not receive a heartbeat from the distributed large model server management module within 3 minutes, it determines that the distributed large model server management module has malfunctioned and notifies the administrators via an alarm work order. After receiving a user request, the distributed large model server management module reports the node task execution status in real time when a task fails, including the request task ID, request time, request content, and the large model's response. The management module at the large model management center receives data on the execution status of large model service requests and saves it to object storage. If the number of failed large model service requests exceeds the threshold, or if there is no response for an extended period of time, the large model service on that node is deemed to be malfunctioning, and administrators are notified.

Images