Large model business indicator monitoring method, device and storage medium

Abstract

The present invention provides a large model business indicator monitoring method, device and storage medium, which relates to the field of large model technology, including: deploying the large model to be monitored in the CGRA platform; monitoring the generation time of the first Token; monitoring the generation time of the Token; monitoring the average number of Tokens; monitoring the total number of requests per second; monitoring the average time of the request; monitoring the total number of long connection requests in the process; transmitting the collected monitoring data to the monitoring system based on the data transmission mechanism provided by the CGRA platform; setting an alarm threshold for each monitored data according to historical data and business needs, and issuing abnormal warning information when the monitored data exceeds the alarm threshold. The technical solution provided by this application can accurately grasp the text generation efficiency of the model, the server load and the richness of the model output content, which helps developers take corresponding measures to enhance the processing efficiency of large models and improve server reliability.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a large model business indicator monitoring method, device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, large models have been widely used in many fields such as natural language processing and computer vision. The CGRA platform has become one of the ideal choices for running large models due to its high configurability and parallel computing capabilities. However, the operation of large models requires huge computing resources, and its complex structure and dynamic business load make it crucial to accurately understand its operation status on the CGRA platform. The existing technology mainly discloses the following three methods for monitoring business indicators of large models: (1) Application of traditional monitoring tools: directly use traditional server monitoring tools to monitor the operation of large models on the CGRA platform. These tools mainly focus on basic monitoring of hardware resources, such as CPU usage, memory usage, etc. However, for the complex business logic and special computing requirements of large models, this monitoring method is too general and cannot deeply understand the operation status of each module within the large model and the detailed changes of business indicators. (2) Simple logging: The existing solution records the execution status of key steps and some business data by adding simple logging statements to the large model code. However, this approach lacks systematicity, and the analysis and integration of logs are difficult. It is difficult to reflect the operating status of the large model in real time. In addition, for large-scale, high-concurrency large model business scenarios, the performance overhead of log records cannot be ignored. (3) Manual adjustment based on experience: In some cases, operation and maintenance personnel rely on experience to manually adjust resource configuration according to the output results of the large model and the overall performance of the platform. This approach lacks accurate data support, and the adjustment process is often blind. It is unable to respond to the dynamic changes of the large model business indicators in a timely and accurate manner, which easily leads to resource waste or performance bottlenecks. In the process of realizing the present invention, the applicant found that the existing technology mainly has the following defects: (1) Insufficient real-time performance: In traditional monitoring methods, due to the delay in data processing and analysis, it may be impossible to obtain the operating status and performance indicators of the large model in a timely manner, thereby affecting the adjustment and optimization of the model. (2) Lack of fine-grained monitoring capabilities: For large model business indicator monitoring is a relatively new field, and different parts may have different performance requirements. Traditional monitoring methods cannot provide sufficiently fine-grained information to support targeted optimization measures for large models. (3) Insufficient scalability and adaptability: As the model size and complexity increase, fixed monitoring strategies may no longer be applicable, requiring frequent manual intervention to adjust monitoring parameters, increasing operation and maintenance costs. (4) Lack of CGRA platform monitoring system: Existing monitoring tools are mostly applicable to general computing environments and cannot fully meet the needs of large models for fine-grained performance data in high-performance computing environments. The dynamic resource allocation and unique scheduling mechanism of the CGRA platform require that the monitoring system must be able to adapt to these changes, but the existing monitoring system does not have this function.

[0003] Therefore, how to effectively monitor the business indicators of large models has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses a large model business indicator monitoring method, device and storage medium, which can accurately grasp the text generation efficiency of the model, the server load and the richness of the model output content, and help developers take corresponding measures to enhance the processing efficiency of large models and improve server reliability.

[0005] The first aspect of the present invention discloses a large model business indicator monitoring method, including: model deployment: deploying the large model to be monitored in the CGRA platform; monitoring the generation time of the first Token: inserting a time recording function at the starting logic code of the large model generated text to obtain the starting timestamp; monitoring the generation time of the Token: inserting a time recording function after the core code block responsible for generating a single Token inside the large model, and obtaining the timestamp at this moment whenever a Token is generated, so as to subsequently use the statistics and calculation functions to calculate the time taken for each Token to be generated from the beginning to the completion; monitoring the average number of Tokens: inserting a counter function at the end of the processing flow of each request, before returning the result to the client, and counting the Tokens generated by this request Total number, so that the statistics and calculation functions can be used later to calculate the average number of Tokens generated according to the request; monitor the total number of requests per second: set a counter function at the function of the large model receiving external requests to record the number of requests per second; monitor the average time of the request: set a time recording function at the function of the large model receiving external requests to record the timestamp of the request arrival, and record the timestamp again when the request is processed and the result is about to be returned, so that the statistics and calculation functions can be used later to calculate the time taken for each request and the average time taken for the request; monitor the total number of long connection requests in process: set a counter function in the logic code block of the large model processing long connection requests, and increase the counter value every time a new long connection is established; when the long connection is closed, reduce the counter value, so that the statistics and calculation functions can be used later to obtain the total number of long connection requests in process; data collection and transmission: during the operation of the large model, the generation time of the first Token, the generation time of the Token, and the average Token are collected at the preset frequency. The collected data will be transmitted to the monitoring system based on the data transmission mechanism provided by the CGRA platform; Anomaly detection and early warning: According to historical data and business needs, alarm thresholds are set for each monitored data, and abnormal warning information is issued when the monitored data exceeds the alarm threshold.

[0006] According to the large model business indicator monitoring method disclosed in the present invention, preferably, the step of data collection and transmission also includes: obtaining monitoring data of hardware resource usage, the hardware resource usage includes at least: CPU usage and memory occupancy; obtaining monitoring data of network resource usage.

[0007] According to the large-model business indicator monitoring method disclosed in the present invention, preferably, it also includes: data processing: decompressing, converting the data format and detecting outliers on the received data in the monitoring system, and storing the processed data in a database or cache for long-term preservation and subsequent analysis.

[0008] According to the large model business indicator monitoring method disclosed in the present invention, preferably, it also includes: monitoring data visualization: using the Echarts chart library to convert the collected monitoring data into a visual chart.

[0009] According to the big model business indicator monitoring method disclosed in the present invention, preferably, it also includes: packaging SDK: packaging the time recording function, counter function and statistical and calculation function into SDK for the big model to call.

[0010] According to the large model business indicator monitoring method disclosed in the present invention, preferably, the abnormal warning information is sent to the mobile terminal via instant messaging software or non-instant messaging software.

[0011] According to the large model business indicator monitoring method disclosed in the present invention, preferably, the large model is a GPT series model.

[0012] The second aspect of the present invention discloses a large model business indicator monitoring device, including: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement a large model business indicator monitoring method such as any of the above technical solutions.

[0013] A third aspect of the present invention discloses a computer-readable storage medium storing a program code, wherein the program code is used to implement a large model business indicator monitoring method as described in any of the above technical solutions.

[0014] The beneficial effects of the present invention include at least: The present invention is highly targeted in monitoring indicators: the first Token generation time, the output Token generation time, etc. are all unique performance parameters of large models, providing clearer and more specific data support for optimizing the operating performance of large models. Secondly, code insertion monitoring is deeply integrated into the model operation process, and key nodes are collected in real time to avoid traditional monitoring delays and deviations. The data is accurate, and the timeliness and credibility are extremely high. Finally, the data transmission technology adapted to the CGRA platform uses the DMA channel and PCIe bus without interfering with CPU operations, ensuring the efficient operation of the monitoring system, and presenting the status of large models in an all-round and high-precision manner, which strongly supports the operation of large models on the CGRA platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic flow chart of a large model business indicator monitoring method according to an embodiment of the present invention is shown.

[0016] Figure 2 A schematic block diagram of a large model business indicator monitoring device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0017] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0019] According to an embodiment of the present invention, a large model business indicator monitoring method is disclosed, comprising:

[0020] Model deployment: deploy the large model to be monitored in the CGRA platform;

[0021] Monitor the generation time of the first token: insert a time recording function into the starting logic code of the text generated by the large model to obtain the starting timestamp;

[0022] Monitor the time it takes to generate a token: Insert a time recording function after the core code block responsible for generating a single token in the large model. Whenever a token is generated, obtain the timestamp at that moment so that the time it takes to generate each token from the beginning to the completion can be calculated using statistics and calculation functions later.

[0023] Monitor the average number of tokens: At the end of each request processing flow, before returning the result to the client, insert a counter function to count the total number of tokens generated by this request, so that the statistics and calculation functions can be used later to calculate the average number of tokens generated according to the request;

[0024] Monitor the total number of requests per second: Set a counter function in the function where the large model receives external requests to record the number of requests per second;

[0025] Monitor the average time taken by requests: Set a time recording function in the function that receives external requests in the large model to record the timestamp of the request arrival. When the request is processed and the result is about to be returned, record the timestamp again so that the statistics and calculation functions can be used later to calculate the time taken by each request and the average time taken by the request.

[0026] Monitor the total number of persistent connection requests being processed: Set a counter function in the logic code block for processing persistent connection requests in the large model. Whenever a new persistent connection is established, increase the counter value. When the persistent connection is closed, decrease the counter value so that the total number of persistent connection requests being processed can be obtained using the statistics and calculation functions later.

[0027] Data collection and transmission: During the operation of the large model, specific data such as the time of first token generation, token generation time, average number of tokens, total number of requests per second, average request time, and total number of long connection requests in process are collected at a preset frequency, and the collected data is transmitted to the monitoring system based on the data transmission mechanism provided by the CGRA platform;

[0028] Data processing: decompress the received data, convert the data format and detect outliers in the monitoring system, and store the processed data in a database or cache for long-term storage and subsequent analysis;

[0029] Visualization of monitoring data: Use Echarts chart library to convert the collected monitoring data into visual charts;

[0030] Anomaly detection and warning: According to historical data and business needs, set alarm thresholds for each monitored data, and issue abnormal warning information when the monitored data exceeds the alarm threshold.

[0031] According to the above embodiment, preferably, it also includes: packaging SDK: packaging the time recording function, counter function and statistical and calculation function into SDK for the large model to call.

[0032] According to the above embodiment, preferably, the step of data collection and transmission also includes: obtaining monitoring data of hardware resource usage, the hardware resource usage at least includes: CPU usage and memory occupancy; obtaining monitoring data of network resource usage.

[0033] According to the above embodiment, preferably, the abnormal warning information is sent to the mobile terminal via instant messaging software or non-instant messaging software.

[0034] According to the above embodiment, preferably, the large model is a GPT series model.

[0035] like Figure 1 As shown, according to another embodiment of the present invention, a specific implementation process of the large model business indicator monitoring method of the above embodiment in an actual application scenario is disclosed, including:

[0036] 1. Determine monitoring parameters and insertion points:

[0037] 1) First Token Generation Time: Insert a monitoring function at the starting logic code of the large model to generate text, usually at the location where the model receives input and starts to generate the first output token. For example, at the entry of the function responsible for processing input data and starting the reasoning process, this is the starting point of the first token generation process, which can accurately capture the starting timestamp.

[0038] 2) Time of generating each output token: Insert a monitoring function after the core code block responsible for generating a single token inside the model. Whenever a token is generated and about to be output, record the time at this time so as to calculate the time taken for the token to be generated from the beginning to the end. This location is usually located in the decoder part of the model, depending on the model architecture. For example, in a Transformer-based decoder, it is inserted after the calculation step of generating each token is completed.

[0039] 3) Average number of tokens generated in a request: At the end of the request processing flow and before returning the result to the client, insert a function to count the total number of tokens generated by this request, and record it in combination with the request identifier for subsequent average calculation. This is usually located at the end of the interface function that the model provides services to the outside world to ensure that all tokens have been generated.

[0040] 4) Total number of requests per second for large model services: At the entry point of the large model service, that is, the function that receives external requests, set a global counter. Whenever a new request comes in, the counter increases by 1. At the same time, start a scheduled task to record the current counter value every second and clear the counter to count the number of requests per second. This entry point is usually the starting point of the interaction between the network communication module and the model to ensure that all requests can be counted.

[0041] Average time consumption of large model service requests: At the entrance of each request to the model service, record the timestamp of the request arrival. When the request is processed and the result is about to be returned, record the timestamp again. The difference between the two is the time consumption of the request. Store these time consumption data in a list or array, and set a function to periodically calculate the average time consumption of all the time consumptions in the list. For example, calculate the average time consumption once every 10 requests are processed. This function can be run regularly in the background.

[0042] 5) The total number of long connection requests being processed by the large model service: In the logic code block of the model processing long connection requests, each time a new long connection is established, the value of a long connection counter is increased; when a long connection is closed, the value of the counter is reduced accordingly. And where the monitoring system needs to obtain the current number of long connections, insert a function to return the current value of this counter, which is usually implemented in the interface function where the monitoring data acquisition module interacts with the model.

[0043] 2. Function design and implementation:

[0044] 1) Time recording function: used to record various time nodes, such as the first token generation time, each token generation time, request arrival and end time, etc. Use the high-precision time acquisition API provided by the system, for example, in C++, you can use the std::chrono library to obtain nanosecond timestamps, and store the obtained time in global variables or specific data structures for subsequent calculations and statistics.

[0045] 2) Counter function: For counting requirements such as the total number of requests and the total number of long connection requests, simple add 1 and subtract 1 functions are designed to ensure the atomic operation of the counter and avoid data conflicts in a multi-threaded environment. In Python, threading.Lock can be used to implement a thread-safe counter, and in C++, std::atomic type can be used to ensure atomic operations.

[0046] 3) Statistics and calculation functions: used to calculate indicators such as the average number of tokens and average time consumption. These functions read the required data from the data storage structure and perform calculations according to the corresponding mathematical formulas. For example, average time consumption = total time consumption / number of requests, average number of tokens = total number of tokens / number of requests. The results are stored in the specified location for subsequent data collection and transmission.

[0047] 3. Package SDK and large model integrated monitoring module:

[0048] The above functions are encapsulated as an SDK for large models to call, and at the same time, the large models are adapted to the CGRA platform (since the existing large models are all based on CUDA, necessary adjustments are required to run on the CGRA platform).

[0049] 4. Data collection and transmission:

[0050] During the model operation, the data collection function is triggered at a certain frequency (for example, every 10 seconds). This function traverses all global variables, numbers and other data storage structures that record business parameters, collects the values ​​of various business parameters at the current moment, and forms a data structure or array.

[0051] When the collected data is transmitted, the efficient data transmission mechanism provided by the CGRA platform is utilized without CPU intervention, which greatly improves the data transmission speed and can quickly move the collected data to the back-end storage or analysis unit of the monitoring system.

[0052] 5. Data processing and visualization:

[0053] At the back end of the monitoring system, the received data is further processed, including decompression, data format conversion, outlier detection, etc. The processed data is stored in a database or cache for long-term preservation and subsequent analysis.

[0054] The visualization interface uses the chart library Echarts to display the statistical traditional business parameters (server indicators and network indicators) and large model business parameters (first token generation time, each output token generation time, etc.) in an intuitive way. For example, a line chart is used to show the change trend of the total number of requests per second over time, and a bar chart is used to compare the average request time in different time periods, etc., to provide operation and maintenance personnel and developers with clear large model operation status information.

[0055] 6. Anomaly detection and early warning:

[0056] According to historical data and business needs, set a reasonable threshold range for each business parameter. For example, if the average request time exceeds a certain set value, it will be considered abnormal, and if the number of requests per second suddenly drops sharply or increases by more than a certain percentage, it will trigger an early warning.

[0057] The collected business parameter values ​​are compared with the thresholds in real time. Once an abnormality is found, the early warning mechanism is activated immediately. The operation and maintenance personnel can be notified via email or text message. At the same time, the abnormal area is highlighted on the visual interface, and the time when the abnormality occurred, parameter values ​​and other information are recorded in detail to facilitate quick troubleshooting.

[0058] In this embodiment, the specific process of function customization and packaging and model business parameter monitoring is mainly disclosed. Function customization and packaging are mainly to implement different monitoring functions for different large model architectures, which is convenient for users to call and integrate. Model business parameter monitoring is to conduct unified monitoring and early warning for the large model access visualization platform that has been adapted.

[0059] like Figure 2 As shown, according to another embodiment of the present invention, a large model business indicator monitoring device 200 is also disclosed, including: a memory 201, used to store program instructions; a processor 202, used to call the program instructions stored in the memory to implement the large model business indicator monitoring method as in the above embodiment.

[0060] According to another embodiment of the present invention, a computer-readable storage medium is disclosed. The computer-readable storage medium stores program code, and the program code is used to implement the large model business indicator monitoring method as described in the above embodiment.

[0061] In summary, according to the technical solutions provided by the above embodiments, the present invention can accurately monitor model performance: monitor the running details of the large model on the CGRA platform, accurately grasp the text generation efficiency of the model by monitoring the parameters such as the first token generation time and the generation time of each output token, and assist developers in evaluating the execution speed of the model architecture and algorithm on the specific platform, so as to optimize the model in a targeted manner and improve the model response capability. It helps to optimize service resource configuration: focus on the monitoring of indicators such as the total number of requests per second for large model services, the average time of requests, and the total number of long connection requests processed, in order to allow operation and maintenance personnel to clearly understand the service load, dynamically allocate CGRA platform resources based on real-time data, avoid idle or overloaded resources, ensure the smoothness and stability of services, and reduce operating costs. Help improve service quality: pay attention to the average number of tokens generated in the request to measure the richness of the model output content, and combine other indicators to comprehensively judge whether the service meets user needs, help optimize the input and output logic of the model, and enhance user satisfaction with the interaction with the large model.

[0062] The present invention can at least achieve the following technical effects:

[0063] 1. Performance improvement: With the continuous monitoring and analysis of key business parameters, the development team can quickly locate the bottleneck of model operation. For example, if the delay in generating the first token in a certain period of time is too high, the problem of decoder code or data input can be promptly checked. After implementing improvement measures, the overall response time can be significantly shortened and the model processing efficiency can be improved.

[0064] 2. Efficient use of resources: Operation and maintenance personnel accurately adjust the allocation of computing, storage and other resources on the CGRA platform based on the total number of requests per second and the average time consumption trend, ensure that key business requests are given priority during peak traffic, and reasonably recycle idle resources during low traffic to reduce unnecessary hardware expansion investment.

[0065] 3. Enhanced service reliability: Through real-time anomaly detection and early warning mechanisms, once an abnormal situation occurs, such as the average request time far exceeding the threshold, the number of requests per second changes suddenly, etc., the operation and maintenance personnel can be notified immediately to intervene and investigate, quickly repair the problem, and minimize the risk of service interruption, ensuring the continuous and stable operation of large model services and reducing business losses caused by failures.

[0066] All or part of the steps in the various methods of the above embodiments can be completed by controlling the relevant hardware through a program, and the program can be stored in a readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large model business indicator monitoring method, characterized in that: include: Model deployment: deploy the large model to be monitored in the CGRA platform; Monitor the generation time of the first token: insert a time recording function at the starting logic code of the text generated by the large model to obtain the starting timestamp; Monitor the time it takes to generate a token: Insert a time recording function after the core code block responsible for generating a single token in the large model. Whenever a token is generated, obtain the timestamp at that moment so that the time it takes to generate each token from the beginning to the completion can be calculated using statistics and calculation functions later. Monitor the average number of tokens: At the end of each request processing flow, before returning the result to the client, insert a counter function to count the total number of tokens generated by this request, so that the statistics and calculation functions can be used later to calculate the average number of tokens generated according to the request; Monitor the total number of requests per second: set a counter function at the function of the large model receiving external requests to record the number of requests per second; Monitor the average time taken by requests: set a time recording function at the function of the large model receiving external requests to record the timestamp of the request arrival. When the request is processed and the result is about to be returned, record the timestamp again so that the statistics and calculation functions can be used to calculate the time taken by each request and the average time taken by the request later. Monitor the total number of persistent connection requests being processed: set a counter function in the logic code block for processing persistent connection requests in the large model, and increase the value of the counter each time a new persistent connection is established; decrease the value of the counter when the persistent connection is closed, so that the total number of persistent connection requests being processed can be obtained using the statistics and calculation functions later; Data collection and transmission: During the operation of the large model, specific data such as the time when the first Token is generated, the time it takes to generate the Token, the average number of Tokens, the total number of requests per second, the average time it takes to request, and the total number of long connection requests being processed are collected at a preset frequency, and the collected data is transmitted to the monitoring system based on the data transmission mechanism provided by the CGRA platform; Anomaly detection and warning: According to historical data and business needs, set alarm thresholds for each monitored data, and issue abnormal warning information when the monitored data exceeds the alarm threshold.

2. The large model business indicator monitoring method according to claim 1 is characterized in that: The step of data collection and transmission also includes: Acquire monitoring data of hardware resource usage, wherein the hardware resource usage includes at least: CPU usage and memory occupancy; Obtain monitoring data on network resource usage.

3. The large model business indicator monitoring method according to claim 1 is characterized in that: Also includes: Data processing: In the monitoring system, the received data is decompressed, converted into data format and processed for outlier detection, and the processed data is stored in a database or cache for long-term preservation and subsequent analysis.

4. The large model business indicator monitoring method according to claim 1 is characterized in that: Also includes: Visualization of monitoring data: Use the Echarts chart library to convert the collected monitoring data into visual charts.

5. The large model business indicator monitoring method according to claim 1 is characterized in that: Also includes: Encapsulating SDK: Encapsulating the time recording function, the counter function and the statistics and calculation function into SDK for calling by the large model.

6. The large model business indicator monitoring method according to any one of claims 1 to 5, characterized in that: The abnormal warning information is sent to the mobile terminal via instant messaging software or non-instant messaging software.

7. The large model business indicator monitoring method according to any one of claims 1 to 5, characterized in that: The large model is a GPT series model.

8. A large-model business indicator monitoring device, characterized in that: include: A memory for storing program instructions; A processor, used to call the program instructions stored in the memory to implement the large model business indicator monitoring method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code is used to implement the large model business indicator monitoring method according to any one of claims 1 to 7.

Images