Large model reasoning service processing method and system, storage medium and product
By configuring agent processes, pre-filled instances, and decoding instances in the large model inference service, stage decoupling is achieved and GPU resources are utilized, solving the problem of insufficient GPU utilization and improving service stability and performance.
Patent Information
- Application Number
- CN202511684387.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-17
- Publication Date
- 2026-02-17
AI Technical Summary
When the large model inference service is decoupled into two stages, preprocessing and decoding, the GPU utilization is not fully utilized, resulting in excessive load, even crashes, insufficient performance, and unstable service.
By configuring proxy processes, pre-filled instances, and decoding instances, the phase decoupling of large model inference services is achieved. GPU resources are used for vector computation and token generation, and a rate limiter is used to smooth and shape burst traffic to ensure the consistency of data processing capabilities.
It improves GPU resource utilization, enhances the stability of multi-node operation, avoids problems such as excessive load and crashes, and improves service performance.
Smart Images
Figure CN121542036A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of large model inference service technology, and in particular to a large model inference service processing method, system, storage medium and product. Background Technology
[0002] Currently, the process of large model inference services can be decoupled into two stages: preprocessing and decoding. The preprocessing stage is mainly implemented by GPUs (Graphics Processing Units), while the decoding stage is mainly implemented by CPUs. Due to the different decoupling efficiencies of the two stages, the data processing speeds of the two stages differ. As a result, the GPU utilization is not fully utilized when large model inference services are decoupled into preprocessing and decoding stages. In addition, the inconsistent capabilities of the vector data generated in the preprocessing stage and the vector data consumed in the decoding stage can lead to excessive load, or even crashes, causing insufficient performance and instability of the inference service. Summary of the Invention
[0003] The main purpose of this application is to provide a method, system, storage medium and product for processing large model inference services, which aims to improve the GPU resource utilization of large model inference services and effectively improve the stability of multi-node operation.
[0004] To achieve the above objectives, this application proposes a method for processing large-model inference services, applied to a large-model inference service system, wherein the large-model inference service system includes an agent process, a pre-filled instance, and a decoding instance, and the method includes: In response to receiving at least one external service request, the proxy process forwards the external service request to the pre-populated instance; The pre-filled instance performs vector calculation for the inference service based on the received service request, and stores the calculated vector in the inference service storage engine. In response to receiving a message that the pre-filled instance has completed vector computation, the proxy process forwards the external service request to the decoding instance; The decoding instance generates a token by calling the vector stored in the inference service storage engine according to the external service request, and returns the generated token to the external requester through the proxy process.
[0005] In one embodiment, the step of the proxy process forwarding the external service request to the pre-populated instance includes: The proxy process modifies the token parameter of the external service request and sends the service request with the modified token parameter to the pre-populated instance.
[0006] In one embodiment, the proxy process modifies the token parameter of the external service request by: The proxy process modifies the token parameter of the external service request to the first value.
[0007] In one embodiment, the proxy process is equipped with a rate limiter, and the method further includes: The external service requests are smoothed and shaped by the rate limiter set on the proxy process, so as to limit the burst traffic within the upper limit of the decoding instance caching and inference service capabilities.
[0008] In one embodiment, the rate limiter is a token bucket algorithm rate limiter, and the step of smoothing and shaping the external service requests through the rate limiter set on the proxy process includes: Based on the number of tokens configured in the rate limiter, tokens are allocated to the external service requests, with one token corresponding to each external service request; External service requests that have not been assigned a token are placed in the waiting queue; External service requests that have been assigned a token are processed using a round-robin strategy.
[0009] In one embodiment, the method further includes: The length of the waiting queue is dynamically adjusted based on the number of external service requests currently being processed and the current instance deployment status, and / or the virtual service resources or instance configuration resources are dynamically adjusted.
[0010] In one embodiment, the method further includes: In response to receiving at least one external service request, the inference service is decoupled into a preprocessing stage and a decoding stage via the KV-Connector. In the preprocessing stage, vector computation of the inference service is performed, and in the decoding stage, token generation of the inference service is performed.
[0011] In one embodiment, the step of storing the computed vector to the inference service storage engine by the pre-filled instance includes: The pre-filled instance stores the calculated vector to the inference service storage engine via the KV-Connector; The step of the decoding instance generating a token by calling the vector stored in the inference service storage engine according to the external service request includes: The decoding instance generates a token by calling the vector stored in the inference service storage engine through the KV-Connector, based on the external service request.
[0012] This application also proposes a large-model inference service system, the system comprising: an agent process, a pre-filled instance, and a decoding instance, wherein: The proxy process is used to forward the external service request to the pre-populated instance in response to receiving at least one external service request; The pre-filled instance is used to perform vector calculation for inference service based on the received service request, and store the calculated vector in the inference service storage engine. The proxy process is also configured to forward the external service request to the decoding instance in response to receiving a message that the pre-filled instance has completed vector calculation; The decoding instance is used to generate a token by calling the vector stored in the inference service storage engine according to the external service request, and to return the generated token to the external requester through the proxy process.
[0013] This application also proposes a large model inference service processing system, the system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large model inference service processing method described above.
[0014] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the large model inference service processing method described above.
[0015] This application also proposes a computer program product comprising a computer program that, when executed by a processor, implements the steps of the large model inference service processing method described above.
[0016] One or more technical solutions proposed in this application have at least the following technical effects: This application provides a method, system, storage medium, and product for processing large model inference services. In response to receiving at least one external service request, the proxy process forwards the external service request to a pre-filling instance. The pre-filling instance performs vector calculations for the inference service based on the received service request and stores the calculated vectors in the inference service storage engine. In response to receiving a message that the pre-filling instance has completed vector calculations, the proxy process forwards the external service request to the decoding instance. The decoding instance, based on the external service request, calls the vectors stored in the inference service storage engine to generate a token and returns the generated token to the external requester through the proxy process. This technical solution, by configuring GPU resources, a pre-filling instance for the large model inference service, a decoding instance for the large model inference service, and a proxy process for the large model inference service, achieves phase decoupling of the large model inference service. The large model inference service is executed in two independent processes, which rely on high-speed communication with the GPU card. This avoids the problem of insufficient inference service performance and instability caused by excessive load or even crashes due to inconsistent vector data generation capabilities of the pre-processing instance and vector data consumption capabilities of the decoding stage. This effectively improves the GPU resource utilization of the large model inference service and enhances the stability of multi-node operation. Attached Figure Description
[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A flowchart illustrating the first embodiment of the large model inference service processing method of this application; Figure 2 This is a schematic diagram of the architecture of the large model inference service system involved in the embodiments of the large model inference service processing method of this application; Figure 3 A schematic diagram of the current limiter principle involved in the embodiment of the large model inference service processing method of this application; Figure 4 This is a schematic diagram of the modules of the large model inference service system according to an embodiment of this application; Figure 5 This is a schematic diagram of the system architecture of the hardware operating environment involved in the large model inference service processing method in the embodiments of this application.
[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0023] The main solution of this application embodiment is as follows: In response to receiving at least one external service request, the proxy process forwards the external service request to the pre-filling instance; the pre-filling instance performs vector calculation for the inference service according to the received service request, and stores the calculated vector in the inference service storage engine; in response to receiving a message that the pre-filling instance has completed vector calculation, the proxy process forwards the external service request to the decoding instance; the decoding instance, according to the external service request, calls the vector stored in the inference service storage engine to generate a token, and returns the generated token to the external requester through the proxy process. This technical solution, by configuring GPU resources, the pre-filling instance of the large model inference service, the decoding instance of the large model inference service, and the proxy process of the large model inference service, achieves phase decoupling of the large model inference service. The large model inference service is executed in two independent processes, which rely on the high-speed communication of the GPU card. A proxy instance with rate limiting function combined with the system's inference capabilities is designed, which effectively improves the GPU resource utilization of the large model inference service and effectively improves the stability of multi-node operation.
[0024] This embodiment takes into account that in the prior art, during the process of large model inference service, the preprocessing stage is mainly implemented by GPU and the decoding stage is mainly implemented by CPU. Due to the different decoupling efficiency of the two stages, the data processing speed of the two stages is different. As a result, when the large model inference service is decoupled into two stages of preprocessing and decoding, the GPU utilization is not fully utilized. In addition, the ability of the preprocessing stage to generate vector data and the ability of the decoding stage to consume vector data are inconsistent, which can lead to excessive load or even crash, causing problems such as insufficient performance and instability of the inference service.
[0025] Therefore, this application provides a solution that can improve the GPU resource utilization of large model inference services and effectively improve the stability of multi-node operation.
[0026] Based on this, the embodiments of this application provide a method for processing large model inference services, referring to... Figure 1 , Figure 1This is a flowchart illustrating the first embodiment of the large-model inference service processing method of this application.
[0027] In this embodiment, the large model inference service processing method includes steps S10 to S40. The following provides a detailed explanation of each step.
[0028] like Figure 1 As shown, the first embodiment of this application proposes a method for processing large model inference services, applied to a large model inference service system. The large model inference service system includes an agent process, a pre-filled instance, and a decoding instance. The method includes: Step S10: In response to receiving at least one external service request, the proxy process forwards the external service request to the pre-populated instance; The technical solution of this embodiment can improve the GPU resource utilization of large model inference services and effectively improve the stability of multi-node operation.
[0029] The execution entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, capable of performing large model inference service-related processing. Examples include tablet computers, personal computers, mobile phones, and other terminals or servers, or an electronic device or large model inference service system capable of performing the aforementioned functions. The following description uses a large model inference service system as an example to illustrate this embodiment.
[0030] In this embodiment, the large model inference service process can be decoupled into a preprocessing stage and a decoding stage. The preprocessing stage performs vector calculations for the inference service, while the decoding stage generates the inference service token. These preprocessing and decoding stages are used to process external service requests.
[0031] In this embodiment, the preprocessing stage and the decoding stage of the large model inference service are executed in two independent processes. The two processes rely on the high-speed communication of the GPU card and use the resources provided by the GPU to realize the corresponding inference service data processing.
[0032] In the large model inference service, GPU resources are mainly responsible for using GPU computing power to perform large model inference calculations, including hardware capabilities and software deployment capabilities such as model loading, prompt word parsing, vector data calculation, and data transmission.
[0033] In this embodiment, the large model inference service system includes an agent process, a pre-filled instance, and a decoding instance, wherein: The pre-filled instance processing stage is the preprocessing stage. For each inference task, after the service request input, the prompt words are converted into vector semantics, the vector calculation of the inference service is performed through the pre-filled instance, and the calculated vector is stored in the inference service storage engine.
[0034] The processing phase of the decoding instance is the decoding phase. For each inference task, the token generation of the inference service is performed. The token is generated by calling the vector prediction stored in the inference service storage engine. The token generation process ends when the termination condition is met, based on the parameters of the original service request, and the output result is returned.
[0035] The proxy process runs through the entire inference service process. For decoupled inference services, external service requests are forwarded through a proxy process. The proxy process modifies the token parameter of the request to a preset value (such as 1) and then forwards it to the pre-fill instance. After the pre-fill instance completes the calculation, it returns a completion message to the proxy process. The proxy process then forwards the original service request to the decoding instance. After the decoding instance completes the token generation, it returns the result to the proxy process. The proxy process then returns the result to the external requester.
[0036] Among them, external requesters can request large model inference services based on their own circumstances, and send external service requests to the large model inference service system.
[0037] Among them, external service requests can be user queries used to request the number of tokens generated at the end.
[0038] In the large model inference service, a token is the basic unit of text, a fragment for the large model to understand. The large model breaks down the input text into smaller, meaningful fragments, which are the tokens. For English, a token might be a word (such as "apple") or a root word (such as "un", "believe", "able"). For Chinese, a single Chinese character is usually considered a token, but complex words or phrases may be broken down into multiple tokens.
[0039] Large models can be large language models (LLMs), which are artificial intelligence models with a large number of parameters. They are trained on large amounts of data using deep learning algorithms to understand and generate natural language. These models can perform various language tasks, such as text generation, translation, question answering, and text summarization.
[0040] In this context, user prompts can be text entered into these large models by the user to guide the model in generating specific outputs. Prompts can range from simple questions or instructions to more complex text containing multiple instructions or contextual information.
[0041] In this embodiment, the user can send a service request to the large model inference service system to request the final number of tokens generated.
[0042] As one implementation method, the large model inference service system can receive external service requests through a proxy process.
[0043] Among them, the large model inference service system can handle multiple external service requests simultaneously.
[0044] As one implementation method, after receiving multiple external service requests, the proxy process can process multiple external service requests in a polling manner to avoid the risk of service timeouts or system crashes caused by request accumulation when running on multiple nodes.
[0045] In one implementation, in response to receiving at least one external service request, the proxy process forwards the external service request to the pre-filled instance, which then performs vector computation for the inference service.
[0046] In one implementation, the external service request may include a token parameter. After receiving the external service request, the proxy process can modify the token parameter of the external service request and send the service request with the modified token parameter to the pre-populated instance.
[0047] The proxy process modifies the token parameter of external service requests primarily to limit the number of tokens a user can request.
[0048] As one implementation method, the proxy process can modify the token parameter of external service requests using the following approach: The proxy process modifies the token parameter of the external service request to a first value. Optionally, the first value can be 1 or other set values, which are not limited in this embodiment.
[0049] As one implementation method, the large model inference service system can also set a maximum number of tokens that can be requested to improve system performance and alleviate GPU resource pressure.
[0050] Step S20: The pre-filled instance performs vector calculation for the inference service based on the received service request, and stores the calculated vector in the inference service storage engine. Specifically, after receiving a service request forwarded by the agent process, the pre-filled instance performs vector calculation for the inference service based on the received service request and stores the calculated vector in the inference service storage engine.
[0051] As one implementation, the pre-filled instance can process multiple service requests simultaneously, perform inference service vector calculations on the received multiple service requests, and then store the calculated vectors in the inference service storage engine.
[0052] Based on GPU resources, the system can set the upper limit of data that the inference service storage engine can store.
[0053] Pre-filled instances perform vector computations for inference services in response to service requests. These vector data are stored in the inference service storage engine. If the stored data exceeds the storage limit of the inference service storage engine, the inference service storage engine will usually clean up historical data to ensure the storage of current data.
[0054] As one implementation method, after completing the vector calculation for the inference service of the service request, the pre-filled instance sends a message to the proxy process indicating that the vector calculation is complete.
[0055] Step S30: In response to receiving a message that the pre-filled instance has completed vector calculation, the proxy process forwards the external service request to the decoding instance; Specifically, after receiving the message from the pre-filled instance indicating that the vector computation is complete, the proxy process forwards the original external service request to the decoding instance, which then generates the token for the inference service during the decoding phase.
[0056] In step S40, the decoding instance, based on the external service request, calls the vector stored in the inference service storage engine to generate a token, and returns the generated token to the external requester through the proxy process.
[0057] Specifically, the decoding instance generates a token by calling the vector stored in the inference service storage engine according to the external service request. The decoding instance ends the token generation process when the termination condition is met, based on the parameters of the original external service request, and sends the output result to the proxy process. The proxy process then returns the generated token to the external requester.
[0058] This embodiment, through the above-described scheme, specifically involves the following steps: upon receiving at least one external service request, the proxy process forwards the external service request to the pre-filling instance; the pre-filling instance performs vector calculations for the inference service based on the received service request and stores the calculated vectors in the inference service storage engine; upon receiving a message that the pre-filling instance has completed vector calculations, the proxy process forwards the external service request to the decoding instance; the decoding instance, based on the external service request, calls the vectors stored in the inference service storage engine to generate a token, and returns the generated token to the external requester through the proxy process. This technical solution, by configuring GPU resources, the pre-filling instance of the large model inference service, the decoding instance of the large model inference service, and the proxy process of the large model inference service, achieves phase decoupling of the large model inference service. Furthermore, the large model inference service is executed in two independent processes, which rely on high-speed communication with the GPU card. This avoids the problem of insufficient inference service performance and instability caused by excessive load or even crashes due to inconsistent vector data generation capabilities of the preprocessing instance and consumption capabilities of the decoding stage. This effectively improves the GPU resource utilization of the large model inference service and enhances the stability of multi-node operation.
[0059] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter.
[0060] In this embodiment, the large model inference service system ensures efficient communication between data storage nodes (i.e., inference service storage engine) and preprocessing nodes (pre-filled instances), and between data storage nodes and decoding nodes (i.e., decoding instances) through corresponding communication media or connectors.
[0061] like Figure 2 As shown, the large model inference service system architecture involved in this embodiment includes: GPU resources, several (e.g., X) pre-filled instances of large model inference services, several (e.g., Y) decoding instances of large model inference services, proxy processes (or proxy instances) of large model inference services, communication media (communication methods implemented through various hardware connections) or connectors between instances, and vector data storage media (storage methods implemented through various storage media, such as the large model inference service storage engine). Through the above-mentioned large model inference service system architecture, the stage decoupling of the large model inference service is realized, effectively solving the problem of improving the GPU resource utilization of the large model inference service and effectively improving the stability of multi-node operation.
[0062] In one implementation, pre-populated instances can communicate with the inference service storage engine via a KV-Connector.
[0063] In one implementation, the decoding instance and the inference service storage engine can communicate via a KV-Connector.
[0064] In one implementation, each pre-populated instance is configured with a Large Model Framework (VLLM) and a Large Model Storage Manager (storageManager). Each pre-populated instance interacts with the distributed storage engine parent node (Master Service) through the Large Model Framework and the Large Model Storage Manager, and interacts with the large model inference service storage engine through the distributed storage engine parent node.
[0065] In one implementation, each decoding instance is configured with a Large Model Framework (VLLM) and a Large Model Storage Manager (storageManager). Each decoding instance interacts with the distributed storage engine parent node (Master Service) through the Large Model Framework and the Large Model Storage Manager, and interacts with the large model inference service storage engine through the distributed storage engine parent node.
[0066] The inference service storage engine communicates with the parent node of the distributed storage engine through a communication medium or connector.
[0067] Specifically, the communication between pre-populated instances and the inference service storage engine is exemplified by the KV-Connector, and the communication between decoding instances and the inference service storage engine is also exemplified by the KV-Connector. As one implementation method, the large model inference service system decouples the inference service into a preprocessing stage and a decoding stage through the KV-Connector, which can improve the first token response speed and throughput of the inference service.
[0068] Specifically, as one implementation, when the large model inference service system receives at least one external service request, it decouples the inference service into a preprocessing stage and a decoding stage through a KV-Connector. In the preprocessing stage, vector calculation for the inference service is performed, and in the decoding stage, token generation for the inference service is performed.
[0069] Specifically, as one implementation, step S10, in response to receiving at least one external service request, the proxy process forwards the external service request to the pre-populated instance, may specifically include: In response to receiving at least one external service request, the agent process decouples the inference service into a preprocessing stage and a decoding stage through the KV-Connector, modifies the token parameter of the external service request, and sends the service request with the modified token parameter to the pre-populated instance.
[0070] Specifically, as one implementation, step S20, where the pre-filled instance performs vector calculation for the inference service based on the received service request, and stores the calculated vector in the inference service storage engine, may specifically include: The pre-filled instance performs vector calculation for the inference service based on the received service request, and stores the calculated vector to the inference service storage engine through the KV-Connector.
[0071] Specifically, as one implementation, step S40, whereby the decoding instance, based on the external service request, calls the vector stored in the inference service storage engine to generate a token, and returns the generated token to the external requester through the proxy process, may specifically include: The decoding instance generates a token by calling the vector stored in the inference service storage engine through the KV-Connector according to the external service request, and returns the generated token to the external requester through the proxy process.
[0072] In this embodiment, through the above scheme, the proxy process decouples the inference service into a preprocessing stage and a decoding stage via the KV-Connector, and modifies the token parameter of the external service request, sending the service request with the modified token parameter to the pre-filling instance. The pre-filling instance performs vector calculation for the inference service based on the received service request, and stores the calculated vector in the inference service storage engine via the KV-Connector. In response to receiving a message that the pre-filling instance has completed vector calculation, the proxy process forwards the external service request to the decoding instance. The decoding instance, based on the external service request, calls the vector generator t stored in the inference service storage engine via the KV-Connector. The generated token is returned to the external requester through the proxy process. The technical solution of this application achieves phase decoupling of the large model inference service by configuring GPU resources, pre-filled instances of the large model inference service, decoding instances of the large model inference service, and proxy processes of the large model inference service. The large model inference service is executed in two independent processes, which rely on the high-speed communication of the GPU card. Through the KV-Connector connector, efficient communication between the data storage node (i.e., the inference service storage engine) and the pre-processing node (pre-filled instance), and between the data storage node and the decoding node (i.e., the decoding instance) is ensured. At the same time, the GPU resource utilization of the large model inference service is effectively improved, and the stability of multi-node operation is effectively improved.
[0073] Based on the first or second embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to the first or second embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The proxy process is equipped with a rate limiter, and the large model inference service processing method further includes: The external service requests are smoothed and shaped by the rate limiter set on the proxy process, so as to limit the burst traffic within the upper limit of the decoding instance caching and inference service capabilities.
[0074] Specifically, during the large model inference service, when the agent process handles external service requests, it can use a rate limiter to smooth and shape the external service requests, limiting burst traffic to the upper limit of the decoding node's cache and service capacity, which greatly improves the operational stability of multiple nodes.
[0075] In one embodiment, the rate limiter can be a token bucket algorithm rate limiter.
[0076] In one embodiment, the system can configure the rate limiter with a corresponding number of tokens based on the actual situation of GPU resources, and one token can correspond to one external service request.
[0077] As one implementation method, the process of smoothing and shaping external service requests through a rate limiter set on the proxy process may include the following technical solutions: First, after receiving an external service request, the proxy process allocates a token to the external service request according to the number of tokens configured in the rate limiter, with each external service request corresponding to one token; if the number of external service requests is too large, the external service requests that have not been allocated a token are placed in a waiting queue. Then, external service requests assigned to tokens are processed using a round-robin strategy.
[0078] Furthermore, in specific implementation, the length of the waiting queue can be dynamically adjusted, and / or the virtual service resources or instance configuration resources can be dynamically adjusted, based on the number of external service requests currently being processed and the current instance deployment status.
[0079] For example, if the number of tokens for the rate limiter is configured to be 10, the proxy process will allocate tokens to 10 of the multiple external service requests received, based on the order in which the requests are received. That is, the first 10 requests can get tokens, and the remaining requests have to wait. In other words, requests that have not received tokens are placed in a waiting queue, and the length of the waiting queue can be customized.
[0080] As one implementation method, to avoid users waiting too long, if there are too many external service requests, the system can remind the backend server through alarms or other means. In this way, the monitoring party can expand the virtual service according to the queue waiting situation to meet the request processing needs.
[0081] For external service requests that have obtained a token, the proxy process can start processing them using a polling strategy.
[0082] In the case of deploying multiple pre-filled instances and decoding instances in the system, the corresponding instance resources can be allocated according to the current request processing situation, and instances can be added adaptively when the number of requests is large to meet the request processing needs.
[0083] It's important to note that without a rate limiter, when there are many external service requests (such as during peak periods), all requests entering the system can cause congestion or even crashes. For example, the proxy process needs to process all requests. The pre-population instance performs vector calculations for the inference service based on the received service requests. The resulting vector data is stored in the inference service storage engine. However, the inference service storage engine has a storage limit. If the stored data exceeds this limit, the inference service storage engine typically cleans up historical data before it's used. This results in the decoding engine not finding the corresponding vector data when calling the inference service storage engine, forcing the system to recalculate the vectors. This repeated vector calculation is not only time-consuming but can also cause system crashes due to insufficient GPU computing power.
[0084] This embodiment, through the above-described scheme, sets a rate limiter in the proxy process to smoothly shape external requests, limiting sudden traffic to the upper limit of the decoding node's cache and service capacity. This can address the problem of single-node request accumulation or crash caused by sudden traffic. Thus, in the phased decoupling process of the large model inference service, by adopting a rate-limiting and node request balancing strategy for the input requests of the proxy process, system congestion or even crashes can be avoided, greatly improving the operational stability of multiple nodes.
[0085] Furthermore, this application also proposes a large-scale model inference service system, such as Figure 4 As shown, the system includes: agent process 10, pre-fill instance 20, and decoding instance 30, wherein: The proxy process 10 is used to forward the external service request to the pre-populated instance 20 in response to receiving at least one external service request; The pre-filled instance 20 is used to perform vector calculation for inference service based on the received service request, and store the calculated vector in the inference service storage engine. The proxy process 10 is also configured to forward the external service request to the decoding instance 30 in response to receiving a message that the pre-filled instance 20 has completed vector calculation; The decoding instance 30 is used to generate a token by calling the vector stored in the inference service storage engine according to the external service request, and to return the generated token to the external requester through the proxy process 10.
[0086] The large model inference service system provided in this application, employing the large model inference service processing method described in the above embodiments, avoids the problem of insufficient inference service performance and instability caused by excessive load or even crashes due to inconsistent capabilities in generating vector data during preprocessing and consuming vector data during decoding. This effectively improves the GPU resource utilization of the large model inference service and enhances the stability of multi-node operation. Compared with existing technologies, the beneficial effects of the large model inference service system provided in this application are the same as those of the large model inference service processing method described in the above embodiments, and other technical features of the large model inference service system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0087] Furthermore, to achieve the above objectives, this application also proposes a large model inference service processing system, the system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large model inference service processing method described above.
[0088] The following is for reference. Figure 5 This document illustrates a structural diagram suitable for implementing a large-model inference service processing system according to embodiments of this application. The large-model inference service processing system in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The large model inference service processing system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0089] like Figure 5As shown, the large model inference service processing system may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory 1002 or a program loaded from storage device 1003 into random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the large model inference service processing system. The processing device 1001, read-only memory 1002, and random access memory 1004 are interconnected via bus 1005. Input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the large model inference service processing system to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a large model inference service processing system with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.
[0090] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0091] The large model inference service processing system provided in this application, employing the large model inference service processing method described in the above embodiments, can avoid the problems of insufficient inference service performance and service instability caused by excessive load or even crashes due to inconsistent capabilities in generating vector data during preprocessing and consuming vector data during decoding. It effectively improves the GPU resource utilization of the large model inference service and enhances the stability of multi-node operation. Compared with the prior art, the beneficial effects of the large model inference service processing system provided in this application are the same as those of the large model inference service processing method provided in the above embodiments, and other technical features of this large model inference service processing system are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0092] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0093] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the large model inference service processing method described above.
[0094] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0095] The aforementioned computer-readable storage medium may be included in the large model inference service processing system; or it may exist independently and not be assembled into the large model inference service processing system.
[0096] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the large model inference service processing system, the large model inference service processing system: in response to receiving at least one external service request, the proxy process forwards the external service request to a pre-filled instance; the pre-filled instance performs vector calculation for the inference service based on the received service request and stores the calculated vector in the inference service storage engine; in response to receiving a message that the pre-filled instance has completed vector calculation, the proxy process forwards the external service request to the decoding instance; the decoding instance, based on the external service request, calls the vector stored in the inference service storage engine to generate a token and returns the generated token to the external requester through the proxy process.
[0097] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0099] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0100] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described large model inference service processing method. This avoids the problem of insufficient inference service performance and instability caused by excessive load or even crashes due to inconsistent capabilities in generating vector data during preprocessing and consuming vector data during decoding. It effectively improves the GPU resource utilization of the large model inference service and enhances the stability of multi-node operation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the large model inference service processing method provided in the above embodiments, and will not be elaborated upon here.
[0101] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the large model inference service processing method described above.
[0102] The computer program product provided in this application can avoid the problems of insufficient inference service performance and service instability caused by excessive load or even crashes due to inconsistent capabilities in generating vector data during preprocessing and consuming vector data during decoding. It effectively improves the GPU resource utilization of large model inference services and significantly enhances the stability of multi-node operation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the large model inference service processing method provided in the above embodiments, and will not be elaborated upon here.
[0103] One or more technical solutions proposed in this application have at least the following technical effects: This application provides a method, system, storage medium, and product for processing large model inference services. In response to receiving at least one external service request, the proxy process forwards the external service request to a pre-filling instance. The pre-filling instance performs vector calculations for the inference service based on the received service request and stores the calculated vectors in the inference service storage engine. In response to receiving a message that the pre-filling instance has completed vector calculations, the proxy process forwards the external service request to the decoding instance. The decoding instance, based on the external service request, calls the vectors stored in the inference service storage engine to generate a token and returns the generated token to the external requester through the proxy process. This technical solution, by configuring GPU resources, a pre-filling instance for the large model inference service, a decoding instance for the large model inference service, and a proxy process for the large model inference service, achieves phase decoupling of the large model inference service. The large model inference service is executed in two independent processes, which rely on high-speed communication with the GPU card. This avoids the problem of insufficient inference service performance and instability caused by excessive load or even crashes due to inconsistent vector data generation capabilities of the pre-processing instance and vector data consumption capabilities of the decoding stage. This effectively improves the GPU resource utilization of the large model inference service and enhances the stability of multi-node operation.
[0104] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A method for processing large-scale model inference services, characterized in that, Applied to a large-model inference service system, the large-model inference service system including an agent process, a pre-filled instance, and a decoding instance, the method includes: In response to receiving at least one external service request, the proxy process forwards the external service request to the pre-populated instance; The pre-filled instance performs vector calculation for the inference service based on the received service request, and stores the calculated vector in the inference service storage engine. In response to receiving a message that the pre-filled instance has completed vector computation, the proxy process forwards the external service request to the decoding instance; The decoding instance generates a token by calling the vector stored in the inference service storage engine according to the external service request, and returns the generated token to the external requester through the proxy process.
2. The method as described in claim 1, characterized in that, The steps of the proxy process forwarding the external service request to the pre-populated instance include: The proxy process modifies the token parameter of the external service request and sends the service request with the modified token parameter to the pre-populated instance.
3. The method as described in claim 2, characterized in that, The proxy process modifies the token parameter of the external service request in the following ways: The proxy process modifies the token parameter of the external service request to the first value.
4. The method as described in claim 2, characterized in that, The proxy process is equipped with a rate limiter, and the method further includes: The external service requests are smoothed and shaped by the rate limiter set on the proxy process, so as to limit the burst traffic within the upper limit of the decoding instance caching and inference service capabilities.
5. The method as described in claim 4, characterized in that, The rate limiter is a token bucket algorithm rate limiter, and the step of smoothing and shaping the external service requests through the rate limiter set on the proxy process includes: Based on the number of tokens configured in the rate limiter, tokens are allocated to the external service requests, with one token corresponding to each external service request; External service requests that have not been assigned a token are placed in the waiting queue; External service requests that have been assigned a token are processed using a round-robin strategy.
6. The method as described in claim 5, characterized in that, The method further includes: The length of the waiting queue is dynamically adjusted based on the number of external service requests currently being processed and the current instance deployment status, and / or the virtual service resources or instance configuration resources are dynamically adjusted.
7. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to receiving at least one external service request, the inference service is decoupled into a preprocessing stage and a decoding stage via the KV-Connector. In the preprocessing stage, vector computation of the inference service is performed, and in the decoding stage, token generation of the inference service is performed.
8. The method as described in claim 7, characterized in that, The steps for storing the computed vector to the inference service storage engine in the pre-filled instance include: The pre-filled instance stores the calculated vector to the inference service storage engine via the KV-Connector; The step of the decoding instance generating a token by calling the vector stored in the inference service storage engine according to the external service request includes: The decoding instance generates a token by calling the vector stored in the inference service storage engine through the KV-Connector, based on the external service request.
9. A large-scale model inference service system, characterized in that, The system includes: a proxy process, a pre-filled instance, and a decoding instance, wherein: The proxy process is used to forward the external service request to the pre-populated instance in response to receiving at least one external service request; The pre-filled instance is used to perform vector calculation for inference service based on the received service request, and store the calculated vector in the inference service storage engine. The proxy process is also configured to forward the external service request to the decoding instance in response to receiving a message that the pre-filled instance has completed vector calculation; The decoding instance is used to generate a token by calling the vector stored in the inference service storage engine according to the external service request, and to return the generated token to the external requester through the proxy process.
10. A large-scale model inference service processing system, characterized in that, The system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large model inference service processing method as described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the large model inference service processing method as described in any one of claims 1 to 8.
12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the large model inference service processing method as described in any one of claims 1 to 8.