Request processing method and device based on large model, electronic equipment and storage medium

Abstract

The invention provides a request processing method and device based on a large model, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, large models and the like. According to the scheme, in response to a received large model reasoning request, a target service level requirement of the large model reasoning request is determined according to request parameters in the large model reasoning request; determining a target service instance pair from at least one candidate service instance pair matched with the target service level requirement; wherein the target service instance pair comprises a target pre-filling service instance determined based on a pre-filling delay and a target decoding service instance determined based on a load; adopting the target pre-filling service instance and the target decoding service instance to respond to the large model reasoning request based on the target service level requirement to obtain a response result; and sending a response result to the client associated with the large model reasoning request.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of deep learning, large models, and more particularly to a request processing method, apparatus, electronic device, and storage medium based on a large model. Background Technology

[0002] With the continuous development of artificial intelligence technology, large models are widely used in fields such as natural language processing, image generation, speech recognition, and code generation. To support the operation of various intelligent applications, large models typically provide capabilities externally in a service-oriented manner, forming an inference service system oriented towards user requests. Within this framework, the client sends a large model inference request containing input content, triggering the backend system to schedule and process the large model inference request, thereby realizing the practical implementation and interactive application of model capabilities. Summary of the Invention

[0003] This disclosure provides a request processing method, apparatus, electronic device, and storage medium based on a large model.

[0004] According to one aspect of this disclosure, a request processing method based on a large model is provided, comprising:

[0005] In response to receiving a large model inference request, the target service level requirement of the large model inference request is determined based on the request parameters in the large model inference request;

[0006] A target service instance pair is determined from at least one candidate service instance pair that matches the target service level requirement; wherein the target service instance pair includes a target pre-filled service instance determined based on pre-filled latency and a target decoding service instance determined based on load.

[0007] The target pre-filled service instance and the target decoding service instance are used to respond to the large model inference request based on the target service level requirements to obtain the response result;

[0008] The response result is sent to the client associated with the large model inference request.

[0009] According to another aspect of this disclosure, a request processing apparatus based on a large model is provided, comprising:

[0010] The first determining module is used to, in response to receiving a large model inference request, determine the target service level requirement of the large model inference request based on the request parameters in the large model inference request;

[0011] The second determining module is configured to determine a target service instance pair from at least one candidate service instance pair that matches the target service level requirement; wherein the target service instance pair includes a target pre-filled service instance determined based on pre-filled latency and a target decoding service instance determined based on load.

[0012] The processing module is used to respond to the large model inference request based on the target service level requirements using the target pre-filled service instance and the target decoding service instance, so as to obtain a response result;

[0013] The sending module is used to send the response result to the client associated with the large model inference request.

[0014] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method proposed in one aspect of this disclosure.

[0018] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in the foregoing aspect of this disclosure.

[0019] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method proposed in the foregoing aspect of this disclosure.

[0020] The large model-based request processing method, apparatus, electronic device, and storage medium disclosed herein, upon receiving a large model inference request, first determine the target service level requirement of the request based on the request parameters. This enables the system to identify the differentiated quality requirements of different requests in terms of latency, throughput, and priority, providing a basis for subsequent resource scheduling decisions. Then, a target service instance pair is selected from candidate service instance pairs matching the target service level, achieving precise alignment between resource allocation and service quality requirements. The target service instance pair includes a target pre-filled service instance selected based on pre-filled latency and a target decoding service instance selected based on load status. This target service instance pair is then used collaboratively to complete the inference task, ensuring that the large model inference request consistently receives high-quality resource services matching its service level requirements throughout the entire inference process, effectively guaranteeing that the service level requirements are met. Finally, the response result is returned to the associated client to ensure service interactivity and availability.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 1 of this disclosure;

[0024] Figure 2 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 2 of this disclosure;

[0025] Figure 3 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 3 of this disclosure;

[0026] Figure 4 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 4 of this disclosure;

[0027] Figure 5 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 5 of this disclosure;

[0028] Figure 6 This is a schematic diagram illustrating the principle of the request processing method based on a large model provided in the embodiments of this disclosure;

[0029] Figure 7 This is a schematic diagram of the traffic scheduling principle provided in the embodiments of this disclosure;

[0030] Figure 8 This is a schematic diagram illustrating the principle of service instance information collection provided in the embodiments of this disclosure;

[0031] Figure 9 This is a schematic diagram illustrating the adjustment principle of the online service example provided in the embodiments of this disclosure;

[0032] Figure 10 This is a schematic diagram of the structure of the request processing device based on a large model provided in Embodiment Six of this disclosure;

[0033] Figure 11 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0034] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0036] With the widespread application of large-scale models in fields such as natural language processing and computer vision, their inference services have become one of the core workloads in cloud computing and edge computing scenarios. However, a stable and highly available online large-scale model inference service system requires a large amount of machine resources to deploy the model service. The requests for inference services fluctuate significantly, exhibiting distinct peaks and troughs. For example, daytime call volume is much higher than nighttime volume, and weekday call volume is much higher than holiday volume, while resource utilization is extremely low during troughs, resulting in overall low resource utilization and wasted resources. Therefore, how to improve the overall utilization of underlying resources while ensuring the stability of the online inference service system has become an urgent technical problem to be solved.

[0037] In related technologies, to address the problem of low resource utilization in large model inference services, there are mainly two implementation schemes:

[0038] (1) Adopt a tidal strategy for inference service resources, and release excess resources for other uses during periods of low traffic, such as model training and data processing.

[0039] (2) By introducing offline batch inference services, all online modules are shared to fill the trough space.

[0040] However, the above solution (1) requires the development of an additional tidal system, which increases the system complexity. The tidal system involves resource withdrawal and resource entry. At the same time, since the startup and graceful exit of large model services are usually long, the resource overhead is large. Once the online traffic surges, it is impossible to expand the resources in time, resulting in the loss of online services. Although the above solution (2) can solve the resource overhead of solution (1) to a certain extent, the inference engine in solution (2) processes large model inference requests with different service level requirements at the same time (also known as online requests and offline requests). Abnormal data of offline tasks may affect the stability of online services, such as a significant increase in the first token generation time (TTFT) and the time per output token (TPOT), or even cause a service avalanche. At the same time, since all of them reuse online modules, but the service level objectives (SLO) of online requests and offline requests are different, it is impossible to achieve higher throughput under lower SLO requirements.

[0041] In view of at least one of the above-mentioned problems, this disclosure proposes a request processing method, apparatus, electronic device and storage medium based on a large model.

[0042] The following description, with reference to the accompanying drawings, describes a request processing method, apparatus, electronic device, and storage medium based on a large model according to embodiments of the present disclosure.

[0043] Figure 1 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 1 of this disclosure.

[0044] This disclosure illustrates the example of a large-model-based request processing method configured in a large-model-based request processing device. This large-model-based request processing device can be applied to any electronic device so that the electronic device can perform large-model-based request processing functions.

[0045] Among them, electronic devices can be any device with computing capabilities, such as computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0046] like Figure 1 As shown, this request processing method based on a large model may include the following steps:

[0047] Step 101: In response to receiving a large model inference request, determine the target service level requirements of the large model inference request based on the request parameters in the large model inference request.

[0048] In order to achieve differentiated scheduling of large model inference requests with different service level requirements, in this embodiment of the disclosure, when a large model inference request is received from a client, the request parameters carried in the request are parsed. The request parameters may include, but are not limited to, request type, maximum generation length, expected delay, priority label, etc., and then the target service level requirement of the request is determined based on the request parameters.

[0049] It should be noted that the target service level requirement may include either a first service level requirement or a second service level requirement, with the first service level requirement being more stringent than the second service level requirement. Correspondingly, the SLO (Service Level Requirement) for the first service level requirement is more stringent, for example, requiring lower TTFT (Total Validated Time To Value) and TPOT (Total Validated Time To Value); the SLO for the second service level requirement is relatively more lenient, allowing for higher TTFT or TPOT.

[0050] Step 102: Determine the target service instance pair from at least one candidate service instance pair that matches the target service level requirement.

[0051] The target service instance pair includes target pre-filled service instances determined based on pre-filled latency and target decoding service instances determined based on load.

[0052] To improve overall resource utilization, one possible approach is to select an optimal target service instance pair from a pre-selected pool of candidate service instance pairs that meet the target service level requirements, given the known target service level requirements for the inference request. This target service instance pair includes a target pre-filled service instance determined based on pre-fill latency and a target decoding service instance determined based on load. For example, the target service instance pair might include the target pre-filled service instance with the shortest pre-fill latency and the target decoding service instance with the lowest load.

[0053] Step 103: Use the target pre-filled service instance and the target decoding service instance to respond to the large model inference request based on the target service level requirements to obtain the response result.

[0054] To improve resource utilization efficiency and inference performance while ensuring service quality, one possible approach is to have the target pre-filling service instance execute the pre-filling phase of the large model inference request. After completion, the generated key-value cache and other intermediate states are passed to the target decoding service instance, which then continues to execute the decoding phase, ultimately completing the entire inference process. It should be noted that the entire process strictly follows the target service level requirements of the request (such as latency, priority, resource guarantee, etc.) to ensure controllable service quality.

[0055] Step 104: Send the response result to the client associated with the large model inference request.

[0056] To ensure that users can obtain the model output in a timely manner, thereby guaranteeing the interactivity and availability of the service, as a possible approach, after the large model completes the processing of the inference request and generates the final response result, the response result is accurately and reliably returned to the client that initially initiated the request via network communication.

[0057] In summary, upon receiving a large model inference request, the system first determines the target service level requirement based on the request parameters. This allows the system to identify the differentiated quality requirements of different requests in terms of latency, throughput, and priority, providing a basis for subsequent resource scheduling decisions. Then, a target service instance pair is selected from candidate service instance pairs matching the target service level, achieving precise alignment between resource allocation and service quality requirements. The target service instance pair includes a target pre-filled service instance selected based on pre-filled latency and a target decoding service instance selected based on load status. This target service instance pair is then used collaboratively to complete the inference task, ensuring that the large model inference request consistently receives high-quality resource services matching its service level requirements throughout the entire inference process. This effectively guarantees that the service level requirements are met. Finally, the response result is returned to the associated client to ensure service interactivity and availability.

[0058] To clearly illustrate how the target service instance pair is determined from at least one candidate service instance pair that matches the target service level requirement in the above embodiments, this disclosure proposes another request processing method based on a large model.

[0059] Figure 2 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 2 of this disclosure.

[0060] like Figure 2 As shown, this request processing method based on a large model includes the following steps:

[0061] Step 201: In response to receiving a large model inference request, determine the target service level requirements of the large model inference request based on the request parameters in the large model inference request.

[0062] Step 202: From the multiple first service instances in the large model resource pool, determine multiple second service instances that match the target service level requirements.

[0063] In this embodiment of the disclosure, the large model resource pool includes a set of all service instances, namely multiple first service instances. The second service instance is the service instance among the multiple first service instances that matches the target service level requirement. For example, if the target service level requirement is the first service level requirement, that is, the SLO index is more stringent, such as requiring lower TTFT and TPOT, then the second service instance is the service instance that matches the first service level requirement, and can also be called an online service instance. Or, if the target service level requirement is the second service level requirement, that is, the SLO index is more lenient, allowing higher TTFT or TPOT, then the second service instance is the service instance that matches the second service level requirement, and can also be called an offline service instance.

[0064] Step 203: Based on the metadata of each second service instance, determine the target service instance pair from each second service instance.

[0065] To improve the accuracy of resource scheduling and the reliability of service quality, as a possible approach, multiple candidate service instance pairs are identified from the second service instance by analyzing the metadata of the second service instance that matches the target service level requirements. Then, the pre-filling latency performance of the pre-filled service instance and the current load status of the decoding service instance in each candidate service instance pair are comprehensively evaluated, and the optimal combination in terms of response speed and resource availability is selected as the final target service instance pair.

[0066] Step 204: Use the target pre-filled service instance and the target decoding service instance to respond to the large model inference request based on the target service level requirements to obtain the response result.

[0067] It should be noted that the execution process of steps 201 and 204 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0068] In summary, by selecting multiple second service instances that match the target service level requirements from multiple first service instances in the large model resource pool, the system can quickly locate a set of service instances with service capabilities from a large number of available resources, avoiding the inclusion of service instances that do not meet the service quality requirements in the scheduling scope, thereby narrowing the search space and reducing scheduling overhead. Furthermore, based on the metadata of each second service instance, the target service instance pair for processing the request is further determined, realizing refined and dynamic instance matching under the premise of meeting the service level requirements, significantly improving the scheduling efficiency, service quality assurance capability, and resource utilization efficiency of the large model inference system.

[0069] To clearly illustrate how the target service instance pair is determined from each second service instance based on the metadata of each second service instance in the above embodiments, this disclosure proposes another request processing method based on a large model.

[0070] Figure 3 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 3 of this disclosure.

[0071] like Figure 3 As shown, this request processing method based on a large model includes the following steps:

[0072] Step 301: In response to receiving a large model inference request, determine the target service level requirements of the large model inference request based on the request parameters in the large model inference request.

[0073] Step 302: From the multiple first service instances in the large model resource pool, determine multiple second service instances that match the target service level requirements.

[0074] Step 303: Based on the metadata of each second service instance, determine at least one candidate service instance pair from each second service instance.

[0075] The candidate service instance pair includes one candidate pre-filled service instance and one candidate decoding service instance.

[0076] To further improve the accuracy of resource scheduling and overall utilization efficiency, in this embodiment of the disclosure, after a set of second service instances that meet the target service level requirements has been selected, a set of candidate service instance pairs that are qualified to participate in subsequent scheduling are further selected based on the metadata of these instances. Each candidate service instance pair includes a candidate pre-filled service instance and a candidate decoding service instance.

[0077] As an example, based on the metadata of each second service instance, multiple candidate service instances are determined from each second service instance; wherein, the multiple candidate service instances include at least one candidate pre-filled service instance and at least one candidate decoding service instance; the at least one candidate pre-filled service instance and the at least one candidate decoding service instance are paired to obtain at least one candidate service instance pair.

[0078] In other words, in order to further improve the reliability and efficiency of subsequent resource matching, based on the metadata of each second service instance, multiple candidate service instances with good operating status and service capabilities are further selected from the second service instances that have met the target service level requirements.

[0079] To improve the accuracy of screening candidate service instances, the metadata includes at least one of the following: status information, memory block information, memory block size information, and identity information.

[0080] Among them, the status information is used to indicate whether the corresponding second service instance is in an available state; the memory block information is used to indicate the memory block resource usage of the corresponding second service instance; the memory block size information is used to indicate the maximum number of tokens allowed to be stored in each memory block of the corresponding second service instance; and the identity information is used to identify the corresponding second service instance.

[0081] For example, based on the metadata of each second service instance, a health assessment is performed on these service instances that have met the target service level requirements, and abnormal service instances are identified and determined. Abnormal service instances refer to instances that are currently in an abnormal operating state such as failure, overload, response timeout, model loading failure, or communication interruption. These abnormal service instances are filtered out from the set of second service instances, and the multiple service instances that remain after filtering are healthy candidate service instances with service capabilities.

[0082] Subsequently, at least one candidate service instance pair is generated by pairing these candidate pre-filled service instances with candidate decoding service instances. Each candidate service instance pair includes a candidate pre-filled service instance for handling parallel computation of input prompt words, and a candidate decoding service instance for generating output token-by-token.

[0083] It should be noted that, in order to ensure the compatibility of model states and the reachability of communication between instances during the inference process, and to effectively avoid service anomalies caused by version mismatch or network isolation, in this embodiment of the disclosure, pairing can be performed based on the version consistency and connection relationship topology of candidate pre-filled service instances and candidate decoding service instances.

[0084] For example, firstly, version pairing is performed on at least one candidate pre-filled service instance and at least one candidate decoding service instance that have been selected. That is, only pre-filled and decoding instances that run the same model version (such as consistent model hash value and version number) are combined into intermediate service instance pairs to ensure that the key-value cache (KV Cache) generated in the pre-filling stage can be correctly parsed and reused in the decoding stage, avoiding inference errors or abnormal outputs due to inconsistent model structure or parameters. Then, at least one candidate service instance pair is further selected from all intermediate service instance pairs that meet version consistency. Among them, there is a path that can establish a communication connection between the candidate pre-filled service instance and the candidate decoding service instance in the candidate service instance pair to ensure efficient communication during the inference process.

[0085] Step 304: Based on the pre-filling delay of the candidate pre-filled service instance and the load of the candidate decoding service instance in each candidate service instance pair, determine the target service instance pair from at least one candidate service instance pair.

[0086] Furthermore, by comprehensively evaluating the pre-fill latency of candidate pre-filled service instances and the load of candidate decoding service instances in each candidate pair, the optimal pair is selected as the target service instance pair to handle the current large model inference request.

[0087] As an example, for any candidate service instance pair, the candidate service instance pair is scored based on the pre-fill latency of the candidate pre-fill service instance and the load of the candidate decoding service instance in the candidate service instance pair to obtain the scoring result of the candidate service instance pair; based on the scoring results of each candidate service instance pair, each candidate service instance pair is sorted to obtain the sorting order of each candidate service instance pair; based on the sorting order, the target service instance pair is determined from each candidate service instance pair; wherein, the target service instance pair has the shortest pre-fill latency of the candidate pre-fill service instance and the minimum load of the candidate decoding service instance.

[0088] In other words, to achieve intelligent resource matching and improve resource utilization, for each generated candidate service instance pair, a score is assigned based on the pre-fill latency of the candidate pre-fill service instance and the load of the candidate decoding service instance. A shorter pre-fill latency indicates stronger processing capability during the pre-filling stage, while the load of the candidate decoding service instance represents the responsiveness and stability during the decoding stage; a lower load is more conducive to continuous and efficient generation. Based on these two dimensions, the system calculates a score for each candidate service instance pair by setting weighting coefficients or a scoring function. The scores of each candidate service instance pair are then ranked to obtain an order; a higher score indicates that the combination better meets the service requirements of low latency and high stability. Finally, the candidate service instance pair with the lowest pre-fill latency and the lowest decoding load is selected as the target service instance pair.

[0089] Step 305: Use the target pre-filled service instance and the target decoding service instance to respond to the large model inference request based on the target service level requirements to obtain the response result.

[0090] Step 306: Send the response result to the client associated with the large model inference request.

[0091] It should be noted that the execution processes of steps 301 to 302 and steps 305 to 306 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit these processes and will not elaborate further.

[0092] In summary, based on the metadata of each second service instance, at least one candidate service instance pair is selected from the second service instances that meet the target service level requirements. Each candidate service instance pair includes one candidate pre-filled service instance and one candidate decoding service instance, avoiding the inclusion of abnormal or functionally mismatched service instances in the scheduling scope. Furthermore, based on a comprehensive evaluation of the pre-filling latency of the pre-filled service instance and the current load of the decoding service instance in each candidate service instance pair, the target service instance pair is determined. This improves resource utilization and the overall throughput of the system, enhancing the stability and scalability of the large-model online inference system while ensuring service quality.

[0093] To clearly illustrate any of the above embodiments, this disclosure proposes another request processing method based on a large model.

[0094] Figure 4 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 4 of this disclosure.

[0095] like Figure 4As shown, this request processing method based on a large model includes the following steps:

[0096] Step 401: Based on the total number of historical inference requests for the same period in the current time period, predict the peak value of the total number of prediction inference requests in future time periods after the current time period.

[0097] To provide a basis for subsequent resource scaling up and down, historical data from the same period (e.g., the total number of large model inference requests from 10:00 to 11:00 AM each day over the past 7 days) is analyzed to identify the patterns of request volume changes within the same time period (e.g., weekday peaks, holiday troughs, etc.). Based on this trend, a prediction algorithm is used to estimate the peak value (i.e., the highest possible concurrency) of the total number of inference requests in a future period.

[0098] Step 402: Based on the peak value, determine the first number of third service instances that match the first service level requirements in the future time period.

[0099] To ensure that the first service level requirement (high service level request) still receives sufficient dedicated resources during peak traffic periods, one possible approach is to analyze the proportion or number of requests belonging to the first service level requirement after obtaining the peak total number of requests for a future period. This is then combined with the processing capacity of a single service instance (e.g., each instance can support 100 high service level requests / second) to calculate the number of service instances specifically serving high service level requests required to meet the peak load, i.e., the first quantity.

[0100] As an example, the actual operating metrics of each first service instance are collected during the current time period; based on the total number of actual inference requests during the current time period, the actual operating metrics and peak values ​​of each first service instance, the first number of third service instances that match the first service level requirements in future time periods is determined.

[0101] In other words, by collecting the actual operational metrics of each first-level service instance during the current time period (such as GPU utilization, memory usage, average response latency, requests per second, and instance health status), a comprehensive understanding of the resource performance and load capacity of the current large model legend requests serving the first service level requirements is obtained, i.e., the resource performance and load capacity serving online requests. Furthermore, by combining the total number of inference requests actually received during the current time period with the peak request volume for future periods predicted based on historical trends or real-time traffic, a comprehensive assessment of the potential load pressure of online requests in the future is conducted. Further, using the actual processing capacity of each first-level service instance (i.e., the true throughput capacity reflected by operational metrics), capacity estimation is performed to dynamically calculate the first number of third-level service instances (online service instances) required to ensure future service quality (such as low latency and high availability) and matching the first service level requirements. Thus, by integrating real-time operational metrics, actual request volume, and predicted peak values, the number of online service instances required for future periods is dynamically determined, ensuring dynamic coordination between resource supply and business requests, thereby improving the intelligent scheduling level, service stability, and resource utilization efficiency of the large model inference system.

[0102] It should be noted that, in order to accurately achieve refined perception and dynamic evaluation of resource performance, the actual operating indicators include at least one of the following: Transactions per Second (TPS), Real-time call volume, First Token Generation Time (TTFT), Processing Time per Output Token (TPOT), Batch Processing Utilization, and Graphics Memory Block Utilization. Specifically, TPS indicates the maximum request processing capacity that the corresponding first service instance can handle per unit time; Real-time call volume indicates the actual request rate processed by the corresponding first service instance; TTFT indicates the time delay for the corresponding first service instance to generate and return the first output token; TPOT indicates the average time for the corresponding first service instance to generate one output token during the decoding phase; Batch Processing Utilization indicates the fill level of the batch processing queue of the corresponding first service instance; and Graphics Memory Block Utilization indicates the proportion of memory block resources used by the corresponding first service instance.

[0103] Step 403: Adjust each first service instance based on the first quantity.

[0104] To improve resource utilization, one possible approach is to dynamically adjust the number of online service instances by calling a predefined interface, based on a prediction of the first number of online service instances in the future time period.

[0105] As an example, determine the difference between the first quantity and the second quantity of actual service instances that match the first service level requirement at the current time; call the setting interface to adjust the number of fourth service instances marked with target labels in each first service instance based on the difference quantity; wherein, the target label is used to indicate that the fourth service instance matches the second service level requirement, and the first service level requirement is higher than the second service level requirement.

[0106] In other words, in a large-scale model inference service system, to dynamically adapt to traffic changes in requests for different service levels, the system first determines the first number (i.e., the predicted number) of service instances required to meet the first service level requirements (such as low latency and high throughput). This number is then compared with the second number (the actual number) of service instances currently available that meet the first service level requirements. The difference between the two (i.e., the gap or redundancy) is calculated; that is, the difference between the predicted number of online service instances in the future period after the current moment and the number of online service instances actually running at the current moment. If the difference is positive, it indicates that there are insufficient resources of service instances (online service instances) matching the first service level requirements in the future period; if it is negative, it indicates that there are excess resources. Then, the system calls a designated interface (e.g., a third-party platform interface) to dynamically adjust the number of fourth service instances (i.e., service instances used to serve large-scale model inference requests for the second service level, also known as offline service instances) in the resource pool that have been labeled with target tags based on this difference.

[0107] To effectively achieve dynamic resource adjustment, in this embodiment, when the first number is greater than the second number, it indicates that there are insufficient online service instance resources. Therefore, a different number of instances will be selected from the fourth service instances that originally matched the second service level requirements, their target tags will be deleted, and they will be upgraded to dedicated instances supporting the first service level request. That is, a different number of offline service instances will be selected from the offline service instances, their target tags (e.g., offline tags) will be deleted, and they will be adjusted to online service instances. When the first number is less than the second number, it indicates that there are excess online service instances. Therefore, a different number of instances will be selected from the fifth service instances that are currently not labeled with target tags (i.e., originally online service instances), their target tags will be added, and they will be downgraded to offline service instances.

[0108] Step 404: In response to receiving a large model inference request, determine the target service level requirements of the large model inference request based on the request parameters in the large model inference request.

[0109] Step 405: From the multiple first service instances in the large model resource pool, determine multiple second service instances that match the target service level requirements.

[0110] Step 406: Based on the metadata of each second service instance, determine the target service instance pair from each second service instance.

[0111] Step 407: Use the target pre-filled service instance and the target decoding service instance to respond to the large model inference request based on the target service level requirements to obtain the response result.

[0112] Step 408: Send the response result to the client associated with the large model inference request.

[0113] It should be noted that the execution process of steps 404 to 408 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0114] In summary, based on the total number of historical inference requests during the current period, the peak value of inference requests in the future period is predicted, enabling the system to anticipate changes in business traffic trends. Furthermore, based on this predicted peak value, the first number of third service instances matching the first service level requirements in the future period is determined, transforming the traffic prediction results into an executable resource demand scale, ensuring that online requests still receive sufficient computing resources during peak periods. Subsequently, dynamic adjustments (such as scaling up or down) are made to each first service instance according to the first number, achieving precise alignment between resource supply and business demand, effectively improving the intelligence level of resource scheduling, and significantly enhancing the stability, scalability, and resource utilization efficiency of the large-model inference system.

[0115] To clearly illustrate how the target pre-filled service instance and the target decoding service instance respond to the large model inference request based on the target service level requirements in the above embodiments to obtain the response result, this disclosure proposes another request processing method based on the large model.

[0116] Figure 5 This is a flowchart illustrating the request processing method based on a large model provided in Embodiment 5 of this disclosure.

[0117] like Figure 5 As shown, this request processing method based on a large model includes the following steps:

[0118] Step 501: In response to receiving a large model inference request, determine the target service level requirements of the large model inference request based on the request parameters in the large model inference request.

[0119] To accurately determine the target service level requirements for large model inference requests, one possible approach is to parse the request parameters to extract the service level identifier and combine it with a preset strategy table to complete the level mapping.

[0120] As an example, the large model inference request is parsed to extract the service level identifier field from the request parameters; based on the service level identifier field, the service level policy table is queried to obtain the target service level requirement of the large model inference request; wherein, the service level policy table includes the mapping relationship between the service level identifier field and the service level requirement.

[0121] In other words, the received large model inference request is parsed to extract the service level identifier field used to identify service quality requirements. Subsequently, based on the value of this field, a predefined service level policy table is queried; this table stores the mapping relationship between service level identifiers and specific service level requirements, thereby determining the target service level requirements corresponding to the request, such as the required response latency range, resource isolation level, scheduling priority, etc.

[0122] Step 502: Determine the target service instance pair from at least one candidate service instance pair that matches the target service level requirement.

[0123] The target service instance pair includes target pre-filled service instances determined based on pre-filled latency and target decoding service instances determined based on load.

[0124] Step 503: Based on the target service level requirements, the target pre-fill service instance pre-fills the input prompt words corresponding to the large model inference request to obtain the key-value cache of the input prompt words.

[0125] In this embodiment of the disclosure, the input prompt in the large model inference request is handed over to the target pre-fill service instance for parallel computation. The instance prioritizes the scheduling of computing resources according to the target service level requirements (such as low latency and high priority) corresponding to the request, quickly completes the attention calculation of the entire input sequence, and generates the corresponding key-value cache. The key-value cache is the intermediate state information (key / value vector) generated by the model during the processing of the prompt, which is used for efficient reuse in the subsequent autoregressive generation stage.

[0126] Step 504: Using the target decoding service instance, based on the target service level requirements and key-value cache, perform decoding processing to obtain the response result.

[0127] Subsequently, after the pre-filling stage is completed, the system passes the generated KV Cache to the target decoding service instance. Based on the cache information, the target decoding service instance generates output tokens one by one (i.e., the decoding process). It should be noted that the target decoding service instance follows the target service level requirements of the request during the decoding process (such as maintaining low response jitter and high stability) to ensure that the generation process is not interrupted or delayed due to resource contention or overload.

[0128] Step 505: Send the response result to the client associated with the large model inference request.

[0129] It should be noted that the execution of steps 502 and 505 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0130] In summary, by employing a target pre-fill service instance to pre-fill input prompts based on target service level requirements, high-performance computing resources can be fully utilized to quickly complete parallel attention computation of the input sequence, generating the corresponding key-value cache, significantly shortening the TTFT, and meeting the stringent low-latency requirements of high service level requests. Subsequently, a target decoding service instance is used to perform decoding processing based on the same service level requirements and the key-value cache, improving generation efficiency and ensuring that the decoding process runs continuously on dedicated resources with low load and high stability, guaranteeing stable and controllable TPOT. Thus, by assigning the pre-filling and decoding stages to service instances that meet service level requirements for collaborative execution, the overall throughput of the system is improved while effectively guaranteeing the end-to-end service quality (SLA) of high-priority requests, significantly enhancing the performance determinism and resource utilization efficiency of the large model inference system.

[0131] In any of the above embodiments, such as Figure 6 As shown, the request processing method based on a large model in this disclosure can be applied to an inference service system, which includes a traffic scheduling module, an engine monitoring module, and a resource elasticity management module. The request processing method based on a large model in this disclosure specifically includes the following steps:

[0132] Step 1: The traffic scheduling module receives traffic requests (including online and / or offline requests) sent by the inference link system;

[0133] Step 2: The traffic scheduling module scores and ranks each service instance, selecting the optimal service instance to handle the corresponding traffic, such as... Figure 7 As shown, the specific steps include the following:

[0134] (1) Filtering phase: First, filter out service instances that do not meet the requirements, such as service instances with abnormal status or interrupted heartbeat;

[0135] (2) Matching stage: Multiple PD instances (candidate prefill service instances and candidate decode service instances) have been selected through the filtering stage. Matching is required to perform possible pairing (candidate service instance pairs), mainly through the consistency of PD version numbers and the topology of PD instance connection relationships.

[0136] (3) Score optimization stage: After the filtering and matching stages, multiple PD pairs have been selected. The Score stage is to select the best PD pair for traffic scheduling. The core idea is to select the P with the shortest queuing prefill time and the D with the smallest number of allocated requests.

[0137] It should be noted that when performing traffic scheduling, the traffic scheduling module makes decisions based on environmental information such as the status of service instances, various utilization metrics, and tag identities collected and reported by the engine monitoring module. Figure 8 As shown, the details are as follows:

[0138] (1) Obtain service instance information: Every 2 seconds, by accessing the interface of the engine module (i.e. the service instance module) on the same machine, obtain the status of the service instance, the number of memory blocks, the size of memory blocks, and other indicators, as well as obtain important identity information from the environment variables;

[0139] (2) Reporting information: After obtaining the information, report the relevant information by calling the traffic scheduling module interface.

[0140] It's important to note that before the traffic scheduling model performs request scheduling, the elastic resource management module collects various data metrics (actual operational metrics) from all service instances in the inference service system's resource pool. This collected data is then used to calculate the number of online service instances needed to achieve the target TPS utilization rate, based on a specific algorithm. The number of online service instances is then adjusted through the deployment platform interface. For service instances that are scaled down, an offline tag is added to indicate that the instance can only handle offline requests. This allows them to process offline requests without restarting the instance, thus achieving isolation of offline service instances. When online resources need to be expanded, the offline tag is removed, and the instance is used to handle online requests. Figure 9 As shown, the details are as follows:

[0141] (1) Metrics collection: Periodically collect the capacity and real-time call volume metrics of the large model's TPS, as well as the TTFT, TPOT, and Batch / Block utilization metrics of each service instance.

[0142] (2) Traffic forecasting: Every minute, based on historical (48H) traffic data (total historical requests), operation and maintenance data, etc., the traffic peak for the next hour is estimated;

[0143] (3) Target instance calculation: Based on the current real-time traffic and the utilization rate of TTFT, TPOT, Batch / Block and traffic forecast, the online service instances in the future time period are calculated.

[0144] (4) Adjustment of online service instances: If the predicted online service instances are inconsistent with the actual online service instances, the deployment platform interface is called to adjust the online service instances;

[0145] (5) Next cycle.

[0146] It should be noted that when the Elastic Resource Management module determines that there are offline service instances in the resource pool, it sends a notification to the inference link system to inform the inference link system that it can send offline requests.

[0147] With the above Figures 1 to 9 Corresponding to the request processing method based on a large model provided in the embodiments, this disclosure also provides a request processing apparatus based on a large model. Since the request processing apparatus based on a large model provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 9 The large-model-based request processing method provided in the embodiments corresponds to the large-model-based request processing apparatus provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0148] Figure 10 This is a schematic diagram of the structure of the request processing apparatus based on a large model provided in Embodiment Six of this disclosure.

[0149] like Figure 10 As shown, the request processing device 1000 based on a large model includes: a first determining module 1010, a second determining module 1020, a processing module 1030, and a sending module 1040.

[0150] The first determining module 1010 is used to determine the target service level requirement of the large model inference request based on the request parameters in the large model inference request in response to receiving the request; the second determining module 1020 is used to determine a target service instance pair from at least one candidate service instance pair that matches the target service level requirement; wherein the target service instance pair includes a target pre-filled service instance determined based on pre-filling delay and a target decoding service instance determined based on load; the processing module 1030 is used to respond to the large model inference request based on the target service level requirement using the target pre-filled service instance and the target decoding service instance to obtain a response result; and the sending module 1040 is used to send the response result to the client associated with the large model inference request.

[0151] As one possible implementation, the second determining module 1020 is used to determine multiple second service instances that match the target service level requirements from multiple first service instances in the large model resource pool; and to determine the target service instance pair from each second service instance based on the metadata of each second service instance.

[0152] As one possible implementation, the second determining module 1020 is used to determine at least one candidate service instance pair from each of the second service instances based on the metadata of each of the second service instances; wherein, the candidate service instance pair includes a candidate pre-filled service instance and a candidate decoding service instance; and to determine a target service instance pair from at least one candidate service instance pair based on the pre-filling delay of the candidate pre-filled service instance and the load of the candidate decoding service instance in each candidate service instance pair.

[0153] As one possible implementation, the second determining module 1020 is used to determine multiple candidate service instances from each second service instance based on the metadata of each second service instance; wherein the multiple candidate service instances include at least one candidate pre-filled service instance and at least one candidate decoding service instance; and to pair the at least one candidate pre-filled service instance and the at least one candidate decoding service instance to obtain at least one candidate service instance pair.

[0154] As one possible implementation, the second determining module 1020 is used to perform version matching on at least one candidate pre-filled service instance and at least one candidate decoding service instance to obtain at least one intermediate service instance pair; wherein the version numbers of the candidate pre-filled service instance and the candidate decoding service instance in the intermediate instance pair are consistent; and at least one candidate service instance pair is determined from the at least one intermediate service instance pair; wherein there is a communication connection path between the candidate pre-filled service instance and the candidate decoding service instance in the candidate service instance pair.

[0155] As one possible implementation, the second determining module 1020 is used to determine abnormal service instances from each of the second service instances based on the metadata of each second service instance; and to filter abnormal service instances from each of the second service instances to obtain multiple candidate service instances to be retained.

[0156] As one possible implementation, the metadata includes at least one of the following: status information; wherein the status information is used to indicate whether the corresponding second service instance is in an available state; memory block information; wherein the memory block information is used to indicate the memory block resource usage of the corresponding second service instance; memory block size information; wherein the memory block size information is used to indicate the maximum number of tokens allowed to be stored in each memory block of the corresponding second service instance; and identity identification information; wherein the identity identification information is used to identify the corresponding second service instance.

[0157] As one possible implementation, the second determining module 1020 is used to score any candidate service instance pair based on the pre-filling latency of the candidate pre-filled service instance and the load of the candidate decoding service instance in the candidate service instance pair, so as to obtain a scoring result for any candidate service instance pair; sorting each candidate service instance pair according to the scoring results of each candidate service instance pair, so as to obtain the arrangement order of the candidate service instance pairs; and determining a target service instance pair from each candidate service instance pair based on the arrangement order; wherein the target service instance pair has the shortest pre-filling latency of the candidate pre-filled service instance and the minimum load of the candidate decoding service instance.

[0158] As one possible implementation, the large model-based request processing device 1000 also includes a prediction module, a third determination module, and an adjustment module.

[0159] The prediction module is used to predict the peak value of the total number of prediction inference requests in future periods after the current period based on the total number of historical inference requests in the same period of the current period; the third determination module is used to determine the first number of third service instances that match the first service level requirements in future periods based on the peak value; and the adjustment module is used to adjust each first service instance based on the first number.

[0160] As one possible implementation, the adjustment module is used to determine the difference between the first quantity and the second quantity of actual service instances that match the first service level requirement at the current time; and to call the setting interface to adjust the number of fourth service instances marked with target labels in each first service instance based on the difference quantity; wherein, the target label is used to indicate that the fourth service instance matches the second service level requirement, and the first service level requirement is higher than the second service level requirement.

[0161] As one possible implementation, the adjustment module is used to delete the target labels of the fourth service instances with the difference in number in response to the first number being greater than the second number; and to label the fifth service instances with the difference in number among the first service instances that have not been labeled with target labels in response to the first number being less than the second number.

[0162] As one possible implementation, the third determining module is used to collect the actual operating indicators of each first service instance in the current time period; based on the total number of actual inference requests in the current time period, the actual operating indicators and peak values ​​of each first service instance, it determines the first number of third service instances that match the first service level requirements in future time periods.

[0163] As one possible implementation, the actual operating metrics include at least one of the following: Transactions per Second (TPS) capacity; wherein TPS capacity indicates the maximum request processing capacity that the corresponding first service instance can handle per unit time; Real-time call volume; wherein real-time call volume indicates the actual request rate processed by the corresponding first service instance; Time-To-Flat (TTFT); wherein TTFT indicates the time delay for the corresponding first service instance to generate and return the first output token; Time-To-Time (TPOT); wherein TPOT indicates the average time for the corresponding first service instance to generate one output token during the decoding phase; Batch processing utilization; wherein batch processing utilization indicates the fill level of the batch processing queue of the corresponding first service instance; and Memory block utilization; wherein Memory block utilization indicates the proportion of memory block resources used by the corresponding first service instance.

[0164] As one possible implementation, the processing module 1030 is used to pre-fill the input prompt words corresponding to the large model inference request based on the target service level requirements using a target pre-fill service instance to obtain the key-value cache of the input prompt words; and to perform decoding processing based on the target service level requirements and the key-value cache using a target decoding service instance to obtain the response result.

[0165] As one possible implementation, the first determining module 1010 is used to parse the large model inference request to extract the service level identifier field from the request parameters in the large model inference request; based on the service level identifier field, it queries the service level policy table to obtain the target service level requirement of the large model inference request; wherein, the service level policy table includes the mapping relationship between the service level identifier field and the service level requirement.

[0166] The large model-based request processing apparatus of this disclosure, upon receiving a large model inference request, first determines the target service level requirement of the large model inference request based on the request parameters. This enables the system to identify the differentiated quality requirements of different requests in terms of latency, throughput, and priority, providing a basis for subsequent resource scheduling decisions. Then, it selects a target service instance pair from candidate service instance pairs matching the target service level, achieving precise alignment between resource allocation and service quality requirements. The target service instance pair includes a target pre-filled service instance selected based on pre-filled latency and a target decoding service instance selected based on load status. This target service instance pair is then used collaboratively to complete the inference task, ensuring that large model inference requests with high service level requirements always receive high-quality resource services matching their level requirements throughout the entire inference process, thereby effectively ensuring that service level requirements are met. Finally, the response result is returned to the associated client to ensure service interactivity and availability.

[0167] To implement the above embodiments, this disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the request processing method based on a large model proposed in any of the above embodiments of this disclosure.

[0168] To implement the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the request processing method based on a large model proposed in any of the above embodiments of this disclosure.

[0169] To implement the above embodiments, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the request processing method based on a large model proposed in any of the above embodiments of this disclosure.

[0170] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0171] Figure 11 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0172] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in ROM (Read-Only Memory) 1102 or loaded from storage unit 1108 into RAM (Random Access Memory) 1103. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. An I / O (Input / Output) interface 1105 is also connected to bus 1104.

[0173] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0174] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the large model-based request processing method described above. For example, in some embodiments, the large model-based request processing method described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the large-model-based request processing method described above can be performed. Alternatively, in other embodiments, computing unit 1101 can be configured to perform the large-model-based request processing method described above by any other suitable means (e.g., by means of firmware).

[0175] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0177] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0179] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0180] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0181] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0182] According to the technical solution of this disclosure, upon receiving a large model inference request, the target service level requirement of the large model inference request is first determined based on the request parameters. This enables the system to identify the differentiated quality requirements of different requests in terms of latency, throughput, priority, etc., providing a basis for subsequent resource scheduling decisions. Then, a target service instance pair is selected from the candidate service instance pairs that match the target service level, achieving precise alignment between resource allocation and service quality requirements. The target service instance pair includes a target pre-filled service instance selected based on pre-filled latency and a target decoding service instance selected based on load status. The target service instance pair is then used to collaboratively complete the inference task, ensuring that the large model inference request with high service level requirements always obtains high-quality resource services that match its level requirements throughout the entire inference process, thereby effectively ensuring that the service level requirements are met. Finally, the response result is returned to the associated client to ensure the interactivity and availability of the service.

[0183] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution proposed in this disclosure can be achieved, and this is not limited herein.

[0184] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A request processing method based on a large model, comprising: In response to receiving a large model inference request, the target service level requirement of the large model inference request is determined based on the request parameters in the large model inference request; A target service instance pair is determined from at least one candidate service instance pair that matches the target service level requirement; wherein the target service instance pair includes a target pre-filled service instance determined based on pre-filled latency and a target decoding service instance determined based on load. The target pre-filled service instance and the target decoding service instance are used to respond to the large model inference request based on the target service level requirements to obtain the response result; The response result is sent to the client associated with the large model inference request.

2. The method according to claim 1, wherein, Determining the target service instance pair from at least one candidate service instance pair that matches the target service level requirement includes: From multiple first service instances in the large model resource pool, determine multiple second service instances that match the target service level requirements; Based on the metadata of each of the second service instances, the target service instance pair is determined from each of the second service instances.

3. The method according to claim 2, wherein, The step of determining the target service instance pair from each of the second service instances based on the metadata of each of the second service instances includes: Based on the metadata of each second service instance, at least one candidate service instance pair is determined from each second service instance; wherein, the candidate service instance pair includes a candidate pre-filled service instance and a candidate decoding service instance; The target service instance pair is determined from the at least one candidate service instance pair based on the prefilling delay of the candidate prefilled service instance and the load of the candidate decoding service instance in each candidate service instance pair.

4. The method according to claim 3, wherein, The step of determining at least one candidate service instance pair from each of the second service instances based on the metadata of each second service instance includes: Based on the metadata of each second service instance, multiple candidate service instances are determined from each second service instance; wherein, the multiple candidate service instances include at least one candidate pre-filled service instance and at least one candidate decoding service instance; The at least one candidate pre-filled service instance and the at least one candidate decoding service instance are paired to obtain at least one candidate service instance pair.

5. The method according to claim 4, wherein, The step of pairing the at least one candidate pre-filled service instance and the at least one candidate decoding service instance to obtain at least one candidate service instance pair includes: Version matching is performed on the at least one candidate pre-filled service instance and the at least one candidate decoding service instance to obtain at least one intermediate service instance pair; wherein the version numbers of the candidate pre-filled service instance and the candidate decoding service instance in the intermediate instance pair are the same; The at least one candidate service instance pair is determined from the at least one intermediate service instance pair; wherein, there is a communication connection path between the candidate pre-filled service instance and the candidate decoding service instance in the candidate service instance pair.

6. The method according to claim 4, wherein, The step of determining multiple candidate service instances from each of the second service instances based on the metadata of each second service instance includes: Based on the metadata of each second service instance, identify the abnormal service instance from each second service instance; The abnormal service instances are filtered from each of the second service instances to obtain a plurality of candidate service instances that are retained.

7. The method according to claim 4, wherein, The metadata includes at least one of the following: Status information; wherein, the status information is used to indicate whether the corresponding second service instance is in an available state; Memory block information; wherein, the memory block information is used to indicate the memory block resource usage of the corresponding second service instance; Memory block size information; wherein, the memory block size information is used to indicate the maximum number of tokens allowed to be stored in each memory block in the corresponding second service instance; Identity identification information; wherein, the identity identification information is used to indicate the second service instance corresponding to the identifier.

8. The method according to claim 3, wherein, The step of determining the target service instance pair from the at least one candidate service instance pair based on the prefilling latency of the candidate prefilled service instance and the load of the candidate decoding service instance in each candidate service instance pair includes: For any candidate service instance pair, the candidate service instance pair is scored based on the prefilling latency of the candidate prefilled service instance and the load of the candidate decoding service instance in the candidate service instance pair to obtain the scoring result of the candidate service instance pair. Based on the scoring results of each candidate service instance pair, the candidate service instance pairs are sorted to obtain the arrangement order of the candidate service instance pairs; Based on the sorting order, the target service instance pair is determined from each of the candidate service instance pairs; wherein the candidate pre-filled service instance in the target service instance pair has the shortest pre-filling latency and the candidate decoding service instance has the minimum load.

9. The method according to claim 2, wherein, Before determining, from the multiple first service instances in the large model resource pool, multiple second service instances that match the target service level requirement, the method further includes: Based on the total number of historical inference requests for the same period in the current time period, predict the peak value of the total number of predictive inference requests in future time periods after the current time period. Based on the peak value, determine the first number of third service instances that match the first service level requirements within the future time period; Based on the first quantity, adjustments are made to each of the first service instances.

10. The method according to claim 9, wherein, The adjustment of each of the first service instances based on the first quantity includes: Determine the difference between the first quantity and the second quantity of actual service instances that match the first service level requirement at the current time; The interface is invoked to adjust the number of fourth service instances labeled with target tags in each of the first service instances based on the number of differences; wherein, the target tag is used to indicate that the fourth service instance matches the second service level requirement, and the first service level requirement is higher than the second service level requirement.

11. The method according to claim 10, wherein, The adjustment of the number of fourth service instances labeled with target tags in each of the first service instances includes: In response to the first quantity being greater than the second quantity, the target label of the fourth service instance with the difference in quantity is deleted; In response to the first quantity being less than the second quantity, the target label is labeled on the fifth service instance in each of the first service instances that has not been labeled with the target label and the difference in quantity.

12. The method according to claim 9, wherein, The step of determining the first number of third service instances matching the first service level requirement within the future time period based on the peak value includes: Collect the actual operating metrics of each of the first service instances during the current time period; Based on the total number of actual inference requests in the current time period, the actual operating metrics of each of the first service instances, and the peak value, determine the first number of third service instances that match the first service level requirements in the future time period.

13. The method according to claim 12, wherein, The actual operating indicators include at least one of the following: Transactions per second (TPS) capacity; wherein, the TPS capacity is used to indicate the maximum request processing capacity that the corresponding first service instance can bear per unit time. Real-time call volume; wherein, the real-time call volume is used to indicate the actual request rate processed by the corresponding first service instance; First token generation delay (TTFT); wherein, the TTFT is used to indicate the time delay for the corresponding first service instance to generate and return the first output token; Processing Time for Each Output Token (TPOT); wherein, TPOT is used to indicate the average time for the corresponding first service instance to generate one output token during the decoding phase; Batch processing utilization; wherein, the batch processing utilization is used to indicate the fill level of the batch processing queue of the corresponding first service instance; Video memory block utilization rate; wherein, the video memory block utilization rate is used to indicate the proportion of memory block resources used by the corresponding first service instance.

14. The method according to any one of claims 1-13, wherein, The step of using the target pre-filled service instance and the target decoding service instance to respond to the large model inference request based on the target service level requirements to obtain a response result includes: The target pre-fill service instance is used to pre-fill the input prompt words corresponding to the large model inference request based on the target service level requirements, so as to obtain the key-value cache of the input prompt words; The target decoding service instance performs decoding processing based on the target service level requirements and the key-value cache to obtain the response result.

15. The method according to any one of claims 1-13, wherein, The step of determining the target service level requirement for the large model inference request based on the request parameters in the large model inference request includes: The large model inference request is parsed to extract the service level identifier field from the request parameters in the large model inference request; Based on the service level identifier field, the service level policy table is queried to obtain the target service level requirement for the large model inference request; wherein, the service level policy table includes the mapping relationship between the service level identifier field and the service level requirement.

16. A request processing apparatus based on a large model, comprising: The first determining module is used to, in response to receiving a large model inference request, determine the target service level requirement of the large model inference request based on the request parameters in the large model inference request; The second determining module is configured to determine a target service instance pair from at least one candidate service instance pair that matches the target service level requirement; wherein the target service instance pair includes a target pre-filled service instance determined based on pre-filled latency and a target decoding service instance determined based on load. The processing module is used to respond to the large model inference request based on the target service level requirements using the target pre-filled service instance and the target decoding service instance, so as to obtain a response result; The sending module is used to send the response result to the client associated with the large model inference request.

17. An electronic device, wherein, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-15.

19. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-15.

Images