Asynchronous scheduling system, method and equipment based on large model reasoning and medium

Abstract

The invention discloses an asynchronous scheduling system, method and device based on large model reasoning and a medium, and relates to the technical field of large model reasoning optimizing.The method comprises the steps that a CPU receives a reasoning request of the t-th round, reasoning task scheduling is executed, a reasoning data object of the t-th round is obtained, the reasoning data object is transmitted to a GPU, reasoning lems of the reasoning request of the t-th round are obtained from the GPU, and the reasoning lems of the reasoning request of the t-th round are obtained; judging whether the reasoning lemma of the tth round of reasoning request contains numerical value information or not, and if yes, updating the tth round of reasoning request according to the reasoning lemma to obtain a tth round of reasoning result; transmitting the reasoning result of the tth round to the terminal; preparing the (t + 1) th round of reasoning task scheduling; when the CPU carries out judgment, the GPU obtains the reasoning data object of the tth round from the CPU, reasoning is carried out based on the reasoning data object of the tth round, and the reasoning lexical element of the reasoning request of the (t + 1) th round is obtained. According to the method, the parallel execution of CPU scheduling and GPU reasoning can be realized while the reasoning precision of a large model is kept, and resources are saved.

Description

Technical Field

[0001] The present application relates to the technical field of large model reasoning optimization, and in particular to an asynchronous scheduling system, method, device and medium based on large model reasoning. Background Art

[0002] In the field of large-model inference optimization technology, traditional large-model inference frameworks built on deep learning frameworks like PyTorch or TensorFlow typically follow a synchronous serial execution model for CPU and GPU tasks. Specifically, during the CPU scheduling phase, the CPU is responsible for receiving inference requests and performing data preprocessing, including word segmentation, tagging, and batching.

[0003] However, due to the Python Global Interpreter Lock (GIL) mechanism, multiple threads within the same process cannot achieve true parallel execution, causing the CPU's scheduling logic to hinder the launch of GPU tasks. During the GPU inference phase, the CPU transfers preprocessed data to the GPU and waits for the GPU to complete the inference calculations. After the GPU calculations are completed, the results are returned to the CPU for post-processing, such as decoding and detokenization. However, due to the limitations of the GIL, there is a serial dependency problem. This means that the CPU cannot simultaneously process the next round of scheduled tasks while the GPU is executing inference. This results in the CPU being idle during GPU calculations and the GPU being idle during CPU scheduling, resulting in wasted resources. Summary of the Invention

[0004] The present application provides an asynchronous scheduling system, method, device and medium based on large-model reasoning, which can realize parallel execution of CPU scheduling and GPU reasoning without affecting the accuracy of large-model reasoning, reduce the GPU forward reasoning data preparation time for each round, and thus save resources.

[0005] To achieve the above objectives, this application adopts the following technical solutions: In a first aspect, the present application provides an asynchronous scheduling system based on large model reasoning, the system comprising a CPU and a GPU; The CPU is configured to receive an inference request for round t, process the inference request for round t according to the inference task scheduling for round t, obtain an inference data object for round t, and transmit the inference data object for round t to the GPU; obtain an inference word element of the inference request for round t from the GPU, determine whether the inference word element of the inference request for round t contains numerical information, and obtain a first judgment result; if the first judgment result indicates that the inference word element of the inference request for round t contains numerical information, update the state of the inference request for round t according to the inference word element of the inference request for round t, and obtain an inference result for round t; transmit the inference result of round t to the terminal; and prepare for the inference task scheduling for round t+1; The GPU is used for obtaining the inference data object of the tth round from the CPU when the CPU determines whether the inference word element of the inference request of the tth round contains numerical information, performing the inference of the tth round based on the inference data object of the tth round, and obtaining the inference word element of the inference request of the t+1th round.

[0006] In some possible implementations, the CPU is further configured to prepare for scheduling the reasoning task for the t+1th round if the first judgment result indicates that the reasoning word of the reasoning request for the tth round does not contain numerical information.

[0007] In some possible implementations, the reasoning data object of the tth round includes multiple reasoning data objects, and the multiple reasoning data objects include the newly received reasoning data object and the reasoning data object of the t-1th round. The CPU transmits the newly received reasoning data object in full to the GPU through the inter-process message queue, and the CPU incrementally transmits the reasoning data object of the t-1th round to the GPU through the inter-process message queue. In some possible implementations, the inference request of the tth round includes n inference requests, and the CPU determines whether the n inference requests are greater than the number m of inference requests in the inference space of the GPU, to obtain a second determination result; If the second judgment result indicates that the number of n inference requests is greater than the number of inferences m in the inference space of the GPU, the inference data objects corresponding to the nm inference requests are stored in the preemption queue; when the resources of the inference space of the GPU are sufficient, the CPU extracts the inference data objects corresponding to the nm inference requests from the preemption queue, and incrementally transmits them to the GPU through the inter-process message queue, and the GPU infers the inference data objects corresponding to the nm inference requests in sequence. In some possible implementations, the CPU is also used to update the inter-process message queue. When the inference word element of the inference request of the tth round contains numerical information, the inter-process message queue is updated according to the numerical information of the inference word element of the inference request of the tth round, and the updated inter-process message queue is transmitted to the GPU. In some possible implementations, the task inference scheduling of the CPU refers to sorting inference requests, updating inference states, and generating inference data objects corresponding to the inference requests.

[0008] In some possible implementations, the CPU is also used to transmit the inference data objects corresponding to the n inference requests to the GPU if the second judgment result indicates that the n inference requests are not greater than the number m of inferences in the inference space of the GPU, and the GPU infers the inference data objects corresponding to the n inference requests in sequence.

[0009] In a second aspect, the present application provides an asynchronous scheduling method based on large model reasoning, the method being applied to a CPU and a GPU, the CPU and the GPU processing in parallel, the method comprising: The CPU receives an inference request for round t, processes the inference request for round t according to the inference task scheduling for round t, obtains an inference data object for round t, and transmits the inference data object for round t to the GPU; obtains an inference word element of the inference request for round t from the GPU, determines whether the inference word element of the inference request for round t contains numerical information, and obtains a first judgment result; if the first judgment result indicates that the inference word element of the inference request for round t contains numerical information, updates the state of the inference request for round t according to the inference word element of the inference request for round t, and obtains an inference result for round t; transmits the inference result of round t to the terminal; and prepares for the inference task scheduling for round t+1; When the GPU determines whether the inference word element of the inference request of the tth round contains numerical information, it obtains the inference data object of the tth round from the CPU, performs the inference of the tth round based on the inference data object of the tth round, and obtains the inference word element of the inference request of the t+1th round.

[0010] In a third aspect, the present application provides a computing device, the computing device comprising the system described in any one of the first aspects.

[0011] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes the system described in any one of the first aspects.

[0012] In a fifth aspect, the present application provides a computer program product, comprising the system described in any one of the first aspects.

[0013] It can be seen from the above technical solution that this application has at least the following beneficial effects: In the present application, the CPU receives the reasoning request of the tth round, processes the reasoning request of the tth round according to the reasoning task scheduling of the tth round, obtains the reasoning data object of the tth round, and transmits the reasoning data object of the tth round to the GPU; obtains the reasoning word element of the reasoning request of the tth round from the GPU, judges whether the reasoning word element of the reasoning request of the tth round contains numerical information, and obtains a first judgment result; if the first judgment result indicates that the reasoning word element of the reasoning request of the tth round contains numerical information, then updates the status of the reasoning request of the tth round according to the reasoning word element of the reasoning request of the tth round, and obtains the reasoning result of the tth round; transmits the reasoning result of the tth round to the terminal, and prepares the reasoning task scheduling of the t+1th round; when the GPU judges whether the reasoning word element of the reasoning request of the tth round contains numerical information, it obtains the reasoning data object of the tth round from the CPU, performs reasoning of the tth round based on the reasoning data object of the tth round, and obtains the reasoning word element of the reasoning request of the t+1th round.

[0014] In existing technologies, the Python thread GIL mechanism prevents the CPU and GPU from working in parallel, resulting in significant idle time for hardware resources, especially the GPU, and reduced inference throughput. This increases critical path latency, as each round of inference must wait for the previous round's "CPU→GPU→CPU" process to complete. This high critical path latency negatively impacts scenarios with high real-time requirements, such as conversational systems. Furthermore, the full potential of multi-core CPUs is limited. Due to the GIL, even with multi-threading, CPU scheduling tasks cannot achieve true concurrent execution. Fourth, scalability is limited. As model size increases or the number of concurrent requests increases, the serial mode becomes a performance bottleneck, making it difficult to improve efficiency by simply increasing hardware resources. This demonstrates that, through the aforementioned process, the present application achieves parallel processing of CPU and GPU execution by separating the CPU and GPU execution processes and optimizing asynchronous scheduling strategies, thereby improving the inference rate of model inference services. This allows for parallel CPU scheduling and GPU inference execution without compromising the inference accuracy of large models, reducing the GPU's data preparation time for each round of forward inference and thus conserving resources.

[0015] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of a technical feature, technical solution or beneficial effect in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of an asynchronous scheduling system based on large model reasoning provided in an embodiment of the present application; Figure 2 A schematic diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The terms "first", "second" and "third" in this application specification and the accompanying drawings are used to distinguish different objects rather than to limit a specific order.

[0018] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0019] To make the description of the following embodiments clear and concise, a brief introduction to the related technologies is first given: Large model technology, a cutting-edge technology in the field of artificial intelligence, has achieved significant progress and breakthroughs in recent years. Based on deep learning, large model technology builds neural network models with tens or even hundreds of millions of parameters, enabling efficient processing of complex tasks and data. These models utilize a "pre-training + fine-tuning" training model. After training on large amounts of data, they can quickly adapt to various downstream tasks, demonstrating strong expressive power and predictive performance.

[0020] With the rapid development of artificial intelligence (AI), large-model inference has become a key research direction in deep learning. Large models, particularly those based on the Transformer architecture, have garnered widespread attention for their exceptional performance in tasks such as natural language processing and computer vision. However, as model size continues to expand, inference efficiency and resource consumption become increasingly problematic, necessitating effective optimization techniques. To improve inference efficiency, reduce computational and memory requirements, meet real-time requirements, and lower deployment costs, large-model inference frameworks have emerged.

[0021] Currently, in the field of large-scale model inference technology, the current large-scale model inference process generally covers the key links of scheduling, inference, obtaining results, and outputting them. After completing one round of operations, the next round of scheduling begins immediately. Specifically, in this process, the model is first loaded, and the trained large model parameters are loaded into processing units such as CPUs and GPUs. Then, the input data is preprocessed to make it conform to the input format requirements of the model, such as adjusting the data size, normalizing the data, or encoding it. Then, the model calculation phase begins, and the input data is forward propagated through multiple layers of the model until the final output is generated. Finally, the output results are post-processed, such as converting the probability output by the model into actual category labels for users to understand and use.

[0022] However, current technology has limitations. In large-model inference scenarios based on the Python language, the Global Interpreter Lock (GIL) mechanism of Python threads causes CPU tasks and GPU inference to wait for each other. The GIL, a mutual exclusion lock in CPython, is designed to ensure that only one thread can execute Python bytecode at a time. This means that even in a multi-threaded environment, Python struggles to achieve true parallel execution of Python code across multiple threads. During large-model inference, when the CPU holds the GIL while performing tasks such as data processing and scheduling, the GPU inference thread is forced to wait, and vice versa. This waiting cycle significantly wastes valuable computing resources. This is especially true for large-model inference tasks that require significant computing resources and high concurrency. It slows inference speed and reduces overall system efficiency, becoming a significant bottleneck hindering performance improvements for large-model inference.

[0023] In view of this, an embodiment of the present application provides an asynchronous scheduling system based on large model reasoning, which includes a CPU and a GPU; The CPU receives the inference request of the tth round, processes the inference request of the tth round according to the inference task scheduling of the tth round, obtains the inference data object of the tth round, and transmits the inference data object of the tth round to the GPU; obtains the inference word element of the inference request of the tth round from the GPU, judges whether the inference word element of the inference request of the tth round contains numerical information, and obtains the first judgment result; if the first judgment result indicates that the inference word element of the inference request of the tth round contains numerical information, then updates the status of the inference request of the tth round according to the inference word element of the inference request of the tth round, and obtains the inference result of the tth round; transmits the inference result of the tth round to the terminal, and prepares the inference task scheduling of the t+1th round; when the GPU judges whether the inference word element of the inference request of the tth round contains numerical information, it obtains the inference data object of the tth round from the CPU, performs inference of the tth round based on the inference data object of the tth round, and obtains the inference word element of the inference request of the t+1th round. As can be seen, through the above process, this application achieves separation of CPU and GPU execution processes and achieves parallel processing of CPU and GPU execution through asynchronous scheduling strategy optimization, thereby improving the inference rate of model inference service. In other words, it can achieve parallel execution of CPU scheduling and GPU inference without affecting the inference accuracy of large models, reducing the GPU forward inference data preparation time for each round, thereby saving resources.

[0024] In order to make the technical solution of this application clearer and easier to understand, the following describes an asynchronous scheduling system based on large model reasoning provided by an embodiment of this application in conjunction with the accompanying drawings. Figure 1 As shown, the figure is a schematic diagram of an asynchronous scheduling system based on large model reasoning provided in an embodiment of the present application.

[0025] The asynchronous scheduling system based on large-model inference includes the CPU and GPU. During actual operation, the CPU and GPU process in parallel. While the GPU is processing a certain round of inference data objects, the CPU can pre-schedule and process the output content of the previous round of GPU execution, reducing waiting time and thus saving resources. The following is a detailed introduction to the technical solution.

[0026] The CPU is used to receive the inference request of the tth round, process the inference request of the tth round according to the inference task scheduling of the tth round, obtain the inference data object of the tth round, and transmit the inference data object of the tth round to the GPU; obtain the inference word element of the inference request of the tth round from the GPU, judge whether the inference word element of the inference request of the tth round contains numerical information, and obtain a first judgment result; if the first judgment result indicates that the inference word element of the inference request of the tth round contains numerical information, then update the status of the inference request of the tth round according to the inference word element of the inference request of the tth round to obtain the inference result of the tth round; transmit the inference result of the tth round to the terminal; and prepare the inference task scheduling of the t+1th round; if the first judgment result indicates that the inference word element of the inference request of the tth round does not contain numerical information, then update the inference request of the tth round according to the inference task scheduling of the t+1th round to obtain the inference request of the t+1th round.

[0027] Inference requests are transmitted to the CPU's scheduling module via offline or HTTP requests. This module is a core component of the operating system kernel and is responsible for determining which process or thread receives CPU resources and when, thereby enabling multitasking and optimizing resource utilization. The CPU scheduling module is primarily responsible for receiving inference requests and allocating computing resources; arranging the execution order of inference tasks to optimize GPU utilization; and implementing pipeline parallelism. For example, after the current round's inference data is transmitted to the GPU, the next round of scheduling begins immediately. The inference request for round t is the result of an update to the inference request for round t-1 based on the inference tokens of the inference request for round t-1, as well as the addition of new inference requests to this round.

[0028] An inference request mainly includes the following parameter variables: Class Schedule_seq_new : { Req_id: str, used to specify the unique id of the current request.

[0029] KV_Cache_Blocks: list, KV Cache list allocated during Attention calculation.

[0030] Sampling_params: class, sampling parameters, including top_k, top_k, etc.

[0031] Prompt_ids: list, the token list obtained after the inference request prompt is segmented.

[0032] Computed_num: int, the number of tokens for this inference request and calculation, equivalent to the number of KV Cache caches.

[0033] } After receiving the inference request for round t, the CPU schedules the inference task for that round. This CPU inference task scheduling involves sorting the inference requests, updating the inference status, and generating the corresponding inference data object. Specifically, inference task scheduling involves parsing the inference request parameters to determine the length of the inference request data, the required inference progress, and other information. It then allocates computing resources to the inference request and generates an inference data object containing the model path, input tensors, and execution configuration. The CPU then obtains the inference data object for round t. This step prepares for subsequent inference tasks, converting the input natural language prompt into a format that the model can process, providing basic data support for the entire inference process.

[0034] An inference data object is constructed and sent to the GPU inference interface. The GPU receives data and returns results via an inter-process message queue. This approach enables data exchange between the CPU and GPU, creating conditions for parallel processing and enabling the GPU to obtain the required data for inference operations. The CPU then immediately prepares to schedule inference tasks for round t+1, without waiting for the GPU's inference results from the current round. This advance scheduling breaks the traditional serial execution model, reduces CPU waiting time, improves CPU resource utilization, and shortens the CPU's execution time in a single round of model inference, thereby reducing the CPU's share of the critical path in the processing flow.

[0035] When preparing for the inference task scheduling for round t+1, it is necessary to determine whether the inference word element of the inference request for round t contains numerical information, and obtain a first judgment result. If the first judgment result indicates that the inference word element of the inference request for round t contains numerical information, the state of the inference request for round t is updated based on the inference word element of the inference request for round t, and the inference result of round t is obtained. The inference result of round t is transmitted to the terminal. If the first judgment result indicates that the inference word element of the inference request for round t does not contain numerical information, the inference task scheduling for round t+1 is prepared. This ensures that the inference result can be processed and output in a timely manner. If not, advance scheduling is performed to continue optimizing resource utilization. At the same time, the number of KVCaches occupied and updated during advance scheduling helps improve the efficiency of GPU forward inference.

[0036] Among technologies related to large-scale model inference, KV Cache (KV Cache), an abbreviation of "Key-Value Cache," is an important caching mechanism used to optimize the model inference process. In large models based on the Transformer architecture, the self-attention mechanism requires a large amount of intermediate data, resulting in high computational complexity and a long computational time. KV Cache was introduced to address this issue. During inference, the model calculates a series of key and value data. KV Cache stores these intermediate calculation results. When the model processes subsequent inputs or epochs, if the same key-value pairs are needed, they can be directly retrieved from the cache without repeated computation, thus saving computing resources and time. For example, in text generation tasks, when processing consecutive words, many key-value pairs are reused. KV Cache allows for quick access and avoids repeated computation.

[0037] Inference speed and resource utilization are critical for large-model inference. Using KV Cache can reduce the model's resource consumption for repeated computations, allowing the model to process input data more quickly and generate inference results. In practical applications, this is crucial for improving system responsiveness and reducing latency, thereby better meeting real-time requirements. When the CPU performs advance scheduling, it occupies and updates the KV Cache count for each request participating in this round of GPU forward inference. This operation further optimizes the inference process, ensuring that the GPU can more efficiently utilize cached data during inference. Working in conjunction with the asynchronous scheduling strategy, it improves overall inference performance.

[0038] In some embodiments, the inference request is "What is the temperature today?", the CPU receives this inference request, schedules the inference task for the inference request, extracts the key information of the inference request to obtain the inference data object, and transmits the inference data object to the GPU. The large model inference in the GPU performs inference based on the inference task object to obtain the inference word element of the inference request "Today's temperature is 28 degrees", and returns the inference word element to the CPU. The CPU receives the inference word element, extracts numerical information from the inference word element, and transmits the numerical information to the terminal, thereby completing an inference. The inference request obtains the inference result, thus ending the inference request, and updating the status of the inference request, and sending the updated status to the GPU through the inter-process message queue.

[0039] The GPU is used to obtain the inference data object of the tth round from the CPU when the CPU determines whether the inference word element of the inference request of the tth round contains numerical information, perform the inference of the tth round based on the inference data object of the tth round, and obtain the inference word element of the inference request of the t+1th round.

[0040] Specifically, when the CPU is determining whether the inference token of the inference request for round t contains numerical information, the GPU will obtain the inference data object for round t from the CPU. This inference data object is generated by the CPU after performing a series of operations on the inference request for round t, such as parameter parsing and computing resource allocation. It contains key information such as the model path, input tensors, and execution configuration. After obtaining the inference data object for round t, the GPU will carry out the inference work for round t based on it, processing and calculating the data in the inference data object according to the established algorithm and model structure. During this process, the GPU will perform layer-by-layer calculations and transformations on the input data, gradually generating inference results; through this series of inference operations, the inference tokens for the inference request for round t+1 are obtained. These inference tokens may be further involved in subsequent processes such as result judgment and task status update, driving the entire inference task forward.

[0041] Specifically, the reasoning data object of the tth round includes multiple reasoning data objects, and the multiple reasoning data objects include the newly received reasoning data object and the reasoning data object of the t-1th round. The CPU transmits the newly received reasoning data object in full to the GPU through the inter-process message queue, and the CPU incrementally transmits the reasoning data object of the t-1th round to the GPU through the inter-process message queue.

[0042] Full data transfer refers to the transmission of the entire dataset each time during the data transfer process. In the case of data synchronization between large-model inference processes, when a new inference request arrives at the CPU for the first time, all relevant variables, including Req_id (the unique ID of the current request), KV Cache list KV_Cache_Blocks allocated during attention calculation, sampling parameters Sampling_params, Prompt_ids (the token list obtained after word segmentation of the inference request prompt), and Computed_num (the number of tokens calculated for this inference request), are initialized into inference data objects and sent to the GPU execution process. The advantage of this approach is that the data is complete and comprehensive, allowing the recipient to obtain complete information for processing. However, the disadvantage is also obvious: the large amount of data transmitted, especially when the data is transferred multiple times and the majority of the content remains unchanged, consumes a lot of bandwidth and resources, increases communication overhead, and reduces transmission efficiency.

[0043] However, when a new inference request is transmitted between processes and data is synchronized, all relevant variables are initialized and sent to the GPU execution process. This ensures that the GPU has complete and accurate data when processing new requests, ensuring the smooth execution of inference tasks and avoiding errors or interruptions caused by missing data.

[0044] Incremental transfer transfers only the data that has changed since the previous step. After an inference request has been scheduled and processed once, subsequent inter-process data transfer and synchronization is performed using incremental transfer. For standard inference tasks, since GPU resources are sufficient for the current round of computation, only the GPU ID corresponding to the newly allocated KV Cache and the number of already calculated tokens need to be updated, eliminating the need to retransmit all data. For preemptive tasks, although the newly allocated KV Cache block list must be transferred in full, the overall data transfer volume is still reduced compared to a full transfer method that transfers all data each time. The advantage of incremental transfer is that it effectively reduces data transmission volume, lowers inter-process communication overhead, improves transmission efficiency, reduces resource waste, and improves system performance while ensuring data consistency.

[0045] The incremental transmission structure is defined as follows: Class Schedule_seq_running : { Req_id: str, used to specify the unique id of the current request.

[0046] New_KV_block_ids: list, as the generation length increases, the ids corresponding to the newly allocated KV Cache.

[0047] Computed_num: int, the number of tokens for this inference request and calculation, equivalent to the number of KV Cache caches.

[0048] } Specifically, the inference requests of the tth round include n inference requests, and the CPU determines whether the n inference requests are greater than the number m of inferences in the inference space of the GPU, and obtains a second judgment result; if the second judgment result indicates that the n inference requests are greater than the number m of inferences in the inference space of the GPU, the inference data objects corresponding to the nm inference requests are stored in the preemption queue; when the resources of the inference space of the GPU are sufficient, the CPU extracts the inference data objects corresponding to the nm inference requests from the preemption queue, and incrementally transmits them to the GPU through the inter-process message queue, and the GPU infers the inference data objects corresponding to the nm inference requests in sequence.

[0049] Inference services that are preempted due to GPU memory resource limitations are added to the preemption queue and prioritized when GPU resources are sufficient. Furthermore, a dedicated incremental transmission structure, Schedule_seq_preempt, is designed for preemption tasks to fully transmit the newly allocated KV Cache block list. This ensures efficient use of GPU resources while ensuring data integrity and accuracy when resuming computations for preempted tasks.

[0050] The incremental transmission structure of the preemptive task is designed as follows: Class Schedule_seq_preempt : { Req_id: str, used to specify the unique id of the current request.

[0051] KV_Cache_block_ids: list, ids corresponding to the newly allocated KV Cache.

[0052] Computed_num: int, the number of tokens for this inference request and calculation, equivalent to the number of KV Cache caches.

[0053] } The CPU also updates the inter-process message queue. When the inference token in the tth round of inference request contains numerical information, the inter-process message queue is updated based on the numerical information in the inference token in the tth round of inference request and transmitted to the GPU. By maintaining a list of finished inference request reqs IDs, or the inter-process message queue, cached inference request data can be released promptly during the GPU execution process. This helps optimize GPU memory management, avoids performance degradation caused by excessive memory usage, and improves GPU resource utilization.

[0054] GPU-side data synchronization mechanism design: 1. For the data inference object corresponding to the new inference request, initialize the data type and assign values to all inference data.

[0055] 2. For the data inference object corresponding to the running inference request, update the number of newly added KV Cache Blockids and computed num.

[0056] 3. For preempt tasks, fully update the KV Cache Block list.

[0057] 4. Release the data inference object corresponding to the cached inference request according to the finished req_id list.

[0058] In existing technologies, inter-process communication is expensive, and the inference content transmitted in each dispatch is often repeated. This application uses an incremental transmission mechanism to only transmit data that has changed compared to the previous step, significantly reducing the data communication time between the CPU and GPU processes and reducing communication overhead.

[0059] Due to the limitation of GPU memory resources, the existing technology will have preemption and recalculation problems in large model reasoning, resulting in repeated transmission of reasoning data. The present application processes preemption tasks by updating the calculation location and KV Cache index, etc., avoiding unnecessary recalculation and improving reasoning efficiency. The present application can perform reasonable resource scheduling according to the status of the reasoning request, such as prioritizing the reasoning requests in the preemption queue, and optimizing the allocation and management of KV Cache during the reasoning process, effectively improving the utilization rate of GPU memory resources, and avoiding task failure or inefficiency caused by unreasonable resource allocation. From the perspective of reasoning request status maintenance, the GPU process and the CPU process ensure the data consistency of the reasoning request. However, the existing technology may not be able to effectively guarantee data consistency when synchronizing data between processing processes. The present application makes up for this shortcoming and improves the reliability of the entire reasoning system.

[0060] Based on the above content, in the prior art, due to the Python thread GIL mechanism, CPU tasks and GPU reasoning wait for each other, resulting in a waste of resources. This application uses an asynchronous scheduling strategy to achieve parallel execution of CPU scheduling and GPU reasoning, which reduces this waiting time and improves resource utilization. By performing parallel data processing on CPU execution and GPU execution, the execution time of CPU execution in a single round of model reasoning process is minimized. The integrated reasoning speed comparison results show that for throughput indicators, this optimization solution can achieve an improvement of 15%-30% in different scenarios, significantly improving the reasoning rate of the model reasoning service. At the same time, it reduces the GPU forward reasoning data preparation time for each round, making the entire reasoning process smoother and more efficient, while not affecting the accuracy of model reasoning, and improving the overall performance of the system. This parallel processing method makes full use of the computing resources of the CPU and GPU, realizes asynchronous operation of the two, and further improves the overall reasoning efficiency.

[0061] Building on this foundation, we've also added incremental transfer and preemptive task methods. When transferring data between processes, only the data that has changed since the previous step is transferred. A full transfer is performed when a new inference request is first scheduled. Subsequent normal inference tasks only need to update the memory ID and calculated token count corresponding to the newly allocated KV Cache. Preemptive tasks also perform only the necessary full transfers for the newly allocated KV Cache block list (still reducing the amount of data compared to a full transfer). This significantly reduces data transfer volume, lowers communication overhead between CPU and GPU processes, and improves data transfer efficiency. When GPU memory resources are limited, preventing some inference tasks in the running queue from being allocated the required KV Cache count, the preemptive task mechanism adds these preempted inference services to the preemptive queue. When GPU resources are sufficient, inference requests in the preemptive queue are prioritized, enabling more efficient allocation of GPU resources to different inference tasks and improving resource utilization. This preemptive task mechanism ensures that all inference tasks have a chance to be processed even under resource constraints, ensuring system stability and reliability and enhancing the robustness of the entire large-model inference system.

[0062] The embodiment of the present application also provides an asynchronous scheduling method based on large model reasoning, which includes: the CPU is used to receive an inference request of the tth round, process the inference request of the tth round according to the inference task scheduling of the tth round, obtain the inference data object of the tth round, and transmit the inference data object of the tth round to the GPU; obtain the inference word element of the inference request of the tth round from the GPU, judge whether the inference word element of the inference request of the tth round contains numerical information, and obtain a first judgment result; if the first judgment result indicates that the inference word element of the inference request of the tth round contains numerical information, then update the state of the inference request of the tth round according to the inference word element of the inference request of the tth round, and obtain the inference result of the tth round; transmit the inference result of the tth round to the terminal; and prepare for the inference task scheduling of the t+1th round; When the GPU determines whether the inference word element of the inference request of the tth round contains numerical information, it obtains the inference data object of the tth round from the CPU, performs the inference of the tth round based on the inference data object of the tth round, and obtains the inference word element of the inference request of the t+1th round.

[0063] Specifically, the CPU is further configured to prepare for scheduling the inference task of the t+1th round if the first judgment result indicates that the inference word of the inference request of the tth round does not contain numerical information.

[0064] Specifically, the reasoning data object of the tth round includes multiple reasoning data objects, and the multiple reasoning data objects include the newly received reasoning data object and the reasoning data object of the t-1th round. The CPU transmits the newly received reasoning data object in full to the GPU through the inter-process message queue, and the CPU incrementally transmits the reasoning data object of the t-1th round to the GPU through the inter-process message queue.

[0065] Specifically, the inference request of the tth round includes n inference requests, and the CPU determines whether the n inference requests are greater than the number m of inference requests in the inference space of the GPU, to obtain a second judgment result; If the second judgment result indicates that the number of n inference requests is greater than the number of inferences m in the inference space of the GPU, the inference data objects corresponding to the nm inference requests are stored in the preemption queue; when the resources of the inference space of the GPU are sufficient, the CPU extracts the inference data objects corresponding to the nm inference requests from the preemption queue, and incrementally transmits them to the GPU through the inter-process message queue, and the GPU infers the inference data objects corresponding to the nm inference requests in sequence.

[0066] Specifically, the CPU is also used to update the inter-process message queue. When the inference word element of the inference request of the tth round contains numerical information, the inter-process message queue is updated according to the numerical information of the inference word element of the inference request of the tth round, and the updated inter-process message queue is transmitted to the GPU.

[0067] Specifically, the task inference scheduling of the CPU refers to sorting inference requests, updating inference states, and generating inference data objects corresponding to the inference requests.

[0068] Specifically, the CPU is also used to transmit the inference data objects corresponding to the n inference requests to the GPU if the second judgment result indicates that the n inference requests are not greater than the inference number m in the inference space of the GPU, and the GPU infers the inference data objects corresponding to the n inference requests in sequence.

[0069] The present application also provides a computing device. Figure 2 As shown, this figure is a schematic diagram of a computing device provided by an embodiment of the present application, wherein the computing device 400 includes a bus 401, a processor 402, a communication interface 403, and a memory 404. The processor 402, the memory 404, and the communication interface 403 communicate with each other via the bus 401.

[0070] The bus 401 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0071] The processor 402 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0072] The communication interface 403 is used for external communication. For example, when the computing device is a first switch, the communication interface 403 can be used for communication between the first switch and the first user terminal, or between the first switch and the second switch.

[0073] Memory 404 may include volatile memory, such as random access memory (RAM). Memory 404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0074] The memory 404 stores executable codes, and the processor 402 executes the executable codes to perform the aforementioned asynchronous scheduling method based on large model reasoning.

[0075] Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk, or magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned asynchronous scheduling method based on large model reasoning.

[0076] The present application also provides a computer program product comprising one or more computer instructions that, when loaded and executed on a computing device, fully or partially generate the process or function described in the present application.

[0077] The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0078] When the computer program product is executed by a computer, the computer performs any of the aforementioned methods for asynchronous scheduling based on large model reasoning. The computer program product may be a software installation package, which can be downloaded and executed on a computer when any of the aforementioned methods for asynchronous scheduling based on large model reasoning is needed.

[0079] The descriptions of the processes or structures corresponding to the above figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0080] The above description is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. An asynchronous scheduling system based on large model reasoning, characterized in that: The system includes a CPU and a GPU; The CPU is configured to receive an inference request for round t, process the inference request for round t according to the inference task scheduling for round t, obtain an inference data object for round t, and transmit the inference data object for round t to the GPU; obtain an inference word element of the inference request for round t from the GPU, determine whether the inference word element of the inference request for round t contains numerical information, and obtain a first judgment result; if the first judgment result indicates that the inference word element of the inference request for round t contains numerical information, update the state of the inference request for round t according to the inference word element of the inference request for round t, and obtain an inference result for round t; transmit the inference result of round t to the terminal; and prepare for the inference task scheduling for round t+1; The GPU is used for obtaining the inference data object of the tth round from the CPU when the CPU determines whether the inference word element of the inference request of the tth round contains numerical information, performing the inference of the tth round based on the inference data object of the tth round, and obtaining the inference word element of the inference request of the t+1th round.

2. The system according to claim 1, wherein: The CPU is further configured to prepare for scheduling the reasoning task for the t+1th round if the first judgment result indicates that the reasoning word element of the reasoning request for the tth round does not contain numerical information.

3. The system according to claim 1, wherein: The reasoning data object of the tth round includes multiple reasoning data objects, and the multiple reasoning data objects include the newly received reasoning data object and the reasoning data object of the t-1th round. The CPU transmits the newly received reasoning data object in full to the GPU through the inter-process message queue, and the CPU transmits the reasoning data object of the t-1th round to the GPU incrementally through the inter-process message queue.

4. The system according to claim 1, wherein: The inference request of the tth round includes n inference requests, and the CPU determines whether the n inference requests are greater than the number m of inference requests in the inference space of the GPU, and obtains a second judgment result; If the second judgment result indicates that the number of n inference requests is greater than the number of inferences m in the inference space of the GPU, the inference data objects corresponding to the nm inference requests are stored in the preemption queue; when the resources of the inference space of the GPU are sufficient, the CPU extracts the inference data objects corresponding to the nm inference requests from the preemption queue, and incrementally transmits them to the GPU through the inter-process message queue, and the GPU infers the inference data objects corresponding to the nm inference requests in sequence.

5. The system according to claim 1, wherein: The CPU is also used to update the inter-process message queue. When the inference word element of the inference request of the tth round contains numerical information, the inter-process message queue is updated according to the numerical information of the inference word element of the inference request of the tth round, and the updated inter-process message queue is transmitted to the GPU.

6. The system according to claim 1, wherein: The CPU task reasoning scheduling refers to sorting reasoning requests, updating reasoning states, and generating reasoning data objects corresponding to the reasoning requests.

7. The system according to claim 4, wherein: The CPU is also used to transmit the inference data objects corresponding to the n inference requests to the GPU if the second judgment result indicates that the n inference requests are not greater than the inference number m in the inference space of the GPU, and the GPU infers the inference data objects corresponding to the n inference requests in sequence.

8. An asynchronous scheduling method based on large model reasoning, characterized in that: The method is applied to a CPU and a GPU, the CPU and the GPU are processed in parallel, and the method includes: The CPU receives an inference request for round t, processes the inference request for round t according to the inference task scheduling for round t, obtains an inference data object for round t, and transmits the inference data object for round t to the GPU; obtains an inference word element of the inference request for round t from the GPU, determines whether the inference word element of the inference request for round t contains numerical information, and obtains a first judgment result; if the first judgment result indicates that the inference word element of the inference request for round t contains numerical information, updates the state of the inference request for round t according to the inference word element of the inference request for round t, and obtains an inference result for round t; transmits the inference result of round t to the terminal; and prepares for the inference task scheduling for round t+1; When the GPU determines whether the inference word element of the inference request of the tth round contains numerical information, it obtains the inference data object of the tth round from the CPU, performs the inference of the tth round based on the inference data object of the tth round, and obtains the inference word element of the inference request of the t+1th round.

9. A computing device, characterized in that Comprising the system according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that Comprising the system according to any one of claims 1-7.

Images