Scheduling method, device, medium, program and electronic device in model inference process

Abstract

The application discloses a scheduling method and device in a model inference process, a medium, a program and an electronic device, relates to the technical field of model inference, and performs necessary initialization on a queue when an initial request is processed, ensures that the request can be correctly managed and tracked when being scheduled, and constructs a pre-filled queue based on a resource limit value and a priority, so that the scheduler can dynamically adjust the batch processing size, avoids resource waste or overload operation, and performs next round inference based on a model inference result, avoids calculation delay caused by untimely result processing in a traditional method, improves the scheduling efficiency, and thus solves the problem of low scheduling efficiency of a model inference service in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model inference, in particular to a scheduling method in a model inference process, a scheduling device in a model inference process, a computer readable storage medium, a computer program product and an electronic device. BACKGROUND

[0002] In the field of large model inference, existing inference acceleration technologies mainly focus on optimizing model structure, operator kernel, graph optimization and compilation, inference engine, and parallel computing. Although these methods can significantly improve the efficiency of a single inference task, they still have the following shortcomings when facing high concurrency requests and resource constraints:

[0003] Existing scheduling mechanisms often have difficulty adjusting resource allocation in real time, especially in continuous batching, where they fail to effectively handle dynamic changes in request queues and resource states, resulting in resource waste or task delays.

[0004] In the process of large model inference, frequent memory access and data copying can significantly increase latency, especially when communicating across devices. Existing technologies often overlook optimizing memory management strategies, resulting in limited overall performance.

[0005] That is, the existing model inference service faces low scheduling efficiency. SUMMARY

[0006] The present application provides a scheduling method in a model inference process, a scheduling device in a model inference process, a computer readable storage medium, a computer program product and an electronic device to at least solve the problem of low scheduling efficiency faced by the existing model inference service.

[0007] The present application provides a scheduling method in a model inference process, comprising: receiving a processed inference request sent by a tokenizer and initializing each queue to obtain an initialized queue, the processed inference request comprising at least one wordpiece; extracting corresponding wordpieces from a request waiting queue according to resource limits and the priority of the inference request to construct a pre-filled queue, the resource limits comprising the maximum number of requests, the upper limit of the number of wordpieces, and the upper limit of the number of low-rank adaptive models, the request waiting queue being one of the initialized queues; scheduling the pre-filled queue and a decoding task corresponding to the pre-filled queue to obtain at least a model inference result.

[0008] The application further provides a scheduling device in a model inference process, comprising: a first processing unit configured to receive a processed inference request sent by a tokenizer and perform initialization processing on each queue to obtain an initialized queue, wherein the processed inference request comprises at least one wordpiece; a second processing unit configured to extract a corresponding wordpiece from a request waiting queue according to a resource limit value and a priority of the inference request to construct a pre-filled queue, wherein the resource limit value comprises a maximum number of requests, an upper limit of the number of wordpieces and an upper limit of the number of low-rank adaptive models, and the request waiting queue is one of the initialized queues; and a third processing unit configured to schedule the pre-filled queue and a decoding task corresponding to the pre-filled queue to obtain at least a model inference result.

[0009] The application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the scheduling method in the model inference process when executed by a processor.

[0010] The application further provides a computer program product, comprising a computer program, and the computer program is configured to implement the steps of the scheduling method in the model inference process when executed by a processor.

[0011] The application further provides an electronic device, comprising: a memory configured to store a computer program; and a processor configured to implement the steps of the scheduling method in the model inference process when executing the computer program.

[0012] According to the application, the queue is initialized when an initial request is processed, so that the request can be correctly managed and tracked during scheduling, and resource allocation errors caused by improper initialization are avoided. The pre-filled queue construction strategy based on the resource limit value and the priority enables the scheduler to dynamically adjust the batch size, avoids resource waste or overload operation, and ensures that high-priority requests are processed preferentially. The processing of the model inference result includes updating the request state, managing and releasing the cache resource, and streaming the result, which prepares for the next stage of inference and avoids the calculation delay caused by untimely result processing in the traditional method, improves the scheduling efficiency, and solves the problem of low scheduling efficiency of the model inference service in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the application, the drawings required in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1A flowchart of a scheduling method in a model inference process provided by an embodiment of the present application is shown in the figure;

[0015] Figure 2 A schematic diagram of a scheduler architecture provided by an embodiment of the present application is shown in the figure;

[0016] Figure 3 A schematic diagram of a scheduler workflow in a single process provided by an embodiment of the present application is shown in the figure;

[0017] Figure 4 A flowchart of a scheduling method in a model inference process provided by an embodiment of the present application is shown in the figure;

[0018] Figure 5 A structural block diagram of a scheduling device in a model inference process provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0020] It should be noted that, in the description of the present application, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0021] In order to make the skilled in the art better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0022] An embodiment of the present application provides a scheduling method in a model inference process, as shown in the figure, the method comprises the following steps: Figure 1

[0023] Step S101, receiving the processed inference request sent by the word segmenter, and initializing each queue to obtain an initialized queue, the processed inference request comprising at least one word unit;

[0024] ​Step S102, extracting corresponding tokens from the request waiting queue according to resource limits and priority of the inference request to construct a pre-populated queue, the resource limits including a maximum number of requests, an upper limit on the number of tokens, and an upper limit on the number of low-rank adaptive models, and the request waiting queue being one of the initialized queues;

[0025] Step S103, scheduling the pre-populated queue and the decoding task corresponding to the pre-populated queue to obtain at least a model inference result.

[0026] Among them, the state in the inference process can also be obtained, such as the query state.

[0027] In the above steps, the inference request is segmented by the segmenter to convert the natural language text into Token sequence (token) that can be understood by the model, preparing for subsequent model processing. When processing the initial request, the queue is initialized as necessary to ensure that the request can be correctly managed and tracked during scheduling, avoiding resource allocation errors caused by improper initialization. This pre-populated queue construction strategy based on resource limits and priority allows the scheduler to dynamically adjust the batch size, avoiding resource waste or overload, while ensuring that high-priority requests are processed first. Processing the model inference result includes updating the request state, managing and releasing cache resources, and streaming the result, preparing for the next stage of inference, avoiding the calculation delay caused by the traditional method due to the delay in processing the result, improving the scheduling efficiency, thereby solving the problem of low scheduling efficiency faced by the model inference service in the prior art.

[0028] In an embodiment of the present application, extracting corresponding tokens from the request waiting queue according to resource limits and priority of the inference request to construct a pre-populated queue includes: creating a pre-populated request adder; sorting the inference requests of the request waiting queue according to the priority to obtain a sorting result; and calling the pre-populated request adder to construct the pre-populated queue according to the sorting result and the resource limits.

[0029] Specifically, by the pre-filling request adder, the system can dynamically filter suitable requests for processing according to the current resource conditions (such as the graphic processor memory, the cache size) and the request priority. In this way, resource waste can be avoided, and it can be ensured that the pre-filling queue scheduled each time can be efficiently executed under the existing resource conditions. The pre-filling request adder constructs the pre-filling queue based on the sorting result and the resource limit value, which can ensure the diversity and complementarity of the requests in the batch, and avoid low efficiency caused by too large differences in input length or resource demand between requests. According to the priority sorting and the resource limit value, the scheduler can flexibly adjust the content of the pre-filling queue, process urgent or high-priority requests, and at the same time ensure that low-priority requests are fairly processed according to the availability of resources. By preferentially processing high-priority requests, a better balance point between user waiting time and system resource utilization can be found, so as to improve the response speed of the overall service and user satisfaction.

[0030] In an embodiment of the present application, after the corresponding word pieces are extracted from the request waiting queue according to the resource limit value and the priority of the inference request to construct the pre-filling queue, the method further comprises: determining whether the number of paths of the low-rank adaptive model of the candidate request and / or whether the current running batch exceeds the resource limit value; in the case that the number of paths of the low-rank adaptive model of the candidate request and / or the current running batch exceeds the resource limit value, determining that the batch is full and terminating the inference; initializing the next round of input and calling the pre-filling request adder to add the inference request to the pre-filling queue.

[0031] Specifically, by checking the number of LoRA (Low-Rank Adaptation, a method for efficient fine-tuning on large pre-trained models. LoRA reduces the number of parameters that need to be adjusted by decomposing the weight matrix of the model into a low-rank matrix, thereby reducing the computational and storage overhead) paths and the resource usage of the current running batch, it can avoid over-allocation of resources to a single request, ensure that resources are more evenly distributed among multiple requests, and thus improve overall resource utilization. Too many LoRA paths can increase the computational complexity of the model and may cause performance degradation. Checking the number of LoRA paths can prevent this situation and ensure that the system does not become a performance bottleneck due to too many LoRA models during operation. In resource-constrained situations, by determining whether the batch is full, the system can dynamically adjust the batch size to avoid inference failures or delays due to insufficient resources, ensuring the stable operation of the system. Terminate inference requests that exceed the resource limit to avoid long sequence requests or resource-intensive requests occupying too many resources, causing other requests to be unable to be processed in time, thereby improving the overall throughput of the system in processing requests. In a parallel computing environment, ensuring the consistency of requests among parallel processes is crucial for correct inference results. Through the above steps, the distribution of requests can be effectively managed to ensure that all parallel processes are processing consistent request data. By avoiding over-allocation of resources to a single request, the waiting time for processing user requests can be reduced, response speed can be improved, and user experience can be enhanced. The steps of initializing the next round of input and calling the pre-filled request adder to add inference requests to the pre-filled queue simplify the scheduling logic, making it more modular and easier to maintain, and also facilitate subsequent optimization and adjustment of the scheduling strategy.

[0032] In an embodiment of the present application, the method further comprises: in the case of a pre-filled task in the last round of inference, determining whether the batch of the current round of inference is empty; in the case that the batch of the current round of inference is empty, fusing the pre-filled task of the last round; in the case that there is an unfinished sub-block pre-filled request, inserting the unfinished sub-block pre-filled request and the corresponding index into the index of the vocabulary value cache, and removing the unfinished sub-block pre-filled request in the batch.

[0033] The present application also provides a specific use scenario for fusing the pre-filled task of the last round: in the field of natural language processing, especially in chat robots or text generation applications, users may continuously input multiple instructions or questions to conduct a conversation. For example, in a customer service chat robot system, a user may first ask a simple question, such as "What is the business hours?", and then continue to ask, "What about holidays?" based on the received answer. This continuous, context-dependent conversation mode requires the inference engine to efficiently handle continuous requests, not just single generation tasks.

[0034] Use case details: In traditional text generation processes, each user request is treated as an independent task, with pre-filling (processing the prompt, which refers to a specific input form used to guide the AI model to generate a certain type of output) and decoding (generating responses). This approach is inefficient in continuous dialogue scenarios, as each request must start from scratch to build a batch, even if the results of the previous request can be utilized by the next request. With the scheduler of this patent, the processing flow of continuous dialogue can be optimized. Specifically, in the case of a pre-filling task in the previous round of reasoning, if there is no new request at the moment, the scheduler will fuse the pre-filling task of the previous round and continue decoding using the existing model state without rebuilding the batch. This avoids repeated calculations, especially for pre-filling steps that require a lot of computing resources (such as the self-attention mechanism of the Transformer model that handles long texts, which completely relies on self-attention mechanisms to handle sequence data, eliminating the dependence on recurrent networks, greatly improving training speed and model effectiveness. The Transformer model does not need to consider the order of elements in the sequence when processing sequence data, but learns the relationship between elements (i.e., attention weights) to achieve understanding and generation of the sequence); in addition, for unfinished chunk pre-filling requests, the scheduler will save these requests and corresponding indexes in the vocabulary value cache and continue to utilize them in subsequent batch processing. This mechanism is particularly suitable for handling long texts or generation tasks involving complex grammatical constraints, as long texts may need to be processed in chunks, and the pre-filling results of each chunk can be cached and reused to avoid repeated calculations.

[0035] The beneficial effects of fusing the pre-filling task of the previous round in specific use cases: Through continuous batch processing and cache reuse mechanisms, repeated pre-filling calculations are reduced, significantly improving the processing speed of continuous dialogue or long text generation; avoids the delay of rebuilding the batch for each request, making the system's response more rapid and the user experience better; more precise in memory and computing resource management, effectively utilizing GPU (Graphics Processing Unit, graphics processor) and CPU (Central Processing Unit, central processing unit, is the main operation and control center of the computer) resources, reducing resource waste. Better support for generation tasks involving long texts and complex grammatical constraints, improving the applicability and efficiency of the model in these scenarios.

[0036] In an embodiment of the present application, the method further comprises: in a case where it is determined that the vocabulary value cache meets the needs of the next round of decoding, reducing the number of slots or computing resources of the vocabulary value cache allocated for new input data; in a case where it is determined that the vocabulary value cache does not meet the needs of the next round of decoding, withdrawing a target reasoning request, the target reasoning request being a reasoning request with a priority lower than a priority threshold; and generating at least part of the output conforming to the grammatical constraint according to the rule of the grammatical constraint.

[0037] Specifically, the inference of large models often requires a large amount of video memory, and reasonable reduction of cache allocation can avoid video memory overflow and ensure that more requests can be processed; dynamic adjustment of resource allocation makes limited computing resources more efficiently serve real-time requests, reducing resource waste; fewer cache slots mean more concurrent opportunities, thereby improving the overall throughput of the system. In the case of resource shortage, withdrawing lower-priority requests helps to optimize system performance. This can release cache space by withdrawing low-priority requests when the KV cache capacity reaches the upper limit, freeing up resources for higher-priority requests; ensure that critical or urgent tasks are not affected by resource limitations and are prioritized for processing and response; through fine request management and resource allocation, avoid service delays or interruptions due to resource competition, and maintain good user experience. The fast generation method based on grammatical constraints can intelligently generate partial prediction results without the need for a complete decoding process. For scenarios with explicit grammatical constraints, such as question answering and code completion, the system can quickly generate preliminary results that conform to the rules, significantly reducing user waiting time. Avoid unnecessary computational overhead, especially in long sequence generation tasks, the partial results generated in advance can significantly reduce the total amount of computation. The introduction of grammatical constraints helps guide the model to generate more reasonable and coherent outputs, improving the quality and usability of the generated content.

[0038] In an embodiment of the present application, the method further comprises: obtaining the occupation of the graphics processor in real time; and adjusting the number of reasoning requests in the pre-filled queue according to the occupation.

[0039] Specifically, by monitoring the real-time occupancy of the GPU, the number of requests in the pre-filling queue can be increased when the GPU load is low, thus making full use of idle computing resources. Conversely, the number of requests is reduced when the GPU is close to full load, avoiding overloading the GPU and causing performance degradation or long task queuing waiting time. Dynamically adjusting the number of requests in the pre-filling queue can optimize the batch size, ensuring that the GPU can achieve optimal efficiency when processing each batch. Smaller batches can be completed faster under high load, while larger batches can more efficiently utilize the parallel computing capabilities of the GPU under low load, thus improving the overall throughput of the system. Reasonably adjusting the number of requests in the pre-filling queue helps to reduce task waiting time, especially when the GPU load fluctuates greatly, it can quickly respond to adjust the request processing strategy and reduce the average delay of user requests.

[0040] In an embodiment of the present application, the method further comprises: determining the similarity of the input features of the current batch and the historical requests to obtain a current similarity; and determining whether to reuse the results in the vocabulary value cache according to the size of the current similarity.

[0041] Specifically, by identifying similar input features, unnecessary calculations can be avoided, especially for large language models, repeated calculations can consume a large amount of computing resources. Reusing the results in the cache can significantly improve inference speed and reduce latency; model inference often involves a large amount of memory read and write, especially the use of KV Cache (Key-Value Cache, a cache mechanism used to store key-value pairs in large model inference, especially in the generation task of autoregressive models such as Transformer). The key (Key) and value (Value) refer to two components in the attention mechanism (Attention Mechanism)). Reusing the calculation results of similar requests reduces the use of memory bandwidth, helping to alleviate memory bottlenecks and improve overall resource utilization; similarity evaluation provides a basis for dynamically adjusting resource allocation. When a high similarity is detected between the current batch and the historical requests, the scheduler can prioritize resource allocation to batches with less difference, ensuring optimal use of resources; in a high-concurrency scenario, reusing cache results based on similarity can improve the system's ability to handle more requests, thus increasing the throughput of the server.

[0042] In an embodiment of the present application, determining whether to reuse the results in the vocabulary value cache according to the size of the current similarity comprises: determining that the results in the vocabulary value cache do not need to be reused when the current similarity is less than a similarity threshold; and determining that the results in the vocabulary value cache need to be reused when the current similarity is greater than or equal to the similarity threshold.

[0043] Specifically, the similarity threshold can be 80%, and when the similarity is greater than or equal to the similarity threshold, it means that the current request shares some context or features with one or more requests in the cache to some extent. By reusing the cache results, repeated calculations on the same or similar data are avoided, saving computing resources and memory consumption, and improving the inference speed of the model. For requests with high similarity, instead of re-executing the entire inference process, the computing results are directly obtained from the cache, which can significantly reduce the response time and improve the user experience. Especially in real-time or high-concurrency scenarios, this strategy is crucial to maintaining low latency of the system. In the self-attention (Self-Attention) and feed-forward neural network (FFN) calculations of the model, the reuse of cache results reduces unnecessary operator calculations, especially in attention algorithms, avoiding the generation of multiple KV Caches, and reducing the computational complexity.

[0044] In an embodiment of the present application, the method further comprises: in the case where the tensor parallel process exists, controlling the tensor parallel process to receive input data directly from the tokenizer; in the case where the data parallel thread exists, splitting the inference request into multiple sub-requests and distributing them to each data parallel thread.

[0045] Specifically, in the TP parallel (Tensor Parallelism, Tensor Parallelism mainly aims at the parallel processing of model parameters. In deep learning models, there are many large weight matrices, and TP parallel splits these weight matrices into multiple parts and allocates them to different GPUs for calculation. For example, a large matrix multiplication operation can be split into multiple small matrix multiplications, which are performed simultaneously on multiple GPUs, and then the results are combined. This method is suitable for cases where the model is very large and a single GPU cannot completely store or efficiently calculate. TP parallel can significantly reduce the memory occupancy of each GPU and speed up the forward propagation and back propagation of the model through parallel calculation), each GPU processes different parts of the model tensor, and they share the same data input. Therefore, having each TP process directly receive input data from the tokenizer avoids additional data transmission and synchronization overhead, especially under high concurrency requests, which greatly reduces the communication burden of the master process and improves the overall communication efficiency. In the DP parallel (Data Parallelism, Data Parallelism mainly aims at the parallel processing of data. In DP parallel, the parameters of the model are consistent on all GPUs participating in the calculation. The data is split into multiple subsets, and each subset is calculated on a different GPU for forward propagation and back propagation. After the calculation is completed, the gradients on each GPU are aggregated, and the model parameters are updated. This method is suitable for cases where the data volume is large and the model can be adapted to a single GPU. DP parallel can improve the throughput of data processing, but needs to solve the problem of gradient synchronization to ensure the correctness of parameter updating), the dataset is split, and each thread processes a different part of the data. By splitting the original request into sub-requests matching the number of DP threads and distributing them to each thread, parallel processing of data is ensured, while also avoiding redundant calculations between threads, improving the utilization efficiency of computing resources.

[0046] In an embodiment of the present application, the method further comprises, in the case of inputting multi-modal data, inputting and combining processing of corresponding text and images.

[0047] Specifically, by integrating text and image inputs in the preprocessing stage, the number of model calls can be reduced, avoiding repetitive preprocessing operations, thereby improving the overall inference speed. For example, in the Visual Question Answering (VQA) task, the problem text and related images are pre-combined, which can be processed in one pass, rather than separately and then merged the results. Integrating multi-modal input can more effectively utilize computing resources, especially when computing on GPU, it can maximize the use efficiency of video memory, avoid resource fragmentation and waste caused by independent processing between modalities. The combined processing of multi-modal input allows the model to extract information from both text and image simultaneously, which is crucial for understanding complex scenarios, emotions or context. For example, in sentiment analysis, the model can jointly determine the sentiment polarity through facial expressions in the image and tone in the text, thereby improving the accuracy of analysis.

[0048] In an embodiment of the present application, the method further comprises: recording the pre-filled statistical information in the form of a log, the pre-filled statistical information including the number of new requests, the number of tokens and the cache hit rate.

[0049] Specifically, the number of new requests helps monitor the load of the system, understanding the busy degree of the server at a specific time, which is very important for predicting resource demand and avoiding overload.

[0050] The number of tokens reflects the size of the input processed by the scheduler, which helps to analyze the effective use of computing resources and identify whether there are too many small batch requests, which may affect the overall processing efficiency. The cache hit rate provides feedback on the effectiveness of the cache strategy, and a high cache hit rate means that the scheduler can effectively reuse previous calculation results, reducing the cost of calculation and improving the response speed.

[0051] This application uses FastAPI and Uvicorn to build a large-scale model inference server. FastAPI is a fast (high-performance) web framework based on Python (a high-level programming language) for building APIs. Uvicorn is a fast ASGI (Asynchronous Server Gateway Interface, a standard interface specification in Python) server that can efficiently run asynchronous Python applications. Using Uvicorn to run the aforementioned FastAPI application combines Uvicorn's high-performance ASGI server with FastAPI's asynchronous programming features based on Python, enabling it to handle a large number of concurrent requests and exhibiting excellent performance.

[0052] The scheduler is the core of the inference server, responsible for coordinating the entire lifecycle management of model inference tasks, including request reception, task scheduling, memory / cache management, inter-process communication (IPC), and performance monitoring. As the "brain" of the distributed inference system, the scheduler is responsible for transforming external requests (generation / embedding) into GPU-executable computation batches and dynamically adjusting resource allocation to optimize throughput and latency.

[0053] The client requests data via the FastAPI interface, which is then processed into token data by the tokenizer and sent to the scheduler. The scheduler schedules workers (parallel worker threads or processes that handle inference tasks) to perform inference on the token data according to the configuration. The resulting token data is sent to the desegmentation manager to be restored into readable words, and then integrated by the tokenizer (a tool or component in natural language processing used to segment input text into smaller units) and sent to the client.

[0054] Scheduler architecture such as Figure 2 As shown ( Figure 2 (The request queue is not shown). It consists of a tokenizer interface, an initialization module, a request processing module, a request queue, a scheduling logic module, an inference scheduling module (located between the scheduling logic module and the inference result processing module), an inference result processing module, and a desegmentation interface module. The scheduler receives requests from the tokenizer, processes them through each module, and finally sends them to the desegmenter.

[0055] The communication interface of the scheduler with the tokenizer and the tokenizer is based on the ZMQ (ZeroMQ, a high-performance asynchronous message library designed to provide fast and flexible communication mechanisms. It is used to implement message passing in distributed applications) communication socket interface, which is used for inter-process communication between inference server components. When there are multiple GPUs, a process is created for each rank (performance ranking or level), and a scheduler is instantiated in the process to schedule the GPU resources of the current rank to complete the distributed inference task. At this time, only the main process will instantiate the communication interface, use the non-blocking receiving function of ZeroMQ to receive the Tokenization request of the tokenizer, and broadcast to other processes, thereby improving communication efficiency and ensuring the consistency of requests of parallel instances.

[0056] The initialization module is the core initialization entry of the scheduler, responsible for configuration parsing, communication initialization, model / memory management component startup, scheduling strategy setting, and other key steps, to ensure that the scheduler can coordinate tensor parallel GPU work processes to handle generation or embedding requests. Parameter parsing and distributed information initialization, such as service configuration, port configuration, GPU number, tensor parallel rank, data parallel rank, and other key configurations. Inter-process communication initialization, which allocates communication sockets for the main process with the tokenizer and tokenizer. Model and tokenizer initialization, first load the model configuration, then load the tokenizer or multi-modal processor according to the configuration. Worker initialization, the worker performs CPU tasks and GPU calculations through asynchronous threads and manages GPU tasks and resources; the initialization of the worker mainly includes parameter parsing and configuration, model configuration loading, model inference service loading, input / output queue initialization, inference service thread startup, and other operations, initializes the memory pool and cache; run state initialization: initializes the request waiting queue, running batch, current batch, and other scheduling-related state variables; scheduling strategy initialization; performance analyzer initialization: initializes the Torch performance analyzer (a tool for analyzing and debugging model performance) related variables for subsequent performance analysis.

[0057] Scheduler application: the scheduler workflow in a single process is as follows Figure 3As shown, an instance of the scheduler is instantiated and the scheduler state is sent to the main process; a double-ended queue is initialized for storing results, decoupling the request reasoning and the reasoning result processing, so that the two processes are overlapped, improving the reasoning efficiency; an infinite loop of reasoning is entered: receiving requests, request processing, obtaining a scheduled batch of requests, reasoning of the request batch, and continuing to receive requests. If the scheduled batch of requests is failed to be obtained, there are two possibilities, one is that there is no request, at this time only some state checks need to be done; the other is the last batch, at this time only the reasoning result needs to be obtained from the queue and processed; if the scheduled batch of requests is successfully obtained, after the processing is completed, it is still necessary to judge whether it is the first batch, if it is the first batch, a virtual scheduled batch needs to be constructed to trigger some processes of the request processing, so that the communication, reasoning and reasoning result processing processes are overlapped, improving the reasoning efficiency.

[0058] The thread starts, the process configuration and parameter initialization, the scheduler is instantiated and the ready state is sent to the parent process, the double-ended queue is initialized, the request is received from the tokenizer, the input request is processed, converted into an internal object, and the scheduled request batch is obtained.

[0059] Among them, in the case of successful acquisition, the request batch is reasoned and the result is stored in the double-ended queue, a virtual request scheduling batch is created to trigger the information acquisition of the next batch, so that the communication process and the reasoning process are overlapped; then it is determined whether it is the last batch, if it is in the case of acquisition failure, it is also directly determined whether it is the last batch; in the case that the current batch is the last batch, the result is obtained from the queue and processed, the current batch is assigned to the last batch, and then the request from the tokenizer is received again; in the case that the current batch is not the last batch, it is determined that the service is idle, the memory check is performed and the state is reset, the current batch is assigned to the last batch, and then the request from the tokenizer is received again.

[0060] The request processing module comprises a request receiving component and a request processing component. The request receiving component is responsible for receiving requests from the front end (such as a word segmenter) and broadcasting the requests to all relevant processes in a tensor parallel (TP) or data parallel (DP) scenario, ensuring the consistency of requests of each parallel process. When receiving a request, a corresponding interface function is used and a request queue is set to avoid blocking when receiving a request, improving concurrent efficiency. When DP exists, the request is further classified into a work request and a control request, optimizing the broadcast efficiency (for example, work requests may be more frequent and need to be synchronized within a smaller parallel group. The request processing component is responsible for distributing the received request categories to the corresponding processing functions and returning the processing results to the caller. Requests and corresponding processing functions appear in pairs and can be expanded according to needs. Currently, the request categories include: word table generation requests (such as generating a response after a user inputs text), embedding vector generation requests (such as obtaining a feature representation of input text), aborting an executing request (such as a user actively canceling a task), opening a session (for long dialogue scenarios, maintaining context state), closing a session (ending a long dialogue and releasing session resources), loading and updating model weights from a disk (such as hot updating model parameters), initializing a weight update group (preparation work for distributed weight synchronization), synchronizing model weights from a distributed system (such as parameter synchronization after multi-node training), releasing memory occupation (such as temporarily releasing video memory to run other tasks), performance analysis requests (collecting performance indicators of model runtime), processing remote procedure call requests (interacting with other service modules), and the like. The request processing component will call the corresponding function to process the corresponding request according to the request category.

[0061] The flowchart of the parallel case is shown in Figure 4 The configuration interface, the global environment is configured, and the server interface is initialized. In the case of existing tensor parallel processes, the tensor parallel processes directly receive input data from the word segmenter. In the case of existing data parallel threads, the inference request is split into multiple sub-requests and distributed to each data parallel thread. In the case of the current node being the master node, a manager process is created and the manager is initialized, and the server is started after all scheduler child processes complete model loading, so that the server is in a state of waiting for access requests. In the case of the current node not being the master node, the server is started after all scheduler child processes complete model loading, so that the server is in a state of waiting for access requests.

[0062] Take the word template generation request processing function as an example, it is responsible for converting the user's generation request into an internal processable request object (essentially a new request data structure that integrates the request information, configuration information, inference information, etc. required by subsequent functional modules), and completes the pre-operation of input verification, multi-modal processing, constraint generation syntax cache initialization, etc. Finally, the legal request is added to the scheduling queue. The main steps are: session validity check and request initialization, when the user sends a generation request, it may carry session parameters, so the historical context can be reused, otherwise a new request is created directly; multi-modal input processing, if the model supports multi-modal input (such as images), it needs to combine image embedding with text input; input length verification, to prevent long input from causing model calculation errors or memory overflow; other rationality verification, such as syntax verification, maximum generation length verification, etc.

[0063] The scheduling logic module: the scheduler receives requests through the communication interface and calls the request receiving component, verifies the legality of the request through the request processing component, and creates a request that can be processed by the subsequent component (essentially a new request data structure that integrates the request information, configuration information, inference information, etc. required by subsequent functional modules), and adds it to the request waiting queue. Model inference is usually performed in batches, and the function of the scheduling logic module is to determine the next batch task to be executed (ScheduleBatch). It will determine whether to prioritize pre-filling (Prefill) or decoding (Decode) tasks based on the current system state (such as running tasks, waiting queues, cache states, etc.), to achieve efficient continuous batching. Specifically: when the last task is a pre-filling task, if the current batch is not empty, integrate the last pre-filling task to achieve continuous batching; if there are unfinished chunk pre-filling requests, retain the unfinished requests and request indexes that have not been chunked, and insert the corresponding Token and KV index into the cache index. In addition, remove the corresponding from the batch and release the cache resources. These unfinished requests will be added to the next batch in the next step. Determine whether the pre-filling request has been completed; if there is a pre-filling processing request, filter the requests that meet the conditions from the waiting queue and generate a new pre-filling batch. If there is no pre-filling processing request, dynamically update the decoding batch that is currently running. In the update process, it will also handle request contraction when there is not enough memory, check and handle jump optimization under syntax constraints, and update the batch state to ensure the stability and resource utilization of the subsequent decoding steps. The scheduling logic mainly exists in the last two steps in the above steps: pre-filling batch generation and decoding batch (decoding batch) update.

[0064] The scheduling module is responsible for filtering requests that meet the conditions from the waiting queue and generating new prefill batches. The prefill batch is used to process the initial input text (such as user input prompt) for the model, preparing for the subsequent decoding (generating output Token) stage. The main functions include: if batch_is_full (batch is full) is true, or the waiting queue is empty and there is no request being chunked (being_chunked_req), return None and do not generate a new batch; calculate the number of running requests (running_bs), if it exceeds the maximum number of running requests (max_running_requests), mark the batch as full and return; use the scheduling strategy to calculate the priority of the requests in the waiting queue (prefix_computed) to determine the processing order. The scheduling strategy includes a tree-aware scheduling strategy and a tree-unaware scheduling strategy. The former relies on tree caching to optimize scheduling, aiming to improve cache utilization, and contains the longest prefix matching strategy, which prioritizes scheduling requests that match the longest prefix with the existing cache, and the depth-first weight strategy, which prioritizes scheduling requests with high node weights (weights are usually determined by factors such as the number of requests in the subtree). The latter contains the common first-come-first-served, longest-output-first, and random strategies. Create a prefill request adder with resource management instance to manage the addition of prefill requests, mainly used to dynamically filter requests that meet resource limits during the prefill stage of generating tasks (initial stage of processing user input prompt), and manage the chunked processing of requests to ensure that batch construction does not exceed memory or Token pool limits; if there is a request being chunked, initialize the next round of input and add it to the prefill batch through the request adder; if there is a LoRA (Low Rank Adaptation) path configuration, check if the number of LoRA paths of the current running batch and candidate requests exceeds the maximum limit per batch to avoid excessive different LoRA models affecting performance; iterate through each request in the waiting queue, check the LoRA number and running request number limit, if it exceeds, mark the batch as full and terminate the loop. Initialize the next round of input for the request, then try to add the request to the prefill batch through the request adder. According to the return result, judge whether to continue adding, if there is no available Token, adjust the state of whether the batch is saturated according to the hierarchical cache configuration; remove the requests that have been added to the prefill batch from the waiting queue, update the state of processing chunked requests; record prefill statistical information, including the number of new requests, Token number, cache hit rate, etc., for monitoring and performance analysis. Use the newly created scheduling batch as the prefill batch, and prepare the parameters and state required for extension (prefill).The batch request container managed by the scheduling batch scheduler stores the high-level scheduling information of a batch of requests and is a key data structure connecting the request scheduling logic and model inference execution. Its core functions are:

[0065] Integrate information of multiple requests (Reqs), uniformly manage resources (such as memory pools, caches), prepare input data (such as Token sequences, sampling parameters) for model inference, and transfer batch metadata between the scheduler and model worker nodes (Workers).

[0066] Hybrid chunk processing: If hybrid chunking is enabled and there is a running batch, merge the running decoding batch into a new batch, supporting hybrid pre-filling and decoding operations.

[0067] The core of the pre-filling batch generation component is to select suitable requests from the waiting queue based on resource limits (such as maximum number of requests, Token number, LoRA quantity) and scheduling strategies to generate new pre-filling batches, providing input for the pre-filling stage of the model.

[0068] Decoding scheduling component: The decoding scheduling component is used to update the current running decoding batch (running_batch). Its core functions are: filtering completed or terminated requests to simplify batch content; checking and handling memory shortage problems (such as triggering request withdrawal); optimizing decoding efficiency (such as quickly generating partial output based on syntax rules); preparing batch data for the next round of decoding (such as updating cache location and sequence length).

[0069] The execution flow of the decoding scheduling component can be divided into five key steps: initialization and state recording, saving the initial number of requests in the batch, which is used to determine whether the batch has been reduced due to filtered requests; filtering invalid requests, including completed, terminated, and withdrawn requests due to memory shortage; memory management, first checking whether the current KV cache is sufficient to support the next round of decoding (considering the draft token buffer for speculative decoding); if the memory is insufficient, preferentially withdrawing long sequence requests (or selecting low priority requests according to the strategy), releasing the cache resources occupied by them; if the memory is sufficient, slowly reduce the proportion of new token allocation (for example, reduce the proportion of new token allocation by a corresponding step), avoiding excessive resource occupation by subsequent requests; if the memory is insufficient, directly update to a more conservative proportion; efficiency optimization (syntax rule quick generation), when the request enables syntax constraint generation, directly generate partial output that meets the constraints through syntax rules, reducing decoding steps and improving efficiency. The quickly generated request will be returned to the waiting queue for the next round of processing.

[0070] Batch status update and preparation, if the number of filtered or withdrawn requests decreases, mark the batch as not full, allow new requests to join the next batch. Complete the necessary initialization for the next round of decoding, including: update the sequence length of each request, allocate new token KV cache slots, integrate sampling parameters.

[0071] The inference scheduling module is to schedule the model worker method to infer the request. The model worker is an abstraction of model inference, including methods for model inference and resource scheduling. The scheduler only needs to call the inference method of the worker to process the request batch, and get the final result.

[0072] The scheduler handles the model calculation result entry method, according to the running phase of the batch (forward mode) to call the corresponding post-processing logic, mainly responsible for: virtual first batch sampling information preparation (trigger subsequent batch sampling logic), avoid subsequent real batch processing blocked due to waiting for initialization information; pre-fill (expand) result cache management and request state synchronization; decoding result state update and resource release (such as request state update after generating Token); idle batch result analysis (such as resource synchronization when overlapping calculation).

[0073] The core function of the pre-filled result processing component is to process the batch calculation results in the pre-filled (Prefill) stage, update the request state, manage the cache, and stream the results. The main steps are: analyze the Logits output by the model and generate Token ID. Update the output state of the request (such as Token generated, completion flag); manage cache (cache completed or incomplete requests); process Logprob and hidden state records (for returning to users); handle the state of chunked requests; stream the results to the Detokenizer.

[0074] The core function of the decoding result processing component is to process the batch calculation results in the decoding stage (Decode), including updating the request state, managing cache resources, recording generated information (such as Logprob, hidden state), and streaming the results. The main steps are: analyze the generated Token, Logprob, and hidden state output by the model; update the output state of the request (such as Token list generated, completion flag); manage KV cache pool resources (release Token cache no longer needed); record detailed information of the generation process (such as Logprob, syntax constraint state); stream the results to the Detokenizer, and finally return them to the user.

[0075] When the inference phase is idle (the system or component is in an idle state and does not perform any activity or processing task), synchronization between different processes is used to ensure that the states of the workers are consistent and to avoid blocking caused by the idling of some workers. At this time, the following needs to be completed: waiting for CUDA event synchronization, which is a tool for measuring and managing parallel computing tasks on a GPU, and ensuring that data copying from the GPU to the CPU is completed.

[0076] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software and a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation.

[0077] Embodiments of the present application also provide a scheduling device in a model inference process, as shown in Figure 5 The device comprises:

[0078] The first processing unit 51 is configured to process the processed inference request sent by the tokenizer and initialize each queue to obtain an initialized queue, and the processed inference request comprises at least one token.

[0079] The second processing unit 52 is configured to extract the corresponding token from the request waiting queue according to the resource limit value and the priority of the inference request to construct a pre-filled queue, wherein the resource limit value comprises a maximum number of requests, an upper limit of the number of tokens, and an upper limit of the number of low-rank adaptive models, and the request waiting queue is one of the initialized queues.

[0080] The third processing unit 53 is configured to schedule the pre-filled queue and the decoding task corresponding to the pre-filled queue to obtain at least a model inference result.

[0081] In the above device, the tokenizer is used to tokenize the received inference request, convert the natural language text into Token sequences (tokens) that can be understood by the model, and prepare for subsequent model processing. When processing the initial request, the queue is initialized as necessary to ensure that the request can be correctly managed and tracked during scheduling, and to avoid resource allocation errors caused by improper initialization. The pre-filled queue construction strategy based on the resource limit value and the priority enables the scheduler to dynamically adjust the batch size, avoid resource waste or overload operation, and ensure that high-priority requests are processed first. The processing of the model inference result includes updating the request state, managing and releasing the cache resources, and streaming the result, which prepares for the next stage of inference and avoids the calculation delay caused by the delay in processing the result in the traditional method, improves the scheduling efficiency, and solves the problem of low scheduling efficiency of the model inference service in the prior art.

[0082] In an embodiment of the present application, the second processing unit comprises: a creating module configured to create a pre-filling request adder; a first processing module configured to sort the inference requests in the request waiting queue according to the priorities, to obtain a sorting result; and a second processing module configured to invoke the pre-filling request adder, and construct a pre-filling queue according to the sorting result and the resource limit value.

[0083] Specifically, through the pre-filling request adder, the system can dynamically select appropriate requests for processing according to the current resource status (such as the memory of the graphics processor, the cache size) and the request priority. In this way, resource waste can be avoided, and it can be ensured that the pre-filling queue scheduled each time can be efficiently executed under the existing resource conditions. The pre-filling request adder constructs the pre-filling queue based on the sorting result and the resource limit value, which can ensure the diversity and complementarity of the requests in the batch, and avoid low efficiency caused by too large differences in input length or resource demand between requests. According to the priority sorting and the resource limit value, the scheduler can flexibly adjust the content of the pre-filling queue, process urgent or high-priority requests, and at the same time ensure that low-priority requests are fairly processed according to the availability of resources. By processing high-priority requests first, a better balance point between user waiting time and system resource utilization can be found, thereby improving the response speed of the overall service and user satisfaction.

[0084] In an embodiment of the present application, the device further comprises: a first determining unit configured to determine whether the number of paths of the low-rank adaptive model of the candidate request and / or whether the current running batch exceeds the resource limit value after extracting the corresponding tokens from the request waiting queue according to the resource limit value and the priority of the inference request to construct the pre-filling queue; a second determining unit configured to determine that the batch is full and terminate the inference in the case that the number of paths of the low-rank adaptive model of the candidate request and / or the current running batch exceeds the resource limit value; and a fourth processing unit configured to initialize the next round of input, and invoke the pre-filling request adder to add the inference request to the pre-filling queue.

[0085] Specifically, by checking the number of LoRA (Low-Rank Adaptation, a method for efficient fine-tuning on large pre-trained models. LoRA reduces the number of parameters that need to be adjusted by decomposing the weight matrix of the model into a low-rank matrix, thereby reducing the computational and storage overhead) path number and resource usage of the current running batch, it can avoid over-allocation of resources to a single request, ensure more uniform distribution of resources among multiple requests, and thus improve overall resource utilization. Excessive number of LoRA model paths can increase the computational complexity of the model and may lead to performance degradation. Checking the number of LoRA paths can prevent this situation and ensure that the system does not become a performance bottleneck due to excessive LoRA models during operation. In resource-constrained situations, by determining whether the batch is full, the system can dynamically adjust the batch size to avoid inference failures or delays due to insufficient resources, ensuring stable operation of the system. Terminating inference requests that exceed resource limits can prevent long sequence requests or resource-intensive requests from occupying too many resources, preventing other requests from being processed in a timely manner, thereby improving the overall throughput of the system in processing requests. In a parallel computing environment, ensuring the consistency of requests among parallel processes is crucial for correct inference results. Through the above steps, the distribution of requests can be effectively managed to ensure that all parallel processes are processing consistent request data. By avoiding over-allocation of resources to a single request, the waiting time for processing user requests can be reduced, improving response speed and enhancing user experience. The steps of initializing the next round of input and calling the pre-filled request adder to add inference requests to the pre-filled queue simplify the scheduling logic, making it more modular and easier to maintain, and also facilitate subsequent optimization and adjustment of the scheduling strategy.

[0086] In an embodiment of the present application, the device further comprises a third determination unit configured to determine whether the batch of the current round of inference is empty in the case that the previous round of inference is a pre-filled task; a fifth processing unit configured to fuse the pre-filled task of the previous round in the case that the batch of the current round of inference is empty; and a sixth processing unit configured to insert the unfinished sub-block pre-filled request and the corresponding index into the index of the vocabulary value cache in the case that there is an unfinished sub-block pre-filled request, and remove the unfinished sub-block pre-filled request in the batch.

[0087] Through continuous batch processing and cache reuse mechanism, repeated pre-filling calculation is reduced, the processing speed of continuous dialogue or long text generation is significantly improved, the delay of re-constructing batch processing every time is avoided, the response of the system is more rapid, and the user experience is better. The management of memory and computing resources is more fine, GPU (Graphics Processing Unit) and CPU (Central Processing Unit) resources are effectively utilized by dynamically adjusting the batch size and cache strategy, and resource waste is reduced. The generation task involving long text and complex syntax constraints can be better supported, and the applicability and efficiency of the model in these scenarios are improved.

[0088] In an embodiment of the present application, the device further comprises: a seventh processing unit configured to reduce the number of slots or computing resources of the vocabulary value cache allocated for new input data if it is determined that the vocabulary value cache meets the needs of the next round of decoding; an eighth processing unit configured to withdraw a target reasoning request if it is determined that the vocabulary value cache does not meet the needs of the next round of decoding, the target reasoning request being a reasoning request with a priority lower than a priority threshold; and a ninth processing unit configured to generate at least part of the output that meets the syntax constraints according to the rules of the syntax constraints.

[0089] Specifically, the inference of large models often requires a large amount of video memory, and reasonable reduction of cache allocation can avoid video memory overflow and ensure that more requests can be processed. Dynamic adjustment of resource allocation enables limited computing resources to serve real-time requests more efficiently, reducing resource waste. Fewer cache slots mean more concurrent opportunities, thereby improving the overall throughput of the system. In the case of resource shortage, withdrawing lower-priority requests helps to optimize system performance. This can release cache space by withdrawing low-priority requests when the KV cache capacity reaches the upper limit, freeing up resources for higher-priority requests; ensures that critical or urgent tasks are not affected by resource limitations and are prioritized for processing and response; through fine request management and resource allocation, service delays or interruptions caused by resource competition are avoided, maintaining good user experience. The fast generation method based on syntax constraints can intelligently generate partial prediction results without the need for complete decoding process. For scenarios with explicit syntax constraints, such as question answering and code completion, the system can quickly generate preliminary results that meet the rules, significantly reducing user waiting time. Avoiding unnecessary computational overhead, especially in long sequence generation tasks, the partial results generated in advance can significantly reduce the total amount of calculation. The introduction of syntax constraints helps guide the model to generate more reasonable and coherent outputs, improving the quality and usability of the generated content.

[0090] In an embodiment of the present application, the device further comprises: an acquisition unit configured to acquire, in real time, an occupancy of the graphics processor; and a tenth processing unit configured to adjust the number of inference requests in the pre-populated queue according to the occupancy.

[0091] Specifically, by monitoring the real-time occupancy of the GPU, the number of requests in the pre-populated queue can be increased when the GPU load is low, thereby making full use of idle computing resources. Conversely, the number of requests is reduced when the GPU is close to full load, avoiding excessive loading of the GPU leading to performance degradation or long task queuing waiting time. Dynamically adjusting the number of requests in the pre-populated queue can optimize the batch size, ensuring that the GPU can achieve optimal efficiency when processing each batch. Smaller batches can be completed faster under high load, while larger batches can more efficiently utilize the parallel computing power of the GPU under low load, thereby improving the overall throughput of the system. Reasonably adjusting the number of requests in the pre-populated queue helps to reduce task waiting time, especially when the GPU load fluctuates greatly, it can quickly respond to adjust the request processing strategy and reduce the average delay of user requests.

[0092] In an embodiment of the present application, the device further comprises: a fourth determination unit configured to determine the similarity between the input features of the current batch and the historical requests to obtain a current similarity; and a fifth determination unit configured to determine whether to reuse the results in the vocabulary value cache according to the size of the current similarity.

[0093] Specifically, by identifying similar input features, unnecessary calculations can be avoided, especially for large language models, repeated calculations can consume a large amount of computing resources. Reusing the results in the cache can significantly improve inference speed and reduce latency; model inference often involves a large amount of memory read and write, especially the use of KV Cache (Key-Value Cache, in large model inference, especially in the generation task of autoregressive models (such as Transformer), it refers to a cache mechanism for storing key-value pairs. The key (Key) and value (Value) here refer to two components in the attention mechanism (Attention Mechanism)). Reusing the calculation results of similar requests reduces the use of memory bandwidth, helping to alleviate memory bottlenecks and improve overall resource utilization; similarity evaluation provides a basis for dynamic adjustment of resource allocation. When a high similarity is detected between the current batch and the historical requests, the scheduler can prioritize resource allocation to batches with less difference, ensuring optimal use of resources; in a high-concurrency scenario, reusing cache results based on similarity can improve the system's ability to handle more requests, thereby increasing the throughput of the server.

[0094] In an embodiment of the present application, the fifth determining unit comprises: a first determining module configured to determine that the result in the vocabulary value cache does not need to be reused when the current similarity is less than the similarity threshold; and a second determining module configured to determine that the result in the vocabulary value cache needs to be reused when the current similarity is greater than or equal to the similarity threshold.

[0095] Specifically, the similarity threshold can be 80%, and when the similarity is greater than or equal to the similarity threshold, it means that the current request shares some context or features with one or more requests in the cache to some extent. By reusing the cache result, repeated calculation of the same or similar data is avoided, the computing resources and memory consumption are saved, and the inference speed of the model is improved. For requests with high similarity, the calculation result is directly obtained from the cache instead of re-executing the entire inference process, which can significantly reduce the response time and improve the user experience. Especially in real-time or high-concurrency scenarios, this strategy is crucial to maintaining low latency of the system. In the self-attention (Self-Attention) and feed-forward neural network (FFN) calculation of the model, the reuse of the cache result reduces unnecessary operator calculation, especially in the attention algorithm, avoiding the generation of multiple KV Caches and reducing the computational complexity.

[0096] In an embodiment of the present application, the device further comprises: an eleventh processing unit configured to control the tensor parallel process to directly receive input data from the tokenizer when the tensor parallel process exists; and a twelfth processing unit configured to split the inference request into multiple sub-requests and distribute them to each data parallel thread when the data parallel thread exists.

[0097] Specifically, in TP parallelism (Tensor Parallelism, which mainly targets the parallel processing of model parameters. In deep learning models, there are many large weight matrices, and TP parallelism splits these weight matrices into multiple parts and allocates them to different GPUs for calculation. For example, a large matrix multiplication operation can be split into multiple small matrix multiplications, which are performed simultaneously on multiple GPUs, and then the results are combined. This method is suitable for cases where the model is very large and a single GPU cannot store or efficiently compute it. TP parallelism can significantly reduce the memory occupancy of each GPU and speed up the forward propagation and back propagation of the model through parallel computation), each GPU processes a different part of the model tensor, and they share the same data input. Therefore, having each TP process directly receive input data from the tokenizer avoids additional data transmission and synchronization overhead, especially under high concurrency requests, which greatly reduces the communication burden of the master process and improves overall communication efficiency. In DP parallelism (Data Parallelism, which mainly targets the parallel processing of data. In DP parallelism, the parameters of the model are consistent across all GPUs participating in the calculation. The data is split into multiple subsets, and each subset is calculated on a different GPU for forward propagation and back propagation. After the calculation is completed, the gradients on each GPU are aggregated, and the model parameters are updated. This method is suitable for cases where the data volume is large, and the model can be adapted to a single GPU. DP parallelism can improve the throughput of data processing, but it needs to solve the problem of gradient synchronization to ensure the correctness of parameter updating), the dataset is split, and each thread processes a different part of the data. By splitting the original request into sub-requests matching the number of DP threads and distributing them to each thread, parallel processing of data is ensured, while also avoiding redundant calculations between threads, improving the utilization efficiency of computing resources.

[0098] In an embodiment of the present application, the apparatus further comprises a thirteenth processing unit configured to, in the case of inputting multi-modal data, perform input combination processing on the corresponding text and image.

[0099] Specifically, by integrating text and image inputs in the preprocessing stage, the number of model calls can be reduced, avoiding repeated preprocessing operations, thereby improving the overall inference speed. For example, in the Visual Question Answering (VQA) task, the problem text and the related image are pre-combined, which can be sent to the model for processing at one time, rather than being processed separately and then merging the results. Integrating multi-modal inputs can more effectively utilize computing resources, especially when computing on a GPU, it can maximize the use efficiency of video memory and avoid resource fragmentation and waste caused by independent processing between modalities. The combined processing of multi-modal inputs allows the model to extract information from both text and image simultaneously, which is crucial for understanding complex scenarios, emotions, or context. For example, in sentiment analysis, the model can jointly determine the sentiment polarity through facial expressions in the image and tone in the text, thereby improving the accuracy of analysis.

[0100] In an embodiment of the present application, the apparatus further comprises a fourteenth processing unit configured to record the pre-filling statistics in a log, the pre-filling statistics comprising the number of new requests, the number of tokens, and the cache hit rate.

[0101] Specifically, the number of new requests helps monitor the load of the system and understand the busy degree of the server at a specific time, which is very important for predicting resource demand and avoiding overload. The number of tokens reflects the size of the input processed by the scheduler, which helps analyze the effective use of computing resources and identify whether there are too many small batch requests that may affect the overall processing efficiency. The cache hit rate provides feedback on the effectiveness of the cache strategy, and a high cache hit rate means that the scheduler can effectively reuse previous calculation results, reducing the calculation cost and improving the response speed.

[0102] The description of the features in the scheduling device corresponding to the embodiments in the model inference process can be referred to the related description of the scheduling method corresponding to the embodiments in the model inference process, which will not be repeated here.

[0103] Embodiments of the present application also provide an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above-described scheduling method embodiments in the model inference process.

[0104] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is configured to execute the steps in the scheduling method embodiment in any of the model inference processes when running.

[0105] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0106] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the steps in the scheduling method embodiment in any of the model inference processes.

[0107] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in the scheduling method embodiment in any of the model inference processes.

[0108] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0109] The above describes in detail the scheduling method in a model inference process, the scheduling device in a model inference process, the computer readable storage medium, the computer program product and the electronic device provided by the present application. The principles and implementation manners of the present application are described by applying specific examples in this paper, and the above description of the examples is only used to help understand the method and its core idea of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A method for scheduling in a model inference process, the method comprising: The method comprises the following steps: receiving a processed inference request sent by a word segmentation device, and initializing each queue to obtain an initialized queue, wherein the processed inference request comprises at least one word piece; extracting the corresponding word piece from a request waiting queue according to resource limits and a priority of the inference request to construct a pre-populated queue, wherein the resource limits comprise a maximum number of requests, an upper limit of the number of word pieces, and an upper limit of the number of low-rank adaptive models, and the request waiting queue is one of the initialized queues; scheduling the pre-populated queue and a decoding task corresponding to the pre-populated queue to obtain at least a model inference result; extracting the corresponding word piece from a request waiting queue according to resource limits and a priority of the inference request to construct a pre-populated queue, comprising: creating a pre-populated request adder; sorting the inference requests in the request waiting queue according to the priority to obtain a sorting result; calling the pre-populated request adder to construct the pre-populated queue according to the sorting result and the resource limits; after extracting the corresponding word piece from a request waiting queue according to resource limits and a priority of the inference request to construct a pre-populated queue, the method further comprises: determining whether the number of paths of the low-rank adaptive model of the candidate request and / or the current running batch exceeds the resource limits; in the case that the number of paths of the low-rank adaptive model of the candidate request and / or the current running batch exceeds the resource limits, determining that the batch is full and terminating the inference; initializing the next round of input, and calling the pre-populated request adder to add the inference request to the pre-populated queue.

2. The method of claim 1, wherein, The method further comprises: in the case that the previous round of inference is a pre-populated task, determining whether the batch of the current round of inference is empty; in the case that the batch of the current round of inference is empty, fusing the pre-populated task of the previous round; in the case that there is an unfinished sub-block pre-populated request, inserting the unfinished sub-block pre-populated request and the corresponding index into the index of the vocabulary value cache, and removing the unfinished sub-block pre-populated request in the batch processing.

3. The method of claim 1, wherein, The method further comprises: in the case that the vocabulary value cache meets the demand of the next round of decoding, reducing the number of slots of the vocabulary value cache or the number of computing resources allocated for new input data; in the case that the vocabulary value cache does not meet the demand of the next round of decoding, withdrawing a target inference request, wherein the target inference request is the inference request with a priority lower than a priority threshold; generating at least part of the output conforming to the syntax constraint according to the rules of the syntax constraint.

4. The method of claim 1, wherein, The method further comprises: obtaining the occupation of the graphics processing unit in real time; adjusting the number of inference requests of the pre-populated queue according to the occupation.

5. The method of claim 1, wherein, The method further comprises: determining the similarity between the input features of the current batch and the historical requests to obtain a current similarity; determining whether to reuse the results in the vocabulary value cache according to the size of the current similarity.

6. The method of claim 5, wherein, Determining whether to reuse the results in the vocabulary value cache according to the size of the current similarity, comprising: In a case where the current similarity is less than the similarity threshold, it is determined that the result in the vocabulary value cache does not need to be reused. In a case where the current similarity is greater than or equal to the similarity threshold, it is determined that the result in the vocabulary value cache needs to be reused.

7. The method of claim 1, wherein, The method further comprises: In a case where the tensor parallel process exists, the tensor parallel process is controlled to directly receive input data from a tokenizer; In a case where the data parallel thread exists, the inference request is split into multiple sub-requests and distributed to each data parallel thread.

8. The method of claim 1, wherein, The method further comprises: In a case where the input is multi-modal data, corresponding text and image are input combined.

9. The method of claim 1, wherein, The method further comprises: Pre-population statistics are recorded in a log, the pre-population statistics including a number of new requests, a number of tokens, and a cache hit rate.

10. A scheduling apparatus in a model inference process, comprising: Comprise: The first processing unit is configured to receive a processed inference request sent by a tokenizer, and to perform initialization processing on each queue to obtain an initialized queue, the processed inference request including at least one token; The second processing unit is configured to extract the token corresponding to the inference request from a request waiting queue according to a resource limit and a priority of the inference request to construct a pre-population queue, the resource limit including a maximum number of requests, an upper limit of the number of tokens, and an upper limit of the number of low-rank adaptive models, and the request waiting queue being one of the initialized queues; The third processing unit is configured to schedule the pre-population queue and a decoding task corresponding to the pre-population queue to obtain at least a model inference result; The second processing unit comprises: a creation module configured to create a pre-population request adder; a first processing module configured to sort the inference requests in the request waiting queue according to the priority to obtain a sorting result; and a second processing module configured to call the pre-population request adder to construct the pre-population queue according to the sorting result and the resource limit. The scheduling device in the model inference process further comprises: a first determination unit configured to determine a number of paths of low-rank adaptive models of a candidate request and / or whether a current running batch exceeds the resource limit after extracting the token corresponding to the inference request from the request waiting queue according to the resource limit and the priority of the inference request to construct the pre-population queue; a second determination unit configured to determine that the batch is full and terminate the inference in a case where the number of paths of low-rank adaptive models of the candidate request and / or the current running batch exceeds the resource limit; and a fourth processing unit configured to initialize a next round of input and add the inference request to the pre-population queue by calling the pre-population request adder.

11. A computer readable storage medium, characterized in that, The computer program stored in the computer-readable storage medium, when executed by a processor, implements the steps of the scheduling method in the model inference process according to any one of claims 1 to 9.

12. A computer program product comprising a computer program, characterized in that, The computer program stored in the computer-readable storage medium, when executed by a processor, implements the steps of the scheduling method in the model inference process according to any one of claims 1 to 9.

13. An electronic device, comprising: Comprise: The memory is configured to store a computer program; The processor is configured to implement the scheduling method in the model inference process according to any one of claims 1 to 9 when executing the computer program.

Images