Request task processing method and electronic device

Abstract

The application discloses a request task processing method and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: determining a load dynamic balance control condition according to the relationship between the total quantity of request tasks allocated to a previous flow task line and the total number of currently scheduled request tasks, and based on a flow execution sequence constraint condition; in the process of allocating the request in a batch queue, if the current flow task line meets the load dynamic balance control condition and does not reach a parallel capacity limit condition, the request task scheduled at the current time is allocated to the current flow task line; if one of them does not meet the condition, the allocation of a new request task to the current flow task line is suspended. The application can solve the problem that the idle computing resources are caused by the flow line air bubble in the related art flow parallel processing task process, realize balanced scheduling of request tasks, and effectively improve the utilization rate of computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a request task processing method and an electronic device. BACKGROUND

[0002] With the wide deployment of large-scale neural network models in various artificial intelligence applications, the inference process thereof has an increasing demand for computing resources. To optimize inference throughput and system latency, related technologies split the model into multiple computing devices in a pipelining parallel manner, and combine batch queues to aggregate and schedule requests. However, when related technologies schedule requests in batch queues, the load of each pipelining stage is uneven, which causes the problem of pipelining bubble of idle computing resources, and reduces the overall efficiency and resource utilization of the system. SUMMARY

[0003] The present application provides a request task processing method and an electronic device, which balance the scheduling of requests in batch queues and effectively improve the utilization of computing resources.

[0004] To solve the above technical problems, the present application provides the following technical solutions:

[0005] In one aspect, the present application provides a request task processing method, comprising:

[0006] adding a received inference request task to a to-be-scheduled request batch queue, extracting each inference request task from the to-be-scheduled request batch queue and assigning the inference request task to each pipelining task line for parallel execution of tasks, determining a load dynamic balance control condition corresponding to a dynamic balance of request allocation of each pipelining task line according to the relationship between the total amount of request tasks allocated to the previous pipelining task line and the total number of currently scheduled request tasks, and based on a pipelining task line execution order constraint condition, in the process of allocation, if the current pipelining task line meets the corresponding load dynamic balance control condition and the parallel capacity limitation condition is not reached at the current time, assigning a target inference request task scheduled at the current time to the current pipelining task line, if the current pipelining task line does not meet the corresponding load dynamic balance control condition or the parallel capacity limitation condition is not reached at the current time, suspending the allocation of new request tasks to the current pipelining task line, wherein the previous pipelining task line includes pipelining task lines from the first pipelining task line to the current pipelining task line.

[0007] The present application also provides an electronic device comprising a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory to implement the steps of the above request task processing method.

[0008] The technical scheme provided by the application has the advantages that by continuously monitoring the dynamic relationship between the cumulative request quantity of the previous sequential flow task line and the global scheduled task number, combining with the flow line execution sequence constraint, the load balancing control condition is constructed in real time; in the task allocation process, when it is detected that the current flow line simultaneously satisfies the load balancing condition and the parallel capacity limit, the new request allocation is allowed, which not only guarantees the dynamic balance of the loads among the flow lines, effectively schedules the computing resources, solves the problem of idle computing resources (i.e. flow line air bubbles) caused by unbalanced request scheduling in the existing inference framework flow parallel strategy, significantly improves the flow line utilization rate, prevents the video memory resource overload through the capacity limit, ensures that the system will not be overloaded in the resource limited environment (such as GPU video memory or computing resource limited), thereby avoiding resource contention, performance degradation or insufficient video memory error; at the same time, the suspension allocation replaces the forced rollback operation, which fundamentally eliminates the program interruption risk caused by unordered rollback, thereby solving the program interruption problem caused by request rollback exception, enabling the system to maintain high throughput while realizing zero service interruption, enhancing the robustness of the inference process and the overall stability of the system. At the same time, it is easy to extend to the current commonly used inference framework. In addition, the application also provides a corresponding implementation electronic device, computer readable storage medium and computer program product for the request task processing method, which further makes the method more practical, and the electronic device, computer readable storage medium and computer program product have corresponding advantages. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical scheme of the present application or related technology, the following will briefly introduce the drawings needed to be used in the embodiment or related technology description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0010] Figure 1 The hardware composition framework diagram applicable to the request task processing method provided by the present application.

[0011] Figure 2 The inference framework flow parallel strategy execution flow diagram in the exemplary scenario provided by the present application.

[0012] Figure 3 The flow diagram of the request task processing method provided by the present application.

[0013] Figure 4 The flow diagram of another request task processing method provided by the present application.

[0014] Figure 5 The structure framework diagram of the request task processing device in the exemplary embodiment provided by the present application.

[0015] Figure 6 An exemplary embodiment structure diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0016] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. Among them, the terms "first", "second", "third" and the like in the specification and the above-mentioned drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. The term "exemplary" means "as an example, embodiment or illustration". Any embodiment described as "exemplary" herein is not necessarily interpreted as superior or better than other embodiments.

[0017] With the rapid development of AI (Artificial Intelligence) technology, and the continuous development of deep learning technology and NLP (Natural Language Processing) technology in AI tasks, large models (such as generative pre-training models) have become tools for solving complex tasks, such as intelligent customer service, machine translation tasks, content generation tasks, etc. The larger the network model size, the higher the computing resources and storage resources required for model training and inference processes. To widely apply large models to solve complex tasks in various industries or multiple technical fields, it is necessary to efficiently utilize limited computing resources to achieve fast and accurate inference task processing. Through optimization algorithms, hardware acceleration and distributed computing and other technical means, large model inference engines (Inference Engine) have emerged, such as vLLM (Vectorized Large Language Model Inference) and SGLang (Structured Generation Language). vLLM is a high-performance library designed for large language model LLM (Large Language Models) inference and service, with the advantages of high efficiency, speed and ease of use. SGLang is a high-performance service framework for large language models and visual language models. These engines not only accelerate the inference process of large models, but also reduce the consumption of computing resources while ensuring the quality of inference, so that large models can be widely applied in more business scenarios.

[0018] The inference engine optimizes the inference throughput and system latency of a large language model through a batch algorithm, which is a technique that uniformly executes multiple operations or requests by packaging them into a single task. This method aggregates discrete requests into batch tasks, thereby functioning in high-performance computing and distributed systems, significantly reducing the frequency of RPC (Remote Procedure Call) calls, reducing kernel context switching and memory copy overhead, thereby effectively reducing the average network latency of the system and ultimately significantly improving the overall throughput. As the parameter quantity of large models continues to rise, the scale of the computing nodes on which they are deployed also grows. In this context, a single parallel strategy has been unable to meet demand, and related technologies use multiple parallel techniques, such as tensor parallelism (Tensor Parallel) and pipeline parallelism (Pipeline Parallel). Among them, pipeline parallelism effectively solves the problem of large models that cannot reside in a single device memory by finely splitting each layer of the model to different computing nodes, and alleviates the communication wall pressure faced by pure data parallelism. However, there is a pipeline bubble in pipeline parallelism, causing idle waiting in the calculation process, so that the computing resources cannot be fully utilized. To solve the problem of pipeline bubble in pipeline parallelism, related technologies implement batch queue processing when scheduling requests, distribute requests to each pipeline according to the batch queue, and combine the ray (a distributed execution framework name) directed acyclic graph (DAG) to optimize the task execution order, reduce idle waiting time, and improve overall throughput.

[0019] However, when related technologies schedule requests for batch queues, there is an imbalance in the load of the pipelines, which can cause inconsistent running times for each pipeline, resulting in pipeline bubbles. For example, a batch queue includes 32 requests, and there are 2 pipeline parallelisms. The batch queue is split into [31, 1] two pipeline processing, 31 requests take 2s to infer, and 1 request takes 1s to infer. At the same time, the first computing device performs inference on 31 requests, and the second computing device performs data on 1 request. The second acceleration component needs to wait for 1s to get the inference result of the first acceleration component before performing subsequent operations. At the next time, the first acceleration component performs inference on 1 request, and the second acceleration component performs data on 31 requests. The first acceleration component needs to wait for 1s to receive new data. For scenarios where the request quantity is less than the highest concurrency of the deployed inference service, the imbalance in the scheduling of batch queue requests can cause long periods of resource idleness, which is more pronounced when the request concurrency is insufficient, resulting in low utilization of computing resources and reducing the overall efficiency of the system.

[0020] In view of this, in order to solve the problem of pipeline bubble caused by unbalanced load of each pipeline stage due to lack of balanced control of request allocation under the flow parallel strategy in the reasoning framework, by dynamically monitoring and adjusting the request allocation of each pipeline, the balanced judgment and resource estimation are added during scheduling to control the number and order of requests entering each pipeline, so that the system overall efficiency and resource utilization are improved. The method not only pursues the balance in quantity, but also couples the pre-evaluation of system resources (such as video memory) to prevent request rollback or service interruption caused by resource competition, so as to improve the utilization of pipeline and the system throughput as a whole.

[0021] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the request task processing method depends, the specific application environment architecture or specific hardware architecture is described herein. The following describes the specific application environment architecture or specific hardware architecture in combination with the execution of the request task processing method. Figure 1 Some possible application scenarios related to the technical solutions of the present application are exemplarily introduced, which can include the following content:

[0022] The multi-element heterogeneous computing system includes multiple general servers and multiple artificial intelligence servers, each of which is equipped with multiple computing devices of different types. For example, the multi-element heterogeneous computing system can include multiple first-type servers each equipped with at least one XPU (heterogeneous computing processor), multiple second-type servers each equipped with at least one GPU (graphics processing unit), and multiple third-type servers each equipped with at least one tensor processing unit. The different servers are connected through an exchange mechanism to build a non-100GB / s RoCEv2 (RDMA over Converged Ethernet Version 2, Remote Direct Memory Access over Converged Ethernet Version 2) lossless network, ensuring the stability of data transmission between the computing devices. The first-type servers can be used for high-parallel pre-padding stage computation, for example, to adapt to input processing of large models. The second-type servers can be used for optimization decoding stage low-latency characteristics, for example, to meet the demand for response of inference request services. The third-type servers can be used as backup computing nodes, for example, to undertake fault migration tasks.

[0023] The multi-element heterogeneous computing system has a management node 1 that receives inference request tasks sent by one or more user terminals 2, aggregates all inference request tasks received within a period of time, such as within 1s, and encapsulates them into a request set. Various servers deploy large model services in an inference framework, such as using a vLLM inference framework and using an LLM large model service, and select two second-type servers to implement flow imbalance load balancing for the current existing batch queue request scheduling. The inference framework divides the inference stage into a pre-filling stage and a decoding stage, and the flow imbalance occurs in the decoding stage. The selected two servers are each deployed with 16 GPUs to execute the inference request tasks issued by the user terminal 2. In the vLLM scheduler, the request task processing method of the application is enabled, the maximum number of single-flow watermarks max_num_pre is set, and the request scheduling strategy is set to first-come-first-served.

[0024] Among them, the existing batch queue request scheduling process is described with 2 computing devices and a parallel scale parameter PP=2: that is, two-stage flow task line GPU0+GPU1, corresponding to two batch queues batch queue 0 and batch queue 1, a token is generated to describe the implementation scheme after two stages, and the existing batch queue scheduling request is as follows Figure 2As shown, first, a queue with a length of PP is constructed to record the current pipelined scheduling requests, and two computing devices are GPUs, the first computing device can be represented as the first computing device G-0 and the second computing device G-1, the requests are first scheduled to the first computing device G-0 for calculation, and then to the GPU1 for calculation, and then according to the inference result, if no end symbol is generated or the maximum sequence generation length is reached, the request is rescheduled to the first computing device G-0, and the above process is repeated until the end symbol is generated or the maximum sequence generation length is reached. Among them, batch queue 0 represents data on the first pipelined (which can be represented as batchqueue (batch queue) 0), when the first pipelined calculation is completed, the intermediate data is sent to the second pipelined (batchqueue 1), and all pipelineds are executed once and the entire large model is executed once. Generally, a large model generates a character sequence every time it is executed. The request scheduling in the batch queue of the inference framework in the related art is random. If the maximum request processing number set when the inference service is deployed is max_num_seqs=128, the minimum value of batch queue 0 is 0, and the maximum value is 128. Due to the randomness of request scheduling, the number of requests in batch queue 0 and batch queue 1 is often not equal. In the extreme case, batch queue 0 is 128, and batch queue 1 is 0, which causes the first computing device G-0 to wait for the second computing device G-1 to execute batch queue 1 after the first computing device G-0 calculates batch queue 0. The first computing device G-0 waits for the second computing device G-1 to process the current task. At this time, the first computing device G-0 cannot process the task, thereby generating a bubble.

[0025] To solve the above problems, the application can balance the number of requests in the queue when scheduling requests, and solve the problem of program interruption caused by request fallback when the video memory space is insufficient during request scheduling. The implementation process is as follows:

[0026] As Figure 3As shown, the management node 1 receives the external request sent by the user terminal 2, taking the visual-language understanding API service as an example, the API gateway receives the image and text request of the user terminal 2, after preprocessing, initiates the inference request to the large model, after the pre-filling stage is completed on the corresponding GPU, the decoding request is generated and sent to the to-be-scheduled request batch queue of the vLLM scheduler. Assume that 20 inference request tasks corresponding to decoding requests have been accumulated in the queue in succession. The scheduler starts to process the requests in the queue in sequence (such as req1, req2,...). For the first request (req1), if the current flow task line meets the corresponding load dynamic balance control condition, and the current time does not reach the parallel capacity limitation condition, the target inference request task scheduled at the current time is allocated to the current flow task line; if the current flow task line does not meet the corresponding load dynamic balance control condition, but as the first request, the system can be specially processed or waited, and finally it is allocated to the flow line 0, and the state is updated. The first 8 requests (req1-req8) are allocated to 8 different flow lines in sequence according to the same method, and the initial balance is realized.

[0027] When scheduling the 9th request (i.e., req9), the algorithm calculates which flow line can still maintain the best balance after receiving it. It simulates the allocation and checks whether the load dynamic balance control condition is still met (i.e., whether the cumulative load of the previous flow line reaches the new expected benchmark) if req9 is allocated to the candidate flow line X. Through this dynamic calculation, req9 is finally allocated to the flow line with the relatively lightest load and meeting the balance condition (such as flow line 0, if its request has been processed for a part). When the number of requests mounted on a flow line (such as flow line 0) reaches max_num_pre=16, the capacity limitation condition is triggered. Subsequent requests will not be allocated to flow line 0 even if they meet the load balance condition, until the request processing is completed and the resource is released. The scheduler may allocate the request to other unfulfilled flow lines, or trigger real-time capacity analysis. If the allocation of a new request at a certain time will lead to serious imbalance (for example, the previous load of all unfulfilled flow lines does not reach the expected benchmark for receiving a new request), the request is returned to the back-off queue and the state is marked. The scheduler continues to process other requests in the queue that may meet the conditions. Later, when the request in the back-off queue meets the conditions due to the change of system state after the completion of the request on some flow lines, they will be rescheduled.

[0028] The request balance of each flow water is realized by the above method, the request scheduling is performed according to the request sequence to each flow water for calculation, and in the process of calculation of the large model and data communication between each board, the intermediate state generated by G-0 first flow water calculation is communicated to G-1, the intermediate state generated by G-1 calculation is communicated to G-2, and until G-N Nth flow water calculation is completed, a sequence is generated, and it is judged whether the sequence contains an end symbol or reaches the maximum sequence output length, if the end symbol is generated or the maximum generation length is reached, the request reasoning request task final result is returned to the user end 2.

[0029] It should be noted that the above application scenarios are only shown for the purpose of facilitating the understanding of the ideas and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario. After introducing the technical solutions of the present application, various non-limiting embodiments of the present application will be described in detail in combination with the drawings and specific embodiments.

[0030] First, please refer to Figure 4 According to the request task processing method provided by the present application, a computer program product can be implemented, which is installed and run on one of the high-performance computing nodes or management nodes in the heterogeneous computing system or a special server outside the heterogeneous computing system, for implementing corresponding processing on the received user reasoning request task. In some embodiments of the method, the request task processing method comprises the following steps:

[0031] S401: Add the received reasoning request task to the to-be-scheduled request batch queue, and extract each reasoning request task from the to-be-scheduled request batch queue and assign it to each flow task line of the parallel execution task.

[0032] Among them, the reasoning request task is a user request that needs to be processed by a large language model or other reasoning framework, which can contain input data, parameter configuration and other information required for reasoning. The to-be-scheduled request batch queue is used to temporarily store a set of reasoning request tasks that have been received but have not been assigned to computing resources, and plays a buffering and unified scheduling role. The flow task line is a parallel processing link formed by splitting each layer of the model to different computing nodes, each flow task line can independently perform part of the reasoning calculation, and multiple flow task lines cooperate to complete the complete reasoning process.

[0033] The to-be-scheduled request batch queue contains a group of inference request tasks that meet a pre-set task processing condition, the total number of inference request tasks is related to the set task processing condition, the task processing condition can be, for example, that inference request tasks received within a period of time, such as 2 seconds, are regarded as a group of inference request tasks, and the task processing condition can also be a threshold value of the number of inference request tasks, such as 100, when the number of received inference request tasks reaches the threshold value of the number of inference request tasks, the inference request tasks are regarded as a group of inference request tasks and added to the to-be-scheduled request batch queue. Each inference request task can be issued by the same user or different users, and the inference request tasks can be the same type of tasks or different types of tasks, as long as an inference model for executing the inference request tasks is deployed on the heterogeneous computing system. The inference request task can be a natural language processing inference task, the natural language processing inference task takes language data in the form of text / speech as input, understands, generates, and analyzes human language through a natural language model deployed on the heterogeneous computing system, and can be applied to human-computer interaction, content processing, information extraction, and the like. The inference request task can also be a computer vision inference task, which takes image / video data as input, understands visual content through a visual model deployed on the heterogeneous computing system, realizes image recognition, target detection, scene analysis, and the like, and can be applied to the fields of industry, security, and medical treatment. The inference request task can also be a speech and audio processing inference task, which takes speech / audio signals as input, in addition to the speech-to-text task and the text-to-speech task included in the natural language task, also includes tasks such as speech emotion recognition and audio event detection for audio semantic understanding, and can be applied to security, medical treatment, entertainment, and the like. The inference request task can also be a recommendation system and personalization inference task, which takes user behavior data and item feature data as input, predicts user preferences through an artificial intelligence model deployed on the heterogeneous computing system, realizes personalized recommendation, and is used in Internet platforms (e-commerce, short video, information). The inference request task can also be a decision and control inference task, which takes environmental data and / or state data as input, outputs decision instructions or control signals through an artificial intelligence model deployed on the heterogeneous computing system, and directly drives device actions, which can be used in industrial automation, autonomous driving, robots, and the like. The inference request task can also be a cross-modal inference task, which refers to an inference task in which the input / output contains multiple modal data such as text, image, speech, and video, and the artificial intelligence model deployed on the heterogeneous computing system realizes semantic alignment and collaborative understanding of different modal data.

[0034] S402: According to the relationship between the total number of request tasks allocated to the previous pipeline task line and the current total number of scheduled request tasks, and based on the flow task line execution sequence constraint condition, determine the load dynamic balance control condition corresponding to the dynamic balance of the request allocation of each flow task line.

[0035] The precedent pipelined task line is all pipelined task lines from the first pipelined task line to the pipelined task line currently being processed, for example, when the second pipelined task line is currently being processed, the precedent pipelined task line includes the first pipelined task line and the second pipelined task line. The load dynamic balancing control condition is a condition for determining whether the current pipelined task line can receive a new request based on the relationship between the total quantity of requests of the precedent pipelined task line and the total number of scheduled requests, in combination with the pipelined execution sequence constraint. Since the inference framework uses KV Cache to cache the values of the keys and values of the history to reduce repeated calculations when performing large model inference, the values of the KV Cache of the history need to be read in the decoding stage. When the KV Cache space is insufficient, a new KV Cache space cannot be allocated for the current request, and the KV Cache values occupied by the request with low priority or late arrival need to be released. Since the requests in the batch queue are out of order, if the current pipelined backtracking request is the last request of the current pipelined task line, an error will occur, which will cause the entire inference service to be interrupted. In order to solve the problem of service interruption caused by request backtracking, the load dynamic balancing control is performed in this step, and if it is not satisfied, the scheduling of new requests is stopped, and subsequent calculations are performed. The pipelined task line execution sequence constraint condition is that the dynamic resource allocation is realized through the pre-set execution sequence, for example, the dynamic resource allocation can be realized by constraining the execution sequence of the pipelined task line through the numbering sequence, for example, the pipelined task lines are processed in strict ascending order of the pipeline index: The constraint condition is relaxed by one level every time the index increases by 1, forming a gradual load lifting, that is, In this way, the first pipelined task line has absolute priority, and the pipelined task lines with low indexes always have priority allocation rights, and subsequent requests are preferentially allocated to pipelined task lines with smaller indexes (such as the first pipelined task line corresponding to batch_index=0), the gradual load lifting prevents some pipelined task lines from being idle for a long time, and realizes the load balancing of the pipelined task lines. The space position index is converted into a time allocation priority based on the execution sequence constraint, and efficient load balancing can be realized through this simple mathematical constraint, while avoiding the uncertainty and system overhead caused by complex scheduling algorithms, which can reduce power consumption.

[0036] In the embodiment, the total number of allocated request tasks of all the preceding pipeline task lines (collectively referred to as the preceding pipeline task lines) from the first pipeline task line (the first pipeline task line) to the current pipeline task line is obtained. At the same time, the total number of scheduled request tasks (the current total number of scheduled request tasks) of all the pipeline task lines in the current system is obtained. According to the relationship between the two, and in combination with the constraint that the pipeline task lines must be executed in sequence (that is, the request must be processed in the current stage on the pipeline task line with a smaller number before entering the next pipeline task line with a larger number), it is determined whether the current pipeline task line is in a balanced state allowing receiving a new request. The determination condition is the load dynamic balancing control condition, and the purpose is to ensure that the cumulative distribution of requests among the pipeline task lines is balanced, and to prevent the accumulation of tasks in one pipeline and the idling of another pipeline.

[0037] S403: In the allocation process, if the current pipeline task line satisfies the corresponding load dynamic balancing control condition and the parallel capacity limitation condition is not met at the current time, the target reasoning request task scheduled at the current time is allocated to the current pipeline task line; if the current pipeline task line does not satisfy the corresponding load dynamic balancing control condition or the parallel capacity limitation condition is not met at the current time, the allocation of new request tasks to the current pipeline task line is suspended.

[0038] The preceding pipeline task lines include the pipeline task lines from the first pipeline task line to the current pipeline task line. The parallel capacity limitation condition is a constraint condition set to avoid overloading of computing resources, and can include the maximum number of request tasks that can be carried by a single pipeline task line and / or the upper limit of the total load of all pipeline task lines. When scheduling requests, not only the capacity upper limit of a single pipeline is considered, but more importantly, it is determined in real time whether the request allocation between the pipeline task lines has reached dynamic balance, so as to avoid the occurrence of an unbalanced state in the allocation process, and to improve the utilization rate of the overall computing resources and the system throughput.

[0039] Taking the example of assigning text generation inference request tasks in the order of increasing flow task line number, an inference framework (such as vLLM) receives a user-initiated text generation inference request task, and adds these tasks to a request batch queue in turn. The execution order constraint of each flow task line is to assign the requests in the order of increasing flow task line number, based on this constraint, the total number of request tasks assigned to the previous flow task line is counted, and the total number of scheduled request tasks is recorded. According to the ratio of the two, the load dynamic balance control condition is determined: when the ratio of the total number of requests of the current flow task line to the total number of scheduled requests meets the preset balance standard, it is determined that the current flow task line meets the load dynamic balance control condition. The target inference request task to be scheduled is extracted from the request batch queue to be scheduled, and it is first determined whether the current flow task line meets the above load dynamic balance control condition, and then it is checked whether the parallel capacity limit condition (such as the maximum number of requests of a single flow is 64, and the maximum number of requests of the total flow is 128) is met. If both conditions are met, the target inference request task is assigned to the current flow task line; if any one condition is not met, the assignment of new requests to the current flow task line is suspended until the condition is met.

[0040] In the technical scheme provided in the embodiment, by continuously monitoring the dynamic relationship between the cumulative request quantity of the previous flow task line and the global scheduled task number, and combining the flow line execution order constraint, the load balancing control condition is constructed in real time; during the task assignment process, when it is detected that the current flow line meets both the load balancing condition and the parallel capacity limit, new requests are allowed to be assigned, which not only ensures the dynamic balance of the loads between the flow lines, but also effectively schedules the computing resources, solves the problem of idle computing resources caused by unbalanced request scheduling in the existing inference framework flow parallel strategy, significantly improves the utilization rate of the flow line, prevents the overload of the video memory resource through the capacity limit, ensures that the system will not be overloaded in the resource-limited environment (such as insufficient GPU video memory or computing resources), thereby avoiding resource contention, performance degradation, or insufficient video memory (i.e. OOM) errors; at the same time, the suspension of assignment replaces the forced rollback operation, which fundamentally eliminates the risk of program interruption caused by unordered rollback, thereby solving the problem of program interruption caused by request rollback exceptions, enabling the system to maintain high throughput while achieving zero service interruption, and enhancing the robustness of the inference process and the overall stability of the system. At the same time, it is easy to extend to the current commonly used inference framework.

[0041] Based on the above embodiment, the present application also provides an exemplary method for determining the load dynamic balance control condition, which quantifies the load dynamic balance control condition as a calculable judgment criterion, making the load balancing control operable and implementable, which can include the following steps:

[0042] The execution order constraint condition is based on filling the pipeline task line with a smaller number as an execution order constraint condition, and the pipeline number correlation adjustment information is determined according to the execution order constraint condition; the pipeline number correlation adjustment information is an adjustment coefficient positively related to the pipeline number; and the request allocation of each pipeline task line reaching the dynamic balance corresponding load dynamic balance control condition is determined according to the total amount of request tasks allocated to the previous pipeline task line, the total number of currently scheduled request tasks, and the pipeline number correlation adjustment information.

[0043] The execution order constraint condition is based on filling the pipeline task line with a smaller number as an execution order constraint condition, and the pipeline number correlation adjustment information is determined according to the execution order constraint condition; the pipeline number correlation adjustment information is an adjustment coefficient positively related to the pipeline number; and the request allocation of each pipeline task line reaching the dynamic balance corresponding load dynamic balance control condition is determined according to the total amount of request tasks allocated to the previous pipeline task line, the total number of currently scheduled request tasks, and the pipeline number correlation adjustment information.

[0044] In the embodiment, the dynamic expected load task baseline of the current global scheduling state of the pipeline is calculated according to the total number of currently scheduled request tasks and the pipeline number correlation adjustment information corresponding to the current pipeline task line. This baseline value represents the lower limit of the number of requests that should be allocated to the current pipeline by the end of the ideal balanced state considering the execution order. Then, the total amount of tasks actually carried by the previous pipeline task line is obtained, that is, the sum of the number of requests allocated to all pipeline task lines from the first pipeline to the previous pipeline. Finally, whether the balance is achieved is determined by comparing the relationship between the total amount of tasks actually carried by the previous pipeline task line and the dynamic expected load task baseline of the current pipeline task line. If the actual load of the previous pipeline task line has reached the expected baseline of the current pipeline task line, it is considered that the current pipeline task line can receive new requests to further balance; otherwise, it is considered that the load of the previous pipeline task line is insufficient, and if the request is allocated to the current pipeline task line, it will exacerbate the imbalance, and therefore the condition is not met.

[0045] As can be seen from the above, the embodiment provides an explicit and quantifiable calculation method for the load dynamic balance control condition, which makes the balance judgment more accurate, ensures that each pipeline task line achieves load balance according to the execution order constraint, avoids the imbalance of request allocation caused by the ambiguity of the balance standard, can more accurately control the distribution of requests, further improves the utilization rate of computing resources, and realizes more optimal dynamic load balance.

[0046] Further, in order to solve the problem that the judgment logic of the load dynamic balance control condition is not clear, which leads to the inability to accurately determine whether the current flow task line has the condition to receive a new request, and further affects the load balancing effect, the embodiment also provides a corresponding solution implementation, which can include the following contents:

[0047] According to the total number of scheduled request tasks and the flow path number correlation adjustment information, the dynamic expected load task reference of each flow task line is determined; according to the total number of request tasks allocated by the previous flow task line at the current time, the actual total amount of tasks carried by the previous flow task line is determined. Correspondingly, according to the relationship between the actual total amount of tasks carried by the previous flow task line and the dynamic expected load task reference of each flow task line, the load dynamic balance control condition corresponding to the dynamic balance of the request allocation of each flow task line is determined: if the actual total amount of tasks carried by the previous flow task line is less than the dynamic expected load task reference of the current flow task line, the current flow task line does not satisfy the load dynamic balance control condition; if the actual total amount of tasks carried by the previous flow task line is greater than or equal to the dynamic expected load task reference of the current flow task line, the current flow task line does not satisfy the load dynamic balance control condition.

[0048] Among them, the dynamic expected load task reference is calculated based on the total number of scheduled requests and the flow path number correlation adjustment information, and is the reference value of the number of requests that the current flow task line should carry, that is, the load reference value that the current flow task line should reach, which is the reference for judging whether the load is balanced. The actual total amount of tasks carried by the previous flow task line is the total number of request tasks that the previous flow task line has actually allocated at the current time. For example, the total number of scheduled requests M=40, the flow task line number n=2, the adjustment coefficient is 1, the dynamic expected load task reference of the current flow task line is calculated as 120 through 40×(2+1), and the actual total amount of tasks carried by the previous flow task line S=110 is calculated through the scheduling log of the inference framework. Comparing S and the dynamic expected load task reference: since 110<120, it is determined that the current flow task line does not satisfy the load dynamic balance control condition, and a new request task is not received temporarily; if the actual total amount of tasks carried by the previous flow task line increases to 120 or more subsequently, it is determined that the load dynamic balance control condition is satisfied.

[0049] As can be seen from the above, the embodiment clearly defines the judgment logic of the load dynamic balance control condition, so that the scheduling system can quickly and accurately determine the load state of the current flow task line; avoids allocating new requests when the load is unbalanced, ensures that the load of each flow task line is always in a balanced state, and improves the overall inference efficiency.

[0050] Further, in order to solve the problem that the dynamic expected load task benchmark calculation method is single and cannot accurately adjust according to the number of pipeline task lines, resulting in mismatch between load distribution and pipeline execution sequence, the embodiment also provides an implementation manner of how to calculate the dynamic expected load task benchmark, which can include the following contents:

[0051] The sum of the number of the current pipeline task line and the target adjustment coefficient is taken as the pipeline number associated adjustment information corresponding to the current pipeline task line, and the product operation result of the total number of the scheduled request tasks and the pipeline number associated adjustment information corresponding to the current pipeline task line is taken as the dynamic expected load task benchmark of the current pipeline task line.

[0052] The number of the pipeline is a serial number for uniquely identifying each pipeline task line, which is sequentially increased from 0, for example, 0, 1, 2, …, and the target adjustment coefficient is a fixed coefficient set for optimizing the calculation precision of the dynamic expected load task benchmark, which can be taken as 1 in the embodiment. The pipeline number associated adjustment information is composed of the sum of the number and the target adjustment coefficient, and is used to correct the influence of the total number of the scheduled requests on the dynamic expected load task benchmark. For example, the target adjustment coefficient is 1, the number of the current pipeline task line is n=3, the pipeline number associated adjustment information corresponding to the current pipeline task line is calculated as n+target adjustment coefficient=3+1=4. The total number of the scheduled request tasks is M=50, and the product operation of the total number of the scheduled request tasks M and the pipeline number associated adjustment information is performed, that is, 50*4=200, so that the dynamic expected load task benchmark of the current pipeline task line is 200.

[0053] As can be seen from the above, the pipeline number associated adjustment information is constructed by combining the number and the target adjustment coefficient, so that the calculation of the dynamic expected load task benchmark is more suitable for the pipeline execution sequence; the load benchmark of different numbered pipeline task lines is adjusted differently, the precision of load balancing is improved, and the utilization efficiency of the computing resources is further optimized.

[0054] In order to solve the problem that the scheduling sequence of the request in the to-be-scheduled request batch queue is chaotic, resulting in disordered request distribution and affecting the load balancing of the pipeline task line, and unable to guarantee the fairness of request processing, the application also provides a determination manner of the scheduling sequence strategy of the to-be-scheduled request batch queue, which can include the following contents:

[0055] According to the enqueue time sequence of each inference request task, a corresponding scheduling request identifier is assigned to each inference request task, and the value of the scheduling request identifier of each inference request task increases with the enqueue time sequence; according to the scheduling request identifier value from small to large sequence, each inference request task is extracted in turn and is assigned to each pipeline task line in turn according to the pipeline task line number.

[0056] The scheduling request identifier is a unique serial number assigned to each inference request task in the to-be-scheduled request batch queue, and is used to identify the order of the request. The order of the time of entering is the order of the inference request task being received and added to the to-be-scheduled request batch queue. In this embodiment, the to-be-scheduled request batch queue receives the inference request task in the order of the time of entering. When the first request task enters, the scheduling request identifier is assigned as 1; when the second request task enters, the scheduling request identifier is assigned as 2; and so on. The scheduling request identifier value increases in the order of the time of entering. The request task is extracted in the order of the scheduling request identifier value from small to large, that is, the request with the identifier 1 is extracted first, and then the request with the identifier 2 is extracted. The request task is assigned in the order of the serial number of the pipeline task line from small to large, that is, the request with the identifier 1 is assigned to the 0th pipeline task line, the request with the identifier 2 is assigned to the 1st pipeline task line, the request with the identifier 3 is assigned to the 0th pipeline task line, the request with the identifier 4 is assigned to the 1st pipeline task line, and the cycle is repeated to complete the assignment.

[0057] As can be seen from the above, the embodiment realizes the ordered scheduling and assignment of the inference request task, and guarantees the fairness of the request processing. The ordered assignment mode and the load balancing control condition work together to avoid the load imbalance caused by disordered scheduling, further improve the load balancing effect of each pipeline task line, and guarantee the stability and efficiency of the inference service.

[0058] In order to solve the problem that the scheduling information is not updated in time after the request is assigned, the subsequent load balancing judgment basis is inaccurate, and then the new request assignment decision is affected, the embodiment also needs to perform a state updating operation after successfully assigning the request, to ensure the real-time and accuracy of the system state information, and keep the internal state of the scheduler synchronized with the actual situation. The operation can include the following contents:

[0059] The total number of scheduled request tasks is updated; and the single pipeline scheduling information is updated according to the updated total number of scheduled request tasks.

[0060] The total number of scheduled request tasks is the total number of request tasks that have been extracted from the to-be-scheduled request batch queue and assigned to each pipeline task line. The single pipeline scheduling information is a data set recording the number of request tasks received by a single pipeline task line, the current processing progress, the remaining capacity and other related information.

[0061] For example, scheduled_running_reqs represents the total number of scheduled request tasks, then len(scheduled_running_reqs) can be added by 1 after successful allocation, and the field pipeline_reqs[batch_index] recording the number of allocated requests of the current pipeline task line is updated to the total number of requests allocated to the pipeline, i.e., len(request list dedicated to the pipeline) or equivalent value.

[0062] It can be understood that before performing inference on the inference request task issued to the user, the inference request tasks arriving within a period of time are aggregated and encapsulated into a request set, and then input into the heterogeneous computing system for PD separation inference. For each inference request task, the pre-population stage and the decoding stage need to be executed in turn to obtain the final task processing result. Among them, the pre-population stage processes the input of the inference request task and generates KV Cache (key value cache), which is relatively computationally intensive; the decoding stage generates the final result step by step according to the existing KV Cache, which is relatively sensitive to memory bandwidth, and the pipeline bubble problem is more, accordingly, the embodiment also clearly requests scheduling balancing for the decoding stage, which can include the following contents:

[0063] The intermediate representation data packet generated by the pre-population stage of the received inference request task is received, and the dynamic parameters required by the decoding stage of the inference request task are obtained; the decoding request task is constructed according to the intermediate representation data packet and the dynamic parameters, and the decoding request task is inserted into the corresponding to-be-scheduled request batch queue according to the inter-layer topology sequence.

[0064] Among them, the pre-population stage is the early stage of the inference process, which performs preprocessing, feature extraction, etc. on the input data to generate an intermediate representation data packet. The intermediate representation data packet is the output result of the pre-population stage, which contains the feature information of the preprocessed input data and is one of the input data of the decoding stage. The decoding stage is the stage of the inference process, which generates the final inference result such as text, answer, etc. based on the intermediate representation data packet and dynamic parameters. Dynamic parameters are configuration parameters required by the decoding stage, such as attention head activation state matrix, cross-layer normalization coefficient, and residual connection path identifier, which can be flexibly adjusted according to user demand or task type. The inter-layer topology sequence is the task insertion order determined according to the dependency relationship of each layer of the model, which ensures that the request task enters the to-be-scheduled request batch queue in order according to the model calculation logic, such as the output of the Nth layer as the input of the N+1th layer, to ensure the dependency relationship of the decoding stage.

[0065] In the embodiment, after the inference framework receives an inference request task, it first enters a pre-filling stage to perform pre-processing operations such as word segmentation and encoding on the input text data to generate an intermediate representation data packet containing data features. The decoding stage dynamic parameters of the inference request task are extracted, the inter-layer topology sequence is determined according to the dependency relationship of each layer of the model, and the decoding request task containing the intermediate representation data packet and the dynamic parameters is inserted into the corresponding to-be-scheduled request batch queue according to the sequence, so that the subsequent allocation can be processed in order according to the model calculation logic.

[0066] As can be seen from the above, the embodiment makes the request tasks entering the to-be-scheduled request batch queue highly match the model calculation logic, improves the processing efficiency of the decoding stage, avoids calculation errors caused by chaotic task insertion order, ensures the accuracy of the inference result, and provides a basis for subsequent ordered allocation and load balancing, further optimizing the overall inference performance.

[0067] Based on the above embodiment, the present application provides a processing method when the request cannot be allocated immediately, which can include the following contents:

[0068] If each flow task line does not meet the load dynamic balance control condition, the target inference request task is returned to the backtracking queue, and a backtracking state identifier is added to the target inference request task. If the target inference request task carries a flow routing mark, the scheduling order of the target inference request task in the backtracking queue is adjusted according to the flow routing mark and the dynamic parameters. If the parallel capacity limit condition is reached at the current time, the target inference request task is returned, and a real-time capacity analysis operation is triggered. When the capacity analysis result supports the construction of a new flow task line, the target inference request task is allocated to the new flow task line.

[0069] In the embodiment, the backtracking queue is a collection of inference request tasks that cannot be allocated due to not meeting the load balance condition or reaching the capacity limit. The backtracking state identifier is identifier information added to the request tasks in the backtracking queue, used to mark the backtracking reason of the task, such as load imbalance and insufficient capacity. The flow routing mark is a mark recording the flow task line number or execution computing device requirement information allocated to the request task before. The real-time capacity analysis operation is an operation triggered when the parallel capacity limit is reached, which analyzes the resource occupation and remaining capacity of each flow task line in real time.

[0070] In the embodiment, if the load dynamic balance control condition is not met, that is, the allocation of the request will cause serious imbalance, the scheduler will back off the target inference request task of the current round of scheduling to a special back-off queue, and mark it with a back-off state identifier. If the request already carries a water flow routing marker (indicating that it belongs to or once belonged to a specific water flow task line or is executed using a high-performance computing device), the request is redirected to the most suitable dedicated hardware water flow line to solve the resource mismatch problem; in combination with the marker and its dynamic parameters, the load is updated in real time to provide the latest computing load, memory pressure and other indicators for all water flow lines, and the queuing order of the request in the back-off queue is adjusted so that the request can be more efficiently rescheduled when the conditions are suitable. If the parallel capacity limit condition is reached (such as single water flow or total water flow full load), the request is also backed off. However, a real-time capacity analysis operation is triggered at this time. The analysis evaluates the computing resources of the current cluster, such as analyzing various indicators such as single water flow pipeline load rate, GPU memory utilization, bandwidth occupation, request queuing delay, etc. to analyze the capacity. The capacity analysis result is a hardware bottleneck, a software bottleneck, and a data bottleneck. If it is a hardware bottleneck, the physical water flow line can be increased, such as activating a backup computing unit; if it is a software bottleneck, the scheduling algorithm can be optimized, such as adjusting the water flow line allocation strategy; if it is a data bottleneck, the memory capacity can be expanded, such as activating a memory pool node. If the analysis result shows that a new water flow task line can be constructed, the system will perform dynamic expansion, and the backed-off request will be allocated to the newly created water flow line.

[0071] As can be seen from the above, the embodiment can avoid the loss of unallocatable request tasks, and ensure the effective processing of user requests. The back-off state identifier and the routing marker are used to optimize the scheduling order of the back-off queue, reduce the request processing delay, and dynamically adjust the allocation strategy based on real-time capacity analysis to fully utilize the computing resources and improve the flexibility and throughput of the system.

[0072] Based on the above embodiment, the present application also provides a solution to prevent misallocation or missed allocation due to single or improper sequence of judgment logic in a complex dynamic scheduling environment, which affects the load balancing effect and resource utilization, and can include the following contents:

[0073] The target inference request task extracted from the to-be-scheduled request batch queue for scheduling at the current time is first identified to determine whether the current water flow task line meets the corresponding load dynamic balance control condition. If the current water flow task line meets the corresponding load dynamic balance control condition, it is identified whether the parallel capacity limit condition is reached. If the parallel capacity limit condition is not reached at the current time, it is identified again whether the current water flow task line meets the corresponding load dynamic balance control condition. If the current water flow task line meets the corresponding load dynamic balance control condition, the target inference request task for scheduling at the current time is allocated to the current water flow task line.

[0074] The target inference request task is the inference request task currently extracted from the to-be-scheduled request batch queue and ready to be allocated. The load dynamic balance control condition is a condition for judging whether the current flow task line can receive a new request, which is determined based on the relationship between the load of the previous flow task line and the total number of scheduled requests. The parallel capacity limitation condition is a constraint condition for limiting the number of requests received by the flow task line to prevent resource overload.

[0075] In the embodiment, first, it is identified whether the current flow task line satisfies the load dynamic balance control condition. If yes, it is further identified whether the parallel capacity limitation condition is reached. If the capacity limitation is not reached, in order to ensure that the system state does not change significantly during the capacity check and cause the balance condition to be destroyed, it is identified again whether the current flow task line satisfies the load dynamic balance control condition. Only if this last check still satisfies, the target inference request task is finally allocated to the current flow task line.

[0076] As can be seen from the above, the three checks of balance, capacity and re-balance ensure the atomicity and correctness of the allocation decision, and prevent making an incorrect allocation due to state update in the judgment gap. Through the rigorous process of double judgment of the load dynamic balance control condition and the intermediate capacity limitation judgment, load imbalance and resource overload caused by incorrect allocation are avoided; it is ensured that the request task allocated to the flow task line meets the balance requirement and is within the capacity limitation, which improves the stability and reliability of system operation.

[0077] Based on the above embodiment, the present application also provides an implementation method for solving the problem that the parallel capacity limitation condition lacks clear judgment standard and calculation basis, which leads to inaccurate identification of whether the capacity limitation is reached, and further causes resource overload or resource waste, which can include the following contents:

[0078] The total number of scheduled request tasks is determined according to the processed data of the request; the total number of scheduled request tasks and the maximum number of request tasks of the single flow task line corresponding to the current flow task line are obtained; if the total number of scheduled request tasks is the same as the maximum number of request tasks of the single flow task line, the single flow task line capacity limitation condition is reached at the current time; if the total number of scheduled request tasks is the same as the upper limit of the total flow line load, the total flow line capacity limitation condition is reached at the current time.

[0079] The single-path water flow capacity limitation condition is a constraint condition of the maximum inference request task quantity that a single water flow task line can receive. The total water flow capacity limitation condition is a constraint condition of the maximum inference request task quantity that all water flow task lines can cumulatively receive. The total quantity of scheduled request tasks is the total quantity of inference request tasks that have been allocated to each water flow task line. The maximum request task quantity of a single water flow is the maximum request task quantity that a single water flow task line can bear, which is determined by hardware resources (such as a display memory) and model configuration. The upper limit of the total water flow line load is the maximum request task quantity that all water flow task lines can cumulatively bear, which is the sum of the maximum request task quantities of each single water flow.

[0080] In the embodiment, it is checked whether the quantity of requests allocated to the current water flow task line (which can be calculated by the distribution of the total quantity of scheduled request tasks in the current water flow line) is equal to the preset maximum request task quantity of a single water flow (which can be represented as max_num_pre), which can be determined by the display memory capacity, model size and KV Cache. If they are equal, the single-path capacity limitation is reached. It is checked whether the total quantity of scheduled request tasks is equal to the upper limit of the total water flow line load. The upper limit is the maximum quantity of requests that the system can simultaneously process according to the total resources (such as the total GPU display memory) and a preset safety factor. The scheduler needs to check these two conditions in sequence, and any one reaching the condition is regarded as reaching the parallel capacity limitation condition.

[0081] As can be seen from the above, the embodiment controls from the single device and global cluster dimensions respectively, clearly defines the judgment standard and calculation basis of the parallel capacity limitation condition, can accurately identify the capacity state of the single-path water flow and the total water flow, avoids resource overload or waste caused by inaccurate capacity judgment, guarantees the efficient operation of the system within the safe capacity range, and provides reliable support for request allocation decision. The single-card display memory overflow and the entire cluster resource overload are effectively prevented, and the stability and quality of the inference service are guaranteed.

[0082] Based on the above embodiment, in order to avoid the problem of resource overload, the application also provides an exemplary determination method of the upper limit of the total water flow line load, which can include the following contents:

[0083] The total quantity of inference request tasks is determined according to the water flow scheduling running information, and the actual total quantity of inference request tasks is determined according to the total quantity of inference request tasks and a preset safety parameter. The upper limit of the total water flow line load is obtained by performing an upward integer division calculation on the actual total quantity of inference request tasks and a parallel scale parameter of the water flow line.

[0084] Wherein, the flow water scheduling operation information is a data set recording scheduling state, allocated request quantity, processing progress and other information of the flow water task line. The total number of inference request tasks is the total number of inference request tasks that have not been allocated in the request batch queue to be scheduled. The preset safety parameter is a redundant parameter set to cope with resource fluctuation in the request processing process, which is hardware friendly, used to adjust the actual total number of inference request tasks, avoid resource overload, avoid pipeline pause and prevent integer overflow. The actual total number of inference request tasks is the number of inference request tasks used to calculate the upper limit of the total pipeline load after considering the preset safety parameter. The pipeline parallel scale parameter is the total number of flow water task lines. The upper limit of the total pipeline load is the maximum number of request tasks that can be carried by all flow water task lines, which is calculated from the actual total number of inference request tasks and the pipeline parallel scale parameter.

[0085] In the embodiment, the total number of inference request tasks to be processed is determined according to the current flow water scheduling operation information (such as the running queue). For system stability, a preset safety parameter (for example, 1, which reserves 10% of the memory) is used to modify the total number to obtain the actual total number of inference request tasks, which is used for more conservative capacity estimation. Then, the actual total number is divided by the pipeline parallel scale parameter (that is, the total number of flow water lines pp_size), and an upward rounding division operation is performed. For example, the upper limit of the total pipeline load can be calculated by the relationship len((actual total number of inference request tasks / pp_size)-1)*pp_size, which ensures that the upper limit of the load is an integer multiple of the number of flow water lines, which meets the principle of balanced distribution.

[0086] As can be seen from the above, the embodiment can make full use of resources, leave a safety margin for system fluctuation, and coordinate with the load balancing goal.

[0087] Based on the above embodiment, the present application also provides a method for determining the pipeline parallel scale parameter (pp_size) to avoid the problem of insufficient use of computing resources or resource allocation conflict, which can include the following contents:

[0088] Obtaining the total number of devices of the decoding stage of the computing device for executing the inference request task; determining the pipeline parallel scale parameter according to the pipeline parallel scale parameter being less than the total number of devices.

[0089] Wherein, the computing device is a hardware device for executing inference request task calculation, such as GPU, XUP and TPU. The total number of devices is the total number of computing devices participating in inference calculation. The pipeline parallel scale parameter is the total number of flow water task lines, that is, the number of parallel processing links.

[0090] In the embodiment, the total number of computing devices performing the inference request task decoding stage is counted, and the currently deployed computing devices are 16 GPUs, and the total number of devices is 16. According to the computing power, memory size and other resource conditions of the computing devices, it is determined that the pipeline parallel scale parameter needs to be less than the total number of devices to ensure that each pipeline task line has sufficient hardware resources to support. Considering the model splitting demand and resource utilization, the pipeline parallel scale parameter is set to 8, that is, 8 pipeline task lines are constructed, and each pipeline task line is allocated 2 GPUs to ensure reasonable resource allocation and no conflict.

[0091] As can be seen from the above, the embodiment matches the pipeline parallel scale parameter with the total number of computing devices, fully utilizes the existing computing resources; avoids resource allocation conflicts caused by setting the parameter too large, or resource waste caused by setting the parameter too small; provides reasonable parallel architecture support for subsequent request allocation and load balancing, and improves the overall inference efficiency.

[0092] To enable those skilled in the art to clearly understand the technical solution of the present invention, the present invention also provides an exemplary implementation method. When performing inference framework request scheduling, scheduling judgment is added. When requesting scheduling, all requests for pipeline scheduling are recorded in the runing (pipeline scheduling operation information) field, and the number of requests for each pipeline scheduling is recorded in the pipeline_reqs (single pipeline scheduling information) field. req_index represents the index of the current scheduling request in running, batch_index represents which pipeline it is currently (the pipeline record field), scheduled_running_reqs (processed request data) represents the field of the number of requests that have been processed during the pipeline scheduling process, pp_size (parallel scale parameter) represents the number of pipeline parallelisms when deploying the inference service, and max_num_pre (maximum number of request tasks for a single pipeline) represents the maximum number of requests for each inference when deploying the inference service, that is, the maximum number of requests for each pipeline. 1) When len(scheduled_running_reqs) == max_num_pre, the maximum number of requests for each pipeline request scheduling is reached, stop scheduling new requests, and perform subsequent calculations; 2) When len(scheduled_running_reqs) == (len(running) + pp_size - 1) / / pp_size, / / represents integer division, the maximum number of requests for each pipeline scheduling has not been reached, and the balanced state of each pipeline is reached, stop scheduling new requests, and perform subsequent calculations. When requesting scheduling, the number of requests in the balanced queue and when solving the problem of insufficient video memory space during request scheduling, the problem of program interruption caused by request fallback is solved. This embodiment can dynamically adjust the number of requests for each pipeline scheduling through the following code and distribute them to each pipeline in an orderly manner, ensuring the balance of each pipeline while avoiding the problem of program interruption caused by request fallback. The code can be:

[0093] while req_index < len(self.running): (indicating an operation to traverse all pending scheduling requests)

[0094] if sum(pipeline_reqs[:batch_index]) < len(scheduled_running_reqs) * (batch_index + 1): (indicating an operation to check load balance)

[0095] break;

[0096] if pipeline_parallel_size > 1: (indicating an operation to check for multi-pipeline specific situations)

[0097] if len(scheduled_running_reqs)==max_num_pre: (indicates the operation of judging whether the current pipeline has reached the maximum capacity)

[0098] break;

[0099] if len(scheduled_running_reqs)==(len(running)+pp_size-1) / / pp_size: (indicates the operation of judging whether the request distribution is balanced, ensuring that the requests are evenly distributed to each pipeline)

[0100] break;

[0101] if sum(pipeline_reqs[:batch_index])<len(scheduled_running_reqs)*(batch_index+1): (indicates the operation of load balancing check)

[0102] break;

[0103] Request parsing and request rollback operations

[0104] scheduled_running_reqs adds a request

[0105] pipeline_reqs[batch_index]=len(scheduled_running_reqs) (indicates the operation of updating the request count of the current pipeline).

[0106] It should be noted that there is no strict execution order between the steps in the present application, as long as the logical order is met, the steps can be executed simultaneously, or according to a certain preset order, Figure 3 and Figure 4 It is only an illustrative way and does not mean that only this execution order can be used.

[0107] The application also provides a corresponding device for the request task processing method, further making the method more practical. The device can be described from the perspective of functional modules and the perspective of hardware. The request task processing device provided by the application is introduced below, which is used to implement the request task processing method provided by the application. In this embodiment, the request task processing device can include or be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to complete the request task processing method disclosed in embodiment one. The program module referred to in this embodiment refers to a series of computer program instruction segments that can complete a specific function, which is more suitable for describing the execution process of the request task processing device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module in this embodiment. The request task processing device described below can be mutually referred to with the request task processing method described above.

[0108] Based on the perspective of functional modules, refer to Figure 5 , Figure 5 The structure diagram of the request task processing device provided in this embodiment is shown in the following figure. The device can include:

[0109] The scheduling module 501 is configured to add the received inference request task to the to-be-scheduled request batch queue, and extract each inference request task from the to-be-scheduled request batch queue and assign it to each flow task line of the parallel execution task.

[0110] The load balancing module 502 is configured to determine the request assignment of each flow task line to reach the load dynamic balancing control condition corresponding to the dynamic load balancing condition according to the relationship between the total amount of request tasks assigned to the previous flow task line and the total number of currently scheduled request tasks, and based on the flow task line execution sequence constraint condition.

[0111] The request assignment module 503 is configured to, in the assignment process, if the current flow task line meets the corresponding load dynamic balancing control condition and the parallel capacity limitation condition is not reached at the current time, assign the target inference request task scheduled at the current time to the current flow task line; if the current flow task line does not meet the corresponding load dynamic balancing control condition or the parallel capacity limitation condition is not reached at the current time, suspend assigning new request tasks to the current flow task line; wherein the previous flow task line includes the flow task lines from the first flow task line to the current flow task line.

[0112] Exemplarily, in some embodiments of the present embodiment, the load balancing module 502 can be further configured to: determine, based on the priority of filling the pipeline task line with a smaller pipeline task line number as an execution sequence constraint condition, and determine the pipeline number correlation adjustment information according to the execution sequence constraint condition; the pipeline number correlation adjustment information is an adjustment coefficient positively correlated with the pipeline task line number; and determine the request allocation of each pipeline task line reaching the dynamic balance corresponding to the load dynamic balance control condition according to the total number of request tasks allocated to the previous pipeline task line, the total number of currently scheduled request tasks, and the pipeline number correlation adjustment information.

[0113] As an exemplary implementation of the above embodiment, the load balancing module 502 can be further configured to: determine the dynamic expected load task reference of each pipeline task line according to the total number of scheduled request tasks and the pipeline number correlation adjustment information; determine the total number of actual carried tasks of the previous pipeline task line as the actual total number of carried tasks of the previous pipeline task line according to the total number of allocated request tasks of the previous pipeline task line at the current time; if the actual total number of carried tasks of the previous pipeline task line is less than the dynamic expected load task reference of the current pipeline task line, the current pipeline task line does not satisfy the load dynamic balance control condition; and if the actual total number of carried tasks of the previous pipeline task line is greater than or equal to the dynamic expected load task reference of the current pipeline task line, the current pipeline task line does not satisfy the load dynamic balance control condition.

[0114] As an exemplary implementation of the above embodiment, the load balancing module 502 can be further configured to: determine the sum of the pipeline number of the current pipeline task line and the target adjustment coefficient as the pipeline number correlation adjustment information corresponding to the current pipeline task line, and determine the product of the total number of scheduled request tasks and the pipeline number correlation adjustment information corresponding to the current pipeline task line as the dynamic expected load task reference of the current pipeline task line.

[0115] Exemplarily, in some other embodiments of the present embodiment, the scheduling module 501 can be further configured to: configure the scheduling order of the inference request task in the request batch queue to be positively correlated with the enqueue time order, assign each inference request task with a corresponding scheduling request identifier according to the enqueue time order of each inference request task, and increase the scheduling request identifier value of each inference request task as the enqueue time order increases; and extract each inference request task in order according to the scheduling request identifier value from small to large, and assign each inference request task to each pipeline task line in order according to the pipeline task line number.

[0116] Exemplarily, in some other embodiments of the present embodiment, the request allocation module 503 can be further configured to: update the total number of scheduled request tasks; and update the single pipeline scheduling information according to the updated total number of scheduled request tasks.

[0117] Exemplarily, in some other embodiments of the present embodiment, the scheduling module 501 can be further configured to receive the intermediate representation data packet generated in the pre-filling stage of the inference request task, and obtain the dynamic parameters required in the decoding stage of the inference request task; construct a decoding request task according to the intermediate representation data packet and the dynamic parameters, and insert the decoding request task into the corresponding to-be-scheduled request batch queue according to the inter-layer topology sequence.

[0118] Exemplarily, in some other embodiments of the present embodiment, the request allocation module 503 can be further configured to: if none of the pipelining task lines meets the load dynamic balance control condition, back up the target inference request task to a back-up queue, and add a back-up state identifier to the target inference request task; if the target inference request task carries a pipelining routing mark, adjust the scheduling order of the target inference request task in the back-up queue according to the pipelining routing mark and the dynamic parameters; if the parallel capacity limit condition is met at the current time, back up the target inference request task and trigger a real-time capacity analysis operation; and when the capacity analysis result supports the construction of a new pipelining task line, direct the target inference request task to the new pipelining task line.

[0119] Exemplarily, in some other embodiments of the present embodiment, the request allocation module 503 can be further configured to: for the target inference request task to be scheduled at the current time extracted from the to-be-scheduled request batch queue, first identify whether the current pipelining task line meets the corresponding load dynamic balance control condition; if the current pipelining task line meets the corresponding load dynamic balance control condition, identify whether the parallel capacity limit condition is met; if the parallel capacity limit condition is not met at the current time, identify again whether the current pipelining task line meets the corresponding load dynamic balance control condition; and if the current pipelining task line meets the corresponding load dynamic balance control condition, allocate the target inference request task to be scheduled at the current time to the current pipelining task line.

[0120] As an exemplary implementation of the above embodiment, the load balance module 502 can be further configured to: determine the total number of scheduled request tasks according to the request processed data; obtain the total number of scheduled request tasks and the maximum number of request tasks of a single pipelining of the current pipelining task line; if the total number of scheduled request tasks is the same as the maximum number of request tasks of a single pipelining, the single pipelining capacity limit condition is met at the current time; and if the total number of scheduled request tasks is the same as the upper limit of the total pipelining line load, the total pipelining capacity limit condition is met at the current time.

[0121] As another exemplary implementation of the above embodiment, the load balancing module 502 can be further configured to: determine a total number of inference request tasks according to the flow scheduling operation information, and determine an actual total number of inference request tasks according to the total number of inference request tasks and a preset safety parameter; and perform an upward rounding division calculation on the actual total number of inference request tasks and the pipeline parallel scale parameter to obtain a total pipeline load upper limit.

[0122] As another exemplary implementation of the above embodiment, the load balancing module 502 can be further configured to: obtain a total number of devices of the computing devices performing the decoding stage of the inference request task; and determine the pipeline parallel scale parameter according to the pipeline parallel scale parameter being less than the total number of devices.

[0123] The request task processing apparatus mentioned above is described from the perspective of functional modules, and further, the present application also provides an electronic device, which is described from the perspective of hardware. Figure 6 The electronic device provided by the embodiment of the present application has the structure shown in the structural schematic diagram in one implementation. The electronic device includes a memory 601 and a processor 602, the memory 601 stores a computer program, and the processor 602 is configured to run the computer program to perform the steps in any of the above request task processing method embodiments.

[0124] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above request task processing method embodiments when running.

[0125] In one exemplary embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0126] The embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps in any of the above request task processing method embodiments.

[0127] The embodiment of the present application also provides another computer program product, which includes a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above request task processing method embodiments.

[0128] The above describes in detail the request task processing method and the electronic device provided by the present application. Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to by each other. The units and algorithm steps of each example described in each disclosed embodiment are executed in an electronic hardware or computer software manner, which depends on the specific application and design constraints of the technical solution. The professional technicians can use different methods to implement the described functions for each specific application, and such implementation should not be considered beyond the scope of the present application. Without departing from the principles of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method of processing a request task, characterized by, The method comprises the following steps: adding the received inference request task to a to-be-scheduled request batch queue, extracting each inference request task from the to-be-scheduled request batch queue and assigning the inference request task to each pipeline task line for parallel execution of the task; determining a load dynamic balance control condition corresponding to a dynamic balance of request assignment of each pipeline task line according to a relationship between a total amount of request tasks assigned to a previous pipeline task line and a current total number of scheduled request tasks and based on a pipeline task line execution sequence constraint condition; in the assignment process, if the current pipeline task line meets the corresponding load dynamic balance control condition and a parallel capacity limitation condition is not met at the current time, assigning a target inference request task scheduled at the current time to the current pipeline task line; if the current pipeline task line does not meet the corresponding load dynamic balance control condition or the parallel capacity limitation condition is met at the current time, suspending assignment of a new request task to the current pipeline task line; wherein the previous pipeline task line comprises pipeline task lines from a first pipeline task line to the current pipeline task line; the load dynamic balance control condition corresponding to the dynamic balance of the request assignment of each pipeline task line is determined according to a relationship between a total amount of request tasks assigned to the previous pipeline task line and a current total number of scheduled request tasks and based on a pipeline task line execution sequence constraint condition, comprising: determining a pipeline number correlation adjustment information according to the execution sequence constraint condition, wherein the pipeline number correlation adjustment information is a regulation coefficient positively correlated with a pipeline number; and determining the load dynamic balance control condition corresponding to the dynamic balance of the request assignment of each pipeline task line according to the total amount of request tasks assigned to the previous pipeline task line, the current total number of scheduled request tasks and the pipeline number correlation adjustment information.

2. The request task processing method according to claim 1, characterized by, determining the load dynamic balance control condition corresponding to the dynamic balance of the request assignment of each pipeline task line according to the total amount of request tasks assigned to the previous pipeline task line, the current total number of scheduled request tasks and the pipeline number correlation adjustment information, comprising: determining a dynamic expected load task benchmark of each pipeline task line according to the current total number of scheduled request tasks and the pipeline number correlation adjustment information; and taking a total amount of request tasks assigned to the previous pipeline task line at the current time as an actual carried task total amount of the previous pipeline task line; if the actual carried task total amount of the previous pipeline task line is less than the dynamic expected load task benchmark of the current pipeline task line, the current pipeline task line does not meet the load dynamic balance control condition; if the actual carried task total amount of the previous pipeline task line is greater than or equal to the dynamic expected load task benchmark of the current pipeline task line, the current pipeline task line meets the load dynamic balance control condition.

3. The request task processing method according to claim 2, wherein determining the dynamic expected load task benchmark of each pipeline task line according to the current total number of scheduled request tasks and the pipeline number correlation adjustment information, comprising: The sum of the number of the current flow task line and the target adjustment coefficient is used as flow path number associated adjustment information corresponding to the current flow task line, and the product of the current scheduled request task total number and the flow path number associated adjustment information corresponding to the current flow task line is used as a dynamic expected load task reference of the current flow task line.

4. The request task processing method according to claim 1, characterized by, The to-be-scheduled request batch queue is configured to have a positive correlation between the scheduling order of the inference request task and the time order of entering the queue. Each inference request task is extracted from the to-be-scheduled request batch queue and assigned to each flow task line of the parallel execution task, including: According to the time order of entering the queue of each inference request task, each inference request task is assigned a corresponding scheduling request identifier, and the scheduling request identifier value of each inference request task increases with the time order of entering the queue; According to the order from small to large of the scheduling request identifier value, each inference request task is extracted in turn and assigned to each flow task line in turn according to the flow task line number.

5. The request task processing method according to claim 1, wherein After assigning the target inference request task scheduled at the current time to the current flow task line, the method further includes: updating the current total number of scheduled request tasks; updating the single flow scheduling information according to the updated total number of scheduled request tasks.

6. The request task processing method according to claim 1, wherein The received inference request task is added to the to-be-scheduled request batch queue, including: receiving the intermediate representation data packet generated in the pre-population stage of the inference request task, and obtaining the dynamic parameters required in the decoding stage of the inference request task; constructing a decoding request task according to the intermediate representation data packet and the dynamic parameters, and inserting the decoding request task into the corresponding to-be-scheduled request batch queue according to the inter-layer topology sequence.

7. The request task processing method according to claim 1, wherein Further comprising: If each flow task line does not meet the load dynamic balance control condition, the target inference request task is rolled back to the rollback queue, and a rollback state identifier is added to the target inference request task, and if the target inference request task carries a flow routing mark, the scheduling order of the target inference request task in the rollback queue is adjusted according to the flow routing mark and the dynamic parameters; If the current time reaches the parallel capacity limit condition, the target inference request task is rolled back, and a real-time capacity analysis operation is triggered; when the capacity analysis result supports the construction of a new flow task line, the target inference request task is assigned to the new flow task line.

8. The method of claim 1 to 7, wherein, If the current flow task line meets the corresponding load dynamic balance control condition and the current time does not reach the parallel capacity limit condition, the target inference request task scheduled at the current time is assigned to the current flow task line, including: The target inference request task scheduled at the current time is extracted from the to-be-scheduled request batch queue, and it is first identified whether the current flow task line meets the corresponding load dynamic balance control condition; If the current flow task line meets the corresponding load dynamic balance control condition, it is identified whether the parallel capacity limit condition is reached; If the current time does not reach the parallel capacity limit condition, it is identified again whether the current flow task line meets the corresponding load dynamic balance control condition; If the current pipeline task line meets the corresponding load dynamic balance control condition, a target reasoning request task scheduled at the current time is allocated to the current pipeline task line.

9. The request task processing method according to claim 8, wherein The parallel capacity limit condition includes a single-path pipeline capacity limit condition and a total pipeline capacity limit condition, and identifying whether the parallel capacity limit condition is reached includes: Determining a current total number of scheduled request tasks according to processed data; Obtaining the current total number of scheduled request tasks and a maximum number of request tasks of the single-path pipeline corresponding to the current pipeline task line; If the current total number of scheduled request tasks is the same as the maximum number of request tasks of the single-path pipeline, the single-path pipeline capacity limit condition is reached at the current time; If the current total number of scheduled request tasks is the same as the total pipeline load upper limit, the total pipeline capacity limit condition is reached at the current time.

10. The method of claim 8, wherein, Before identifying whether the parallel capacity limit condition is reached, further including: Determining a total number of reasoning request tasks according to pipeline scheduling operation information, and determining an actual total number of reasoning request tasks according to the total number of reasoning request tasks and a preset safety parameter; Performing an upward integer division calculation on the actual total number of reasoning request tasks and a pipeline parallel scale parameter to obtain the total pipeline load upper limit.

11. The method of claim 10, wherein, Before identifying whether the parallel capacity limit condition is reached, further including: Obtaining a total number of devices of a computing device in a decoding stage of executing the reasoning request task; Based on the pipeline parallel scale parameter being less than the total number of devices, determining the pipeline parallel scale parameter.

12. An electronic device, comprising: Including: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the request task processing method according to any one of claims 1 to 11.

Images