Reasoning request task processing method and device, equipment, medium, product and heterogeneous system

Abstract

The application discloses a reasoning request task processing method and device, equipment, medium, product and a heterogeneous system, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring reasoning task information corresponding to each reasoning request task and computing resource information of a heterogeneous computing system; determining target computing equipment of a pre-filling stage and a decoding stage for executing each reasoning request task and corresponding task start time based on the condition that the processing cost of each reasoning request task is the lowest under the condition that each task processing requirement information is met, according to the reasoning task information and the computing resource information; sending a task allocation result as a task allocation request to the heterogeneous computing system; and each target computing equipment executes corresponding tasks according to the task start time of the task allocation request. The application can solve the problem that related technologies cannot simultaneously consider task processing requests and low execution costs, and can allocate computing equipment for each reasoning task in the most cost-effective manner under the condition that task processing requirements are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a reasoning request task processing method and device, equipment, medium, product and a heterogeneous system. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the scale of network models processing artificial intelligence related tasks is getting larger and larger. In order to accelerate the task execution efficiency of large-scale network models, related technologies based on PD (Prefill-Decode, prefill-decode) separate reasoning method split the model and deploy it to multiple computing devices of a heterogeneous computing platform. However, in order to meet the reasoning request task processing request, it is difficult to execute the user's reasoning request task with lower cost as much as possible. SUMMARY

[0003] The present application provides a heterogeneous computing system and its reasoning request task processing method, device, electronic equipment, non-volatile storage medium and computer program product, which can reasonably allocate each computing device of the heterogeneous computing system under the condition of meeting the reasoning request task processing requirements of the reasoning task request, and realize the execution of the arrived reasoning request task with lower cost as much as possible.

[0004] To solve the above technical problems, the present application provides the following technical solutions:

[0005] In one aspect, the present application provides a reasoning request task processing method, comprising:

[0006] obtaining reasoning task information corresponding to each reasoning request task and computing power resource information of a heterogeneous computing system executing each reasoning request task; determining target computing devices of a prefill stage and a decode stage of each reasoning request task and corresponding task start time based on the condition that the processing cost of each reasoning request task is lowest under meeting each task processing requirement information according to the reasoning task information and the computing power resource information, to obtain a task allocation result; sending the task allocation result as a task allocation request to the heterogeneous computing system, so that each target computing device executes the corresponding task according to the task start time of the task allocation request; wherein the reasoning task information includes resource requirement information and task processing requirement information; the computing power resource information includes the computing ability and execution cost of each computing device of the heterogeneous computing system.

[0007] In another aspect, the present application provides a reasoning request task processing device, comprising:

[0008] The data acquisition module is used to acquire inference task information corresponding to each inference request task, as well as computing resource information of the heterogeneous computing system executing each inference request task; wherein, the inference task information includes resource requirement information and task processing requirement information; the computing resource information includes the computing power and execution cost of each computing device in the heterogeneous computing system.

[0009] The task optimization module is used to determine the target computing devices and corresponding task start times for executing the pre-filling and decoding stages of each inference request task based on the information of each inference task and computing resources, and based on the condition of achieving the lowest processing cost for each inference request task while meeting the processing requirements of each task, so as to obtain the task allocation results.

[0010] The task assignment and execution module is used to generate task assignment requests based on the task assignment results and send the task assignment requests to the heterogeneous computing system so that each target computing device can execute the corresponding task according to the task start time of the task assignment request.

[0011] The present invention also provides an electronic device, including a memory and a processor, wherein the processor is configured to implement the steps of any of the above-described inference request task processing methods when executing a computer program stored in the memory.

[0012] The present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of any of the above-described inference request task processing methods.

[0013] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the above-described inference request task processing methods.

[0014] Finally, the present invention also provides a heterogeneous computing system, including multiple types of computing devices and processors, with each type of computing device connected to the processor; the processor executes a computer program stored in the memory to implement the steps of any of the above-mentioned inference request task processing methods.

[0015] The advantages of the technical solution provided by this invention are that the inference task information clarifies the resource requirements (such as the computational load of the P / D stage) and task processing requirements (such as request priority and time limit) of the inference request task, while the computing power resource information quantifies the computing power (such as execution speed) and execution cost of heterogeneous devices. This avoids the randomness and inaccuracy of matching devices based solely on experience in traditional allocation. By specifying the task start time, it ensures that the D stage of the same inference request task starts after the P stage and the total time does not exceed the upper limit of the task requirements. This supplements the accuracy of resource allocation from the time dimension, avoids the task queuing timeout problem caused by device occupation conflicts, and ultimately ensures that the P / D stage of each inference request task received by the heterogeneous computing system can be matched with a computing device that is performance-adapted, cost-controllable, and time-feasible. This avoids the problem of high-priority requests timeout due to low-priority resources being occupied and the waste of resources caused by the mismatch between device performance and the computational load of the request. It ensures that all inference request tasks can be executed efficiently while meeting their own processing requirements. Furthermore, the goal is to minimize the execution cost of each inference request task, while adhering to the constraints of meeting task processing requirements. This avoids blindly selecting the most expensive high-end equipment to meet the time requirements of high-priority requests, which would lead to inflated costs, and also avoids selecting underperforming equipment to reduce costs, which would lead to request timeouts. This achieves the global optimization of the total processing cost of all inference request tasks. The cost optimization effect is particularly significant for scenarios with multiple types of inference request tasks (such as high / low priority, large / small computational load), effectively reducing the operating cost of heterogeneous computing systems. By optimizing task startup times, data transmission congestion between computing devices is effectively reduced, ensuring the continuous and reliable operation of heterogeneous computing systems. This is especially suitable for scenarios with large-scale concurrent inference requests.

[0016] Furthermore, the present invention also provides corresponding implementation devices, electronic devices, non-volatile storage media, and computer program products for the fault handling method, further making the method more practical, and the devices, electronic devices, non-volatile storage media, and computer program products have corresponding advantages. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the present invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the hardware framework applicable to the reasoning request task processing method provided by the present invention.

[0019] Figure 2This is a flowchart illustrating a reasoning request task processing method provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the framework of the heterogeneous computing system provided by the present invention in an exemplary application scenario.

[0021] Figure 4 This is a schematic diagram illustrating the deployment of PD split inference in a heterogeneous computing system provided by the present invention.

[0022] Figure 5 This is a structural framework diagram of an exemplary embodiment of the reasoning request task processing device provided by the present invention.

[0023] Figure 6 This is a structural diagram of an exemplary embodiment of the electronic device provided by the present invention.

[0024] Figure 7 This is a structural framework diagram of an exemplary embodiment of the heterogeneous computing system provided by the present invention.

[0025] Figure 8 This is a flowchart illustrating the reasoning request task processing method provided by the present invention in an exemplary application scenario. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. In this specification and the aforementioned drawings, the terms "first," "second," "third," "fourth," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0027] With the rapid development of artificial intelligence (AI) technology, an increasing number of technical fields are using trained and deployed AI network models to solve business problems, such as Large Language Models (LLMs). To meet users' high-precision business processing needs, the scale of network models processing AI-related tasks is growing, leading to increasingly higher demands on computing and storage resources. To accelerate the inference speed of large-scale network models, distributed inference has emerged. Distributed inference involves splitting a large-scale network model and deploying the split models onto multiple hardware devices with computing and storage resources. One such method is P (Prefill) and D (Decode) separation inference, which splits the prefill (P stage) and decoding (D stage) stages of the inference process and deploys them to different computing devices. For example, deploying the prefill and decoding stages on different graphics processors or instances, and optimizing scheduling and parallel strategies for each stage, eliminates competition between stages and significantly improves throughput and response speed.

[0028] Heterogeneous computing systems integrate computing devices of various types and performance levels on the same or multiple physical hosts within a single distributed computing environment, such as a multi-host heterogeneous computing system. Each physical host can be defined as a computing node, and computing devices with different performance levels collaborate to complete distributed inference tasks for large-scale network models. However, because the computing performance of each device in a multi-host heterogeneous computing system varies, and their operating costs also differ, related technologies cannot achieve the lowest possible cost while still meeting user task processing requests.

[0029] For example, one related technology allows the client to maintain parameters in an efficient fine-tuning method and then fine-tunes the neural network on the server through edge-cloud collaboration, enabling the server-side neural network to provide personalized services at a lower cost. However, this method does not specifically consider the current process of separating product development (PD) in large-scale model inference, nor does it take into account the heterogeneous nature of computing centers, thus limiting its cost optimization capabilities. Another related technology first uses one language model to generate an initial idea for task execution, and then uses another language model for correction and optimization. Based on a dual-model collaborative reflection approach, the second language model uses real-time scoring and correction of the inference path generated by the first language model to promptly adjust for errors and ensure the authenticity of the inference chain. Although it can generate high-quality inference data efficiently and at low cost, it also does not specifically consider the current process of separating PD in large-scale model inference, nor does it take into account the heterogeneous nature of computing centers, thus limiting its cost optimization capabilities. Another related technology provides multiple standardized functions through multiple standardized inference interfaces and loads large models into GPU (graphics processing unit) hardware drivers to perform model inference calculations. It utilizes a distributed design to deploy multiple large models with different parameter amounts to multiple GPU graphics card hardware at the same time. However, this method does not consider how to meet the user's requirements for inference request tasks.

[0030] Therefore, this invention rationally deploys large model inference request tasks with separate PD (Programmable Detection) operations in heterogeneous computing systems. While simultaneously meeting the task requirements of inference request tasks, such as request priority and time consumption, it executes arriving inference request tasks at the lowest possible cost, achieving a reasonable allocation of inference request tasks in heterogeneous computing systems. The specific application environment architecture or specific hardware architecture upon which the inference request task processing method depends is described below. Figure 1 Examples of possible application scenarios related to the technical solutions of this invention are provided below:

[0031] A multi-type heterogeneous computing system comprises multiple general-purpose servers and multiple artificial intelligence (AI) servers. Each AI server is equipped with multiple different types of computing devices. For example, a multi-type heterogeneous computing system may include 18 Type 1 servers equipped with 4 GPUs each, 12 Type 2 servers equipped with 4 XPUs (heterogeneous computing processors), and 8 Type 3 servers equipped with 4 Tensor Processing Units (TPUs). These servers are interconnected via switches to build a lossless network with a speed of at least 100GB / s using RoCEv2 (RDMA over Converged Ethernet Version 2) to ensure stable data transmission between the computing devices. Type 1 servers can be used, for example, for computation in the high-parallel pre-filling stage, adapting to the input processing of large models. Type 2 servers can be used, for example, to optimize low-latency characteristics in the decoding stage, meeting the service response requirements of inference requests. Type 3 servers can serve as backup computing nodes, undertaking fault migration tasks.

[0032] In addition to its own management node 1, the heterogeneous computing system can select a high-performance server as task optimization node 2. The management node 1 is used to receive inference request tasks sent by one or more user terminals 3, and aggregate all inference request tasks received within a certain period of time, such as 1 second, and encapsulate them into a request set. Then, it triggers the task optimization node 2 to allocate corresponding execution devices and execution time to each inference request task in the request set. The management node 1 will collect the computing power and execution cost of each computing device in the heterogeneous computing system, store it as computing power resource information of the heterogeneous computing system in a designated location, such as a shared space, or send it to the task optimization node 2 for storage in the local storage device of the task optimization node 2. Whenever a user-issued inference request task is received, the task parameter data corresponding to that inference request task is obtained, such as the computational load of the inference request task in the pre-filling and decoding stages, the upper limit of the execution time of the inference request task, and the processing priority of the request. When the pre-set task processing conditions are met, such as the number of inference request tasks contained in the request set reaching a specified number, or the timeout period reaching a specified task collection period, the task parameter data corresponding to each inference request task and the total number of inference request tasks contained in the request set are packaged as inference task information and sent to task optimization node 2. Upon receiving the inference task information, task optimization node 2 triggers the allocation of corresponding execution devices and execution time for each inference request task in the request set, so that the execution cost of all inference request tasks is kept as low as possible.

[0033] Task optimization node 2 calculates the execution time and cost of each inference request task on different computing devices: For example, inference request task 1 takes 0.000512 seconds in the P phase on computing device A100, with a cost of 0.0002048 yuan; the D phase takes 0.000256 seconds, with a cost of 0.0001024 yuan, and the total cost for completing the entire inference request task is 0.0003072 yuan. Tasks are allocated according to their priority from highest to lowest. Inference request task 1, with the highest priority, is first assigned to computing device A100 for the P phase, and then to computing device X100 for the D phase. For other priority tasks, all computing device combinations are enumerated. Each combination has one computing device for the P phase and one for the D phase; the two phases can also use the same computing device. The computing device combination that satisfies the condition of not exceeding the maximum task time and having the lowest cost is selected as the initial allocation result, and the computing device task execution table is updated. The iteration count is set to 1000, the initial temperature to 1000, and the cooling rate to 0.95. In each iteration, by exchanging the computing devices of the inference request tasks and adjusting the start time, a candidate allocation result is obtained while ensuring that the time consumption of all inference request tasks meets the task requirements. The optimal allocation result for the current round is determined by comparing the candidate allocation result of the current round, the allocation result of the previous round, and the current optimal allocation result. This process is repeated until the total number of iterations reaches 1000, and the optimal allocation result determined in the last round is taken as the task allocation result. Task optimization node 2 sends the final task allocation result as a task allocation request to management node 1. Management node 1 schedules the processing of all inference request tasks according to the task allocation request and feeds back the task processing results of each inference request task to the corresponding user terminal 3. This achieves the allocation of execution time and execution position for the P and D phases for each inference request task, minimizing the execution cost of all inference request tasks.

[0034] It should be noted that the above application scenarios are only shown to facilitate understanding of the ideas and principles of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario. After introducing the technical solution of the present invention, various non-limiting embodiments of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Please see first. Figure 2 According to the inference request task processing method provided by the present invention, it can be implemented as a computer program product, which can be installed and run on one of the high-performance computing nodes or management nodes in a heterogeneous computing system or on a dedicated server outside the heterogeneous computing system, to perform corresponding processing on the received user inference request tasks. Heterogeneous computing systems include... Figure 3As shown, it includes multiple heterogeneous computing devices, such as computer accelerator cards from different manufacturers and / or accelerator cards with different performance levels. These computing devices can communicate within the same physical host, such as a server, or between different physical hosts. In this heterogeneous computing system, such as... Figure 4 As shown, before executing inference on user-issued inference request tasks, multiple inference request tasks arriving within a certain period are aggregated and encapsulated into a request set, which is then input into a heterogeneous computing system for PD-separated inference. For each inference request task, a pre-filling stage and a decoding stage need to be executed sequentially to obtain the final task processing result. The pre-filling stage processes the input of the inference request task and generates a KV Cache (key-value cache), which is relatively computationally intensive; the decoding stage gradually generates the final result based on the existing KV Cache, which is relatively memory bandwidth sensitive. Each computing device can handle different stages of multiple inference request tasks simultaneously according to its own performance capacity, and each computing device has different computing capabilities for inference request tasks, resulting in different costs. In practical applications, due to different user payment levels, the request priority of each inference request task may differ. Simultaneously, each inference request task also has an upper limit on inference time, requiring completion of the entire inference request task within a specified time. This embodiment deploys and allocates stages for user-issued inference request tasks in a loaded heterogeneous computing system, while also optimizing costs. In some embodiments of this method, the method includes the following steps:

[0036] S201: Obtain the inference task information corresponding to each inference request task, as well as the computing power resource information of the heterogeneous computing system executing each inference request task.

[0037] Each inference request task is a set of inference request tasks that meet pre-set task processing conditions. The total number of inference request tasks is related to the set task processing conditions. Task processing conditions may include treating inference request tasks received within a time period, such as 2 seconds, as a group of inference request tasks, allocating corresponding task execution time and execution devices to them, or setting a threshold for the number of inference request tasks, such as 100. When the number of received inference request tasks reaches the threshold, these inference request tasks are treated as a group and this step is executed. Inference request tasks can be issued by the same user or by different users, and can be of the same type or different types, as long as the inference model executing these inference request tasks is deployed on a heterogeneous computing system. Inference request tasks can be inference tasks in the field of natural language processing (NLP). NLP inference tasks take "language data in text / speech form" as input, and use a natural language model deployed on a heterogeneous computing system to understand, generate, and analyze human language, which can be applied to scenarios such as human-computer interaction, content processing, and information extraction. Inference request tasks can also be inference tasks in the field of computer vision. These tasks take "image / video data" as input and use visual models deployed on heterogeneous computing systems to understand visual content, achieving functions such as image recognition, object detection, and scene analysis. They can be applied in fields such as industry, security, and healthcare. Inference request tasks can also be inference tasks in the field of speech and audio processing. These tasks take "speech / audio signals" as input. In addition to speech-to-text and text-to-speech tasks included in natural language processing, they also include "audio semantic understanding" tasks such as speech emotion recognition and audio event detection. They can be applied in scenarios such as security, healthcare, and entertainment. Inference request tasks can also be inference tasks in the fields of recommendation systems and personalization. These tasks take "user behavior data and item feature data" as input and use artificial intelligence models deployed on heterogeneous computing systems to predict user preferences, achieving personalized recommendations. They can be used on internet platforms (such as e-commerce, short videos, and news). Inference request tasks can also be inference tasks in the decision-making and control domain. These tasks take "environmental data and state data" as input and output "decision instructions" or "control signals" through artificial intelligence models deployed on heterogeneous computing systems to directly drive equipment actions. They can be used in scenarios such as industrial automation, autonomous driving, and robotics. Inference request tasks can also be cross-modal inference tasks. These tasks refer to inference tasks where the input / output includes multiple modalities of data such as "text, images, voice, and video." They achieve "semantic alignment" and "collaborative understanding" of different modalities of data through artificial intelligence models deployed on heterogeneous computing systems.

[0038] The inference task information comprises a set of parameters including the total number of inference request tasks, the resource requirements of each inference request task, and task processing information. Each inference request task can have its own dedicated inference task information. Resource requirement information represents the computational load of the inference request task during the execution phase, reflecting the required computing resources. This resource requirement information can be, for example, the computational load of the inference request task in the P and D phases. Task processing requirement information represents the user's execution requirements for the inference request task, such as the maximum task execution time limit and task execution priority. Of course, task processing requirement information can also include the type or type of computing device specified by the user for executing the task; this does not affect the implementation of the invention. Computing resource information reflects the amount of computing resources that each computing device in the heterogeneous computing system can provide and the cost required to execute the task. For example, computing resource information includes the computing power (such as the execution speed in the P / D phase and the number of parallel processing requests) and execution cost of each computing device in the heterogeneous computing system. The execution cost of each computing device can be determined by comprehensively analyzing hardware characteristics, operating status, usage scenarios, and management and maintenance throughout the entire lifecycle. For example, when determining execution costs, one can consider the inherent costs of the hardware itself (such as procurement and depreciation costs). A significant portion is the energy consumption cost during operation, especially in high-load scenarios. Energy consumption costs can be determined based on power consumption characteristics, actual operating load rate, electricity price, and power supply stability. Additionally, one can consider computing efficiency (i.e., cost per unit of computing power) and task adaptability (i.e., whether computing performance matches task requirements). Computing efficiency can be calculated as actual output computing power / rated computing power. If computing power is not fully utilized or does not match the task, it will lead to inflated execution costs. The lower the computing power utilization rate, the higher the execution cost per unit of task. This applies to different types of... Different inference tasks (such as large language model inference and computer vision inference) have different adaptability. Poor adaptability will lead to wasted computing power and indirectly increase the execution cost. For example, GPUs are good at parallel computing-intensive tasks (such as the P stage of LLM, which requires large-scale matrix operations), and the computing power utilization rate can reach more than 80% when performing such tasks. On the other hand, CPUs (central processing units) are good at serial logic processing. When performing the P stage of LLM, the computing power utilization rate is only 20% due to insufficient parallel capability. If lightweight inference tasks (such as image classification, which has a small amount of computation and a lot of serial logic) are assigned to CPUs instead of GPUs, the waste of GPU computing power can be avoided and the unit task cost can be reduced.

[0039] For example, collecting inference task information involves estimating the computational load of each inference request task in the P phase (e.g., the number of input tokens × the computational load of a single token in the P phase) and the D phase (e.g., the maximum length of the output token × the computational load of a single token in the D phase) through tests or historical data; determining the upper limit of request execution time based on user input or expert definitions, and prioritizing requests according to user level; and counting the number of requests in a single aggregated request set. Collecting computing resource information involves obtaining the unit time execution cost (i.e., total cost / execution time) of each computing device through full-load testing; and obtaining the execution speed of each computing device in the P and D phases and the number of inference requests it can handle simultaneously through actual deployment testing.

[0040] S202: Based on the inference task information and computing resource information, and based on the condition of achieving the lowest processing cost for each inference request task while meeting the processing requirements of each task, determine the target computing devices and corresponding task start times for executing the pre-filling and decoding stages of each inference request task, so as to obtain the task allocation results.

[0041] The target computing device is the optimal computing device matched for the P and D phases of each inference request task, which must meet the task processing requirements and have the lowest implementation cost. The task start time is the time when the P and D phases of the inference request task begin execution on the corresponding computing device, ensuring that the D phase starts after the P phase is completed, and that the total time does not exceed the upper limit. This step uses the constraints of "meeting the processing requirements of all inference request tasks, such as being lower than the time limit and priority requirements, and ensuring that each computing device is not overloaded" and the objective of "lowest total processing cost of all inference requests". It can use any optimization algorithm in related technologies to calculate the target computing device corresponding to the P and D phases of each inference request task and the task start time of each phase, forming the task allocation result.

[0042] For example, in scenarios with large-scale requests and multiple constraints, a genetic algorithm can be used, drawing on the mechanisms of "natural selection, crossover, and mutation" in biological evolution, to calculate the target computing devices corresponding to the P and D phases of each inference request task, as well as the task start time of each phase. Each "task allocation scheme corresponding to an inference request task" is encoded as a "chromosome" (e.g., each gene corresponds to the device selection and start time of the P / D phases for a single request). The optimal task allocation result is then selected through multiple generations of iteration: Initializing the population: N feasible candidate allocation results are randomly generated (which must satisfy "D phase starts after P phase, device is not overloaded, and total execution time is less than or equal to the upper limit"), forming the initial population; Fitness calculation: The goal is to "minimize the total execution cost of all requests". The algorithm calculates the fitness of each candidate allocation result (e.g., fitness = 1 / total execution cost; a higher value indicates a better solution); selection operation: filters candidate allocation results with high fitness to enter the next generation (e.g., selects the top 50% of solutions in the population); crossover operation: randomly pairs selected candidate allocation results and exchanges some genes (e.g., exchanges the device allocation information of 5 requests in two solutions) to generate new candidate allocation results; mutation operation: randomly modifies the genes of some candidate allocation results with a low probability (e.g., 1%-5%) (e.g., adjusts the start time or device selection of a request) to avoid population homogenization; iteration termination: when the number of iterations reaches a preset value (e.g., 100 generations) or the fitness of the optimal solution has not improved for 10 consecutive generations, the current optimal allocation result is output.

[0043] For example, in scenarios with high real-time requirements, particle swarm optimization can be used to calculate the target computing devices corresponding to the P and D phases of each inference request task, as well as the task start time of each phase. Each "allocation result" is regarded as a "particle," which moves in the "allocation result space." By learning the optimal experience of itself and the swarm, the particle adjusts its movement direction and gradually approaches the global optimum: Particle initialization: M particles are randomly generated, each particle corresponding to an allocation result. An initial position (i.e., initial allocation result) and velocity (i.e., adjustment range) are set for each particle; Individual optimum and global optimum update: The fitness (total execution cost) of each particle is calculated, and the historical optimal position (self-optimal position) of each particle is recorded. The algorithm calculates the optimal solution achieved by the individual and the global optimal position of the group (the optimal solution among all particles); Particle position update: Based on the individual optimal, the global optimal, and the current velocity, adjust the position and velocity of the particles (e.g., velocity = inertia weight × current velocity + cognitive factor × (individual optimal - current position) + social factor × (global optimal - current position)) to generate a new allocation result; Constraint verification: The new allocation result must satisfy "execution order according to priority order, equipment not overloaded, total time ≤ time limit", otherwise readjust the velocity and position; Iteration termination: When the fitness of the global optimal solution fluctuates less than or equal to 0.1% for 5 consecutive generations, or the number of iterations reaches a preset value (e.g., 50 generations), output the global optimal allocation result.

[0044] S203: Send the task allocation result as a task allocation request to the heterogeneous computing system so that each target computing device can execute the corresponding task according to the task start time of the task allocation request.

[0045] Once the task allocation result is determined in the previous step, it can be encapsulated into a task allocation request and sent to the heterogeneous computing system. The heterogeneous computing system will allocate the P-stage and D-stage of each inference request task to the corresponding target computing device. Each target computing device will execute the P-stage and D-stage tasks of the corresponding inference request in sequence according to the start time of the task allocation result, complete the inference process of all inference request tasks, and obtain the final task processing result.

[0046] In the technical solution provided in this embodiment, the inference task information clarifies the resource requirements (such as the computational load of the P / D stage) and processing requirements (such as priority and time limit) of the inference request task. The computing power resource information quantifies the computing power (such as execution speed) and execution cost of heterogeneous devices, avoiding the randomness and inaccuracy of "matching devices based solely on experience" in traditional allocation. By clarifying the task start time, it ensures that the D stage of the same inference request task starts after the P stage and the total time does not exceed the upper limit of the task requirements. This supplements the accuracy of resource allocation from the time dimension, avoids the task queuing timeout problem caused by device occupation conflicts, and ultimately ensures that the P / D stage of each inference request task received by the heterogeneous computing system can be matched with a computing device that is performance-adapted, cost-controllable, and time-feasible. This avoids the problem of high-priority requests timeout due to low-priority resources being occupied and the waste of resources caused by the mismatch between device performance and the computational load of the request, ensuring that all inference request tasks can be executed efficiently while meeting their own processing requirements. Furthermore, the goal is to minimize the execution cost of each inference request task, while adhering to the constraints of task processing requirements. This avoids blindly selecting the most expensive high-end equipment to meet the time requirements of high-priority requests, which would lead to inflated costs, and also avoids selecting underperforming equipment to reduce costs, which would lead to request timeouts. This achieves the global optimization of the total processing cost of all inference request tasks. The cost optimization effect is particularly significant for scenarios with multiple types of requests (such as high / low priority, large / small computational load), reducing the operating cost of heterogeneous computing systems. By scheduling task startup times in an orderly manner, data transmission congestion between devices is reduced, ensuring the continuous and reliable operation of heterogeneous computing systems. This is especially suitable for scenarios with large-scale concurrent inference requests.

[0047] In the above embodiments, no limitations are made on how to determine the target computing devices and corresponding task start times for the pre-filling and decoding phases of each inference request task. This embodiment also provides an exemplary process for determining the task allocation results, which may include the following:

[0048] Determine the execution time and cost required for each inference request task to be processed using each computing device; determine the computing devices and task start times for each inference request task to execute its pre-filling and decoding phases according to the task processing requirements; and determine the initial allocation result based on the selection criteria of the lowest execution cost, execution time not exceeding the longest execution time of the corresponding task, and no overload of each computing device; obtain multiple candidate allocation results by adjusting the computing devices and / or task start times of each inference request task in the initial allocation result; select the allocation result that achieves the lowest processing cost for each inference request task while satisfying the task processing requirements from the initial allocation result and each candidate allocation result.

[0049] In this embodiment, execution time is the duration required for the P / D phase of a single inference request task to complete on a certain computing device, which can be determined by "the computational load of this phase / the execution speed of the corresponding phase on the computing device". Execution cost is the cost incurred by executing the P / D phase of a single inference request task on a certain computing device, which can be calculated by combining execution time, computing device unit time cost, and request priority discount. The initial allocation result is a preliminary resource allocation scheme that meets the task processing requirements and has a low cost. The candidate allocation result is a number of alternative resource allocation schemes formed after adjusting the computing devices or start times in the initial allocation result. The maximum task execution time is the time limit defined by the user when issuing the inference request task, that is, the total time from the start of the P phase to the completion of the D phase must not exceed this value. Computing device overload refers to the number of parallel inference request tasks allocated to the computing device within a certain time period exceeding its maximum processing capacity.

[0050] In this embodiment, the execution time of each inference request task in the P and D phases on each computing device can be calculated first: P phase time = request P phase computation amount / device P phase execution speed, D phase time = request D phase computation amount / device D phase execution speed. The execution cost of each inference request task in the P and D phases on each computing device is then calculated: P / D phase cost = P / D phase execution time × unit time cost × quantified value of execution order cost determined by priority. Requests are sorted from high to low priority, with high-priority requests processed first; all P and D phase computing device combinations (total) are enumerated for each request. (K is the total number of computing devices). The system filters for combinations where "Phase D starts after Phase P, total execution time is less than or equal to the longest task execution time, and devices are not overloaded." The combination with the lowest execution cost is selected as the initial allocation result for the request, and the task occupancy table of each computing device is updated synchronously. The initial allocation result is adjusted to generate candidate allocation results: new allocation results are generated by re-enumerating the computing devices and start times used in the P and D phases of a single inference request task, swapping the computing devices in the P phase of two inference request tasks, and adjusting the start time of a single inference request task. If the new allocation result satisfies the constraints of "total execution time less than or equal to the upper limit and devices not overloaded," it is used as a candidate allocation result. The total execution cost of the initial allocation result and all candidate allocation results is compared, and the solution that satisfies the constraints and has the lowest total cost is selected as the final task allocation result.

[0051] As shown above, this embodiment ensures the basic feasibility and low cost of resource allocation through initial allocation, and breaks through local optima through global optimization, further reducing the total execution cost. The entire process verifies task processing requirements and device capacity constraints to ensure that the allocation scheme can successfully execute the corresponding tasks, avoiding timeouts or device overload issues.

[0052] To avoid high-priority requests potentially timeing out due to low-priority requests occupying resources, thus failing to meet the service requirements of users at different levels, and to address the issue of invalid allocation schemes being easily generated due to the lack of explicit constraints when enumerating device combinations, this invention also provides an exemplary implementation based on the above embodiments:

[0053] Based on the task processing requirements, the priority of each inference request task is determined, and the execution order of each inference request task is determined from high to low priority. According to the execution order, for the first inference request task, the corresponding computing device is first assigned to the pre-filling stage, and then to the corresponding decoding stage. The remaining inference request tasks are then assigned devices according to the execution order: a computing device is randomly assigned to the pre-filling and decoding stages of each inference request task. If the start time of the decoding stage of the same inference request task is after the end time of the corresponding pre-filling stage, and the total execution time and start-up time of the pre-filling and decoding stages of the same inference request task do not exceed the longest task execution time, then the current device assignment result is a candidate device assignment result. From the candidate device assignment results of the same inference request task, the candidate device assignment result with the lowest execution cost is selected as the corresponding device assignment result.

[0054] The request priority is a quantified value representing the priority of the processing order of inference request tasks. A higher value indicates a higher priority, requiring resources to be allocated first. The request execution order is the order in which inference request tasks are allocated resources based on the priority ranking, with higher-priority requests being allocated computing devices for the P and D phases first. The candidate device allocation result is a device combination scheme that satisfies the condition that "the D phase starts after the P phase, and the total execution time is less than or equal to the longest task execution time" for a single inference request task.

[0055] In this embodiment, the priority quantification values ​​of all inference request tasks are extracted and sorted from high to low to determine the execution order of the requests; if the priorities are the same, they can be sorted by the request reception time. High-priority requests are allocated first: For the highest-priority request in the ranking, all available computing devices are enumerated for its P phase, and computing devices with idle capacity are selected. The execution time and cost of each computing device executing the P phase are calculated. Then, based on the computing devices selected for the P phase, all available computing devices are enumerated for the D phase, ensuring that the D phase starts after the P phase is completed, and the total execution time is less than or equal to the longest task execution time. Remaining requests are processed according to the execution order: For each unallocated inference request task, computing devices are randomly allocated for its P and D phases, and the constraints (D phase starts after P phase, total execution time is less than or equal to the longest task execution time) are verified. If these constraints are met, the task is marked as a candidate device allocation result. The scheme with the lowest execution cost is selected from all candidate device allocation results to determine the device allocation result for that inference request task.

[0056] As can be seen from the above, this embodiment can ensure that high-priority requests can obtain resources first, filter candidate device allocation results through constraints, avoid invalid enumeration, improve allocation efficiency, and at the same time ensure that the scheme meets the time consumption requirements.

[0057] Considering that the task occupancy status of devices is not updated in real time when allocating resources, it is easy for the same device to be assigned multiple tasks at the same time, causing device overload, affecting the execution efficiency of inference tasks, or even causing task failure. Based on the above embodiments, the present invention also includes the following:

[0058] Obtain the task execution table for each computing device in the heterogeneous computing system; the task execution table should at least record the start execution time and task execution time corresponding to different tasks; update each task execution table according to the request execution order, the device allocation result of each inference request task, and the task execution time of the pre-filling phase and decoding phase of each inference request task; when different target tasks are processed in the same time period, the device allocation result is re-determined for each target task.

[0059] The task execution table records task occupancy information at the computing device level, including the start and execution times of each inference request task on that device, providing a clear picture of the device's load status at different time periods. A target task refers to multiple inference request tasks allocated to the same computing device within the same time period; this situation can lead to device overload.

[0060] In this embodiment, an independent task execution table is created for each computing device in the heterogeneous computing system. The task execution table is initialized, meaning there are no task records in the initial state. The task execution table is then updated: according to the request execution order, combined with the device allocation result and P / D stage execution time for each request, the start execution time and execution duration of the request are entered into the task execution table of the corresponding computing device. After each entry, the task execution table is verified: if the number of tasks recorded within the same time period exceeds the device's maximum parallel processing capacity, it is determined that the device is overloaded, and the tasks involved are the target tasks. The target tasks are then reallocated: the device combinations in the P and D stages are re-enumerated for the target tasks, and candidate solutions that "do not cause device overload and meet the time requirements" are selected. The solution with the lowest cost is chosen to update the device allocation result, and the task execution table is corrected simultaneously.

[0061] As can be seen from the above, this embodiment tracks the load status of each computing device in real time to avoid device overload, ensure the stable operation of the heterogeneous computing system, and promptly corrects the device allocation results under overload conditions by updating the table, ensuring that all inference request tasks can be executed on available devices and reducing the task failure rate.

[0062] The above embodiments do not limit how to select the optimal allocation result. This embodiment also provides an exemplary implementation method, which may include the following:

[0063] The total number of iterations is determined based on the total number of inference request tasks. The total cost of the current task is determined based on the execution costs of each inference request task in the pre-filling and decoding phases, respectively, according to the initial allocation results. New candidate allocation results are obtained by randomly adjusting the computing devices and / or task startup times of each inference request task. If the new candidate allocation results satisfy the conditions that the total execution time of the pre-filling and decoding phases and the startup execution time of the same inference request task do not exceed the longest task execution time, and the capacity of each computing device is not overloaded, then the new total task cost corresponding to the new candidate allocation result is calculated. If the new total task cost is less than the current total task cost, the new candidate allocation result replaces the initial allocation result. If the new total task cost is less than the optimal total task cost, the optimal total task cost is updated to the new total task cost to complete one iteration. The new candidate allocation result is used as the initial allocation result to execute the next iteration, and the iteration count is updated. If the current iteration count reaches the total number of iterations, the allocation result corresponding to the current optimal total task cost is used as the task allocation result.

[0064] The total number of iterations refers to the total number of times the allocation scheme is adjusted during the global optimization process, determined based on the total number of inference request tasks. For example, the total number of iterations = 10 × the total number of inference request tasks. The current total task cost refers to the sum of the execution costs of all inference request P and D phases under the current allocation scheme. The new candidate allocation result refers to the new resource allocation scheme formed after adjusting the initial allocation result. The optimal total task cost refers to the minimum total cost among all feasible allocation schemes that appear during the iteration process.

[0065] In this implementation, iteration parameter initialization can be performed first: determine the iteration number L based on the total number of inference request tasks N, such as L = 10 × N. Calculate the current total task cost of the initial allocation result and set it as the initial optimal total task cost. One iteration process includes: adjusting the initial allocation result to generate new candidate allocation results; randomly adjusting the computing devices or startup times of some requests; verifying whether the new scheme meets the requirements of "total execution time less than or equal to the longest task execution time and no device overload"; if so, calculate the new total task cost. If the new total task cost is less than the current total task cost, replace the initial allocation result with the new candidate allocation result; if the new total task cost is less than the optimal total task cost, update the optimal total task cost to the new total task cost, and complete this iteration. Use the updated new candidate allocation result as the initial allocation result for the next iteration, and increment the iteration number by 1; if the iteration number reaches the total number of iterations, output the allocation result corresponding to the optimal total task cost as the final task allocation result.

[0066] As shown above, this embodiment optimizes the allocation scheme through multiple iterations, continuously approaching the global optimal solution. It determines that, while satisfying priority and maximum time requirements, the execution cost of all inference request tasks is kept as low as possible. The number of iterations is determined based on the total number of inference request tasks, balancing optimization effectiveness and computational efficiency while avoiding ineffective iterations.

[0067] For example, this embodiment also provides various random adjustment methods to generate as many candidate allocation schemes as possible, which may include the following:

[0068] Randomly select a target inference request task and redetermine the new device allocation result for the target inference request task; randomly select a first target inference request task and a second target inference request task and exchange the computing devices of the first target inference request task and the second target inference request task; randomly select a target inference request task and a perturbation value, calculate the task start time of the target inference request task and the sum of the perturbation value, and update the task start time of the target inference request task using the maximum value between the sum and the preset value.

[0069] The target inference request task refers to a single, randomly selected inference request task that requires adjustment of the resource allocation scheme. The new device allocation result is a new allocation scheme determined after re-enumerating the computing device combinations for the target inference request task. The first target inference request task and the second target inference request task are two different inference request tasks randomly selected for exchanging computing devices in the P-phase. The perturbation value is a random value used to adjust the task start time, following a uniform distribution. U represents a uniform distribution, and y is a user-defined time constant, such as 3 seconds.

[0070] For example, re-enumerate the devices and times for a single request: Randomly select a target inference request task, and according to the method for determining the device allocation results for the remaining inference request tasks described in the above embodiments, re-enumerate all device combinations in the P and D phases of the target inference request task, filter candidate schemes that meet the constraints, and select the scheme with the lowest cost as the new device allocation result. Swap the P-phase devices of two requests: Randomly select two different inference request tasks (such as the first and second target inference request tasks), and swap the computing devices corresponding to their P-phases; verify whether the swap satisfies "total time less than or equal to the upper limit, and devices are not overloaded". If satisfied, a new candidate allocation result is formed. Adjust the start time of a single request: Randomly select a target inference request task, generate a timer that conforms to the specified parameters, and then adjust the start time of the request. perturbation value (e.g., y = 3 seconds); Calculate the new startup time = max(0, original startup time + ... To ensure the startup time is non-negative, the adjustment is checked to see if the constraints are met. If they are met, a new candidate allocation result is formed. Each adjustment uses only one of the three methods mentioned above, selected randomly to ensure the diversity of allocation scheme adjustments.

[0071] As can be seen from the above, this embodiment expands the exploration space of allocation schemes through diverse adjustment methods, effectively avoiding the optimization process from getting trapped in local optima. Each adjustment method takes into account the constraints, ensuring the feasibility of the new candidate allocation results.

[0072] Furthermore, if the total cost of the new task is greater than or equal to the total cost of the current task, the cost difference between the total cost of the new task and the total cost of the current task is calculated, and the negative value of the ratio of the cost difference to the temperature value of the current iteration is used as the independent variable of the exponential function. The exponential function is used as the probability value of accepting the new candidate allocation result to replace the initial allocation result. The temperature value of the next iteration is determined based on the cooling rate value and temperature value of the current iteration.

[0073] Among them, cost difference This is the difference between the total cost of the new candidate allocation and the total cost of the current allocation. The temperature value is a parameter in the simulated annealing algorithm; its initial value can be set to, for example, 1000, and it is gradually decreased with iterations to control the probability of accepting inferior solutions. The cooling rate value is a coefficient used to reduce the temperature value (e.g., ...). After each iteration, the temperature value = current temperature value × cooling rate value. Acceptance probability value. To calculate the probability of accepting the new candidate assignment with the higher total cost, the relationship is as follows: T represents the current temperature value.

[0074] In this embodiment, if the total cost of the new task in the new candidate allocation result is greater than or equal to the total cost of the current task, the cost difference ΔE is calculated as: total cost of the new task - total cost of the current task. Based on Calculate the acceptance probability and generate a random number between 0 and 1. If the random number is..., then accept the new candidate allocation result and use it to replace the initial allocation result; If the new solution fails, the original allocation result is retained. After each iteration, the next temperature value is calculated as: "Next temperature value = Current temperature value × Cooling rate value (e.g., ...)". Update the temperature value to gradually reduce the probability of accepting the different solution.

[0075] As shown above, the high temperature value in the early stage of iteration in this embodiment allows for the acceptance of some suboptimal solutions, expanding the scope of solution exploration; the low temperature value in the later stage of iteration focuses on optimizing better solutions, balancing exploration and convergence. Simulated annealing effectively avoids local optima and increases the probability of determining the globally optimal allocation scheme.

[0076] Based on the above embodiments, the present invention also provides a method for determining the execution time required for each inference request task to be processed by each computing device, which may include the following:

[0077] Based on the inference task information, determine the computational load required for the pre-filling and decoding phases of each inference request task; based on the computing power resource information, determine the execution speed of each computing device; for each inference request task, based on the computational load required for the pre-filling phase of the target inference request task and the execution speed of each computing device, determine the execution time of each computing device for the pre-filling phase of the target inference request task; based on the computational load required for the decoding phase of the target inference request task and the execution speed of each computing device, determine the execution time of each computing device for the decoding phase of the target inference request task.

[0078] The computational load of the pre-filling phase refers to the number of floating-point operations (FLOPs) required for the inference request task to complete in the P phase, reflecting the computational load of this phase. The computational load of the decoding phase refers to the number of floating-point operations (FLOPs) required for the inference request task to complete in the D phase. Execution speed refers to the number of floating-point operations (FLOPS) that a computing device can complete per unit of time, divided into P-phase execution speed and D-phase execution speed.

[0079] In this embodiment, the P-stage computational cost (Wp) and D-stage computational cost (Wd) of each inference request task are extracted from the inference task information. The P-stage computational cost of the i-th inference request task can be expressed as: The computational complexity of stage D can be expressed as: Extract the P-stage execution speed of each computing device from the computing resource information. Execution speed of Phase D The execution speed of stage P of the k-th computing device can be expressed as: The execution speed of phase D can be expressed as For inference request task i, the execution time of stage P on computing device k can be: The execution time in phase D can be By iterating through all inference request tasks and computing devices, and calculating the execution time for all "request-device-stage" combinations, an execution schedule can be generated.

[0080] As shown above, this embodiment accurately quantifies the execution time of each request at different stages on different devices, providing an accurate basis for verifying the time consumption of resource allocation. The execution time table clearly presents the processing efficiency of each computing device, facilitating the rapid selection of device combinations that meet the time consumption requirements.

[0081] Based on the above embodiments, the present invention also provides a method for determining the execution cost required for each inference request task to be processed using each computing device, which may include the following:

[0082] The execution cost of each computing device is determined based on computing resource information; the request priority of each inference request task is determined based on inference task information; and the execution cost of each computing device for each inference request task is determined based on the execution time of each computing device for each inference request task, the request priority of each inference request task, and the execution cost of each computing device.

[0083] Request priority is a quantified value representing the processing priority of inference request tasks (e.g., level 1-5, with level 5 being the highest). Execution cost refers to the total cost for the computing device to execute a certain stage or the entire stage of the inference request task, which can be determined by combining execution time, device unit time cost, and request priority.

[0084] For example, the execution cost required for each inference request task to be processed using different computing devices can be determined in stages: The execution cost of the pre-filling stage of each inference request task is determined based on the execution time of the pre-filling stage, the request priority of each inference request task, and the execution cost of each computing device; the execution cost of the decoding stage of each inference request task is determined based on the execution time, the request priority of each inference request task, and the execution cost of each computing device. Here, the pre-filling stage execution cost refers to the cost incurred by the execution of inference request task P on a certain computing device. The decoding stage execution cost refers to the cost incurred by the execution of inference request task D on a certain computing device.

[0085] For example, the implementation process of incorporating priority quantification into cost calculation may include: determining the execution order cost quantification value of each inference request task based on the total number of priority levels and the request priority quantification value of each inference request task; and determining the execution cost of each computing device for each inference request task based on the execution time of each computing device for each inference request task, the execution order cost quantification value of each inference request task, and the unit time execution cost of each computing device. Here, the total number of priority levels refers to the total number of priority levels for inference request tasks (e.g., if the total number of priority levels is 5, it corresponds to priority levels 1-5). The request priority quantification value is a numerical value used to characterize the level of priority (e.g., level 5 corresponds to a quantification value of 5, and level 1 corresponds to a quantification value of 1). The execution order cost quantification value is a discount coefficient calculated based on priority and used to adjust the execution cost.

[0086] As a convenient implementation method, the execution cost calculation formulas for the P-stage and D-stage can be pre-stored. The execution costs for each inference request task on each computing device are then determined by calling these formulas. The P-stage execution cost calculation formula can be expressed as follows: The calculation relationship for the execution cost in Phase D can be expressed as follows: ,in, Let $\frac{i}{p}$ represent the execution cost of inference request task $i$ in stage P on computing device $k$. Let $\frac{i}{d}$ represent the execution cost of inference request task $i$ in stage D on computing device $k$. This represents the execution cost of computing device k. Indicates the total number of priority levels. This represents the request priority quantization value for inference request task i.

[0087] For example, extracting the unit time execution cost of each computing device from computing resource information. Extract the priority of each request from the inference task information. Determine the total number of priority levels x. For each inference request task, iterate through all computing devices, call the P-stage execution cost calculation formula to calculate the P-stage execution cost, and the D-stage execution cost calculation formula to calculate the D-stage execution cost. The total cost of a single inference request task under a certain device combination (e.g., k1 executes the P-stage, k2 executes the D-stage) is equal to the sum of the P-stage cost on k1 and the D-stage cost on k2.

[0088] As shown above, this embodiment quantifies priorities to make the calculation of execution costs for high-priority requests more aligned with business needs, facilitating the allocation of high-quality resources to high-priority requests. Quantifying priorities into cost adjustment coefficients makes the impact of priorities on resource allocation quantifiable and controllable. Higher-priority requests have smaller quantified execution order costs, resulting in lower execution costs, making it easier to allocate resources to them first while ensuring cost optimization goals are achieved. Calculating execution costs across multiple dimensions accurately reflects cost differences between different devices and requests, providing data support for low-cost allocation schemes. Furthermore, breaking down execution costs into stages precisely identifies stages and devices with high costs, facilitating targeted optimization and providing a basis for matching low-cost devices to the P and D stages, further reducing the total execution cost of individual requests.

[0089] Based on the above embodiments, the present invention further clarifies the data content included in the reasoning task information, and correspondingly, the process of obtaining the reasoning task information may include:

[0090] Obtain the computational requirements of each inference request task in the pre-filling and decoding stages as resource requirement information; obtain the longest execution time of each inference request task and determine the corresponding request priority based on the user level of each inference request task; use the longest execution time of each inference request task and the priority of each request as task processing requirement information.

[0091] The resource requirement information refers to the quantitative indicators of the computing resources required to execute the inference request task, including at least the computational load of the P and D phases. The maximum task execution time is the maximum allowable time from the start of the inference request task in the P phase to the completion of the D phase; i.e., the time limit. User levels, such as those based on user payment or service agreements, are used to determine the priority of the inference request task. The maximum task execution time can be determined by collecting user input of the time requirement through a user interface, or by a default value defined by technical experts based on the business scenario. Request priority can be determined by establishing a mapping relationship between user levels and priorities (e.g., the highest priority user corresponds to priority 5, and ordinary users correspond to priority 1), with priority determined based on the user level to which the request belongs.

[0092] For example, this embodiment can also accurately calculate the computational load of the pre-filling and decoding stages through test tasks, which may include the following: sending a reasoning test task belonging to the same task type as the reasoning request task to the target computing device of the heterogeneous computing system; obtaining the first test computational load of the target computing device executing the pre-filling stage of the reasoning test task through the reasoning model of the reasoning request task, and the second test computational load corresponding to the decoding stage of the reasoning test task; determining the computational load required for a single lexical length in the pre-filling and decoding stages based on the test lexical length of the reasoning test task, the first test computational load, and the second test computational load; determining the computational load required for each reasoning request task in the pre-filling and decoding stages based on the lexical length of each reasoning request task and the computational load required for a single lexical length in the pre-filling and decoding stages.

[0093] The inference test task is a test inference task of the same type as the inference request task issued by the user, used to measure the computational cost per unit word length. The target computing device is a computing device with performance parameters completely identical or nearly identical to the source computing device executing the inference request task, used to execute the inference test task. The first test computational cost is the total computational cost (in FLOPs) of the pre-filling phase of the inference test task executed by the target computing device. The second test computational cost is the total computational cost (in FLOPs) of the decoding phase of the inference test task executed by the target computing device. The test word length refers to the number of input words in the inference test task (e.g., 1000). The computational cost per unit word length refers to the computational cost required per unit word in the pre-filling / decoding phase, equal to the test computational cost divided by the test word length.

[0094] For example, to test the computational cost of obtaining a single token in phase P: Input a test request with a token length of A on a computing device, execute phase P, and use tools to collect the computational cost. Calculate the computational cost of a single token as total computational cost / A; the actual computational cost of the request in phase P = number of input tokens × computational cost of a single token. To test the computational cost of obtaining a single token in phase D: Input a test request, execute phase D, collect the computational cost, and record the number of output tokens B. The computational cost of a single token = total computational cost / B; the actual computational cost of the request in phase D = maximum length of output tokens × computational cost of a single token.

[0095] In this embodiment, for the inference model deployed in a heterogeneous computing system to execute the inference request task of S201, a computing device can be selected to receive the inference test task. The inference test task is an inference test request with a token (lexical) length of A. This request is input to the computing device. Within the heterogeneous computing system, a first target computing device is selected for the prefilling stage of the inference test task, and a second target computing device is selected for the decoding stage. The prefill process is executed on the first target computing device, and during execution, tools such as PyTorch Profiler are used to collect the first test computational cost (FLOPs). Then, the computational cost required for each token in the prefill stage is obtained by dividing the first test computational cost by A (this process does not need to be repeated). After obtaining the above information, the computational cost of the inference request task in stage P can be obtained by multiplying the number of tokens input to the inference request task by the computational cost required for each token in the prefill stage. Furthermore, various other methods, such as mathematical formulas, can be used to estimate the computational cost of this process, which will not be elaborated here. Similarly, the decoding phase is performed on the second target computing device. During execution, tools such as PyTorch Profiler are used to collect the second test computational cost (FLOPs) of this process, and the number of output tokens B is recorded. Then, the computational cost required for each token in the decoding phase is obtained by dividing the second test computational cost by B (this process does not need to be repeated). Subsequently, by multiplying the maximum length of the tokens output by the inference request task by the computational cost of each token, the computational cost of each inference request task in phase D can be estimated more conservatively. If the estimated length is too short, the allocation of this algorithm may cause some requests to fail to complete within the specified time.

[0096] For example, select the target computing device in a heterogeneous computing system and deploy the same LLM model as the actual inference request; generate an inference test task of the same type as the actual request, and determine the input token length (test lexicon length) of the test task. During the P phase of the test task, use the PyTorch profiler tool to collect the first test computation; during the D phase of the test task, collect the second test computation and record the number of output tokens. The computation cost per lexicon in the P phase = first test computation cost / test lexicon length; the computation cost per lexicon in the D phase = second test computation cost / number of output tokens. For each actual inference request, the P phase computation cost = input lexicon length × single lexicon P phase computation cost; the D phase computation cost = maximum output lexicon length × single lexicon D phase computation cost.

[0097] As shown above, this embodiment ensures the accuracy of computational load in the P and D phases through standardized testing methods, providing a reliable basis for resource allocation. Prioritization based on user levels and determination of time limits based on user input / expert definitions ensure that task processing requirements align with actual business scenarios. Furthermore, standardized testing calibrates the computational load of individual tokens, eliminating estimation errors caused by differences in request token lengths. Calculating the actual request computational load based on test data yields high accuracy, providing a reliable foundation for subsequent execution time and cost calculations.

[0098] Based on the above embodiments, the present invention further clarifies the data content included in the computing power resource information, and correspondingly, the process of obtaining inference task information may include:

[0099] Obtain the unit time execution cost of each computing device in the heterogeneous computing system; obtain the number of parallel request tasks processed by each computing device in the heterogeneous computing system and the execution speed corresponding to the pre-filling stage and the decoding stage respectively.

[0100] Among these, execution cost per unit time refers to the cost incurred by a computing device operating at full load per unit time (e.g., 1 second), including energy consumption and hardware depreciation. The number of parallel request tasks processed refers to the number of inference requests that a computing device can process simultaneously, i.e., device capacity. Execution speed refers to the number of floating-point operations (FLOPS) completed by the computing device per unit time, divided into P-stage and D-stage.

[0101] In this embodiment, computing resource information may include the unit-time execution cost of each computing device, obtained by dividing the unit-time execution cost of each computing device by cost / time. The execution speed of each computing device in the P phase can be obtained by actually deploying and testing on the computing device. The execution speed of each computing device in the D phase can also be obtained by actually deploying and testing on the computing device. The number of inference requests that each computing device can handle simultaneously can also be obtained by actually deploying and testing on the computing device. All this information will not be collected frequently; the relevant information for each type of computing device only needs to be collected once and can be reused repeatedly. For example, a computing device in a heterogeneous computing system is selected and fully loaded to execute LLM inference tasks for a preset duration (e.g., 1 hour); the total execution cost (e.g., energy consumption costs, hardware depreciation and amortization) during this period is calculated; the unit-time execution cost = total execution cost / preset duration. The number of parallel inference requests executed on this computing device is gradually increased until the device response latency exceeds a threshold or a task fails; the maximum number of parallel requests before the threshold is recorded as the number of parallel request tasks processed by this device. Deploy test tasks, executing the P and D phases respectively. Use a profiler tool to collect the number of floating-point operations completed per unit time, which will be used as the execution speed of the P and D phases. Repeat the above steps for all computing devices in the heterogeneous computing system to complete the collection of computing resource information.

[0102] For example, the process of determining the execution cost per unit time can be as follows: Calculate the execution cost of each computing device in a heterogeneous computing system executing tasks at full load; determine the execution cost per unit time for each computing device based on the duration of its full-load execution task and the corresponding execution cost. Here, "full-load execution task" refers to the state where the computing device runs inference tasks at maximum load, at which point the device's resource utilization reaches its peak. Execution cost refers to all costs incurred by the computing device during task execution, including energy consumption costs, hardware depreciation costs, and maintenance costs. The duration of the full-load execution task refers to the duration for which the computing device maintains a full-load state while executing tasks.

[0103] In this embodiment, the unit-time execution cost of each computing device can be obtained by fully loading the computing device, such as filling its memory, for a period of time, recording the execution cost during that period (e.g., cost converted from energy consumption), and then dividing the cost by time. For example, deploying an LLM inference task to the target computing device and adjusting the request concurrency to achieve a CPU and GPU resource utilization rate of over 95% (full load state). Recording the start and end times of the fully loaded task execution, calculating the duration of the fully loaded task execution; and calculating the total execution cost during this period, including energy consumption cost (obtained by multiplying electricity consumption by the unit electricity price); hardware depreciation cost (amortized based on the total value and lifespan of the device during this period); and maintenance cost (allocated based on maintenance personnel man-hours, equipment maintenance costs, etc.). Unit-time execution cost = total execution cost / full-load task execution duration. The above statistics are performed on all computing devices in the heterogeneous computing system to form a table of unit-time execution costs for each device.

[0104] As can be seen from the above, this embodiment ensures the accuracy and consistency of computing resource information through standardized testing and statistical methods. The complete computing resource information covers cost, performance, and capacity dimensions, providing data support for multi-dimensional optimization of resource allocation. In addition, the cost statistics under full-load testing closely match the actual operating scenarios of the equipment, and the execution cost data per unit time has high accuracy, making the cost optimization of resource allocation more in line with the actual business cost structure.

[0105] It should be noted that there is no strict order of execution between the steps in this invention. As long as they conform to the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 2 This is just an illustrative example and does not mean that this is the only possible execution order.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0107] This invention also provides a corresponding apparatus for the inference request task processing method, further enhancing the method's practicality. The apparatus can be described from both a functional module perspective and a hardware perspective. The following describes the inference request task processing apparatus provided by this invention, which is used to implement the inference request task processing method provided by this invention. In this embodiment, the inference request task processing apparatus may include or be divided into one or more program modules. These program modules are stored in a storage medium and executed by one or more processors to complete the inference request task processing method disclosed in Embodiment 1. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing a specific function, and is more suitable than the program itself for describing the execution process of the inference request task processing apparatus in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The inference request task processing apparatus described below can be referred to in correspondence with the inference request task processing method described above.

[0108] From the perspective of functional modules, see Figure 5 , Figure 5 This is a structural diagram of the inference request task processing device provided in this embodiment under one specific implementation. The device may include:

[0109] The data acquisition module 501 is used to acquire inference task information corresponding to each inference request task, as well as computing power resource information of the heterogeneous computing system executing each inference request task; wherein, the inference task information includes resource requirement information and task processing requirement information; the computing power resource information includes the computing power and execution cost of each computing device in the heterogeneous computing system.

[0110] The task optimization module 502 is used to determine the target computing devices and corresponding task start times for executing the pre-filling stage and decoding stage of each inference request task based on the information of each inference task and computing resources, and based on the condition of achieving the lowest processing cost of each inference request task while meeting the processing requirements of each task, so as to obtain the task allocation result.

[0111] The task assignment and execution module 503 is used to generate a task assignment request based on the task assignment result and send the task assignment request to the heterogeneous computing system so that each target computing device can execute the corresponding task according to the task start time of the task assignment request.

[0112] For example, in some embodiments of this embodiment, the task optimization module 502 can also be used to: determine the execution time and execution cost required for each inference request task to be processed by each computing device; determine the computing device and task start time for each inference request task to execute its respective pre-filling stage and decoding stage according to the task processing requirement information, and determine the initial allocation result based on the selection conditions that the execution cost of each inference request task is the lowest, the execution time does not exceed the longest execution time of the corresponding task, and each computing device is not overloaded; obtain multiple candidate allocation results by adjusting the computing device and / or task start time of each inference request task in the initial allocation result; and select the allocation result that achieves the lowest processing cost for each inference request task under the condition of satisfying the task processing requirement information from the initial allocation result and each candidate allocation result.

[0113] As an exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: determine the priority of each inference request task according to the task processing requirement information, and determine the request execution order corresponding to each inference request task from high to low priority; according to the request execution order, for the first inference request task, first determine the corresponding computing device for the corresponding pre-filling stage, and then determine the corresponding computing device for the corresponding decoding stage; according to the request execution order, determine the device allocation result for the remaining inference request tasks in sequence: randomly allocate a computing device to the pre-filling stage and the decoding stage of each inference request task; if the task start execution time of the decoding stage of the same inference request task is after the task end time of the corresponding pre-filling stage, and the total duration of the task execution time and start execution time of the pre-filling stage and the decoding stage of the same inference request task does not exceed the longest task execution time, then the current device allocation result is the candidate device allocation result; from the candidate device allocation results of the same inference request task, select the candidate device allocation result with the lowest execution cost as the corresponding device allocation result.

[0114] As an exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: obtain the task execution table of each computing device in the heterogeneous computing system; the task execution table at least records the start execution time and task execution time corresponding to the execution of different tasks; update each task execution table according to the request execution order, the device allocation result of each inference request task, and the task execution time of the pre-filling stage and the decoding stage of each inference request task; when there are different target tasks being processed in the same time period, the device allocation result is re-determined for each target task.

[0115] As another exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: determine the total number of iterations based on the total number of inference request tasks; determine the current total task cost based on the execution costs of each inference request task in the pre-filling and decoding stages according to the initial allocation results; obtain new candidate allocation results by randomly adjusting the computing devices and / or task startup times of each inference request task; if the new candidate allocation results satisfy the conditions that the total execution time of the pre-filling and decoding stages of the same inference request task does not exceed the longest execution time of the task, and the capacity of each computing device is not overloaded, then calculate the new total task cost corresponding to the new candidate allocation results; if the new total task cost is less than the current total task cost, then use the new candidate allocation results to replace the initial allocation results; if the new total task cost is less than the optimal total task cost, then update the optimal total task cost to the new total task cost to complete one iteration; use the new candidate allocation results as the initial allocation results to execute the next iteration, and update the iteration count; if the current iteration count reaches the total number of iterations, then use the allocation result corresponding to the current optimal total task cost as the task allocation result.

[0116] As an exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: randomly select a target inference request task, and redetermine the new device allocation result for the target inference request task; randomly select a first target inference request task and a second target inference request task, and exchange the computing devices of the first target inference request task and the second target inference request task; randomly select a target inference request task and a perturbation value, calculate the task start time of the target inference request task and the sum of the perturbation value, and update the task start time of the target inference request task using the maximum value between the sum and a preset value.

[0117] As another exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: if the total cost of the new task is greater than or equal to the total cost of the current task, calculate the cost difference between the total cost of the new task and the total cost of the current task, and use the negative value of the ratio of the cost difference to the temperature value of the current iteration as the independent variable of the exponential function, and use the exponential function as the probability value of accepting the new candidate allocation result to replace the initial allocation result; determine the temperature value of the next iteration based on the cooling rate value and temperature value of the current iteration.

[0118] As another exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: determine the amount of computation required for the pre-filling stage and the decoding stage of each inference request task based on the inference task information; determine the execution speed of each computing device based on the computing power resource information; for each inference request task, determine the execution time of each computing device for executing the pre-filling stage of the target inference request task based on the amount of computation required for the pre-filling stage of the target inference request task and the execution speed of each computing device; and determine the execution time of each computing device for executing the decoding stage of the target inference request task based on the amount of computation required for the decoding stage of the target inference request task and the execution speed of each computing device.

[0119] As another exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: determine the execution cost of each computing device based on computing resource information; determine the request priority of each inference request task based on inference task information; and determine the execution cost of each computing device executing each inference request task based on the execution time of each computing device executing each inference request task, the request priority of each inference request task, and the execution cost of each computing device.

[0120] As an exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: determine the execution cost of the pre-filling stage of each inference request task by each computing device based on the execution time of the pre-filling stage of each inference request task executed by each computing device, the request priority of each inference request task, and the execution cost of each computing device; and determine the execution cost of the decoding stage of each inference request task executed by each computing device based on the execution time of the decoding stage of each inference request task executed by each computing device, the request priority of each inference request task, and the execution cost of each computing device.

[0121] As another exemplary implementation of the above embodiments, the task optimization module 502 can also be used to: determine the execution order cost quantization value of each inference request task based on the total number of priority levels and the request priority quantization value of each inference request task; and determine the execution cost of each computing device executing each inference request task based on the execution time of each computing device executing each inference request task, the execution order cost quantization value of each inference request task, and the unit time execution cost of each computing device.

[0122] For example, in some other embodiments of this embodiment, the data acquisition module 501 can also be used to: acquire the amount of computation required by each inference request task in the pre-filling stage and the decoding stage, respectively, as resource requirement information; acquire the longest execution time of each inference request task, and determine the corresponding request priority according to the user level corresponding to each inference request task; and use the longest execution time of each inference request task and each request priority as task processing requirement information.

[0123] As an exemplary implementation of the above embodiments, the data acquisition module 501 can also be used to: send a reasoning test task belonging to the same task type as the reasoning request task to the target computing device of the heterogeneous computing system; acquire the first test computation amount of the target computing device executing the pre-filling stage of the reasoning test task through the reasoning model of the reasoning request task, and the second test computation amount corresponding to the decoding stage of the reasoning test task; determine the computation amount required for a single word length in the pre-filling stage and the decoding stage based on the test word length of the reasoning test task, the first test computation amount, and the second test computation amount; and determine the computation amount required for each reasoning request task in the pre-filling stage and the decoding stage based on the word length of each reasoning request task and the computation amount required for a single word length in the pre-filling stage and the decoding stage.

[0124] For example, in some other embodiments of this embodiment, the data acquisition module 501 can also be used to: acquire the unit time execution cost of each computing device in the heterogeneous computing system; acquire the number of parallel request tasks processed by each computing device in the heterogeneous computing system and the execution speed corresponding to the pre-filling stage and the decoding stage respectively.

[0125] As an exemplary implementation of the above embodiments, the data acquisition module 501 can also be used to: calculate the execution cost of each computing device in the heterogeneous computing system performing tasks at full load; and determine the unit time execution cost of each computing device based on the full load execution time of each computing device and the corresponding execution cost.

[0126] The inference request task processing device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. The electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program, and the processor 602 is configured to run the computer program to perform the steps in any of the above-described embodiments of the reasoning request task processing method.

[0127] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described inference request task processing method embodiments when it runs.

[0128] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0129] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described inference request task processing method embodiments.

[0130] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described inference request task processing method embodiments.

[0131] Finally, this invention also provides a heterogeneous computing system, see [link to relevant documentation]. Figure 7 The system may include multiple types of computing devices 701 and a processor 602, with each type of computing device connected to the processor 602. The processor 602 executes the computer program stored in the memory, implementing the steps in any of the above-described inference request task processing embodiments. The computing devices 701 have different computing performances, and these computing devices 701 collectively constitute heterogeneous computing resources in the heterogeneous computing system. Each computing device 701 can be inserted into different computing nodes (such as servers) or into the same computing node, without affecting the implementation of the present invention. The processor 602 can be deployed on one of the high-performance computing nodes or management nodes in the heterogeneous computing system, or on a dedicated server outside the heterogeneous computing system, without affecting the implementation of the present invention.

[0132] To enable those skilled in the art to more clearly understand how heterogeneous computing systems allocate appropriate computing devices and startup times for multiple inference request tasks, and to minimize the execution cost of each inference request task while satisfying its task requirements, the following may be included:

[0133] In this embodiment, the computer program executing the inference request task processing method is encapsulated into four functional modules: an inference task information collection module, a heterogeneous computing system information collection module, an allocation scheme optimization module, and a task distribution module. The allocation scheme optimization module has an input receiving layer, an execution layer, and an output layer. The receiving layer receives corresponding data sent by the inference task information collection module and the heterogeneous computing system information collection module. The task distribution module receives output data from the output layer. The inference task information collection module can be deployed on the computing nodes in the heterogeneous computing system that receive inference request tasks issued by users. The heterogeneous computing system information collection module can be deployed on the management node of the heterogeneous computing system. The allocation scheme optimization module can be deployed on the high-performance computing nodes in the heterogeneous computing system, and the task distribution module can also be deployed on the management node.

[0134] Among them, such as Figure 8 As shown, the inference task information collection module is used to obtain inference task information issued by users through collection or testing, and then sends it to the allocation scheme optimization module. The heterogeneous computing system information collection module is used to obtain information about heterogeneous computing systems through collection or testing, and then sends it to the allocation scheme optimization module. Based on the collected information, the allocation scheme optimization module calculates information such as the device where each inference request task is located in the P and D phases, and the task start time, and sends the final task allocation result to the task distribution module. Based on the task allocation result received from the allocation scheme optimization module, the task distribution module generates a task allocation request instructing each inference request task to be migrated to the corresponding target computing device and that each computing device executes the corresponding task at the specified start time. The management node obtains the task identifier corresponding to each inference request task, and through this task identifier and the task allocation request, distributes each inference request task to the corresponding target computing device. Each target computing device executes the corresponding task according to the task start time of the task allocation request, that is, according to the deployed execution method, the arriving inference request set is piped through the P and D phases.

[0135] In this embodiment, the input receiving layer of the allocation scheme optimization module receives a set of inference task information parameters and a set of computing resource information parameters. The inference task information parameter set includes the different computational loads (CLOSE) for each inference request task in the pre-filling and decoding stages. This indicates the computational cost during the pre-filling stage. (This represents the computational cost of the decoding phase) and the maximum execution time of inference request task i. Request priority ( (High-priority tasks are processed first) and the total number of inference request tasks. The set of computing resource information parameters may include the unit time execution cost of the k-th computing device. Execution speed in the P phase and D phase respectively , ),capacity The execution layer of the allocation scheme optimization module encapsulates the execution cost calculation formulas for the P-stage and D-stage, as well as the allocation scheme optimization algorithm. The goal and output of the allocation scheme optimization algorithm are: to output the allocation details for each request (which hardware is used for P, which for D, and the start time of task execution) and the total cost, while minimizing the total cost, without timeouts or exceeding hardware capacity. The allocation scheme optimization algorithm is a cost optimization algorithm based on preemption followed by optimization. First, a relatively high-quality initial solution (i.e., the initial allocation scheme) is determined through a heuristic algorithm. Then, an optimization search algorithm is used to optimize the initial solution, ultimately obtaining the optimal deployment scheme through the search. The implementation process of the allocation scheme optimization algorithm may include:

[0136] The first preprocessing step is to calculate the execution time and cost of each inference request task on each computing device in the heterogeneous computing system, and to create a task execution table for each computing device, recording the busy time period for each request.

[0137] The second initial allocation step: For each request, select the cheapest hardware combination that can complete on time, in descending order of priority (which combination to use for the P phase and which for the D phase). During the check, ensure that the computing devices are not overloaded (e.g., by checking for overlap in the task execution table). The allocation process can be as follows: Sort all arriving inference requests in descending order of priority (higher priority requests acquire resources first), allocate the P phase to high-priority requests first, then allocate the D phase to high-priority requests. For the currently highest priority request that is not allocated, try all hardware combinations, each combination containing one P-phase computing device and one D-phase computing device (one type of hardware combination). In this case, where K is the total number of computing devices (typically, a heterogeneous computing system won't have too many computing devices, so this method can be used), the request is assigned to the earliest start time when the computing device has available capacity. During the trial, it is ensured that the D phase of the request can only be executed after the P phase has finished, and the total time (i.e., P phase execution time + D phase execution time + request start time) is calculated. If the total time exceeds the maximum execution time of the task, this combination is skipped, and other assignments are tried. For this request, the lowest-cost record is selected, and the task execution table is updated, recording the start times of the P and D phases for each inference request task. Iterate through the requests until all requests have been allocated.

[0138] The third global optimization step: Use simulated annealing to randomly change computing devices or shift time. If the new solution is better, record it and use it. Even if the cost is slightly higher, it may help avoid pitfalls.

[0139] A. Initialization process:

[0140] Setting the initial temperature, cooling rate, and number of iterations can be, for example, as follows: , , , For the current allocation solution, the total cost E(s) can be expressed as: ,in, The identifiers for the computing devices allocated to the P and D phases of inference request task i are denoted by cost, and N represents the total number of inference request tasks.

[0141] B. Calculation process of the new solution:

[0142] Generate neighborhood solutions It is generated randomly through one of the following three perturbation operations:

[0143] First method: Randomly select requests Re-enumerate the deployment computing devices and times for phases P and D. Second method: Randomly select requests. The computing device in the P phase of the exchange reasoning request task i. The computing device for the P phase of the inference request task j. Third image: Randomly selected request , , , max represents the maximum value, and y is a user-defined time constant.

[0144] If all requests meet the time requirement that P time + D time + task start time do not exceed the corresponding maximum task execution time, and there is no calculation of device capacity overload, If feasible, then , For the total cost corresponding to the new allocation scheme, then: if (That is, if the new solution has a lower cost), then accept it directly: Check if the optimal solution has been updated. This represents the minimum total cost corresponding to the current optimal allocation scheme, i.e., if ,but , Otherwise (i.e.) ), then by probability accept: .

[0145] In this step, a better solution is allowed ( ), and by temperature The probability of relying on a poor solution is accepted in order to escape local optima. During the search process, the best solution is updated whenever a new solution is better than the historical best.

[0146] C. Update parameters: , , For the current iteration number, if If the conditions are met, output the allocation result of each request in the best solution and the total cost. Otherwise, return to step B and continue iterating.

[0147] In this step, the temperature is multiplied by the cooling rate after each iteration. By gradually reducing the probability of accepting different solutions, the process from exploration to convergence is achieved.

[0148] As can be seen from the above, in this embodiment, for a multi-functional heterogeneous computing system composed of computing devices with different performance and from different manufacturers, the computing devices in the system are rationally allocated and deployed in the PD phase according to the inference task information of the inference request task and the heterogeneous computing power resource information, so as to minimize the cost of processing inference requests in the heterogeneous computing system. Furthermore, in the heterogeneous computing system, based on multiple inference load information input within a period of time, and considering load factors such as request latency requirements and request priorities, the execution time and execution position of the P and D phases are allocated to each request, so as to minimize the execution cost of all inference requests.

[0149] The foregoing has provided a detailed description of a heterogeneous computing system and its inference request task processing method, apparatus, electronic device, non-volatile storage medium, and computer program product provided by the present invention. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Whether the units and algorithm steps of the various examples described in the disclosed embodiments are executed in electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, and such implementations should not be considered beyond the scope of the present invention. Several improvements and modifications can be made to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for processing inference request tasks, characterized in that, include: The computational requirements of each inference request task in the pre-filling and decoding stages are obtained as resource requirement information. The longest execution time of each inference request task is obtained, and the corresponding request priority is determined according to the user level of each inference request task. The longest execution time of each inference request task and the request priority are used as task processing requirement information to obtain the inference task information corresponding to each inference request task. Obtain computing power resource information of the heterogeneous computing system executing each inference request task; wherein, the computing power resource information includes the computing power and execution cost of each computing device in the heterogeneous computing system; Based on the inference task information and the computing resource information, and based on the condition of achieving the lowest processing cost for each inference request task while satisfying the processing requirements of each task, the target computing devices and corresponding task start times for executing the pre-filling and decoding stages of each inference request task are determined to obtain the task allocation results. The task allocation result is sent as a task allocation request to the heterogeneous computing system so that each target computing device executes the corresponding task according to the task start time of the task allocation request. This includes determining the target computing devices and corresponding task start times for the pre-filling and decoding phases of each inference request task, including: Determine the execution time and cost required to process each inference request task using each computing device; Based on the task processing requirements, determine the computing devices and task start times for the respective pre-filling and decoding phases of the corresponding inference request tasks. Based on the selection criteria of the lowest execution cost, execution time not exceeding the longest execution time of the corresponding task, and no overload of computing devices, determine the initial allocation results. By adjusting the computing devices and / or task startup times of each inference request task in the initial allocation result, multiple candidate allocation results are obtained; from the initial allocation result and each candidate allocation result, the allocation result that achieves the lowest processing cost for each inference request task while satisfying the processing requirements of each task is selected.

2. The reasoning request task processing method according to claim 1, characterized in that, Based on the task processing requirements, determine the computing devices and task start times for the respective pre-filling and decoding phases of the corresponding inference request tasks, including: The request priority of the corresponding inference request task is determined based on the task processing requirement information, and the request execution order of each inference request task is determined from high to low according to the request priority. According to the order of request execution, for the first inference request task, first determine the corresponding computing device for the corresponding pre-filling stage, and then determine the corresponding computing device for the corresponding decoding stage. According to the order of request execution, determine the device allocation results for the remaining inference request tasks in sequence: randomly allocate a computing device to the pre-filling stage and the decoding stage of each inference request task. If the task start execution time of the decoding stage of the same inference request task is after the task end time of the corresponding pre-filling stage, and the total execution time and start execution time of the pre-filling stage and the decoding stage of the same inference request task do not exceed the longest task execution time, then the current device allocation result is the candidate device allocation result; from the candidate device allocation results of the same inference request task, select the candidate device allocation result with the lowest execution cost as the corresponding device allocation result.

3. The reasoning request task processing method according to claim 2, characterized in that, Also includes: Obtain the task execution table of each computing device in the heterogeneous computing system; The task execution table records at least the start time and task execution time for different tasks. Update the task execution table according to the request execution order, the device allocation result of each inference request task, and the task execution time of the pre-filling phase and the decoding phase of each inference request task; If different target tasks are processed within the same time period, the device allocation results will be recalculated for each target task.

4. The reasoning request task processing method according to claim 1, characterized in that, By adjusting the computing device and / or task startup time of each inference request task in the initial allocation result, including: The total number of iterations is determined based on the total number of inference request tasks; The total cost of the current task is determined based on the execution costs of each inference request task in the pre-filling and decoding phases in the initial allocation results. A new candidate allocation result is obtained by randomly adjusting the computing device and / or task startup time of each inference request task; if the new candidate allocation result satisfies the condition that the total execution time of the pre-filling stage and the decoding stage of the same inference request task does not exceed the longest execution time of the task, and the capacity of each computing device is not overloaded, then the total cost of the new task corresponding to the new candidate allocation result is calculated. If the total cost of the new task is less than the total cost of the current task, the initial allocation result is replaced by the new candidate allocation result. If the total cost of the new task is less than the total cost of the optimal task, the total cost of the optimal task is updated to the total cost of the new task to complete one iteration. The new candidate allocation result is used as the initial allocation result to execute the next iteration, and the iteration count is updated. If the current iteration count reaches the total number of iterations, then the allocation result corresponding to the current optimal total task cost will be used as the task allocation result.

5. The reasoning request task processing method according to claim 4, characterized in that, By randomly adjusting the computing device and / or task startup time for each inference request task, including: Randomly select a target inference request task and redetermine the new device allocation result for the target inference request task; Randomly select a first target inference request task and a second target inference request task, and exchange the computing devices of the first target inference request task and the second target inference request task; Randomly select a target inference request task and a perturbation value, calculate the task start time of the target inference request task and the sum of the perturbation values, and update the task start time of the target inference request task using the maximum value between the sum and a preset value.

6. The reasoning request task processing method according to claim 4, characterized in that, After calculating the total cost of the new task corresponding to the new candidate allocation result, the method further includes: If the total cost of the new task is greater than or equal to the total cost of the current task, then the cost difference between the total cost of the new task and the total cost of the current task is calculated. The negative value of the ratio of the cost difference to the temperature value of the current iteration is used as the independent variable of the exponential function, and the exponential function is used as the probability value of accepting the new candidate allocation result to replace the initial allocation result. The temperature value for the next iteration is determined based on the cooling rate and temperature values ​​of the current iteration.

7. The reasoning request task processing method according to claim 1, characterized in that, Determine the execution time required for each inference request task to be processed using each computing device, including: The computational load required for the pre-filling and decoding phases of each inference request task is determined based on the inference task information. The execution speed of each computing device is determined based on the computing resource information. For each inference request task, the execution time of the pre-filling stage of the target inference request task is determined by the amount of computation required for the pre-filling stage of the target inference request task and the execution speed of each computing device; the execution time of the decoding stage of the target inference request task is determined by the amount of computation required for the decoding stage of the target inference request task and the execution speed of each computing device.

8. The reasoning request task processing method according to claim 1, characterized in that, Determine the execution cost required to process each inference request task using each computing device, including: The execution cost of each computing device is determined based on the computing resource information. The request priority of each inference request task is determined based on the inference task information; The execution cost of each inference request task is determined based on the execution time of each computing device in executing each inference request task, the request priority of each inference request task, and the execution cost of each computing device.

9. The reasoning request task processing method according to claim 8, characterized in that, Determine the execution cost required to process each inference request task using each computing device, including: The execution cost of the pre-filling phase of each inference request task is determined based on the execution time of the pre-filling phase of each inference request task executed by each computing device, the request priority of each inference request task, and the execution cost of each computing device. The execution cost of the decoding phase of each inference request task is determined based on the execution time of the decoding phase of each inference request task executed by each computing device, the request priority of each inference request task, and the execution cost of each computing device.

10. The reasoning request task processing method according to claim 8, characterized in that, Determine the execution cost required to process each inference request task using each computing device, including: Based on the total number of priority levels and the request priority quantification value of each inference request task, determine the execution order cost quantification value of each inference request task; The execution cost of each inference request task is determined based on the execution time of each computing device in executing each inference request task, the quantified value of the execution order cost of each inference request task, and the unit time execution cost of each computing device.

11. The reasoning request task processing method according to claim 1, characterized in that, Obtain the computational requirements of each inference request task in the pre-filling and decoding stages, including: The inference test task, which belongs to the same task type as the inference request task, is sent to the target computing device of the heterogeneous computing system. The first test computation amount for the pre-filling stage of the inference test task is obtained by the target computing device through the inference model of the inference request task, and the second test computation amount corresponding to the decoding stage of the inference test task; Based on the test lexical length of the inference test task, the computational cost of the first test, and the computational cost of the second test, determine the computational cost required for a single lexical length in the pre-filling stage and the decoding stage, respectively. Based on the lexical length and the computational cost required for each inference request task in the pre-filling and decoding stages, respectively, determine the computational cost required for each inference request task in the pre-filling and decoding stages.

12. The reasoning request task processing method according to any one of claims 1 to 10, characterized in that, Obtain computing resource information of heterogeneous computing systems executing each inference request task, including: Obtain the unit time execution cost of each computing device in the heterogeneous computing system; Obtain the number of parallel request tasks processed by each computing device in the heterogeneous computing system and the execution speed corresponding to the pre-filling stage and the decoding stage, respectively.

13. The reasoning request task processing method according to claim 12, characterized in that, Obtaining the unit time execution cost of each computing device in the heterogeneous computing system includes: The execution cost of each computing device in the heterogeneous computing system performing tasks at full load is calculated. The unit time execution cost of each computing device is determined based on the full-load execution time of each computing device and the corresponding execution cost.

14. A reasoning request task processing device, characterized in that, include: The data acquisition module is used to acquire the amount of computation required by each inference request task in the pre-filling stage and the decoding stage, as resource requirement information. Obtain the longest execution time for each inference request task, and determine the corresponding request priority based on the user level corresponding to each inference request task; use the longest execution time and the priority of each request task as task processing requirement information to obtain the inference task information corresponding to each inference request task. Obtain computing resource information of heterogeneous computing systems that execute each inference request task; The computing resource information mentioned above includes the computing power and execution cost of each computing device in the heterogeneous computing system; The task optimization module is used to determine the target computing devices and corresponding task start times for executing the pre-filling and decoding stages of each inference request task based on the information of each inference task and the computing power resource information, and based on the condition of achieving the lowest processing cost of each inference request task while meeting the processing requirements of each task, so as to obtain the task allocation result. The task assignment and execution module is used to generate a task assignment request based on the task assignment result and send the task assignment request to the heterogeneous computing system so that each target computing device executes the corresponding task according to the task start time of the task assignment request. The task optimization module is further configured to: determine the execution time and execution cost required for each inference request task to be processed by each computing device; determine the computing devices and task start time for the corresponding inference request task to execute their respective pre-filling and decoding stages according to the task processing requirements information; and determine the initial allocation result based on the selection conditions that the execution cost of each inference request task is the lowest, the execution time does not exceed the longest execution time of the corresponding task, and each computing device is not overloaded. By adjusting the computing devices and / or task startup times of each inference request task in the initial allocation result, multiple candidate allocation results are obtained; from the initial allocation result and each candidate allocation result, the allocation result that achieves the lowest processing cost for each inference request task while satisfying the processing requirements of each task is selected.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the inference request task processing method as described in any one of claims 1 to 13 when executing the computer program.

16. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the inference request task processing method as described in any one of claims 1 to 13.

17. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the inference request task processing method according to any one of claims 1 to 13.

18. A heterogeneous computing system, characterized in that, It includes various types of computing devices and processors, with each type of computing device connected to the processor; The processor, when executing a computer program stored in the memory, implements the steps of the inference request task processing method as described in any one of claims 1 to 13.

Images