Inference acceleration method, device and equipment

Abstract

The invention relates to a reasoning acceleration method, device and equipment, and relates to the technical field of artificial intelligence, and the method, device and equipment adopt the technical scheme provided by the embodiment of the invention to realize the dynamic adjustment of resources through a set continuous batch processing algorithm after a task request is analyzed by setting a dynamic scheduling module, so that the resource utilization rate is improved. The technical problem that in the prior art, the scheduling capacity is limited, and consequently computing power is wasted is solved. Wherein the scheduling capability is improved by improving a continuous batch processing algorithm by taking minimization of total delay and maximization of resource utilization rate as a joint optimization target.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular, to a method, device, and equipment for accelerating inference. Background Art

[0002] An inference accelerator is a hardware or software tool used to improve the computing efficiency of the inference stage of an AI model (i.e., predicting actual data after the model is deployed). They significantly improve the running speed of the model on terminal devices or servers by optimizing computing resource allocation, parallel processing, reducing latency and power consumption, etc.

[0003] Inference accelerators in related technologies include GPUs, NPUs, and ASICs. Among them, GPUs accelerate inference calculations through their parallel computing capabilities and SIMD architectures, while NPUs optimize neural network operations using specialized hardware designs and a certain degree of flexibility. ASICs rely on highly customized and efficient resource utilization to provide top-level performance and power consumption advantages for specific inference tasks. These three inference accelerators each have their own advantages in different application scenarios. However, they all use static resource allocation to cope with the load fluctuations of various types of tasks, resulting in limited scheduling capabilities and wasted computing power.

[0004] Therefore, how to solve the technical problem of limited scheduling capabilities of inference accelerators and wasted computing power has become an urgent problem to be solved in the prior art. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method, device, and equipment for accelerating inference to overcome the technical problem of limited scheduling capabilities of current inference accelerators and wasted computing power.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions: On the one hand, an inference acceleration method includes: Responding to a task request, parsing the task type of the task in the task request; wherein, the task type includes a pre-filling task, a decoding task, and a retrieval task; Traversing all tasks in the task queue, calculating resource requirements, where the resource requirements include the maximum allowable delay of the task, the computing power requirement of the task, the computing power requirement of the resource, the maximum computing power of the resource, and the current load rate of the resource; wherein, the task queue is a set of tasks corresponding to all current task requests; Based on the resource requirements, with the joint optimization goal of minimizing the total delay and maximizing the resource utilization rate, calculating the allocation relationship between each task and heterogeneous resources, and removing the completed task requests to obtain a reorganized task queue; wherein, the heterogeneous resources include a CPU cluster, a GPU cluster, and an NPU cluster; Obtain the current prediction result of the time series analysis model, update the reorganized task queue according to the current prediction result; allocate heterogeneous resources to the tasks in the updated reorganized task queue according to the task type and resource allocation strategy of each task.

[0007] Optionally, the allocation of heterogeneous resources to the tasks in the updated reorganized task queue includes: Allocate the pre-filled tasks in the updated reorganized task queue to the NPU cluster, and allocate the decoding tasks in the updated reorganized task queue to the CPU cluster.

[0008] Optionally, the resource allocation strategy further includes: Give priority to allocating NPUs during peak periods, and enable the CPU cluster to complete low-priority tasks during off-peak periods.

[0009] Optionally, it further includes: when the priority of a new task is higher than that of any current task, give priority to resource allocation for this new task.

[0010] Optionally, it further includes: Obtain the resource utilization rate of each hardware resource. When the CPU core idle rate is greater than or equal to the limit threshold, transfer part of the KVCache to the CPU memory.

[0011] Optionally, it further includes: Convert the model operator into an instruction set supported by the target hardware through middleware; Call the general acceleration library to execute the fused operator; Use the SIMD instruction set of the CPU to parallel process lightweight calculations in the decoding stage, and offload similar types of calculations to the CPU for execution.

[0012] Optionally, it further includes: Obtain the historical task load data for a preset time period; Input the task load data into the time series prediction model to obtain a prediction result; wherein, the prediction result is the task load situation within a future set duration.

[0013] Optionally, with minimizing the total delay and maximizing the resource utilization rate as the joint optimization objective, the objective function is constructed as:

[0014] where Di is the maximum allowable delay of task Ti; Ci is the computing power requirement of task Ti; C j max is the maximum computing power of resource Rj; Lj is the current load rate of resource Rj; α, β are weight coefficients; x ij is a binary decision variable indicating whether task Ti is allocated to resource Rj.

[0015] On the other hand, an inference acceleration device includes: A dynamic scheduling module, configured to respond to a task request, parse the task type of the task in the task request; wherein, the task type includes a pre-filling task, a decoding task, and a retrieval task; traverse all tasks in the task queue, calculate resource requirements, and the resource requirements include the maximum allowable delay of the task, the computing power requirement of the task, the computing power requirement of the resource, the maximum computing power of the resource, and the current load rate of the resource; wherein, the task queue is a set of tasks corresponding to all current task requests; based on the resource requirements, with the joint optimization goal of minimizing the total delay and maximizing the resource utilization rate, calculate the allocation relationship between each task and heterogeneous resources, and eliminate the completed task requests to obtain a reorganized task queue; wherein, the heterogeneous resources include a CPU cluster, a GPU cluster, and an NPU cluster; obtain the current prediction result of the time series analysis model, and update the reorganized task queue according to the current prediction result; according to the task type of each task and the resource allocation strategy, allocate heterogeneous resources to the tasks in the updated reorganized task queue; A cross-hardware acceleration middleware, configured to convert model operators into an instruction set supported by the target hardware through the middleware; call a general acceleration library to execute the fused operator; utilize the SIMD instruction set of the CPU to parallel process lightweight calculations in the decoding stage, and offload similar types of calculations to the CPU for execution; A resource monitoring module, configured to monitor the utilization rate of each hardware resource and feedback the utilization rate of each hardware resource to the dynamic scheduling module; A scenario optimization module, configured to customize an optimization strategy according to the characteristics of the AGI or RAG scenario.

[0016] On the other hand, an inference acceleration device includes a processor and a memory, and the processor is connected to the memory: Wherein, the processor is configured to call and execute a program stored in the memory; The memory is configured to store the program, and the program is at least used to execute the inference acceleration method described in any one of the above.

[0017] The technical solutions described in the embodiments provided by the present invention have at least the following beneficial effects: By adopting the technical solution provided by the embodiment of the present invention, through the setting of the dynamic scheduling module, after parsing the task request, through the set continuous batch processing algorithm, the dynamic adjustment of resources is realized, and the technical problem of limited scheduling ability and wasted computing power in the prior art is solved. Among them, through the improvement of the continuous batch processing algorithm with the joint optimization goal of minimizing the total delay and maximizing the resource utilization rate, the scheduling ability is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0019] Figure 1 It is a schematic structural diagram of an inference acceleration device provided by an embodiment of the present invention; Figure 2 It is a schematic flow diagram of an inference acceleration method based on a computing power platform provided by an embodiment of the present invention; Figure 3 It is a schematic structural diagram of an inference acceleration device provided by an embodiment of the present invention. Detailed implementation manners

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will describe the technical solutions of the present invention in detail. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation manners obtained by those of ordinary skill in the art without creative efforts fall within the scope protected by the present invention.

[0021] As recorded in the background art, an inference accelerator is a hardware or software tool used to improve the computing efficiency of the inference stage of an AI model (i.e., predicting actual data after the model is deployed). They significantly improve the running speed of the model on terminal devices or servers by optimizing computing resource allocation, parallel processing, reducing latency and power consumption, etc.

[0022] The inference accelerators in the related art include GPUs, NPUs, and ASICs. Among them, GPUs accelerate inference calculations through their parallel computing capabilities and SIMD architectures, while NPUs optimize neural network operations using specialized hardware designs and a certain degree of flexibility. ASICs rely on highly customized and efficient resource utilization to provide top-level performance and power consumption advantages for specific inference tasks. These three inference accelerators each have their own advantages in different application scenarios. However, they all use static resource allocation to cope with the load fluctuations of various types of tasks, resulting in limited scheduling capabilities and wasted computing power.

[0023] That is to say, traditional static batching processes a fixed number of requests at a time and does not release resources until all requests are completed, which easily leads to resource waste and latency accumulation.

[0024] Therefore, how to solve the technical problem of limited scheduling ability of the inference accelerator, resulting in wasted computing power, has become an urgent problem to be solved in the existing technology.

[0025] Among them, when the GPU (Graphics Processing Unit) accelerates inference computing, it mainly utilizes its powerful parallel computing ability. Compared with the traditional CPU (Central Processing Unit), the GPU has more computing cores. For example, a high-end GPU can have thousands of processing units, while a CPU generally has only a few to dozens of cores. These numerous cores can perform multiple types of calculations simultaneously. In a deep neural network, this parallelism can be used to process multiple neurons or multiple data samples simultaneously. For example, during the training or inference process of a Convolutional Neural Network (CNN), the calculations in the convolutional layer and the fully connected layer (such as matrix multiplication operations) can be decomposed into multiple small tasks, and then these small tasks are distributed to multiple cores of the GPU, and each core processes different rows or columns simultaneously, thus greatly improving the computing speed.

[0026] NPU (Neural Network Processor): Specialized hardware design, the NPU is a dedicated hardware designed specifically for artificial intelligence neural network inference and training. It is different from general-purpose computing hardware (such as CPUs and GPUs), and its hardware circuit structure is specifically optimized for common neural network operations. For example, during the forward propagation process of deep learning, the matrix dot product operation between the weight matrix and the input is a key step. Specialized hardware modules will be designed in the NPU to execute these matrix operations. Such hardware modules can efficiently process a large number of data dot product and accumulation operations, thereby reducing the frequent transfer of data between the memory and the processor and improving the overall computing efficiency. Flexibility and customizability, some NPUs have a certain degree of flexibility and programmability. They may contain reconfigurable computing units or instruction sets inside, and these units can be adjusted for different neural network architectures and algorithms. For example, for different convolutional neural networks (such as AlexNet, ResNet, etc.) or recurrent neural networks (such as LSTM), the NPU can optimize the data flow and processing order by adjusting the combination of internal processing units or multi-level cache strategies. This can further improve the inference speed and reduce power consumption while ensuring the accuracy of the neural network.

[0027] ASIC (Application Specific Integrated Circuit): Advantages of high customization. ASIC is an integrated circuit customized for specific computing tasks. When used for AI accelerated inference computing, ASIC can be highly optimized for specific application scenarios and algorithms. For example, in some specific tasks such as speech recognition, image recognition, or natural language processing, ASIC can directly solidify the most commonly used operations in these tasks (such as FFT transformation, derivation of finite state machines, etc.) into the hardware circuit. In this way, when the system needs to perform these operations, ASIC can complete complex computing processes in a very short time, without the need to execute a large number of instructions like a traditional CPU. Fixed function and efficient resource utilization. The function of a single specific ASIC chip is often fixed, which means it can only efficiently process the tasks it is designed for. For these specific tasks, the resource utilization efficiency of ASIC is very high. For example, the chip design of ASIC can include dedicated small storage units, and the size and data flow path of these storage units are closely coupled with the computing process of the task. In this way, during data transmission and processing, memory conflicts and delays can be avoided, thereby accelerating the data flow speed of the inference process and improving the overall inference performance.

[0028] In summary, GPUs accelerate inference computing through their parallel computing capabilities and SIMD architectures, NPUs optimize neural network operations using specialized hardware designs and a certain degree of flexibility, while ASICs rely on high customization and efficient resource utilization to provide top-level performance and power consumption advantages for specific inference tasks. These three inference accelerators each have their own advantages in different application scenarios. The mainstream large model inference acceleration hardware, with its parallel computing capabilities and customized architectures, has significantly improved the processing efficiency of specific tasks, but also brought new technical challenges. For example: 1. Ecological closedness: The coexistence of the ecological barriers of proprietary hardware (such as GPUs) and the fragmentation of domestic chips hinders cross-platform efficient development; 2. Insufficient dynamic scheduling ability: Static resource allocation is difficult to cope with the load fluctuations of AGI and RAG type tasks, resulting in waste of computing power; 3. Lack of full-link optimization: There is a lack of collaborative optimization tools in the entire process from model compression to hardware resource scheduling, and the end-to-end efficiency improvement is limited.

[0029] 4. Low hardware resource utilization: Existing mainstream inference servers often configure high-end multi-core CPUs, more than 100 central processing unit cores supporting SIMD functions, and more than 1TGB of DRAM memory. These computing and storage resources are not fully utilized, resulting in waste of computing resources.

[0030] Based on this, an embodiment of the present invention provides an inference acceleration method, device and equipment based on a computing power platform to at least solve the technical problem of insufficient dynamic scheduling capability mentioned above.

[0031] Figure 1 A schematic diagram of a reasoning acceleration device provided by an embodiment of the present invention is shown in FIG. Figure 1 , the device provided by the embodiment of the present invention may include: The dynamic scheduling module 11 is used to respond to the task request and parse the task type of the task in the task request; wherein the task type includes a pre-filling task, a decoding task, and a retrieval task; Traverse all tasks in the task queue and calculate resource requirements, where the resource requirements include the maximum allowable delay of the task, the computing power requirement of the task, the computing power requirement of the resource, the maximum computing power of the resource, and the current load rate of the resource; wherein the task queue is a collection of tasks corresponding to all current task requests; Based on the resource requirements, the allocation relationship between each task and the heterogeneous resources is calculated with the joint optimization goals of minimizing the total delay and maximizing the resource utilization, and the completed task requests are removed to obtain a reorganized task queue; wherein the heterogeneous resources include CPU clusters, GPU clusters, and NPU clusters; The current prediction result of the time series analysis model is obtained, and the reorganized task queue is updated according to the current prediction result; and heterogeneous resources are allocated to the tasks in the updated reorganized task queue according to the task type and resource allocation strategy of each task.

[0032] Among them, the task request can be AGI / RAG and other task requests, where AGI (Artificial General Intelligence) refers to an intelligent system with general cognitive capabilities at the human level, capable of autonomous learning and reasoning in a variety of untrained tasks, rather than being limited to a single field (such as current dedicated AI models). RAG (Retrieval-Augmented Generation): Enhances the ability of the generation model by retrieving external knowledge bases, making the generated content more accurate and factual.

[0033] After the task request is obtained, the task request is parsed to obtain the task type of the task, for example, the task type is a pre-filling task, a decoding task, or a retrieval task.

[0034] Reorganize the task batches through a continuous batch algorithm, dynamically propose completed requests (such as sequences with end markers generated), and release idle resources; according to the time series analysis model, predict the load situation in the future for a period of time, and allocate heterogeneous resources to the tasks in the updated and reorganized task queue according to the task type and resource allocation strategy of each task.

[0035] Specifically, the task queue can be: Q = {T1, T2,..., Tn}, hardware resource pool: R = {CPU, NPU, GPU} Resource allocation mapping table: M(Ti→Rj).

[0036] a. Traverse the tasks in Q, calculate their resource requirements, and the resource requirements include the maximum allowable delay of the task, the computing power requirement of the task, the computing power requirement of the resource, the maximum computing power of the resource, and the current load rate of the resource.

[0037] b. Use the Hungarian algorithm to solve the optimal allocation, and the objective function is to minimize the weighted sum of the total delay and maximize the resource utilization rate.

[0038] In this embodiment, taking minimizing the total delay and maximizing the resource utilization rate as the joint optimization goal, unify them into a weighted objective function, and the specific formula is as follows:

[0039] x ij ∈ {0, 1} is a binary decision variable, indicating whether task Ti is allocated to resource Rj; Tij: The estimated execution time (including communication overhead) of task Ti on resource Rj.

[0040] Di: The maximum allowable delay of task Ti; Ci: The computing power requirement of task Ti (TFLOPS); C j max : The maximum computing power of resource Rj (such as the peak computing power of NPU); Lj: The current load rate of resource Rj (0 - 100%); α, β: Weight coefficients (satisfying α + β = 1), used to balance the importance of delay and resource utilization rate.

[0041] Finally, obtain the allocation situation of tasks and heterogeneous resources through the formula. Due to real-time updates, real-time insertion of new requests, and elimination of completed tasks, a dynamic batch is formed, and the reorganized task queue is obtained after reorganization.

[0042] Obtain the current prediction result of the time series analysis model, update the reorganized task queue according to the current prediction result; allocate heterogeneous resources to the tasks in the updated reorganized task queue according to the task type and resource allocation strategy of each task.

[0043] It can be understood that in the technical solution provided by the embodiments of the present invention, through the set continuous batch processing algorithm, the dynamic adjustment of resources is realized, and the technical problem of limited scheduling ability and wasted computing power in the prior art is solved.

[0044] In some embodiments, it further includes: Obtain the historical task load data of a preset time period; Input the task load data into the time series prediction model, and predict the prediction result; wherein, the prediction result is the task load situation within a set future duration.

[0045] It can be understood that by adopting the technical solution provided by this embodiment, through the time series prediction model, the prediction of the task load situation is realized, so as to guide the pre-allocation.

[0046] In some embodiments, the allocation of heterogeneous resources to the tasks in the updated reorganized task queue includes: Allocate the pre-filled tasks in the updated reorganized task queue to the NPU cluster, and allocate the decoding tasks in the updated reorganized task queue to the CPU cluster.

[0047] It can be understood that by adopting the technical solution provided by this embodiment, the pre-allocation of resources is realized.

[0048] In some embodiments, the resource allocation strategy further includes: Give priority to allocating NPU during peak periods, and enable the CPU cluster to complete low-priority tasks during off-peak periods.

[0049] It can be understood that by adopting the technical solution provided by this embodiment, it is possible to perform tidal scheduling: give priority to allocating NPU during peak periods, and enable the CPU cluster to complete low-priority tasks during off-peak periods, thereby improving the adaptation accuracy.

[0050] In some embodiments, it further includes: when the priority of a new task is higher than that of any current task, give priority to allocating resources to this new task.

[0051] If the latency constraint of the new task Tk is higher than that of the tasks in the current batch, preempt the resources of low-priority tasks.

[0052] It can be understood that by adopting the technical solution provided by the embodiments of the present invention, it supports the task preemption and queuing mechanism, and gives priority to processing high-real-time requests.

[0053] The cross-hardware acceleration middleware 12 is used to convert model operators into an instruction set supported by the target hardware through the middleware; call the general acceleration library to execute the fused operators; utilize the SIMD instruction set of the CPU to parallelize the lightweight calculations in the decoding stage, and offload similar types of calculations to the CPU for execution.

[0054] It can be understood that by adopting the technical solution provided by the embodiment of the present invention, the barriers of different hardware can be removed. For example, the cross-hardware acceleration middleware may include: Unified instruction set conversion layer: Convert model operators (such as matrix multiplication, attention mechanism) into underlying instructions supported by domestic chips such as Ascend and Cambricon.

[0055] General acceleration library: Provide cross-hardware optimized operators (such as fused LayerNorm+GeLU) to reduce the number of video memory copies.

[0056] Runtime adaptation engine: Dynamically load the driver frameworks of different hardware (such as Ascend CANN, Cambricon BANG) to support mixed-precision inference.

[0057] The resource monitoring module 13 is used to monitor the utilization rate of each hardware resource (such as the activity of CPU cores, video memory occupancy rate), and feedback the utilization rate of each hardware resource to the dynamic scheduling module. Thus, if the idle rate of CPU cores > 30%, then trigger video memory-memory collaborative calculation (such as transferring part of the KVCache to CPU memory).

[0058] The scenario optimization module 14 is used to customize optimization strategies according to the characteristics of AGI or RAG scenarios. For example, it may include: Retrieval-generation collaboration unit: In the RAG task, allocate the retrieval stage to the large-memory CPU cluster and the generation stage to the high-computing-power NPU to avoid resource contention.

[0059] Video memory compression unit: Adopt dynamic quantization technology (such as INT8→FP16 fallback) to compress the video memory occupancy of KVCache in the decoding stage.

[0060] Based on a general inventive concept, the embodiment of the present invention also provides an inference acceleration method based on a computing power platform.

[0061] Figure 2 For the flowchart of an inference acceleration method based on a computing power platform provided by the embodiment of the present invention, refer to Figure 2 The method provided by the embodiment of the present invention may include the following steps: Step S101, in response to a task request, parse the task type of the task in the task request; wherein, the task type includes a pre-fill task, a decoding task, and a retrieval task; Step S102, traverse all tasks in the task queue and calculate resource requirements, wherein the resource requirements include the maximum allowable delay of the task, the computing power requirement of the task, the computing power requirement of the resource, the maximum computing power of the resource, and the current load rate of the resource; wherein the task queue is a collection of tasks corresponding to all current task requests; Step S103: Based on the resource requirements, the allocation relationship between each task and the heterogeneous resources is calculated with the joint optimization goals of minimizing the total delay and maximizing the resource utilization, and the completed task requests are removed to obtain a reorganized task queue; wherein the heterogeneous resources include a CPU cluster, a GPU cluster, and an NPU cluster; Step S104, obtaining the current prediction result of the time series analysis model, and updating the reorganized task queue according to the current prediction result; and allocating heterogeneous resources to the tasks in the updated reorganized task queue according to the task type and resource allocation strategy of each task.

[0062] In some embodiments, the allocating heterogeneous resources to the tasks in the updated reorganized task queue includes: The pre-filling tasks in the updated reorganized task queue are allocated to the NPU cluster, and the decoding tasks in the updated reorganized task queue are allocated to the CPU cluster.

[0063] In some embodiments, the resource allocation strategy further includes: During peak hours, NPUs are allocated first, and during off-peak hours, CPU clusters are enabled to complete low-priority tasks.

[0064] In some embodiments, it also includes: when the priority of the new task is higher than any current task, resources are allocated to the new task first.

[0065] In some embodiments, it also includes: Get the resource utilization of each hardware resource. When the CPU core idle rate is greater than or equal to the limit threshold, transfer part of the KVCache to the CPU memory.

[0066] In some embodiments, it also includes: Convert model operators into instruction sets supported by target hardware through middleware; Call the general acceleration library to execute the fusion operator; The SIMD instruction set of the CPU is used to parallelize lightweight computations in the decoding stage, and similar types of computations are offloaded to the CPU for execution.

[0067] In some embodiments, it also includes: Get historical task load data for a preset time period; Input the task load data into a time series prediction model to obtain a prediction result, where the prediction result is the task load situation within a set future time period.

[0068] In some embodiments, with minimizing the total delay and maximizing the resource utilization rate as the joint optimization objective, the objective function is constructed as:

[0069] where Di is the maximum allowable delay of task Ti; Ci is the computing power requirement of task Ti; C j max is the maximum computing power of resource Rj; Lj is the current load rate of resource Rj; α, β are weight coefficients; x ij is.

[0070] Regarding the method in the above embodiments, the specific ways of performing operations in each step have been described in detail in the embodiments related to the device, and will not be elaborated here.

[0071] The present invention also provides an inference acceleration device for implementing the above method embodiments. Figure 3 is a schematic structural diagram of an inference acceleration device provided by an embodiment of the present invention. As Figure 3 shown, the device in this embodiment includes a processor 21 and a memory 22, and the processor 21 is connected to the memory 22. Among them, the processor 21 is used to call and execute the program stored in the memory 22; the memory 22 is used to store the program, and the program is at least used to execute the inference acceleration method in the above embodiments.

[0072] The specific implementation scheme of the inference acceleration device provided by the embodiments of the present application can refer to the implementation manners of the inference acceleration methods in any of the above embodiments, and will not be elaborated here.

[0073] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be seen in the same or similar content in other embodiments.

[0074] It should be noted that in the description of the present invention, terms such as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise stated, the meaning of "a plurality of" is at least two.

[0075] Any process or method description depicted in the flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations where functions may be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0076] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0077] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program. The said program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0078] In addition, in each embodiment of the present invention, the functional units can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0079] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc.

[0080] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0081] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for accelerating inference, characterized in that, Including: In response to a task request, parsing the task type of the task in the task request; wherein, the task type includes a pre-filling task, a decoding task, and a retrieval task; Traversing all tasks in the task queue, calculating resource requirements, where the resource requirements include the maximum allowable delay of the task, the computing power requirement of the task, the computing power requirement of the resource, the maximum computing power of the resource, and the current load rate of the resource; wherein, the task queue is a set of tasks corresponding to all current task requests; Based on the resource requirements, with the joint optimization goal of minimizing the total delay and maximizing the resource utilization rate, calculating the allocation relationship between each task and heterogeneous resources, and removing the completed task requests to obtain a reorganized task queue; wherein, the heterogeneous resources include a CPU cluster, a GPU cluster, and an NPU cluster; Obtaining the current prediction result of the time series analysis model, updating the reorganized task queue according to the current prediction result; and allocating heterogeneous resources to the tasks in the updated reorganized task queue according to the task type and resource allocation strategy of each task.

2. The method according to claim 1, wherein The allocating heterogeneous resources to the tasks in the updated reorganized task queue includes: Allocating the pre-filling tasks in the updated reorganized task queue to the NPU cluster, and allocating the decoding tasks in the updated reorganized task queue to the CPU cluster.

3. The method according to claim 1, wherein The resource allocation strategy further includes: Giving priority to allocating NPUs during peak periods, and enabling the CPU cluster to complete low-priority tasks during off-peak periods.

4. The method according to claim 1, characterized in that Also including: When the priority of a new task is higher than that of any current task, giving priority to allocating resources to this new task.

5. The method according to claim 1, wherein Also including: Obtaining the resource utilization rate of each hardware resource, and when the CPU core idle rate is greater than or equal to the limit threshold, transferring part of the KVCache to the CPU memory.

6. The method according to claim 1, characterized in that, Also including: Converting the model operator into an instruction set supported by the target hardware through middleware; Invoking the general acceleration library to execute the fused operator; Using the SIMD instruction set of the CPU to parallel process the lightweight calculations in the decoding stage and offloading similar types of calculations to the CPU for execution.

7. The method according to claim 1, wherein Also including: Obtaining the historical task load data for a preset time period; Inputting the task load data into the time series prediction model to obtain a prediction result; wherein, the prediction result is the task load situation within a future set duration.

8. The method according to claim 1, characterized in that With the joint optimization goal of minimizing the total delay and maximizing the resource utilization rate, constructing the objective function as: Among them, Di is the maximum allowable delay of task Ti; Ci is the computing power requirement of task Ti; C j max is the maximum computing power of resource Rj; Lj is the current load rate of resource Rj; α, β are weight coefficients; x ij is a binary decision variable indicating whether task Ti is allocated to resource Rj.

9. An inference acceleration device, characterized in that, Including: A dynamic scheduling module is used to respond to a task request and parse the task type of the task in the task request; wherein the task type includes pre-filled tasks, decoding tasks, and retrieval tasks; traverse all tasks in the task queue and calculate resource requirements, wherein the resource requirements include the maximum allowable delay of the task, the computing power requirements of the task, the computing power requirements of the resource, the maximum computing power of the resource, and the current load rate of the resource; wherein the task queue is a collection of tasks corresponding to all current task requests; based on the resource requirements, with minimizing the total delay and maximizing the resource utilization as the joint optimization goals, calculate the allocation relationship between each task and the heterogeneous resources, and remove the completed task requests to obtain the reorganized task queue; wherein the heterogeneous resources include CPU clusters, GPU clusters, and NPU clusters; obtain the current prediction results of the time series analysis model, and update the reorganized task queue according to the current prediction results; according to the task type and resource allocation strategy of each task, allocate heterogeneous resources to the tasks in the updated reorganized task queue; Cross-hardware acceleration middleware is used to convert model operators into instruction sets supported by target hardware through middleware; call the general acceleration library to execute fusion operators; use the CPU's SIMD instruction set to parallelize lightweight calculations in the decoding stage, and offload similar types of calculations to the CPU for execution; The resource monitoring module is used to monitor the utilization of each hardware resource and feed back the utilization of each hardware resource to the dynamic scheduling module; The scenario optimization module is used to customize the optimization strategy based on the AGI or RAG scenario characteristics.

10. An inference acceleration device, characterized in that, The invention comprises a processor and a memory, wherein the processor is connected to the memory: Wherein, the processor is used to call and execute the program stored in the memory; The memory is used to store the program, and the program is at least used to execute the reasoning acceleration method described in any one of claims 1-8.

Images