Task scheduling method and device based on heterogeneous computing, storage medium and equipment

Abstract

The present application relates to a kind of task scheduling method, device, storage medium and equipment based on heterogeneous computing, the method includes: obtaining the inference task of large language model;In the execution process of inference task, at least based on memory access intensity and current inference stage, identify memory access intensive operator and computing intensive operator in inference task;Memory access intensive operator is distributed to the first computing unit for executing memory intensive task, and computing intensive operator is distributed to the second computing unit for executing computing intensive task;The estimated execution time that two computing units respectively execute corresponding task is obtained, and the data transmission time between them;According to estimated execution time and data transmission time, with the overall execution delay of inference task as target to be minimized, the task start time of first computing unit and second computing unit is determined respectively;Control first computing unit and second computing unit execute corresponding task according to task start time asynchronously in parallel.

Description

Technical Field

[0001] This invention relates to the field of large model inference optimization technology, specifically to a task scheduling method, apparatus, storage medium, and device based on heterogeneous computing. Background Technology

[0002] Existing GPU-PIM heterogeneous computing solutions typically employ static or coarse-grained task partitioning and data placement strategies. Specifically, these solutions predetermine task allocation during model compilation or loading, for example, statically assigning all GEMV operators to PIM for execution, while reserving all GEMM operators for execution on the GPU. This static partitioning approach completely ignores the actual computational and memory access load changes during inference. Regarding task scheduling, existing solutions often employ a simple pipeline model: the GPU completes a portion of the computation, transfers data to PIM, and then waits for PIM to complete its computation before continuing execution. This serial scheduling method ignores the speed differences between GPU and PIM, as well as data transfer latency. In terms of data management, existing solutions tend to statically fix critical data such as KV-Cache as a whole in the storage of a single device, lacking the ability to dynamically optimize based on actual access patterns.

[0003] The aforementioned static approach has many drawbacks, resulting in insignificant performance gains for heterogeneous systems.

[0004] First, static task partitioning cannot adapt to the dynamic changes in computation and memory access load during LLM inference. In the Prefill phase, the GPU is under heavy load with a large number of computationally intensive tasks, while PIM may be idle due to a lack of suitable tasks. In the Decoding phase, although there are a large number of memory-intensive tasks suitable for PIM processing, the GPU still needs to execute these tasks due to the pre-defined static partitioning and will be stuck in a waiting state due to frequent memory accesses, resulting in a serious waste of computing resources.

[0005] Secondly, the simple pipeline scheduling mechanism causes a lot of waiting and synchronization overhead between devices, resulting in so-called pipeline bubbles, where one device is idle while waiting for another device to complete its task, which reduces the overall parallelism and throughput.

[0006] Furthermore, a static data placement strategy means that when different devices need to access the same data, cross-device data transfer must be performed, which not only increases data transfer overhead but also negates the inherent advantages of the PIM architecture in reducing memory access latency.

[0007] Ultimately, these problems collectively lead to low efficiency in heterogeneous computing power collaboration, making it difficult to meet the actual requirements of high throughput and low latency for large model inference. Summary of the Invention

[0008] This application discloses a task scheduling method, apparatus, storage medium, and device based on heterogeneous computing, aiming to solve the technical problems existing in the prior art. The present invention adopts the following technical solution:

[0009] According to a first aspect of this application, a task scheduling method based on heterogeneous computing is provided, the method comprising:

[0010] Obtain the inference task of a large language model, wherein the inference task includes multiple operators;

[0011] During the execution of the inference task, memory-intensive operators and computation-intensive operators in the inference task are identified, based at least on memory access intensity and the current inference stage.

[0012] The memory-intensive operator is assigned to a first computing unit, and the computation-intensive operator is assigned to a second computing unit. The first computing unit is used to execute the task corresponding to the memory-intensive operator, and the second computing unit is used to execute the task corresponding to the computation-intensive operator.

[0013] The estimated execution time of the first computing unit and the second computing unit for executing their respective tasks, as well as the data transmission time between the computing units, are obtained.

[0014] Based on the estimated execution time and the data transmission time, with the goal of minimizing the overall execution delay of the inference task, the task start time of the first computing unit and the second computing unit is determined respectively.

[0015] The first computing unit and the second computing unit are controlled to execute the corresponding tasks asynchronously and in parallel according to the task start time.

[0016] In one possible implementation, the method further includes:

[0017] Establish a unified memory address space for the first computing unit and the second computing unit;

[0018] During the execution of the inference task, at least according to the data access requirements of the memory-intensive operator and the computation-intensive operator, the data frequently accessed by the memory-intensive operator is dynamically stored in the first storage area corresponding to the first computing unit, and the data frequently accessed by the computation-intensive operator is dynamically stored in the second storage area corresponding to the second computing unit.

[0019] Based on the execution requirements of the operator, the data is prefetched or migrated between the first storage area and the second storage area;

[0020] Based on the unified memory address space, data sharing and transmission between the first computing unit and the second computing unit are realized, and data consistency between the first storage area and the second storage area is maintained.

[0021] In one possible implementation, during the execution of the inference task, identifying memory-intensive operators and computationally intensive operators in the inference task, at least based on memory access intensity and the current inference stage, includes:

[0022] Obtain the memory access and computational operation characteristics of the operator;

[0023] Based on the memory access operation characteristics and the computation operation characteristics, the memory access density of the operator is determined;

[0024] Based on the current inference stage, determine a threshold for distinguishing between the memory-intensive operator and the computation-intensive operator;

[0025] When the memory access intensity is greater than the determination threshold, the corresponding operator is identified as the memory access intensive operator;

[0026] When the memory access intensity is less than or equal to the determination threshold, the corresponding operator is identified as the computationally intensive operator.

[0027] In one possible implementation, the first computing unit is an in-memory computing unit, and the first storage area is a memory area that the in-memory computing unit can directly access.

[0028] The second computing unit is a graphics processor, and the second storage area is the video memory of the graphics processor.

[0029] In one possible implementation, obtaining the estimated execution time of the corresponding tasks performed by the first computing unit and the second computing unit, and the data transmission time between the computing units, includes:

[0030] Obtain the feature parameters of the inference task, wherein the feature parameters include at least the current sequence length;

[0031] Based on a preset performance prediction model, and according to the feature parameters and the characteristics of the memory-intensive operator, the first estimated execution time of the first computing unit executing the task corresponding to the memory-intensive operator is predicted.

[0032] Based on the performance prediction model, and according to the feature parameters and the characteristics of the computationally intensive operator, the second estimated execution time for the second computing unit to execute the task corresponding to the computationally intensive operator is predicted.

[0033] Based on the performance prediction model, the data transmission time between the first computing unit and the second computing unit is predicted according to the amount of data that needs to be transmitted between the first computing unit and the second computing unit.

[0034] In one possible implementation, determining the task start time of the first computing unit and the second computing unit, based on the estimated execution time and the data transmission time, with the goal of minimizing the overall execution latency of the inference task, includes:

[0035] Based on the first estimated execution time, the second estimated execution time, and the data transmission time, the task execution time difference between the first computing unit and the second computing unit is determined.

[0036] Based on the task execution time difference, the task start time of the first computing unit and the second computing unit is determined to achieve task execution time alignment between the first computing unit and the second computing unit, thereby minimizing the overall execution delay of the inference task.

[0037] In one possible implementation, controlling the first computing unit and the second computing unit to asynchronously and in parallel execute the corresponding tasks according to the task start time includes:

[0038] At the task startup time of the first computing unit, the first computing unit is triggered to start executing the task corresponding to the memory-intensive operator;

[0039] At the task startup time of the second computing unit, the second computing unit is triggered to start executing the task corresponding to the computationally intensive operator;

[0040] The first computing unit and the second computing unit each execute their respective tasks independently without waiting for the other to complete, thus achieving asynchronous parallel execution.

[0041] According to a second aspect of this application, a task scheduling device based on heterogeneous computing is provided, comprising:

[0042] The task acquisition module is used to acquire the inference task of the large language model, wherein the inference task includes multiple operators;

[0043] An operator identification module is used to identify memory-intensive operators and computation-intensive operators in the inference task during the execution of the inference task, based at least on memory access intensity and the current inference stage, wherein the memory access intensity characterizes the ratio of memory access operations to computation operations during the execution of the operator.

[0044] The task allocation module is used to allocate the memory-intensive operator to a first computing unit and the computation-intensive operator to a second computing unit. The first computing unit is used to execute the task corresponding to the memory-intensive operator, and the second computing unit is used to execute the task corresponding to the computation-intensive operator.

[0045] The performance prediction module is used to obtain the estimated execution time of the first computing unit and the second computing unit for executing the corresponding tasks, as well as the data transmission time between the computing units;

[0046] The scheduling decision module is used to determine the task start time of the first computing unit and the second computing unit respectively, based on the estimated execution time and the data transmission time, with the goal of minimizing the overall execution delay of the inference task;

[0047] The task execution control module is used to control the first computing unit and the second computing unit to asynchronously and in parallel execute the corresponding tasks according to the task start time.

[0048] According to a third aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the heterogeneous computing-based task scheduling method as described in any of the preceding claims.

[0049] According to a fourth aspect of this application, a computer device is provided, the computer device including the heterogeneous computing-based task scheduling device as described above.

[0050] The beneficial effects of the technical solution provided in this application include at least the following:

[0051] This application provides a task scheduling method based on heterogeneous computing. By dynamically identifying memory-intensive and computation-intensive operators during the execution of inference tasks, and allocating memory-intensive operators to the first computing unit and computation-intensive operators to the second computing unit, the method achieves precise matching between operators and hardware characteristics. This dynamic identification and allocation mechanism can give full play to the hardware advantages of different computing units. Compared with the method of fixedly allocating all operators to a single computing unit, it can significantly improve the efficiency of hardware resource utilization.

[0052] By obtaining the estimated execution time of the first and second computing units for their respective tasks, as well as the data transfer time between computing units, and determining the task start time of each computing unit with the goal of minimizing the overall execution latency of the inference task, the performance bottleneck problem caused by the difference in execution speed of different computing units in a heterogeneous computing environment is solved. Through precise control of the task start time, the task execution time of the first and second computing units is aligned, effectively eliminating pipeline bubbles and avoiding idle waiting of computing resources.

[0053] By controlling the first and second computing units to asynchronously and in parallel execute their corresponding tasks according to a determined task start time, the limitations of the traditional serial execution mode are broken. The two computing units execute their corresponding tasks independently without waiting for the other to complete, thus achieving true asynchronous parallel execution. This asynchronous parallel mechanism can make full use of the parallel processing capabilities of heterogeneous computing environments and significantly reduce the overall execution latency compared to synchronous execution.

[0054] Furthermore, by establishing a unified memory address space for the first and second computing units, and dynamically managing data storage locations according to the data access needs of memory-intensive and computation-intensive operators, the data management challenge in heterogeneous computing environments is solved. By dynamically storing frequently accessed data by memory-intensive operators in the first storage area corresponding to the first computing unit, and frequently accessed data by computation-intensive operators in the second storage area corresponding to the second computing unit, and supporting prefetching and migration of data between different storage areas, data transmission overhead is effectively reduced while ensuring data consistency.

[0055] This technical solution, based on a pre-defined performance prediction model, can accurately predict the execution time and data transmission time of different computing units, providing a reliable basis for scheduling decisions. By constructing an overall execution latency model for inference tasks and determining the task execution time difference and task start time based on this model, precise optimization of the scheduling strategy is achieved. This model-based adaptive scheduling mechanism can dynamically adjust the scheduling strategy according to the real-time characteristics of the inference task, exhibiting stronger adaptability and superior performance compared to static scheduling methods.

[0056] This technical solution is preferably applicable to inference scenarios of large language models. By dynamically identifying operator types based at least on memory access intensity and the current inference stage, it can accurately capture the differences in computational characteristics at different stages of large language model inference. In the Prefill stage, computationally intensive operators dominate, fully utilizing the high computing power advantage of the second computing unit (GPU); in the Decoding stage, memory-intensive operators dominate, fully utilizing the high memory access bandwidth advantage of the first computing unit (PIM). This adaptive scheduling strategy for different inference stages can continuously optimize performance throughout the inference process, significantly reducing end-to-end latency in large language model inference. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below, forming part of the present invention. The illustrative embodiments of the present invention and their descriptions explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0058] Figure 1 A flowchart of a task scheduling method based on heterogeneous computing provided in one embodiment of the present invention;

[0059] Figure 2 This is a flowchart illustrating the implementation of unified memory management in one embodiment of the present invention.

[0060] Figure 3 This is a structural block diagram of a task scheduling device based on heterogeneous computing provided in one embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. In the description of this invention, it should be noted that the term "or" is generally used to include the meaning of "and / or," unless otherwise expressly stated. In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly.

[0062] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0063] Inference in Large Language Models (LLMs) typically consists of two phases: a prefill phase and a decoding phase. In the prefill phase, the LLM processes the complete input sequence in parallel, primarily performing General Matrix Multiply (GEMM) operations, a computationally intensive task. In the decoding phase, the model generates output tokens one by one. This phase involves extensive read and write operations on the key-value cache (KV-Cache), performing General Matrix-Vector Multiplication (GEMV) operations, essentially a memory-intensive task.

[0064] Traditional Graphics Processing Units (GPUs), with their powerful parallel computing capabilities, excel at handling large-scale matrix operations in the Prefill stage. However, when dealing with memory-intensive tasks in the Decoding stage, GPUs face a severe memory wall problem. Because GPU computation speed is much faster than memory access speed, when performing GEMV operations that require frequent access to the KV-Cache, GPU computing units spend a significant amount of time in an idle state waiting for data, leading to a significant decrease in computing resource utilization. The repeated data transfer between GPU computing units and video memory generates substantial latency overhead, becoming a major bottleneck restricting LLM inference performance.

[0065] To overcome the memory access bottleneck of traditional GPU architectures, Processing-In-Memory (PIM) architectures have emerged. PIM achieves tight coupling between computation and storage by integrating computing units near or inside memory, significantly reducing data access latency and making it particularly suitable for memory-intensive tasks. However, PIM architectures have lower general-purpose computing and parallel processing capabilities than GPUs, resulting in poor performance when handling computationally intensive tasks. Therefore, combining GPUs and PIM to build heterogeneous computing systems, fully leveraging their complementary advantages, has become an important development direction for accelerating LLM inference.

[0066] Existing GPU-PIM heterogeneous computing solutions typically employ static or coarse-grained task partitioning and data placement strategies. Overcoming the shortcomings of uneven heterogeneous computing power utilization and high data collaboration overhead caused by static and coarse-grained task partitioning and data placement in existing solutions, and thus achieving efficient collaborative acceleration and high throughput with low latency for large model inference, has become an urgent technical problem to be solved.

[0067] like Figure 1 As shown, to address the problems existing in the prior art, one embodiment of this application provides a task scheduling method based on heterogeneous computing, which can be applied to computer devices. The method may include:

[0068] Step S101: Obtain the inference task of the large language model. The inference task includes multiple operators.

[0069] In some implementations, inference tasks refer to the processing by which a large language model generates output results based on input data. Specifically, inference tasks can be text generation tasks, question answering tasks, dialogue tasks, translation tasks, or other natural language processing tasks. Inference tasks typically include two main stages: a prefill stage and a decoding stage. The prefill stage processes the input sequence and generates initial output, while the decoding stage generates subsequent output tags one by one.

[0070] In some implementations, the multiple operators in the inference task include, but are not limited to, matrix multiplication operators, attention mechanism operators, normalization operators, activation function operators, and embedding lookup operators. Different operators have different computational and memory access characteristics. For example, matrix multiplication operators usually have high computational density, while embedding lookup operators and certain operations in attention mechanisms usually have high memory access density.

[0071] In some implementations, multiple operators of an inference task are organized according to a predetermined computation graph or execution flow. When acquiring an inference task, the computer device can simultaneously acquire the computation graph information of the inference task. This computation graph information describes the dependencies and data flow between the various operators. Based on this computation graph information, the computer device can determine the execution order of the operators and the data transfer relationships between them, providing a foundation for subsequent operator identification and task scheduling.

[0072] In some implementations, the inference task also includes task-related contextual information, such as feature parameters like the current input sequence length, the length of the generated output sequence, and the batch size. These feature parameters affect the computational and memory access costs of each operator, thereby influencing the operator's memory access intensity and execution performance on different computing units. The computer device can acquire these feature parameters along with the inference task for subsequent operator identification and performance prediction.

[0073] Step S102: During the execution of the inference task, at least based on the memory access intensity and the current inference stage, identify memory-intensive operators and computation-intensive operators in the inference task.

[0074] In some implementations, step S102 specifically includes: acquiring the memory access operation characteristics and computation operation characteristics of the operator; determining the memory access density of the operator based on the memory access operation characteristics and computation operation characteristics; determining a judgment threshold for distinguishing between memory-intensive operators and computation-intensive operators according to the current inference stage; identifying the corresponding operator as a memory-intensive operator when the memory access density is greater than the judgment threshold; and identifying the corresponding operator as a computation-intensive operator when the memory access density is less than or equal to the judgment threshold.

[0075] In some implementations, the memory access operation characteristics include the number of memory access operations required during operator execution, the amount of data accessed in memory, and the memory access pattern. For example, for an embedding lookup operator, its memory access operation characteristics may include the number of embedding vectors to be searched, the dimension of each embedding vector, and the number of times the embedding table is accessed.

[0076] In some implementations, the computational operation characteristics include the number of arithmetic operations required during operator execution, the complexity of the operation, etc. For example, for a matrix multiplication operator, its computational operation characteristics may include the dimension of the matrix, the number of multiplication operations to be performed, the number of addition operations, etc.

[0077] In some implementations, when determining the memory access density of an operator based on memory access and computation characteristics, it can be determined by calculating the ratio of memory access operations to computation operations. That is, memory access density characterizes the proportional relationship between memory access operations and computation operations during operator execution. For example, the total number of bytes accessed during operator execution can be compared with the total number of floating-point operations to obtain a memory access density index. A higher memory access density indicates that the operator requires more memory access operations relative to computation operations during execution, and is therefore more suitable for execution on computing units with high memory bandwidth. Conversely, a lower memory access density indicates that computation operations dominate, and is therefore more suitable for execution on computing units with high computational capabilities.

[0078] In some implementations, the decision threshold is determined based on the characteristics of the current inference stage. Since different stages of inference in a large language model have different computational characteristics, the decision threshold needs to be dynamically adjusted according to the current inference stage. For example, in the Prefill stage, the input sequence is long, and computationally intensive operators such as matrix multiplication have a large computational cost. In this case, a higher decision threshold can be set so that more operators are identified as computationally intensive operators. In the Decoding stage, only one tag is generated each time, the sequence length is short, and the memory access overhead of memory-intensive operators such as attention mechanisms and embedding lookups is relatively more prominent. In this case, a lower decision threshold can be set so that more operators are identified as memory-intensive operators.

[0079] In some implementations, the decision threshold can be preset based on experimental data or performance analysis results and stored in the system configuration. When the computer device performs an inference task, it reads the corresponding decision threshold from the configuration according to the current inference stage.

[0080] In other implementations, the decision threshold can also be dynamically calculated based on the characteristic parameters of the current inference task. For example, the decision threshold can be determined by a preset threshold calculation formula or lookup table based on parameters such as the current sequence length and batch size.

[0081] In some implementations, the identification of memory-intensive operators can be pre-analyzed before the inference task is executed, or it can be analyzed in real time during the inference task. For known large language model architectures, the memory-intensive characteristics of various operators at different inference stages can be analyzed in advance to establish a mapping relationship between operator type and memory-intensiveness. During the inference task, the corresponding memory-intensiveness can be directly found based on the operator type and the current inference stage, and compared with a judgment threshold. For novel or custom operators, real-time analysis can be performed upon first encounter to calculate and identify their memory-intensiveness.

[0082] In some implementations, the same operator may be identified as different types at different inference stages. For example, when processing long sequences in the Prefill stage, the attention mechanism operator may exhibit strong computationally intensive characteristics due to its large matrix dimension; while when processing a single tag in the Decoding stage, the smaller matrix dimension leads to more prominent memory access overhead, potentially resulting in memory-intensive characteristics. Therefore, this embodiment of the application, by combining dynamic identification with the current inference stage, can more accurately grasp the real-time computational characteristics of the operator, thereby achieving better task scheduling.

[0083] Step S103: Assign memory-intensive operators to the first computing unit and computation-intensive operators to the second computing unit. The first computing unit is used to execute the tasks corresponding to the memory-intensive operators, and the second computing unit is used to execute the tasks corresponding to the computation-intensive operators.

[0084] In some implementations, the first computing unit is an in-memory computing unit, or PIM, which is more suitable for executing memory-intensive operators due to its high memory bandwidth and low memory access latency; the second computing unit is a graphics processing unit, or GPU, which has powerful parallel computing capabilities and high throughput, and is therefore suitable for executing computationally intensive operators.

[0085] In some implementations, the operator allocation process is performed dynamically based on the operator identification results. Since the same operator may be identified as different types at different inference stages, it may be allocated to different computing units at different inference stages. This dynamic allocation mechanism can flexibly adjust the execution position of operators according to the real-time characteristics of the inference task, ensuring that each operator is executed on the most suitable computing unit.

[0086] In some implementations, before assigning operators to specific computing units, different allocation schemes can be evaluated based on a preset performance prediction model. Specifically, for operators with memory access intensity close to a threshold, their execution time in the first and second computing units can be predicted respectively, and the corresponding data transfer overhead can be considered to select the computing unit with the shorter expected execution time for allocation. For operators with memory access intensity significantly higher or lower than the threshold, allocation can be made directly based on the memory access intensity judgment result without additional performance prediction evaluation, thereby reducing scheduling overhead.

[0087] Step S104: Obtain the estimated execution time of the first computing unit and the second computing unit for executing their respective tasks, as well as the data transmission time between the computing units.

[0088] In some implementations, step S104 specifically includes: obtaining feature parameters of the inference task, the feature parameters including at least the current sequence length; based on a preset performance prediction model, predicting a first estimated execution time for the first computing unit to execute the task corresponding to the memory-intensive operator, according to the feature parameters and the characteristics of the memory-intensive operator; based on the performance prediction model, predicting a second estimated execution time for the second computing unit to execute the task corresponding to the computationally intensive operator, according to the feature parameters and the characteristics of the computationally intensive operator; and based on the performance prediction model, predicting the data transmission time between the first computing unit and the second computing unit, according to the amount of data to be transmitted between the first computing unit and the second computing unit.

[0089] In some implementations, the feature parameters of the inference task, in addition to the current sequence length, may also include parameters that affect the execution performance of operators, such as batch size, length of generated output sequences, hidden layer dimension of the model, and number of attention heads. These feature parameters determine the specific computational and memory access scale of each operator, thereby affecting the execution time of the operator on different computing units. The computer device acquires these feature parameters in real time during the execution of the inference task, providing input for performance prediction.

[0090] In some implementations, the performance prediction model is a prediction model built based on performance benchmark test data. Specifically, it can be implemented using a lookup table. That is, performance benchmark tests are performed in advance on the first and second computing units under different combinations of feature parameters and operator configurations to measure the actual execution time, and the test results are stored in a performance lookup table. When performing performance prediction, the matching or closest entry is searched in the performance lookup table according to the current feature parameters and operator characteristics to obtain the corresponding estimated execution time. For parameter combinations not covered in the lookup table, the execution time can be estimated by interpolation.

[0091] In some implementations, when multiple tasks are executed in parallel, the performance prediction model may also include a regression model to more accurately predict execution time. When multiple tasks are executed in parallel, resource contention may exist between different tasks, leading to an increase in the execution time of a single task compared to independent execution. The regression model can be trained based on historical execution data to learn performance interference patterns during multi-task parallelism and adjust the estimated execution time according to the current task parallelism.

[0092] In some implementations, the characteristics of memory-intensive operators include the operator type, the amount of data the operator needs to access, and the access pattern. Based on these characteristics and feature parameters, the performance prediction model can predict the time required for the operator to execute on the first computing unit. Similarly, the characteristics of computationally intensive operators include the operator type, the amount of computation the operator needs to perform, and the matrix dimension. Based on these characteristics, the performance prediction model predicts the time required for the operator to execute on the second computing unit.

[0093] In some implementations, the prediction of data transmission time is based on the data transmission bandwidth between the first and second computing units and the amount of data to be transmitted. The amount of data to be transmitted depends on the data dependencies between operators and the operator allocation scheme. The performance prediction model determines the time required for data transmission based on the amount of data to be transmitted and the transmission bandwidth between the first and second computing units. In some implementations, the prediction of data transmission time may also consider factors such as the overhead of the transmission protocol and the data transmission initiation delay.

[0094] In some implementations, although steps S102 and S103 identify and initially allocate operators based on memory access intensity and the current inference stage, for some boundary operators with memory access intensity close to the decision threshold, the allocation decision may need to be further combined with performance prediction results. The computer device can use a performance prediction model to predict the execution time of the boundary operators on the first and second computing units respectively, and take into account the corresponding data transfer overhead, thereby selecting an allocation scheme with lower overall overhead.

[0095] In some implementations, the estimated execution time and data transmission time obtained in step S104 will be used for scheduling decision optimization in subsequent steps. By accurately predicting the task execution time of the first and second computing units, as well as the data transmission time between them, the computer device can reasonably arrange the task start times of the two computing units, achieve task execution time alignment, maximize the advantages of parallel execution, and minimize the overall execution delay of the inference task.

[0096] Step S105: Based on the estimated execution time and data transmission time, and with the goal of minimizing the overall execution delay of the inference task, determine the task start time of the first computing unit and the second computing unit respectively.

[0097] In some implementations, step S105 specifically includes: constructing an overall execution delay model for the inference task based on the first estimated execution time, the second estimated execution time, and the data transmission time; determining the task execution time difference between the first computing unit and the second computing unit according to the overall execution delay model; and determining the task start time of each of the first computing unit and the second computing unit according to the task execution time difference, so as to achieve task execution time alignment between the first computing unit and the second computing unit and minimize the overall execution delay of the inference task.

[0098] In some implementations, the overall execution latency model is used to describe the total time from the start of an inference task to the completion of all operators. The overall execution latency is influenced by several factors, including the time required for the first computational unit to execute memory-intensive operators, the time required for the second computational unit to execute compute-intensive operators, and the data transfer time between the two computational units. Since the first and second computational units can execute their respective tasks in parallel, the overall execution latency is not simply the sum of these individual times; rather, it depends on the execution timing of the two computational units and the data dependencies between them.

[0099] In some implementations, when constructing the overall execution latency model, it is necessary to consider the dependencies between operators and the execution order constraints. Specifically, based on the computation graph of the inference task, it can be determined which operators have data dependencies and how these dependencies affect the execution timing of the task. Based on these dependencies, the overall execution latency model can be represented as a timing model containing multiple execution stages, each stage corresponding to one or more operators that can be executed in parallel.

[0100] In some implementations, the task execution time difference refers to the time difference between the completion times of the tasks of the first computing unit and the second computing unit. In order to achieve task execution time alignment, even if the two computing units complete their tasks as simultaneously as possible, it is necessary to calculate a suitable start time difference based on the estimated execution time.

[0101] Specifically, the goal of task execution time alignment is to reduce the idle waiting time of computing resources. If the completion times of two computing units are significantly different, the computing unit that completes its task first will have to wait for the other computing unit to complete before it can proceed with subsequent processing, resulting in a waste of computing resources. By adjusting the start time to achieve execution time alignment, the two computing units can enter the next round of processing as soon as possible after completing their respective tasks, thereby improving the utilization rate of computing resources and reducing the overall execution latency.

[0102] Step S106: Control the first computing unit and the second computing unit to execute the corresponding tasks asynchronously and in parallel according to the task start time.

[0103] In some implementations, step S106 specifically includes: triggering the first computing unit to start executing the task corresponding to the memory-intensive operator at the task startup time of the first computing unit; triggering the second computing unit to start executing the task corresponding to the computationally intensive operator at the task startup time of the second computing unit; the first computing unit and the second computing unit execute their respective tasks independently without waiting for the other to complete, so as to achieve asynchronous parallel execution.

[0104] In some implementations, asynchronous parallel execution refers to the first and second computing units starting their tasks independently at their respective start times and proceeding independently during execution without strict synchronization. Unlike synchronous parallel execution, asynchronous parallel execution allows the two computing units to execute their tasks at different speeds, with necessary synchronization only occurring at nodes where data dependencies exist. This execution mode can more fully utilize the computing resources of the two computing units and avoid resource idleness caused by forced synchronization.

[0105] In some implementations, the computer device sends task start instructions to a first computing unit and a second computing unit, respectively. When the task start time for the first computing unit is reached, a start instruction is sent to the first computing unit, triggering it to retrieve the task corresponding to the memory-intensive operator from its task queue and begin execution. Similarly, when the task start time for the second computing unit is reached, a start instruction is sent to the second computing unit, triggering it to begin executing the task corresponding to the compute-intensive operator. Both computing units begin execution immediately upon receiving the start instruction, without blocking each other.

[0106] The completion of asynchronous parallel execution is marked by both computing units completing all tasks in their respective task queues and all necessary data transfers being completed. The computer device determines whether the inference task has been fully completed by tracking the execution status of each operator. When the last operator completes execution, the inference task ends, and the computer device can either output the final inference result or proceed to the next inference iteration.

[0107] like Figure 2 In some implementations, the above-described task scheduling method based on heterogeneous computing also includes the following steps:

[0108] Step S201: Establish a unified memory address space for the first computing unit and the second computing unit.

[0109] In some implementations, a unified memory address space refers to a first computing unit and a second computing unit sharing the same virtual address space. This allows the two computing units to access data using the same memory addresses without the need for complex address mapping and translation. By establishing a unified memory address space, computer devices can simplify data sharing mechanisms, enabling operators allocated on different computing units to transparently access the required data, thereby reducing programming complexity and data management overhead.

[0110] In some implementations, the establishment of a unified memory address space can be based on unified virtual addressing technology. The operating system or runtime system of the computer device configures a unified virtual address mapping mechanism for the first computing unit and the second computing unit, mapping virtual addresses to actual physical storage locations. The first storage region corresponding to the first computing unit and the second storage region corresponding to the second computing unit correspond to different physical storage devices, but they appear as a continuous or unified address range in the virtual address space, so that upper-layer applications and operators do not need to be aware of the differences in the underlying physical storage.

[0111] Step S202: During the execution of the inference task, at least according to the data access requirements of the memory-intensive operators and the computation-intensive operators, the data frequently accessed by the memory-intensive operators is dynamically stored in the first storage area corresponding to the first computing unit, and the data frequently accessed by the computation-intensive operators is dynamically stored in the second storage area corresponding to the second computing unit.

[0112] In some implementations, the first storage region is a memory tightly coupled to the first computing unit, such as a storage array within the in-memory computing unit or near-memory memory. The first storage region has the characteristics of high memory bandwidth and low access latency, making it suitable for storing data frequently accessed by memory-intensive operators.

[0113] In some implementations, the second storage region is a memory associated with the second computing unit, such as the video memory of a graphics processor. The second storage region has a large capacity and access characteristics suitable for parallel computing, making it suitable for storing data required by computationally intensive operators.

[0114] Specifically, during the execution of inference tasks, the computer device tracks the access frequency and access patterns of data by each operator, identifies which data is frequently accessed by memory-intensive operators and which data is frequently accessed by computation-intensive operators. For data frequently accessed by memory-intensive operators, such as KV-Cache data, the computer device dynamically migrates or preloads it into the first storage area, enabling the first computing unit to access this data with high bandwidth and low latency when executing memory-intensive operators. For data frequently accessed by computation-intensive operators, such as the input and intermediate results of large-scale matrix operations, the computer device stores it in the second storage area so that the second computing unit can access it efficiently.

[0115] In some implementations, certain data may be accessed by different types of operators at different stages of the inference task, and the computer device dynamically adjusts the storage location of the data according to the current inference stage and the type of operator to be executed.

[0116] Step S203: Based on the execution requirements of the operator, the data is prefetched or migrated between the first storage area and the second storage area.

[0117] In some implementations, data prefetching refers to transferring data from one storage area to another before the operator actually needs it, thus hiding data transfer latency. Based on analysis of the inference task execution flow and operator dependencies, the computer device predicts which data the upcoming operator will need to access and triggers data prefetching in advance. For example, if it is predicted that the first computing unit is about to execute a memory-intensive operator, and the data required by that operator is currently stored in the second storage area, the data is prefetched into the first storage area before the operator executes, ensuring that the data is ready when the operator executes and avoiding waiting during execution.

[0118] In some implementations, data migration refers to moving the primary storage location of data from one storage area to another based on changes in data access patterns. Unlike prefetching, migration typically involves a persistent change in the location of data, rather than just a temporary copy.

[0119] Step S204: Based on a unified memory address space, data sharing and transmission between the first computing unit and the second computing unit are realized, and data consistency between the first storage area and the second storage area is maintained.

[0120] In some implementations, data sharing means that the first computing unit and the second computing unit can access the same data through a unified memory address space without explicitly copying the data. When an operator generates output data on one computing unit, subsequent operators on another computing unit can directly access that data through its virtual address.

[0121] In some implementations, data consistency maintenance ensures that when data is modified in one storage area, copies in other storage areas reflect the latest modification or can correctly identify that a copy has become invalid.

[0122] In some implementations, a unified memory manager coordinates data sharing, transfer, and consistency maintenance between a first storage region and a second storage region. The unified memory manager maintains data location information, access permissions, and consistency status, responds to data access requests from both the first and second computing units, and triggers data prefetching, migration, or consistency synchronization operations as needed. Through a centralized management mechanism, the unified memory manager optimizes the placement and movement of data between the two storage regions, ensuring efficient and correct data access.

[0123] In one specific embodiment of this application, a task scheduling method based on heterogeneous computing is provided. Taking the execution of the self-attention mechanism in the Decoding stage of the domestic large language model Qwen as an example, the following is an illustration:

[0124] In the Decoding stage of the Qwen model, the size of the KV-Cache increases significantly when the sequence length is large, causing the General Matrix-Vector Multiplication (GEMV) operation in this stage to become a typical memory-intensive bottleneck. In this embodiment, after the computer device receives the inference input Token-i, it first performs performance prediction on the operators in the self-attention mechanism through a performance prediction model.

[0125] Specifically, the computer device acquires the memory access and computational operation characteristics of each operator in the attention mechanism. For the GEMV multiplication operator involving the V matrix and weights W_v, its memory access characteristics include the amount of data to be accessed in the large-scale KV-Cache and the number of accesses. Its computational operation characteristics include the number of multiplication and addition operations to be performed. Based on these characteristics, the computer device calculates the memory access intensity of the GEMV operator, that is, the ratio of memory access operations to computational operations.

[0126] Since we are currently in the Decoding phase, only one token is generated at a time, resulting in a relatively short sequence length. At this stage, the amount of KV-Cache data that the GEMV operator needs to access is far greater than the actual computational cost, leading to high memory access intensity. Simultaneously, based on the characteristic that the current inference phase is the Decoding phase, the computer device determines a threshold to distinguish between memory-intensive and computationally intensive operators. During the Decoding phase, because memory access overhead is relatively more prominent, the threshold is set relatively low to identify more operators with obvious memory access characteristics as memory-intensive operators.

[0127] The computer device compares the memory access intensity of the GEMV operator with the decision threshold and finds that its memory access intensity is significantly greater than the decision threshold. Therefore, the GEMV multiplication operator is identified as a memory-intensive operator. Correspondingly, the Softmax operation operator in the self-attention mechanism mainly involves a large number of exponential and normalization operations. The computational operation is dominant, and the memory access intensity is lower than the decision threshold. Therefore, it is identified as a computationally intensive operator.

[0128] After identifying the operator type, the computer equipment further verifies the rationality of task allocation through a performance prediction model. The performance prediction module predicts the execution time of the GEMV operator on both the GPU and PIM architecture based on characteristic parameters such as the current sequence length and batch size. The prediction results show that, because the GEMV task requires frequent access to large-scale KV-Cache data, execution on the GPU will result in a large number of computing units being idle due to the memory wall problem. The PIM architecture, however, can complete the task with higher memory bandwidth and lower memory access latency. Therefore, offloading the GEMV task to the PIM architecture can yield significant time gains.

[0129] Based on the operator identification results and performance prediction results, the computer device assigns the memory-intensive GEMV multiplication operator to the first computing unit (PIM) for execution, while reserving the computationally intensive Softmax operation operator for execution on the second computing unit (GPU).

[0130] After task allocation is completed, the computer device initiates a dynamic data placement strategy. Specifically, the unified memory management mechanism identifies the data that will be accessed frequently by memory-intensive operators based on the GEMV task's access requirements for the V matrix and weights W_v. Therefore, before the GEMV task is executed, the computer device prefetches key data blocks of the V matrix from main memory or GPU memory and dynamically stores them in the first storage area (PIM local storage) corresponding to the first computing unit. This ensures that the data resides in the local storage of the PIM, thereby avoiding remote data access during execution.

[0131] After data prefetching is complete, the computer device precisely controls the task execution timing of the GPU and PIM based on a cooperative scheduling strategy. According to the estimated execution time and data transfer time, the computer device determines the respective task start times for the Softmax operation on the GPU and the GEMV task on the PIM. Specifically, when the GPU begins executing the Softmax operation, the computer device simultaneously starts the GEMV task on the PIM in advance, enabling the PIM to begin reading the V matrix data from local storage and performing calculations before the GPU completes the Softmax operation.

[0132] Since the V matrix is ​​pre-stored in PIM's local storage, PIM can perform GEMV operations directly on the local storage without waiting for cross-device data transfer. This asynchronous parallel execution method eliminates the GPU's memory access wait time and avoids the overhead of transferring large amounts of data between the GPU memory and PIM storage.

[0133] After PIM completes the GEMV calculation, it writes the result back to the first storage area. Based on the data sharing mechanism of a unified memory address space, the GPU can directly read the PIM calculation result from the first storage area through a unified virtual address, without the need for complex address mapping and data copying. The GPU then continues to perform subsequent layer normalization (LayerNorm) and other operations, ultimately completing the Token-i inference.

[0134] Throughout the inference process, the unified memory management mechanism continuously maintains data consistency between GPU memory and PIM storage. When the GPU needs to access data in PIM storage, the unified memory manager is responsible for coordinating data access to ensure that the GPU can read the latest data. For other data that may be accessed by both the GPU and PIM, the unified memory manager dynamically adjusts the data storage location according to the access pattern and performs small data migration when necessary (the migration granularity is related to the KV dimension of the model, and the magnitude can be referenced from the memory paging mechanism, such as 2MB) to minimize cross-device data transfer overhead.

[0135] Compared with the prior art, this embodiment can produce at least the following beneficial effects:

[0136] First, overall inference performance is significantly improved. By using a dynamic operator identification mechanism based on memory access intensity and the current inference stage, combined with adaptive cooperative scheduling of the performance prediction model, the task execution of GPU and PIM achieves precise time alignment, eliminating waiting and synchronization overhead between heterogeneous devices. During the execution of the self-attention mechanism, GPU and PIM can work asynchronously and in parallel efficiently, effectively optimizing and hiding the bottlenecks in the entire inference process, thereby reducing end-to-end inference latency.

[0137] Secondly, hardware resource utilization is improved. By dynamically identifying memory-intensive and computationally intensive operators during the decoding stage and performing precise task allocation, the GPU can focus on computationally intensive tasks such as Softmax, fully leveraging its high parallelism advantage; while PIM focuses on memory-intensive tasks such as GEMV, fully leveraging its low memory latency and high memory bandwidth advantage. This dynamic, fine-grained task allocation based on the characteristics of the current inference stage avoids idle computing units caused by frequent data waiting when the GPU is executing memory-intensive tasks, thus improving the utilization of each computing unit in a heterogeneous computing environment.

[0138] Furthermore, data access overhead is significantly reduced. Through a dynamic data placement strategy, frequently accessed V matrix and KV-Cache data are dynamically stored in PIM local storage. This transforms remote data access, which originally required PCIe or other interconnect buses, into PIM's near-end memory computation. This data placement optimization reduces the amount of large-scale data transferred between GPU memory and PIM storage, lowers data transfer latency, and enables PIM to complete memory-intensive tasks with higher efficiency.

[0139] Furthermore, by unifying the memory address space and data consistency maintenance mechanism, the complexity of data management in heterogeneous computing environments is simplified. GPUs and PIMs can transparently access shared data through a unified virtual address space without explicit data copying and address translation operations, reducing programming complexity and runtime overhead, while ensuring the correctness and consistency of data access.

[0140] like Figure 3 One embodiment of this application provides a task scheduling device based on heterogeneous computing, which can be applied in a computer device. The device includes:

[0141] The task acquisition module 310 is used to acquire the inference task of the large language model, which includes multiple operators.

[0142] The operator identification module 320 is used to identify memory-intensive operators and computation-intensive operators in the inference task during the execution of the inference task, based at least on memory access intensity and the current inference stage.

[0143] The task allocation module 330 is used to allocate memory-intensive operators to the first computing unit and computation-intensive operators to the second computing unit. The first computing unit is used to execute the tasks corresponding to the memory-intensive operators, and the second computing unit is used to execute the tasks corresponding to the computation-intensive operators.

[0144] The performance prediction module 340 is used to obtain the estimated execution time of the first computing unit and the second computing unit for executing their respective tasks, as well as the data transmission time between the computing units.

[0145] The scheduling decision module 350 is used to determine the task start time of the first computing unit and the second computing unit respectively, based on the estimated execution time and data transmission time, with the goal of minimizing the overall execution delay of the inference task.

[0146] The task execution control module 360 ​​is used to control the first computing unit and the second computing unit to execute the corresponding tasks asynchronously and in parallel according to the task start time.

[0147] In some implementations, the operator identification module 320 is further used for:

[0148] Obtain the memory access and computational operation characteristics of the operator;

[0149] The memory access density of the operator is determined based on the characteristics of memory access operations and computational operations.

[0150] Based on the current inference stage, determine the threshold for distinguishing between memory-intensive operators and computation-intensive operators;

[0151] When the memory access intensity is greater than the judgment threshold, the corresponding operator is identified as a memory access intensive operator;

[0152] When the memory access intensity is less than or equal to the judgment threshold, the corresponding operator is identified as a computationally intensive operator.

[0153] In some implementations, the performance prediction module 340 is further configured to:

[0154] Obtain the feature parameters of the inference task; the feature parameters must include at least the current sequence length.

[0155] Based on the preset performance prediction model, and according to the feature parameters and the characteristics of memory-intensive operators, the first estimated execution time of the first computing unit executing the task corresponding to the memory-intensive operator is predicted.

[0156] Based on the performance prediction model, according to the feature parameters and the characteristics of the computationally intensive operator, the second estimated execution time of the second computing unit to execute the corresponding task of the computationally intensive operator is predicted;

[0157] Based on the performance prediction model, the data transmission time between the first computing unit and the second computing unit is predicted according to the amount of data that needs to be transmitted between the first computing unit and the second computing unit.

[0158] In some implementations, the scheduling decision module 350 is also used for:

[0159] Based on the first estimated execution time, the second estimated execution time, and the data transmission time, the task execution time difference between the first computing unit and the second computing unit is determined.

[0160] Based on the task execution time difference, the task start time of the first computing unit and the second computing unit is determined to achieve task execution time alignment between the first computing unit and the second computing unit, thereby minimizing the overall execution delay of the inference task.

[0161] In some implementations, the task execution control module 360 ​​is further configured to:

[0162] At the start time of the task in the first computing unit, the first computing unit is triggered to start executing the task corresponding to the memory-intensive operator;

[0163] At the task startup time of the second computing unit, the second computing unit is triggered to start executing the task corresponding to the computationally intensive operator;

[0164] The first and second computing units execute their respective tasks independently without waiting for the other to complete, thus achieving asynchronous parallel execution.

[0165] In some implementations, the heterogeneous computing-based task scheduling device further includes a unified memory management module 370, used for:

[0166] Establish a unified memory address space for the first computing unit and the second computing unit;

[0167] During the execution of the inference task, at least according to the data access needs of the memory-intensive operator and the computation-intensive operator, the data frequently accessed by the memory-intensive operator is dynamically stored in the first storage area corresponding to the first computing unit, and the data frequently accessed by the computation-intensive operator is dynamically stored in the second storage area corresponding to the second computing unit.

[0168] Based on the execution requirements of the operator, the data is prefetched or migrated between the first storage area and the second storage area;

[0169] Based on a unified memory address space, data sharing and transmission between the first computing unit and the second computing unit are realized, and data consistency between the first storage area and the second storage area is maintained.

[0170] It should be noted that the aforementioned heterogeneous computing-based task scheduling device and the aforementioned heterogeneous computing-based task scheduling method originate from the same inventive concept, and are based on the same technical principles and solutions. Therefore, the technical details, parameter settings, implementation methods, etc., of the heterogeneous computing-based task scheduling device can be referred to the relevant description of the aforementioned heterogeneous computing-based task scheduling method, and will not be repeated here. Those skilled in the art should understand that any implementation method of the heterogeneous computing-based task scheduling method can be applied to the heterogeneous computing-based task scheduling device, and vice versa.

[0171] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the heterogeneous computing-based task scheduling method described above.

[0172] One embodiment of this application provides a computer device, which includes the above-described task scheduling device based on heterogeneous computing.

[0173] It should be noted that the task scheduling device based on heterogeneous computing provided in the above embodiments is only illustrated by the division of the above functional modules when performing task scheduling. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the task scheduling device based on heterogeneous computing can be divided into different functional modules to complete all or part of the functions described above.

[0174] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0175] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

Claims

1. A task scheduling method based on heterogeneous computing, characterized in that, The method includes: Obtain the inference task of a large language model, wherein the inference task includes multiple operators; During the execution of the inference task, memory-intensive operators and computation-intensive operators in the inference task are identified based on memory access intensity and the current inference stage. The memory-intensive operator is assigned to a first computing unit, and the computation-intensive operator is assigned to a second computing unit. The first computing unit is used to execute the task corresponding to the memory-intensive operator, and the second computing unit is used to execute the task corresponding to the computation-intensive operator. The estimated execution time of the first computing unit and the second computing unit for executing their respective tasks, as well as the data transmission time between the computing units, are obtained. Based on the estimated execution time and the data transmission time, with the goal of minimizing the overall execution delay of the inference task, the task start time of the first computing unit and the second computing unit is determined respectively. The first computing unit and the second computing unit are controlled to execute their respective tasks asynchronously and in parallel according to the task start time. Specifically, during the execution of the inference task, identifying memory-intensive and computationally-intensive operators in the inference task based on memory access intensity and the current inference stage includes: Obtain the memory access operation characteristics and computation operation characteristics of the operator; determine the memory access intensity of the operator based on the memory access operation characteristics and the computation operation characteristics; determine a judgment threshold for distinguishing the memory-intensive operator and the computation-intensive operator according to the current inference stage; when the memory access intensity is greater than the judgment threshold, identify the corresponding operator as the memory-intensive operator; when the memory access intensity is less than or equal to the judgment threshold, identify the corresponding operator as the computation-intensive operator.

2. The task scheduling method based on heterogeneous computing according to claim 1, characterized in that, The method further includes: Establish a unified memory address space for the first computing unit and the second computing unit; During the execution of the inference task, according to the data access requirements of the memory-intensive operator and the computation-intensive operator, the data frequently accessed by the memory-intensive operator is dynamically stored in the first storage area corresponding to the first computing unit, and the data frequently accessed by the computation-intensive operator is dynamically stored in the second storage area corresponding to the second computing unit. Based on the execution requirements of the operator, the data is prefetched or migrated between the first storage area and the second storage area; Based on the unified memory address space, data sharing and transmission between the first computing unit and the second computing unit are realized, and data consistency between the first storage area and the second storage area is maintained.

3. The task scheduling method based on heterogeneous computing according to claim 2, characterized in that, The first computing unit is an in-memory computing unit, and the first storage area is a memory area that the in-memory computing unit can directly access. The second computing unit is a graphics processor, and the second storage area is the video memory of the graphics processor.

4. The task scheduling method based on heterogeneous computing according to claim 1, characterized in that, The step of obtaining the estimated execution time of the first computing unit and the second computing unit for executing their respective tasks, and the data transmission time between the computing units, includes: Obtain the feature parameters of the inference task, including the current sequence length; Based on a preset performance prediction model, and according to the feature parameters and the characteristics of the memory-intensive operator, the first estimated execution time of the first computing unit executing the task corresponding to the memory-intensive operator is predicted. Based on the performance prediction model, and according to the feature parameters and the characteristics of the computationally intensive operator, the second estimated execution time for the second computing unit to execute the task corresponding to the computationally intensive operator is predicted. Based on the performance prediction model, the data transmission time between the first computing unit and the second computing unit is predicted according to the amount of data that needs to be transmitted between the first computing unit and the second computing unit.

5. The task scheduling method based on heterogeneous computing according to claim 4, characterized in that, The step of determining the task start time of the first computing unit and the second computing unit respectively, based on the estimated execution time and the data transmission time, with the goal of minimizing the overall execution latency of the inference task, includes: Based on the first estimated execution time, the second estimated execution time, and the data transmission time, the task execution time difference between the first computing unit and the second computing unit is determined. Based on the task execution time difference, the task start time of the first computing unit and the second computing unit is determined to achieve task execution time alignment between the first computing unit and the second computing unit, thereby minimizing the overall execution delay of the inference task.

6. The task scheduling method based on heterogeneous computing according to claim 5, characterized in that, The control of the first computing unit and the second computing unit to asynchronously and in parallel execute the corresponding tasks according to the task start time includes: At the task startup time of the first computing unit, the first computing unit is triggered to start executing the task corresponding to the memory-intensive operator; At the task startup time of the second computing unit, the second computing unit is triggered to start executing the task corresponding to the computationally intensive operator; The first computing unit and the second computing unit each execute their respective tasks independently without waiting for the other to complete, thus achieving asynchronous parallel execution.

7. A task scheduling device based on heterogeneous computing, characterized in that, include: The task acquisition module is used to acquire the inference task of the large language model, wherein the inference task includes multiple operators; The operator identification module is used to identify memory-intensive operators and computation-intensive operators in the inference task based on memory access intensity and the current inference stage during the execution of the inference task. The operator identification module is also used to: acquire the memory access operation characteristics and computation operation characteristics of the operator; Based on the memory access operation characteristics and the computation operation characteristics, the memory access intensity of the operator is determined; according to the current inference stage, a judgment threshold for distinguishing between the memory access-intensive operator and the computation-intensive operator is determined. When the memory access intensity is greater than the determination threshold, the corresponding operator is identified as the memory access intensive operator; when the memory access intensity is less than or equal to the determination threshold, the corresponding operator is identified as the computation intensive operator. The task allocation module is used to allocate the memory-intensive operator to a first computing unit and the computation-intensive operator to a second computing unit. The first computing unit is used to execute the task corresponding to the memory-intensive operator, and the second computing unit is used to execute the task corresponding to the computation-intensive operator. The performance prediction module is used to obtain the estimated execution time of the first computing unit and the second computing unit for executing the corresponding tasks, as well as the data transmission time between the computing units; The scheduling decision module is used to determine the task start time of the first computing unit and the second computing unit respectively, based on the estimated execution time and the data transmission time, with the goal of minimizing the overall execution delay of the inference task; The task execution control module is used to control the first computing unit and the second computing unit to asynchronously and in parallel execute the corresponding tasks according to the task start time.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the task scheduling method based on heterogeneous computing as described in any one of claims 1-6.

9. A computer device, characterized in that, The computer device includes the task scheduling device based on heterogeneous computing as described in claim 7.

Images