Heterogeneous processor systems, task scheduling methods, electronic devices and storage media

Abstract

This disclosure provides a heterogeneous processor system, a task scheduling method, an electronic device, and a storage medium. The heterogeneous processor system includes a first chip, a second chip, and a memory chip. The first chip is a graphics processing unit (GPU) chip, and the second chip is a tensor stream processor (TSP) chip. The second chip and the first chip are stacked and integrated using a hybrid bonding technology. The memory chip is stacked and integrated with the first chip to provide data storage. The first chip is used to schedule computational tasks, and the first and second chips collaboratively execute the computational tasks. This technical solution allows the GPU and TSP chips to work together to execute computational tasks, significantly improving hardware efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a heterogeneous processor system, a task scheduling method, an electronic device, and a storage medium. Background Technology

[0002] In the current inference process of large language models, the Graphics Processing Unit (GPU) schedules and executes computational tasks through a hardware scheduler, and hides memory access latency and reduces the burden on software developers through a hierarchical memory architecture. GPUs have a large number of cores, can handle a large number of parallel tasks well, and have good flexibility and versatility, making them the dominant hardware for AI model inference. However, GPU architecture design has many uncertainties and a deep memory hierarchy; for AI inference tasks that are more concerned with latency and the memory bottleneck of large language model inference, high hardware efficiency cannot be achieved. Summary of the Invention

[0003] This disclosure provides a heterogeneous processor system, a task scheduling method, an electronic device, and a storage medium that can utilize graphics processor chips and tensor stream processor chips to perform computing tasks, thereby significantly improving hardware efficiency.

[0004] According to one aspect of this disclosure, a heterogeneous processor system is provided, comprising:

[0005] The first chip, which is a graphics processor chip;

[0006] The second chip, which is a tensor stream processor chip, is stacked and integrated with the first chip using a hybrid bonding technology.

[0007] A memory chip, stacked and integrated with the first chip, is used to provide data storage;

[0008] The first core is used to schedule computing tasks, and the first core and the second core work together to execute the computing tasks.

[0009] Optionally, the second core includes:

[0010] At least one tensor processing engine for performing matrix operations;

[0011] The vector engine is used to perform vector operations;

[0012] Static random access memory (SRAM) is used to store data.

[0013] Optionally, the first chip includes multiple computing units; the static random access memory is configured as a memory for sharing data among the multiple computing units and provides a high-bandwidth data access interface for the first chip.

[0014] According to one aspect of this disclosure, a task scheduling method for a heterogeneous processor system is proposed, the heterogeneous processor system including a first core and a second core stacked by hybrid bonding, the first core being a graphics processing unit (GPU) core and the second core being a tensor stream processor (TSU) core, the first core being used to schedule computational tasks, and the first core and the second core co-executing the computational tasks, the method comprising:

[0015] Retrieve the subtasks to be executed in the computation task;

[0016] The task type of the subtask is identified and processed;

[0017] If the subtask is identified as a memory-restricted type, it will be scheduled to be executed in the second core.

[0018] If the task is identified as computationally restricted, it is scheduled to be executed in the first core.

[0019] Optionally, the computational task is a large language model inference task; the memory-restricted subtask includes an attention mechanism computation task, and the computationally restricted subtask includes a feedforward network computation task or a reduction operation task.

[0020] Optionally, scheduling the attention mechanism computation task to be executed by the second core includes:

[0021] The input data in the memory connected to the first core is moved to the static random access memory of the second core;

[0022] The second core performs matrix operations on the input data in the static random access memory, obtains the operation result, and sends barrier arrival marker information to the first core;

[0023] If all the barrier arrival marker information received by the first core indicates that the preset task synchronization conditions are met, the calculation result is transferred from the static random access memory back to the memory connected to the first core.

[0024] Optionally, the input data is moved to the static random access memory, and the calculation result is moved back to the memory connected to the first core, which is executed by the replication engine in the first core or by the replication engine in the second core.

[0025] Optionally, the first chip includes a plurality of computing units, each of which includes a workgroup scheduler;

[0026] The step of transferring input data from the memory connected to the first chip to the static random access memory of the second chip includes:

[0027] The workgroup scheduler sends data replication instructions to the replication engine.

[0028] In response to the data copy instruction, a data migration operation is performed through the copy engine.

[0029] Optionally, the second core performs matrix operations, including:

[0030] The workgroup scheduler in the first core sends a matrix operation trigger command to the tensor processing engine of the second core.

[0031] In response to the matrix operation trigger instruction, the tensor processing engine performs matrix multiplication and accumulation operations on the data in the static random access memory.

[0032] Optionally, scheduling the reduction operation task to be executed in the first chip includes:

[0033] Multiple computing units of the first core compute the local reduction results in parallel and store each of the local reduction results in the static random access memory of the second core.

[0034] One of the computing units in the first chip loads all the local reduction results from the static random access memory and performs the final reduction operation.

[0035] Optionally, the task scheduling method further includes:

[0036] The first chip loads data from the static random access memory of the second chip to perform single-instruction multi-threaded operation, and stores the operation result back to the static random access memory;

[0037] The first chip sends an instruction word to the second chip to trigger the second chip to stream the operation results in the static random access memory.

[0038] Optionally, the instruction word sent by the first core to the second core is a very long instruction word, which is used to instruct the second core to perform loading, matrix multiplication and accumulation, vector addition and storage operations in sequence.

[0039] Optionally, the first chip loads data from the static random access memory of the second chip for single-instruction multi-threaded operation, including:

[0040] The thread bundle scheduler in the first core sends a single-instruction multi-threaded instruction to the arithmetic logic unit.

[0041] In response to the single instruction multithreaded instruction, the arithmetic logic unit reads data from the static random access memory and performs the operation, and writes the operation result back to the static random access memory.

[0042] Optionally, before the first chip sends an instruction to the second chip to trigger streaming processing, the method further includes:

[0043] The intermediate data in the note storage of the first core is moved to the static random access memory of the second core using a copy engine.

[0044] According to one aspect of this disclosure, an electronic device is proposed, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program is executed by the processor to implement the task scheduling method as described above.

[0045] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores one or more programs, which can be executed by one or more processors to implement the task scheduling method described above.

[0046] This disclosure discloses a heterogeneous processor system, task scheduling method, electronic device, and storage medium. The heterogeneous processor system includes a first chip, a second chip, and a memory chip. The first chip is a graphics processing unit (GPU) chip, and the second chip is a tensor stream processor (TSP) chip. The first and second chips are stacked and integrated using a hybrid bonding technique. The memory chip is stacked and integrated with the first chip and is used to provide data storage. The first chip is used to schedule computational tasks, and the first and second chips can collaboratively execute related computational tasks. In this disclosure, the heterogeneous processor system includes a GPU chip and a TSP chip. The GPU chip has strong computational capabilities, and the TSP chip has high bandwidth transmission capabilities. Therefore, by having the GPU chip and the TSP chip collaboratively execute computational tasks, it is possible to better adapt to various stages of large language model inference and improve hardware efficiency.

[0047] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0048] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0049] Figure 1 This is an architecture diagram of a heterogeneous processor system according to an embodiment of this disclosure;

[0050] Figure 2 This is a detailed architecture diagram of a heterogeneous processor system according to an embodiment of this disclosure;

[0051] Figure 3 This is a detailed architecture diagram of a heterogeneous processor system according to another embodiment of this disclosure;

[0052] Figure 4 This is a detailed structural diagram of a tensor stream processor chip according to an embodiment of this disclosure;

[0053] Figure 5 This is a detailed structural diagram of a tensor stream processor chip according to another embodiment of this disclosure;

[0054] Figure 6 This is a main flowchart of a task scheduling method according to an embodiment of the present disclosure;

[0055] Figure 7 This is a flowchart illustrating how attention mechanism computation tasks are scheduled to a tensor stream processor chip, according to one embodiment of this disclosure.

[0056] Figure 8 This is a flowchart of data transfer according to an embodiment of this disclosure;

[0057] Figure 9 This is a flowchart illustrating the execution of matrix operations by a tensor stream processor chip according to an embodiment of this disclosure;

[0058] Figure 10 This is a schematic diagram illustrating the process of performing an attention mechanism computation task according to an embodiment of this disclosure;

[0059] Figure 11 This is a schematic diagram illustrating the process of performing an attention mechanism computation task according to another embodiment of this disclosure;

[0060] Figure 12 This is a flowchart of an embodiment of the present disclosure performing a reduction operation task;

[0061] Figure 13 This is a schematic diagram illustrating the process of performing a reduction operation task according to an embodiment of this disclosure;

[0062] Figure 14This is a flowchart illustrating the execution of streaming processing according to an embodiment of this disclosure;

[0063] Figure 15 This is a flowchart illustrating the execution of a single-instruction multithreaded operation according to an embodiment of this disclosure;

[0064] Figure 16 This is a flowchart illustrating the process preceding the triggering of streaming processing in one embodiment of this disclosure;

[0065] Figure 17 This is a schematic diagram illustrating the process of performing stream processing according to an embodiment of this disclosure;

[0066] Figure 18 This is a schematic diagram illustrating the process of performing streaming processing according to another embodiment of this disclosure;

[0067] Figure 19 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0069] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0070] Graphics Processing Unit (GPU): A microprocessor specifically designed for processing graphics and image computations. Originally designed to accelerate computer graphics rendering, GPUs have expanded far beyond this scope with technological advancements. Initially designed to accelerate 2D and 3D graphics rendering, improving the performance of games and professional graphics software, GPUs possess tens of thousands of cores, enabling them to process massive amounts of data simultaneously, making them ideal for parallel computing tasks. With technological advancements, GPUs are no longer limited to graphics processing; they are now used for a variety of general-purpose computing tasks. GPUs play a crucial role in deep learning, machine learning, and artificial intelligence because they can rapidly process large amounts of data, accelerating the training and inference of neural networks.

[0071] Tensor Streaming Unit (TSU): A Tensor Streaming Unit is a core architecture for AI accelerators. It employs a unique functional slicing microarchitecture design, transforming the organization of traditional multi-core processors from "horizontally homogeneous" to "vertically heterogeneous"—the chip is divided into different functional slices in the vertical direction, including instruction control units, on-chip storage, matrix multiplication units, vector execution units, and data rearrangement units, etc. Each slice is specifically responsible for a particular type of computational task, making it particularly suitable for real-time inference of large language models and low-latency AI application scenarios.

[0072] Computing Chip: A computing chip is an unpackaged, small chip diced from a silicon wafer. It contains integrated circuits that perform specific functions. A computing chip is typically composed of one or more computing chips packaged together. The functionality of a computing chip is decomposed into multiple smaller, modular, and functionally independent computing chips. These computing chips are integrated onto a packaging substrate using packaging technology to ultimately form a complete processor. In this embodiment, the graphics processor chip is a computing chip capable of implementing graphics processor-related functions; the tensor stream processor chip is a computing chip capable of implementing tensor stream processor-related functions.

[0073] Three-Dimensional Dynamic Random Access Memory (3D-DRAM): 3D-DRAM is a memory chip that stacks traditional two-dimensional planar random access memory cells vertically using advanced semiconductor manufacturing technology to form a multi-layer structure. 3D-DRAM represents the inevitable direction of random access memory cell technology development, aiming to break through existing technological bottlenecks through vertical dimension. It is not a single technology, but a technological category, encompassing everything from commercially available 3D integration to future cell-stacked random access memory cells. The development of this technology will directly determine the performance and energy efficiency of future computing systems.

[0074] High Bandwidth Memory (HBM): HBM is a cutting-edge memory chip technology developed to solve the "memory wall" bottleneck. Its core innovation lies in vertically stacking multiple DRAM chips using through-silicon vias (TSVs) and packaging them together with processors such as GPUs or CPUs on a silicon interposer, thereby creating an ultra-wide data bus (typically 1024-bit or 2048-bit) within a very small physical space. HBM places a substrate between the stacked memory dies and computing chips, and data transmission between the memory dies and computing chips needs to be carried out through the substrate.

[0075] Dynamic Random Access Memory (DRAM): DRAM is the most common type of main memory in computers. It requires periodic refreshing to maintain data and offers high storage density and low cost. DRAM uses capacitors to store data; a charged capacitor represents a "1", and a discharged capacitor represents a "0". However, because capacitors can leak current, charge loss and data loss occur, necessitating periodic refreshing. The refresh operation is controlled by the memory controller, which periodically reads the data from each storage cell and immediately rewrites it to restore the capacitor's charge. This process is transparent to the computer system and usually does not affect normal operation.

[0076] Static Random Access Memory (SRAM): SRAM is a volatile memory based on the flip-flop principle. It maintains the data state through a latch circuit composed of transistors (typically requiring six transistors to store one bit). As long as power is not interrupted, it can statically store information without the need for periodic refresh operations like DRAM. SRAM's extremely fast access speed allows it to keep synchronized with the clock frequency of the CPU or GPU, achieving nanosecond-level read and write latency. Therefore, it is often used as a level 1, level 2, or level 3 cache inside the processor, as well as register files and on-chip memory arrays.

[0077] Compute Unit (CU): In GPU architecture, a compute unit is the basic unit for performing computational tasks. When a workgroup runs on a GPU, it is actually executing on a compute unit. A compute unit is the basic unit on a GPU used to perform parallel computing tasks; it contains a number of processing elements that can execute multiple threads in parallel.

[0078] Execution unit (EU): The execution unit is the smallest programmable execution core in the GPU. It is a multi-threaded single instruction multiple data processor responsible for executing general-purpose computational instructions such as integer, floating-point, and logic. It is a complete processor core that includes instruction pipeline, register file and multi-threaded context.

[0079] Tensor Processing Engine (TPE): A Tensor Processing Engine is a collective term for a class of technologies designed specifically for efficient tensor operations. It can be a dedicated processor that achieves disruptive performance improvements through customized hardware architecture; or a compiler and runtime system that enables high-performance machine learning inference on general-purpose hardware through software stack optimization. Both share the common goal of addressing the ever-increasing demand for massive tensor operations in artificial intelligence computing, representing important practices of the "domain-specific computing" concept at different levels.

[0080] Vector Engine (VE): A vector engine is a high-performance processor core design that performs parallel operations on multiple data elements in an array (or vector) simultaneously with a single instruction, significantly improving computational efficiency in fields such as scientific computing, artificial intelligence, and signal processing. Unlike scalar engines, which can only process one pair of data at a time, it is equipped with a large number of dedicated vector registers and powerful pipelining units, enabling it to efficiently execute complex computational tasks.

[0081] Copy Engine (CE): The main purpose of a copy engine is to improve the efficiency of moving data between different storage tiers, especially in scenarios where data needs to be frequently copied between global memory and local memory. A copy engine is an engine that optimizes data copying operations at the workgroup level. It improves data transfer efficiency and overall computing performance through asynchronous data transfer and intelligent data movement strategies.

[0082] Workgroup Scheduler: Workgroup-level instruction scheduling is a key component in GPU architecture used to optimize thread execution. In a GPU, a workgroup (WG) is a collection of threads that can share certain resources during execution, such as shared memory. WGIS is responsible for scheduling the execution of instructions within these workgroups to improve execution efficiency and resource utilization.

[0083] Thread Bundle Instruction Scheduler: The thread bundle instruction scheduler is a key hardware component inside the computing unit. In each clock cycle, the load selects one or more instructions from a set of ready thread bundles and issues them to the execution unit. It is the core mechanism for GPUs to achieve latency hiding and fine-grained multithreaded parallelism.

[0084] Arithmetic Logic Unit (ALU): A core digital circuit within a computer's central processing unit (CPU) or other computing chip (such as a GPU). It is the processor's "computing brain," responsible for performing all arithmetic and logical operations. The ALU is the execution engine of a digital computing device; it receives data and operation commands, performs the actual arithmetic and logical calculations through its internal circuitry, and outputs results and status information.

[0085] Register File (RF): Also known as the register stack, the register file is an array of registers within the GPU used to temporarily store operands, intermediate results, and variables during instruction execution. It is the fastest storage level in the processor and the closest to the arithmetic units. It is typically composed of high-speed static random access memory (SRAM) cells.

[0086] Scratchpad memory (SPM) is a high-speed memory located on the processor chip. Unlike cache, which is automatically managed by hardware, SPM is directly and explicitly controlled by software (compiler or programmer). As on-chip memory, it is typically composed of SRAM and has access speeds comparable to cache.

[0087] Barrier Objects: Barrier objects are a synchronization mechanism typically used in multithreaded or concurrent programming to control the execution order of multiple threads or processes, ensuring they execute synchronously at a certain point. The main function of a barrier object is to make a group of threads "wait" at a certain point until all threads have reached that point, and only then are they allowed to continue execution.

[0088] In the current inference process of large language models, the Graphics Processing Unit (GPU) schedules and executes computational tasks through a hardware scheduler, and hides memory access latency and reduces the burden on software developers through a hierarchical memory architecture. GPUs have a large number of cores, can handle a large number of parallel tasks well, and have good flexibility and versatility, making them the dominant hardware for AI model inference. However, GPU architecture design has many uncertainties and a deep memory hierarchy; for AI inference tasks that are more concerned with latency and the memory bottleneck of large language model inference, high hardware efficiency cannot be achieved.

[0089] Based on this, this disclosure proposes a heterogeneous processor system, a task scheduling method, an electronic device, and a computer-readable storage medium, which can utilize graphics processor chips and tensor stream processor chips to perform computing tasks in a coordinated manner, thereby significantly improving hardware efficiency.

[0090] System architecture description applied in the embodiments of this disclosure

[0091] Figure 1This is a heterogeneous processor system according to an embodiment of the present disclosure. The heterogeneous processor system includes a graphics processor chip 100, a tensor stream processor chip 200, and a memory chip 300. The graphics processor chip 100 is stacked and integrated with the tensor stream processor chip 200 through hybrid bonding. The memory chip 300 is stacked and integrated with the graphics processor chip 100, so that either the graphics processor chip 100 or the tensor stream processor chip 200 can read data from the memory chip 300.

[0092] It is worth noting that in the process of processing computing tasks, the heterogeneous processor system needs to use the graphics processor chip 100 to schedule related computing tasks, and uses the graphics processor chip 100 and the tensor stream processor chip 200 to execute related computing tasks in a coordinated manner. This allows the computing power of the graphics processor chip and the bandwidth transmission capability of the tensor stream processor chip to be utilized during the execution of computing tasks, making the execution of computing tasks faster and more efficient, and improving hardware efficiency.

[0093] Reference Figure 2 and Figure 3 When the memory chip 300 is a high-bandwidth memory 310, the graphics processor chip 100 and the high-bandwidth memory 310 are integrated into one unit. The graphics processor chip 100 and the high-bandwidth memory 310 can also be connected to the tensor stream processor chip 200 through hybrid bonding. When the memory chip 300 is a three-dimensional dynamic random access memory 320, the graphics processor chip 100 is connected to the tensor stream processor chip 200 through hybrid bonding. The tensor stream processor chip 200 can also be connected to the three-dimensional dynamic random access memory 320 through hybrid bonding.

[0094] It is worth noting that in some embodiments of this disclosure, the memory chip 300 can be 3D-DRAM or HBM. Both 3D-DRAM and HBM vertically stack multiple layers of chips using through-silicon via (TSV) technology, achieving extremely high storage density and ultra-large capacity per unit area. Furthermore, the 3D structure significantly shortens the physical distance between the data processing unit and the storage unit, and by introducing a wide data bus, it significantly improves data throughput, effectively overcoming the bandwidth bottleneck of traditional memory. Due to short-distance interconnects and optimized I / O drives, both can transmit data with lower power consumption, achieving a higher energy efficiency ratio. Through the above configuration, the memory chip 300 can not only store large amounts of data, but also allows the graphics processor chip 100 and the tensor stream processor chip 200 to quickly read data from the memory chip 300.

[0095] It is worth noting that the graphics processor chip 100 is connected to the tensor stream processor chip 200 via hybrid bonding. Hybrid bonding is a packaging technology used for three-dimensional chip integration. Its core lies in eliminating the need for traditional microbumps or solder balls, and instead achieving direct electrical and mechanical bonding between upper and lower chip layers in the same process step through an atomically flat metal-dielectric interface. This bump-free direct copper-to-copper interconnect method can achieve sub-micron or even nanometer-level interconnect spacing, thereby providing interconnect density far exceeding that of traditional technologies per unit area, significantly shortening signal transmission paths and reducing power consumption. Through the above configuration, the graphics processor chip 100 and the tensor stream processor chip 200 can rapidly transmit data.

[0096] Reference Figure 4 In some embodiments of this disclosure, a tensor stream processor chip 200 may include at least one tensor processing engine 210, a vector engine 230, and a static random access memory (SRAM) 220. The tensor processing engine 210 can perform matrix operations; the vector engine 230 can perform vector operations; the SRAM 220 can store data, and the data in the SRAM 220 of the tensor stream processor chip 200 can also be read and utilized by the graphics processor chip 100.

[0097] Reference Figure 5 In some embodiments of this disclosure, a tensor stream processor chip 200 may include a tensor processing engine 210, a vector engine 230, a replication engine 240, and a static random access memory 220, the tensor stream processor chip 200 being relative to Figure 3 The tensor stream processor chip 200 includes a replication engine 240. The replication engine 240 is capable of processing data replication and transfer.

[0098] It is worth noting that the graphics processor chip 100 can contain multiple computing units. Since the graphics processor chip 100 is connected to the tensor stream processor chip 200 via hybrid bonding, and the tensor stream processor chip 200 contains a static random access memory (SRAM) 220, the SRAM 220 can be configured as a memory for sharing data among the multiple computing units in the graphics processor chip 100. Furthermore, because the SRAM 220 has a high access speed and can be synchronized with the GPU's clock frequency to achieve nanosecond-level read / write latency, the SRAM 220 can provide a high-bandwidth data access interface for the graphics processor chip 100. The computing units in the graphics processor chip 100 can quickly read data from the SRAM 220, improving the efficiency of the graphics processor chip 100 in reading data from the SRAM 220.

[0099] Overall Implementation of the Task Scheduling Method in the Embodiments of this Disclosure

[0100] This disclosure proposes a task scheduling method applied to a heterogeneous processor system described in the above embodiments. The heterogeneous processor system includes a first chip and a second chip. The first chip is a graphics processing unit (GPU) chip, and the second chip is a tensor stream processor (TSP) chip. The GPU chip is connected to the TSP chip via a hybrid bonding stacking method. The GPU chip is used to schedule computational tasks, and the GPU chip and the TSP chip can collaboratively execute computational tasks. (Refer to...) Figure 6 Task scheduling methods include:

[0101] Step S201: Obtain the subtasks to be executed in the computation task;

[0102] Step S202: Identify and process the task type of the subtask;

[0103] Step S203: If the subtask is identified as a memory-restricted type, then the subtask is scheduled to be executed in the second core.

[0104] In step S204, if the task is identified as computationally restricted, the subtask is scheduled to be executed in the first core.

[0105] Specifically, in this disclosure, the task scheduling method proposed in the embodiments of this disclosure is applied to a heterogeneous processor system. The heterogeneous processor system includes a first chip and a second chip. The first chip is a graphics processing unit (GPU) chip, and the second chip is a tensor stream processor (TSP) chip. The GPU chip is connected to the TSP chip via a hybrid bonding stacking method, and the GPU chip can be used to schedule computational tasks. The GPU chip and the TSP chip can collaboratively execute related computational tasks. Furthermore, the first chip is also integrated with a memory chip stack. Because the first chip and the second chip can be connected via a hybrid bonding stacking method, both the first chip and the second chip can read relevant data from the memory chip. Since the second chip also contains static random access memory (SRAM), the first chip can also read data from the SRAM in the second chip.

[0106] In step S201, during the task scheduling process, the heterogeneous processor system acquires a computing task, which includes multiple subtasks to be executed. Subsequently, it is necessary to identify the task type of each subtask in the computing task in order to classify and schedule the corresponding subtasks.

[0107] It is worth noting that the computational task in this embodiment can be a large language model inference task, and the large language model inference task includes multiple different subtasks, and each subtask corresponds to a task type. For subtasks with different task attribute types, they can be scheduled to different cores for execution, making good use of the data execution advantages of different cores, thereby making the execution of the computational task faster and more reasonable.

[0108] In step S202, when the heterogeneous processor system receives a computing task, it needs to identify the task type of each subtask to be executed in the computing task. In order to identify the task type of each subtask, the corresponding subtask can be scheduled according to the identified task type, so that the execution of the computing task can be faster and more accurate.

[0109] In step S203, the task type of the subtask is identified. If the task type of the corresponding subtask is identified as memory-restricted, the corresponding subtask can be scheduled to the tensor stream processor chip to utilize the strong bandwidth transmission capability of the tensor stream processor chip to overcome the memory-restricted characteristics of the corresponding subtask and improve the task execution efficiency of the computing task.

[0110] It is worth noting that due to the complex memory hierarchy of the graphics processing unit (GPU), there is a relative latency issue during data reading. For each stage of large language model inference, the decoding and attention calculations rely solely on the GPU, resulting in significant data reading latency. Therefore, decoding and attention mechanism calculations in large language model inference are memory-constrained tasks. Because the SRAM in the TensorStream Processor (TSP) chip has extremely fast access speeds, it can synchronize with the CPU or GPU clock frequency, achieving nanosecond-level read / write latency. Therefore, when subtasks are memory-constrained, the TSP chip can be used to execute the corresponding subtasks, overcoming the memory-constrained problem.

[0111] In step S204, the task type of the subtask is identified. If the task type of the corresponding subtask is identified as computationally limited, the corresponding subtask can be scheduled to the graphics processor chip to utilize the strong computing power of the graphics processor chip to overcome the computationally limited characteristics of the corresponding subtask, thereby improving the task execution efficiency of the computing task.

[0112] It is worth noting that because GPUs possess strong computational capabilities, they can be utilized to execute computationally-constrained subtasks, fully leveraging their computational power. For each stage of large language model inference, pre-filling and feedforward network layer computations both rely on the computational power of GPUs; therefore, pre-filling and feedforward network computations are both computationally-constrained tasks. In cases where subtasks are computationally-constrained, GPUs can be used to execute them, thus fully utilizing their computational power to overcome computational limitations.

[0113] In steps S201 to S204, during task scheduling, the heterogeneous processor system acquires a computing task, which includes multiple subtasks to be executed. Then, the task type of each subtask to be executed within the computing task is identified. If the task type of a subtask is identified as memory-restricted, it can be scheduled to a TensorStream Processor (TSP) chip to utilize the TSP chip's strong bandwidth transmission capability and overcome the memory-restricted nature of the subtask. Similarly, if the task type of a subtask is identified as computationally-restricted, it can be scheduled to a Graphics Processing Unit (GPU) chip to utilize the GPU chip's strong computing power and overcome the computationally-restricted nature of the subtask.

[0114] For example, during the execution of a computing task, a heterogeneous processor system can first obtain the subtasks to be executed in the computing task, and the subtasks obtained this time include a first subtask and a second subtask; then, the task types of the first subtask and the second subtask are identified and processed respectively; if the first subtask is identified as a memory-restricted type, the first subtask can be scheduled to the tensor stream processor chip; if the second subtask is identified as a computation-restricted type, the second subtask can be scheduled to the graphics processor chip.

[0115] In some embodiments of this disclosure, the computation task can be a large language model inference task, which may include a pre-filling task, a decoding computation task, an attention mechanism computation task, a feedforward network computation task, and a reduction operation task; wherein, the pre-filling task, the feedforward network computation task, and the reduction operation task are all computationally restricted subtasks; and the decoding computation task and the attention mechanism computation task are both memory-restricted subtasks.

[0116] It is worth noting that if the subtask is identified as a pre-filling task, a feedforward network computation task, or a reduction operation task, the corresponding subtask can be scheduled to the graphics processor chip to overcome the limitation of computation; if the subtask is identified as a decoding computation task or an attention mechanism computation task, the corresponding subtask can be scheduled to the tensor stream processor chip to overcome the limitation of memory access.

[0117] In one embodiment, refer to Figure 7 Scheduling the attention mechanism computation task to be executed in the second core can include:

[0118] Step S301: Transfer the input data in the memory connected to the first core to the static random access memory of the second core.

[0119] In step S302, the second core performs matrix operations on the input data in the static random access memory, obtains the operation result, and sends barrier arrival marker information to the first core.

[0120] Step S303: If all barrier arrival marker information received by the first core indicates that the preset task synchronization conditions are met, the calculation result is transferred from the static random access memory back to the memory connected to the first core.

[0121] In step S301, during the process of scheduling the attention mechanism computation task to be executed by the second core, the input data in the memory connected to the first core needs to be moved to the static random access memory of the second core. Subsequently, the tensor processing engine of the second core can be used to perform matrix operations on the relevant data.

[0122] It is worth noting that the memory connected to the first chip in this embodiment is the same memory chip stacked with the first chip disclosed in the above embodiments. Furthermore, during the process of moving input data from the memory connected to the first chip to the static random access memory of the second chip, the input data is the target data required for subsequent matrix operations, and it does not mean that all data in the memory connected to the first chip is moved to the static random access memory of the second chip.

[0123] For example, during the process of scheduling the attention mechanism computation task to be executed in the second chip, it is necessary to read data from the storage area

[012] -

[020] in the HBM connected to the graphics processor chip, and move the data in the storage area

[012] -

[020] in the HBM to the static random access memory of the tensor stream processor chip, in order to prepare for the subsequent execution of the attention mechanism computation task.

[0124] In step S302, after the input data in the memory connected to the first core is transferred to the static random access memory of the second core, the second core can perform matrix operation processing on the input data in the static random access memory to obtain the corresponding operation results; and after each operation result is calculated, barrier arrival marker information is sent to the first core to prepare for subsequent task synchronization.

[0125] It is worth noting that during the process of the second core performing matrix operations on the input data in the static random access memory, different tensor processing engines in the second core can be used to perform matrix operations. Furthermore, when each tensor processing engine performs matrix operations and obtains the corresponding operation result, it will send the corresponding barrier arrival marker information to the first core.

[0126] It is worth noting that the second core performs matrix operations on the input data in the static random access memory, thereby obtaining the operation result, and storing the obtained operation result in the static random access memory in the second core.

[0127] In step S303, if all barrier arrival marker information received by the first core indicates that the preset task synchronization conditions are met, the calculation result can be transferred from the static random access memory back to the memory connected to the first core. Specifically, during matrix operations using the second core, barrier arrival marker information is sent to the first core after each calculation result is obtained. Furthermore, if all barrier arrival marker information received by the first core meets the preset task synchronization conditions, the calculation result stored in the static random access memory can be transferred back to the memory connected to the first core.

[0128] It is worth noting that after the first core receives the barrier arrival marker information sent by the second core, it can perform an increment operation. If the calculated value obtained by the increment operation reaches a preset threshold, it can be determined that the task synchronization condition is met. Subsequently, the calculation result stored in the static random access memory can be transferred back to the memory connected to the first core.

[0129] For example, the tensor stream processor chip has two tensor processing engines. During the matrix operation, both tensor processing engines need to perform matrix multiplication and accumulation operations. After completing the corresponding matrix multiplication and accumulation operation, each tensor processing engine sends the corresponding barrier arrival marker information to the graphics processor chip. It is set that when two barrier arrival marker information are received, it can be determined that all tensor processing engines in the tensor stream processor chip have completed the matrix multiplication and accumulation operation. Then, the operation result stored in the static random access memory of the tensor stream processor chip can be moved back to the data memory connected to the graphics processor chip.

[0130] In steps S301 to S303, during the process of scheduling the attention mechanism computation task to be executed by the second core, the input data in the memory connected to the first core first needs to be moved to the static random access memory of the second core. The second core can then perform matrix operations on the input data in the static random access memory to obtain the corresponding computation results. And when each computation result is obtained, barrier arrival marker information is sent to the first core. When all the barrier arrival marker information received by the first core indicates that the preset task synchronization conditions are met, the computation results can be moved from the static random access memory back to the memory connected to the first core.

[0131] For example, during the process of scheduling the attention mechanism computation task to the tensor stream processor chip for execution, data needs to be read from the storage area

[001] -

[016] in the 3D-DRAM connected to the graphics processor chip, and the data in the storage area

[001] -

[016] in the 3D-DRAM is moved to the static random access memory of the tensor stream processor chip; then the tensor stream processor chip can perform matrix operation processing on the input data in the static random access memory to obtain the corresponding operation result, and store the operation result obtained by the matrix operation in the static random access memory; when each operation result is obtained, barrier arrival marker information is sent to the graphics processor chip, and the graphics processor chip is set to indicate that the task synchronization condition is met when it receives 6 barrier arrival marker information; subsequently, when the graphics processor chip receives 6 barrier arrival marker information, it can move the operation result stored in the static random access memory back to the 3D-DRAM connected to the graphics processor chip.

[0132] In one embodiment, the process of moving input data to static random access memory and moving the results of matrix operations back to the memory connected to the first core can be performed by the replication engine in the first core or by the amplitude engine in the second core.

[0133] It is worth noting that both the graphics processing unit (GPU) chip and the tensor stream processor (TRF) chip can be equipped with a copy engine. Therefore, when moving input data from the memory connected to the GPU chip to the static random access memory (SRAM) in the TRF chip, or when moving computation results stored in the SRAM in the TRF chip back to the memory connected to the GPU chip, this can be achieved using either the copy engine in the GPU chip or the copy engine in the TRF chip. This configuration makes the data transfer process more flexible.

[0134] In one embodiment, refer to Figure 8 The first core includes multiple computing units, each including a workgroup scheduler that moves input data from the memory connected to the first core to the static random access memory of the second core. This can include:

[0135] Step S401: Send a data replication command to the replication engine through the workgroup scheduler;

[0136] In step S402, in response to the data copying instruction, a data transfer operation is performed through the copying engine.

[0137] Specifically, in this disclosure, the graphics processor chip proposed in the embodiments of this disclosure includes multiple computing units, each computing unit is provided with a workgroup scheduler, and when the input data in the memory connected to the graphics processor chip is moved to the static random access memory of the tensor stream processor chip, the execution can be triggered by the workgroup scheduler in the computing unit.

[0138] In step S401, when the input data in the memory connected to the graphics processor chip is moved to the static random access memory of the tensor stream processor chip, a data copy command can be sent to the copy engine through the workgroup scheduler to trigger the copy engine to perform subsequent data transfer processing.

[0139] It is worth noting that during the process of sending data copying instructions to the copying engine through the workgroup scheduler, the copying engine can be either the copying engine in the graphics processor chip or the copying engine in the tensor stream processor chip; there is no limitation here.

[0140] In step S402, after the copy engine receives the data copy instruction sent by the workgroup scheduler, it can perform subsequent data transfer operations in response to the data copy instruction, thereby transferring the input data from the memory connected to the graphics processor chip to the static random access memory of the tensor stream processor chip. The copy engine can be either the copy engine within the graphics processor chip or the copy engine within the tensor stream processor chip.

[0141] In steps S401 and S402, when transferring input data from the memory connected to the graphics processor chip to the static random access memory (SRAM) of the tensor stream processor chip, a data copy command can be sent to the copy engine through the workgroup scheduler to trigger the copy engine to perform subsequent data transfer processing. After receiving the data copy command from the workgroup scheduler, the copy engine, in response to the data copy command, can perform subsequent data transfer operations to transfer the input data from the memory connected to the graphics processor chip to the SRAM of the tensor stream processor chip. This technical solution effectively achieves data transfer processing.

[0142] It is worth noting that whether it is moving input data from the memory connected to the graphics processor chip to the static random access memory of the tensor stream processor chip, or moving the operation results from the static random access memory of the tensor stream processor chip back to the memory connected to the graphics processor chip, the scheduling and triggering process can be performed by the workgroup scheduler in the graphics processor chip.

[0143] For example, when moving input data from the memory connected to the graphics processor chip to the static random access memory of the tensor stream processor chip, the workgroup scheduler in the graphics processor chip sends a data copy instruction to the copy engine in the tensor stream processor chip. Then, in response to the received data copy instruction, the copy engine in the tensor stream processor chip can move the input data from the memory connected to the graphics processor chip to the static random access memory of the tensor stream processor chip.

[0144] In one embodiment, refer to Figure 9 The second core performs matrix operations, which may include:

[0145] Step S501: Send a matrix operation trigger command to the tensor processing engine of the second core through the workgroup scheduler in the first core.

[0146] In step S502, in response to the matrix operation trigger instruction, the tensor processing engine performs matrix multiplication and accumulation operations on the data in the static random access memory.

[0147] In step S501, when it is necessary to trigger the tensor stream processor chip to perform matrix operations, the workgroup scheduler in the graphics processor chip first sends a matrix operation trigger instruction to the tensor processing engine in the tensor stream processor chip to trigger the tensor processing engine in the tensor stream processor chip to perform subsequent matrix multiplication and accumulation operations.

[0148] It is worth noting that the tensor stream processor chip contains a tensor processing engine, which allows matrix operations to be performed based on the tensor processing engine in the tensor stream processor chip.

[0149] In step S502, when the tensor processing engine in the tensor stream processor chip receives a matrix operation trigger instruction, in order to respond to the matrix operation trigger instruction, the tensor processing engine in the tensor stream processor chip can perform matrix multiplication and accumulation operations on the data previously moved to the static random access memory to realize the matrix operation operation.

[0150] In steps S501 to S502, when it is necessary to trigger the tensor stream processor chip to perform matrix operations, a matrix operation trigger command is first sent to the tensor processing engine in the tensor stream processor chip through the workgroup scheduler in the graphics processor chip. When the tensor processing engine in the tensor stream processor chip receives the matrix operation trigger command, in order to respond to the matrix operation trigger command, the tensor processing engine in the tensor stream processor chip can perform matrix multiplication and accumulation operations on the data previously moved to static random access memory. Through the above technical solution, matrix operations can be made more reliable.

[0151] In one embodiment, refer to Figure 10The graphics processing core includes a computing unit containing a workgroup scheduler and a replication engine. The tensor stream processing core includes two tensor processing engines, two static random access memories (SRAMs), and one vector processing engine. In the first step, the workgroup scheduler in the computing unit sends data replication operation instructions to the replication engines within the computing unit. In the second step, the replication engines within the computing unit can copy the target execution data from high-bandwidth memory to the SRAM of the tensor stream processing core. In the third step, after all the target execution data has been copied to the SRAM of the tensor stream processing core, the workgroup scheduler in the computing unit sends a matrix multiplication and accumulation operation instruction to the tensor stream processing core to trigger the tensor processing engines in the tensor stream processing core to perform matrix multiplication and accumulation operations using the target execution data in the SRAM. In the fourth step, the tensor processing engines in the tensor stream processing core... The engine will execute the matrix multiplication and accumulation operation according to the matrix multiplication and accumulation operation instructions to obtain the tensor processing result, and store the tensor processing result in static random access memory. In the fifth step, after the tensor processing engine completes the corresponding matrix multiplication and accumulation operation, it will send barrier arrival information to the graphics processing core. In the sixth step, after the graphics processing core receives all the barrier arrival information sent by the tensor processing engine, it can be determined that the matrix multiplication and accumulation operation has been completed, and the workgroup scheduler in the computing unit sends the data copy operation instruction to the copy engine inside the computing unit again. In the seventh step, the copy engine inside the computing unit copies the tensor processing result stored in static random access memory to high-bandwidth memory.

[0152] In one embodiment, refer to Figure 11The graphics processing core includes a computing unit containing a workgroup scheduler. The tensor stream processing core includes two tensor processing engines, two static random access memories (SRAMs), one vector processing engine, and two copy engines. In the first step, the workgroup scheduler in the computing unit sends data copy operation instructions to the copy engines within the tensor stream processing core. In the second step, the copy engines within the tensor stream processing core copy the target execution data from high-bandwidth memory to the SRAM within the tensor stream processing core. In the third step, after all the target execution data has been copied to the SRAM within the tensor stream processing core, the workgroup scheduler in the computing unit sends a matrix multiplication and accumulation operation instruction to the tensor stream processing core to trigger the tensor processing engines within the tensor stream processing core to perform matrix multiplication and accumulation operations using the target execution data from the SRAM. In the fourth step, the tensor processing engines within the tensor stream processing core... The engine will execute the matrix multiplication and accumulation operation according to the matrix multiplication and accumulation operation instructions to obtain the tensor processing result, and store the tensor processing result in static random access memory. In the fifth step, after the tensor processing engine completes the corresponding matrix multiplication and accumulation operation, it will send barrier arrival information to the graphics processing core. In the sixth step, after the graphics processing core receives all the barrier arrival information sent by the tensor processing engine, it can be determined that the matrix multiplication and accumulation operation has been completed, and the workgroup scheduler in the computing unit sends the data copy operation instruction to the copy engine inside the tensor stream processing core again. In the seventh step, the copy engine inside the tensor stream processing core copies the tensor processing result stored in static random access memory to high-bandwidth memory.

[0153] In one embodiment, refer to Figure 12 Scheduling the reduction operation task to be executed in the first core can include:

[0154] Step S601: Multiple computing units of the first core compute the local reduction results in parallel and store each local reduction result in the static random access memory of the second core.

[0155] In step S602, a computing unit in the first core loads all local reduction results from the static random access memory and performs the final reduction operation.

[0156] In step S601, during the process of scheduling the reduction operation task to be executed in the first chip, multiple computing units in the graphics processor chip calculate the local reduction results in parallel, and then store the calculated local reduction results in the static random access memory of the tensor stream processor chip to prepare for the subsequent final reduction operation.

[0157] It is worth noting that the graphics processing unit (GPU) chip and the tensor stream processor (TRF) chip are connected by a hybrid bonding method, and the TRF chip is equipped with static random access memory (SRAM). Therefore, the multiple local reduction results obtained by multiple computing units in the GPU chip performing local reduction operations in parallel can be stored in the TRF chip's SRAM, which greatly speeds up the data reading efficiency.

[0158] In step S602, during the final reduction operation, a computing unit in the graphics processor chip can load all local reduction results from the static random access memory, and then perform the final reduction operation on all local reduction results to obtain the final reduction result.

[0159] It is worth noting that reduction operations can be used for summation, finding the maximum value, finding the minimum value, logical AND, logical OR, and counting.

[0160] In steps S601 to S602, during the process of scheduling the reduction operation task to be executed in the first chip, multiple computing units in the graphics processor chip calculate the local reduction results in parallel, and then store the calculated local reduction results in the static random access memory of the tensor stream processor chip. Next, one computing unit in the graphics processor chip can load all the local reduction results from the static random access memory, and then perform the final reduction operation on all the local reduction results to obtain the final reduction result. Through the above technical solution, the strong computing power of the graphics processor chip can effectively complete the corresponding reduction operation task.

[0161] For example, the graphics processing unit (GPU) chip includes three computing units: a first computing unit, a second computing unit, and a third computing unit. During a reduction operation, the maximum value needs to be determined. During a local reduction operation, the first computing unit receives data 5 and 8, the second computing unit receives data 9 and 7, and the third computing unit receives data 1 and 5. During the local reduction operation, the first computing unit selects 8 as the local reduction result, the second computing unit selects 9 as the local reduction result, and the third computing unit selects 5 as the local reduction result. Then, the first computing unit transfers data 8 to the static random access memory (SRAM) of the TensorStream Processor (TSP) chip, the second computing unit transfers data 9 to the SRAM of the TSP chip, and the third computing unit transfers data 5 to the SRAM of the TSP chip. Next, the third computing unit reads data 8, 9, and 5 from the SRAM and performs the final reduction operation, using data 9 as the final reduction result, thus completing the corresponding data reduction operation.

[0162] In one embodiment, refer to Figure 13The graphics processing core includes computation units, each containing a workgroup scheduler and an execution unit. The execution unit includes a thread bundle instruction scheduler, an arithmetic logic unit, and a register file. The tensor stream processing core includes two tensor processing engines, two static random access memories (SRAMs), and one vector processing engine. In the first step, each computation unit of the graphics processing core reads target execution data from the data memory, and each computation unit performs data reduction operations in parallel on the read target execution data, thereby obtaining multiple data reduction operation results. These multiple data reduction operation results are then transferred to the SRAM. In the second step, after transferring the multiple data reduction operation results to the SRAM, any computation unit in the graphics processing core can read all data reduction operation results from the SRAM and perform data reduction operations on all read data reduction operation results to obtain the final data reduction operation result.

[0163] In one embodiment, refer to Figure 14 Task scheduling methods may also include:

[0164] Step S701: The first chip loads data from the static random access memory of the second chip to perform single-instruction multi-threaded operation, and stores the operation result back to the static random access memory.

[0165] In step S702, the first chip sends an instruction word to the second chip to trigger the second chip to perform streaming processing on the operation results in the static random access memory.

[0166] In step S701, the graphics processor chip can load data from the static random access memory of the tensor stream processor chip to perform single-instruction multi-threaded operations, thereby obtaining the corresponding operation results. Then, the operation results can be stored back into the static random access memory of the tensor stream processor chip to prepare for subsequent streaming processing.

[0167] It is worth noting that during single-instruction multithreaded computation, the graphics processor chip can load data from the static random access memory of the tensor stream processor chip, then perform single-instruction multithreaded computation, and subsequently store the corresponding computation results back to the static random access memory of the tensor stream processor chip.

[0168] In step S702, after the graphics processor chip loads data from the static random access memory of the tensor stream processor chip to perform single-instruction multi-threaded operation and stores the operation result back to the static random access memory, the graphics processor chip can send an instruction word to the tensor stream processor chip to trigger the second chip to stream the operation result in the static random access memory.

[0169] In steps S701 to S702, the graphics processing unit (GPU) chip can load data from the static random access memory (SRAM) of the tensor stream processor (TRF) chip to perform single-instruction multithreaded computation, thereby obtaining the corresponding computation result. The computation result can then be stored back into the SRAM of the TRF chip. Next, the GPU chip can send an instruction word to the TRF chip to trigger the second chip to perform streaming processing on the computation result in the SRAM. Through this technical solution, the GPU chip and the TRF chip cooperate to complete the corresponding streaming processing.

[0170] In one embodiment, the instruction word sent by the graphics processor chip to the tensor stream processor chip is a very long instruction word (VLSI). The VLSI is used to instruct the tensor stream processor chip to perform loading, matrix multiplication and accumulation, vector addition and storage operations in sequence, thereby realizing streaming processing.

[0171] For example, during streaming processing, the workgroup scheduler sends instruction words to the TensorStreaming Processing Core to enable it to execute streaming computation tasks. First, matrix multiplication and accumulation operations are performed based on the tensor processing engine within the TensorStreaming Processing Core to obtain the tensor processing result. Then, the operation result is loaded from the static random access memory (SRAM), and the tensor processing result and the operation result are superimposed using the vector processing engine to obtain the data superposition result. After obtaining the data superposition result by superimposing the tensor processing result and the operation result using the vector processing engine, the data superposition result can be transferred to the SRAM of the TensorStreaming Processing Core through the vector processing engine to complete the streaming processing.

[0172] In one embodiment, refer to Figure 15 The first chip loads data from the static random access memory of the second chip to perform single-instruction multithreaded operations, which may include:

[0173] Step S801: Send a single-instruction multi-threaded instruction to the arithmetic logic unit through the thread bundle scheduler in the first core.

[0174] In step S802, in response to a single instruction multithreaded instruction, the arithmetic logic unit reads data from static random access memory and performs the operation, and writes the operation result back to static random access memory.

[0175] In step S801, during the process of the graphics processor chip loading data from the static random access memory of the tensor stream processor chip for single instruction multithreaded operation, the single instruction multithreaded instruction is first sent to the arithmetic logic unit through the thread bundle scheduler in the graphics processor chip to trigger the arithmetic logic unit in the graphics processor chip to execute subsequent single instruction multithreaded instructions.

[0176] In step S802, when the arithmetic logic unit receives a single instruction multithreaded instruction, it can respond to the single instruction multithreaded instruction by reading data from the static random access memory and performing the operation. Then, it writes the operation result back to the static random access memory of the tensor stream processing core to prepare data for subsequent streaming processing.

[0177] In steps S801 to S802, during the process of the graphics processing unit (GPU) loading data from the static random access memory (SRAM) of the tensor stream processor (TSP) chip for single-instruction multithreaded computation, the GPU first sends a single-instruction multithreaded instruction to the arithmetic logic unit (ALU) through the thread bundle scheduler in the GPU. Upon receiving the single-instruction multithreaded instruction, the ALU responds by reading data from the SRAM and performing the computation. The computation result is then written back to the SRAM of the tensor stream processor chip, providing a data foundation for subsequent streaming processing within the tensor stream processor chip.

[0178] In one embodiment, refer to Figure 16 Before the first chip sends a command to the second chip to trigger streaming processing, it may also include:

[0179] Step S901: The intermediate data in the note storage of the first core is transferred to the static random access memory of the second core through the copy engine.

[0180] In step S901, before the graphics processor chip sends an instruction to the tensor stream processor chip to trigger streaming processing, the intermediate data in the graphics processor chip's note memory can be moved to the tensor stream processor chip's static random access memory via the copy engine to provide a data foundation for subsequent streaming processing.

[0181] It is worth noting that the copy engine can be either the copy engine within the graphics processing unit (GPU) chip or the copy engine within the tensor stream processor (TSP) chip. Specifically, the sticky note storage within the GPU chip can be used to store intermediate data. The copy engine then moves this intermediate data from the sticky note storage to the static random access memory (SRAM) of the TSP chip, preparing it for subsequent streaming processing.

[0182] In one embodiment, refer to Figure 17 The graphics processing kernel includes a computing unit, which contains a workgroup scheduler and an execution unit. The execution unit includes a thread bundle instruction scheduler, an arithmetic logic unit, and a register file. The tensor stream processing kernel includes two tensor processing engines, two static random access memories, and one vector processing engine. In the first step, the thread bundle instruction scheduler in the execution unit of the computation unit sends a single-instruction multithreaded instruction to the arithmetic logic unit in the execution unit. In the second step, in response to the single-instruction multithreaded instruction, the target execution data can be read from the static random access memory (SRAM) in the tensor stream processing kernel into the register file of the execution unit through the corresponding execution unit. In the third step, the arithmetic logic unit in the execution unit can perform arithmetic logic operations on the target execution data in the register file to obtain the corresponding data execution result. In the fourth step, after the arithmetic logic unit in the execution unit performs arithmetic logic operations on the target execution data in the register file to obtain the corresponding data execution result, the execution unit can transfer the obtained data execution result to the SRAM of the tensor stream processing kernel. In the fifth step, after the execution unit transfers the data execution result to the SRAM of the tensor stream processing kernel, the workgroup scheduler can send an instruction word to the tensor stream processing kernel so that the tensor stream processing kernel can use the data execution result stored in the SRAM to execute subsequent stream computation tasks.

[0183] In one embodiment, refer to Figure 18The graphics processing core includes a computing unit, which contains a workgroup scheduler, an execution unit, and a note-based memory. The execution unit includes a thread-binding instruction scheduler, an arithmetic logic unit, and a register file. The tensor stream processing core includes two tensor processing engines, two static random access memories (SRAMs), and one vector processing engine. In the first step, the workgroup scheduler in the computing unit sends a data copy operation instruction to the copy engine in the tensor stream processing core, enabling the copy engine to read and copy data from the SRAM. In the second step, upon receiving the data copy operation instruction, the copy engine copies the target execution data from the SRAM to the note-based memory in the computing unit. In the third step, the thread-binding instruction scheduler in the execution unit sends a data load instruction to the note-based memory, thereby loading the target execution data from the note-based memory into the register file. In the fourth step, the thread-beam instruction scheduler in the execution unit sends a data overlay instruction to the arithmetic logic unit (ALU), enabling the ALU to perform single-instruction multi-threaded computation using the target execution data in the register file, obtaining the data overlay result, and storing the result in the register file. In the fifth step, upon obtaining the data overlay result, the thread-beam instruction scheduler sends a data transfer instruction to the register file to transfer the data overlay result from the register file to the note memory. In the sixth step, the workgroup scheduler in the computation unit sends a data copy operation instruction to the copy engine to perform the data overlay result copying operation. In the seventh step, upon receiving the data copy operation instruction, the copy engine copies the data overlay result from the note memory to the static random access memory (SRAM). In the eighth step, after transferring the data overlay result to the SRAM in the tensor stream processing kernel, the workgroup scheduler sends an instruction word to the tensor stream processing kernel, enabling the tensor stream processing kernel to use the data overlay result stored in the SRAM to execute subsequent stream computation tasks.

[0184] This disclosure also provides an electronic device 1900, comprising:

[0185] At least one processor, and,

[0186] A memory that is communicatively connected to at least one processor; wherein,

[0187] The memory stores instructions that are executed by at least one processor to implement a task scheduling method as described in any of the above embodiments of the present disclosure when the at least one processor executes the instructions.

[0188] The following is combined Figure 19The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 1910, a memory 1920, an input / output interface 1930, a communication interface 1940, and a bus 1950.

[0189] The processor 1910 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0190] The memory 1920 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1920 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1920 and is invoked by the processor 1910 to execute the task scheduling method of the embodiments of this disclosure.

[0191] The input / output interface 1930 is used to implement information input and output.

[0192] The communication interface 1940 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0193] Bus 1950 transmits information between various components of the device (e.g., processor 1910, memory 1920, input / output interface 1930, and communication interface 1940);

[0194] The processor 1910, memory 1920, input / output interface 1930 and communication interface 1940 are connected to each other within the device via bus 1950.

[0195] This disclosure also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the task scheduling method of the above embodiments, which will not be described in detail here.

[0196] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0197] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0198] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0199] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. It should also be understood that the various implementation methods provided in this disclosure can be arbitrarily combined to achieve different technical effects.

[0200] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A heterogeneous processor system, characterized in that, include: The first chip, which is a graphics processor chip; The second chip, which is a tensor stream processor chip, is stacked and integrated with the first chip using a hybrid bonding technology. A memory chip, stacked and integrated with the first chip, is used to provide data storage; The first core is used to schedule computing tasks, and the first core and the second core work together to execute the computing tasks.

2. The heterogeneous processor system according to claim 1, characterized in that, The second core includes: At least one tensor processing engine for performing matrix operations; The vector engine is used to perform vector operations; Static random access memory (SRAM) is used to store data.

3. The heterogeneous processor system according to claim 2, characterized in that, The first chip includes multiple computing units; the static random access memory is configured as a memory for sharing data among the multiple computing units and provides a data access interface for the first chip.

4. A task scheduling method for heterogeneous processor systems, characterized in that, The heterogeneous processor system includes a first core and a second core stacked by hybrid bonding. The first core is a graphics processing unit (GPU) core, and the second core is a tensor stream processor (TSU) core. The first core is used to schedule computational tasks, and the first and second cores collaboratively execute the computational tasks. The method includes: Retrieve the subtasks to be executed in the computation task; The task type of the subtask is identified and processed; If the subtask is identified as a memory-restricted type, it will be scheduled to be executed in the second core. If the task is identified as computationally restricted, it is scheduled to be executed in the first core.

5. The method according to claim 4, characterized in that, The computational task is a large language model inference task; the memory-restricted subtasks include attention mechanism computation tasks, and the computationally restricted subtasks include feedforward network computation tasks or reduction operation tasks.

6. The method according to claim 5, characterized in that, Scheduling the attention mechanism computation task to be executed by the second core particle includes: The input data in the memory connected to the first core is moved to the static random access memory of the second core; The second core performs matrix operations on the input data in the static random access memory, obtains the operation result, and sends barrier arrival marker information to the first core; If all the barrier arrival marker information received by the first core indicates that the preset task synchronization conditions are met, the calculation result is transferred from the static random access memory back to the memory connected to the first core.

7. The method according to claim 6, characterized in that, The input data is moved to the static random access memory, and the calculation result is moved back to the memory connected to the first core, which is executed by the replication engine in the first core or by the replication engine in the second core.

8. The method according to claim 6, characterized in that, The first chip includes multiple computing units, and each computing unit includes a workgroup scheduler; The step of transferring input data from the memory connected to the first chip to the static random access memory of the second chip includes: The workgroup scheduler sends data replication instructions to the replication engine. In response to the data copy instruction, a data migration operation is performed through the copy engine.

9. The method according to claim 6, characterized in that, The second core performs matrix operations, including: The workgroup scheduler in the first core sends a matrix operation trigger command to the tensor processing engine of the second core. In response to the matrix operation trigger instruction, the tensor processing engine performs matrix multiplication and accumulation operations on the data in the static random access memory.

10. The method according to claim 5, characterized in that, Scheduling the reduction operation task to be executed in the first chip includes: Multiple computing units of the first core compute the local reduction results in parallel and store each of the local reduction results in the static random access memory of the second core. One of the computing units in the first chip loads all the local reduction results from the static random access memory and performs the final reduction operation.

11. The method according to claim 4, characterized in that, Also includes: The first chip loads data from the static random access memory of the second chip to perform single-instruction multi-threaded operation, and stores the operation result back to the static random access memory; The first chip sends an instruction word to the second chip to trigger the second chip to stream the operation results in the static random access memory.

12. The method according to claim 11, characterized in that, The instruction word sent by the first core to the second core is a very long instruction word, which is used to instruct the second core to perform loading, matrix multiplication and accumulation, vector addition and storage operations in sequence.

13. The method according to claim 11, characterized in that, The first chip loads data from the static random access memory of the second chip to perform single-instruction multithreaded operations, including: The thread bundle scheduler in the first core sends a single-instruction multi-threaded instruction to the arithmetic logic unit. In response to the single instruction multithreaded instruction, the arithmetic logic unit reads data from the static random access memory and performs the operation, and writes the operation result back to the static random access memory.

14. The method according to claim 11, characterized in that, Before the first chip sends a command to the second chip to trigger streaming processing, the process further includes: The intermediate data in the note storage of the first core is moved to the static random access memory of the second core using a copy engine.

15. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. The program is executed by the processor to implement the task scheduling method as described in any one of claims 4 to 14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the task scheduling method as described in any one of claims 4 to 14.

Images