Instruction scheduling method and device, electronic equipment and medium

Abstract

The invention provides an instruction scheduling method and apparatus, an electronic device and a medium. The method comprises the steps of obtaining a target instruction through a first instruction scheduling unit; identifying a target unit for executing the target instruction, the target unit being one of an execution unit or a first functional unit; under the condition that the target unit is the first function unit, a target tensor block corresponding to the target instruction and a target working group corresponding to the target tensor block are determined, and the target working group is used for indicating the memory address range of the target tensor block; and scheduling the target instruction to the target working group through the first instruction scheduling unit, scheduling the target working group to the target unit, reading the target tensor block based on the memory address range by the target unit, and executing the target instruction on the target tensor block. Based on the instruction scheduling method disclosed by the invention, the adaptation degree between a hardware layer and block-based GPU programming languages such as triton and the like can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to an instruction scheduling method and device, an electronic device, and a medium. Background Art

[0002] Traditional graphics processing units are often equipped with multiple computing units, each of which is respectively equipped with multiple execution units, and the data processing process of the graphics processing unit is completed by the execution unit. With the development of artificial intelligence technology, in order to accelerate the efficiency of neural network model training and reasoning, the graphics processing unit in the prior art will integrate some dedicated hardware modules dedicated to performing specific data processing tasks in the computing unit, such as tensor core (TensorCore), copy engine (Copy Engine), etc. When certain specific tasks need to be performed, these hardware modules can be used directly to complete them, and these dedicated hardware can support the execution of corresponding data processing tasks with a higher degree of parallelism. At the same time, in order to simplify the difficulty of GPU programming, there are some block-based GPU programming languages ​​in the prior art, such as the triton programming language. Based on these programming languages, users can flexibly decompose the tensor data to be processed into multiple tensor blocks of specified sizes, and then distribute these tensors to each computing unit for execution.

[0003] In the prior art, the computing unit schedules instructions to the corresponding threads through the instruction scheduler in the execution unit, and then schedules the threads loaded with instructions to the corresponding hardware modules for execution to complete the corresponding data processing tasks, while the execution unit often schedules instructions in units of thread bundles, that is, each time an instruction is scheduled, a single instruction is scheduled to each thread in a single thread bundle, and then the thread bundle is scheduled to the hardware unit for execution. That is, in the prior art, although the upper-level application decomposes the data processing tasks and the corresponding tensor data into the granularity specified by the user and then distributes them to the computing units for execution; however, at the hardware layer, the computing units will still execute the tasks issued by the upper-level application at the granularity of thread bundles, and the number of threads contained in a single thread bundle is often fixed, which makes the granularity of instruction scheduling at the hardware layer often inconsistent with the task decomposition granularity specified by the user. At this time, the hardware layer often needs to coordinate the execution of a single subtask assigned by the upper-level application by scheduling the same instruction to multiple thread bundles respectively. Since the instruction scheduling and execution processes of different thread bundles are independent of each other, this results in the user actually needing to perform additional programming to manage the operation of thread bundles when writing the kernel. For example, users need to set up synchronization mechanisms between thread warps to prevent unpredictable situations such as race conditions between thread warps; when different hardware modules need to execute different instructions on different tensor blocks at the same time, the instruction scheduler needs to schedule multiple instructions to multiple thread warps at the same time. At this time, users need to perform Warp specialization to indicate which instructions each thread warp is used to execute so that the execution unit can accurately schedule instructions to the corresponding thread warp. This instruction scheduling mode is poorly compatible with block-based GPU programming languages ​​such as Triton. Summary of the invention

[0004] The embodiments of the present disclosure provide an instruction scheduling method, an electronic device, and a storage medium, which can improve the compatibility between the hardware layer and block-based GPU programming languages ​​such as Triton.

[0005] According to one aspect of the present disclosure, an instruction scheduling method is proposed, which is applied to a computing unit, wherein the computing unit includes a first instruction scheduling unit, at least one execution unit and at least one first functional unit, wherein the first functional unit includes at least one of a copy engine, a tensor core and a scalar processing unit, and the method includes: obtaining a target instruction through the first instruction scheduling unit; identifying a target unit for executing the target instruction, wherein the target unit is one of the execution unit or the first functional unit; when the target unit is the first functional unit, determining a target tensor block corresponding to the target instruction and a target workgroup corresponding to the target tensor block, wherein the target workgroup is used to indicate a memory address range of the target tensor block; scheduling the target instruction to the target workgroup through the first instruction scheduling unit, and scheduling the target workgroup to the target unit, so that the target unit reads the target tensor block based on the memory address range and executes the target instruction on the target tensor block.

[0006] Optionally, the first instruction scheduling unit is provided with a corresponding instruction cache area, and the computing unit further includes a command processor. Before acquiring the target instruction through the first instruction scheduling unit, the method further includes: acquiring a target command, the target command being used to instruct the computing unit to start a target kernel corresponding to the target command, the target kernel being composed of a plurality of instructions; parsing the target command through the command processor to determine the target kernel to be started, determining a plurality of instructions to be executed based on the target kernel, and writing the instructions to be executed into the instruction cache area of ​​the first instruction scheduling unit; acquiring the target instruction through the first instruction scheduling unit includes: acquiring the target instruction from the corresponding instruction cache area through the first instruction scheduling unit.

[0007] Optionally, the target kernel includes at least a main function kernel part, a scalar kernel part and a vector kernel part, the scalar kernel part includes multiple preset scalar operation kernels, the vector kernel part includes multiple preset vector operation kernels, and the main function kernel part calls the scalar operation kernel and the vector operation kernel through a calling instruction.

[0008] Optionally, the identifying of the target unit for executing the target instruction includes: identifying an instruction type corresponding to the target instruction, the instruction type being used to characterize a type of data processing task corresponding to the target instruction, the execution unit and each of the first functional units corresponding to different instruction types respectively; and determining the target unit for executing the target instruction based on the instruction type.

[0009] Optionally, before identifying the target unit for executing the target instruction, the method also includes: creating a first instruction set, the first instruction set including multiple preset instructions, the target instruction is one of the preset instructions; dividing the preset instructions into multiple instruction subsets, each instruction subset including at least one preset instruction, the preset instructions in a single instruction subset corresponding to the same instruction type; identifying the instruction type corresponding to the target instruction includes: identifying the instruction subset corresponding to the target instruction to determine the instruction type corresponding to the target instruction.

[0010] Optionally, the instruction type includes at least one of the following: a tensor operation instruction, which indicates an instruction executed by the tensor core; a scalar operation instruction, which indicates an instruction executed by the scalar processing unit; a vector operation instruction, which indicates an instruction executed by the execution unit; a data copy instruction, which is an instruction executed by the copy engine.

[0011] Optionally, before determining the target tensor block corresponding to the target instruction and the target workgroup corresponding to the target tensor block, the method also includes: obtaining workgroup creation parameters, the workgroup creation parameters including workgroup size and workgroup number parameters, the workgroup size characterizing the size of the tensor block corresponding to a single workgroup in each dimension; creating multiple workgroups based on the workgroup number parameter and assigning a workgroup identifier to each of the workgroups, and dividing the tensor to be processed into multiple tensor blocks based on the workgroup size, wherein the workgroups and the tensor blocks correspond one to one.

[0012] Optionally, each of the execution units includes a second instruction scheduling unit, and the method further includes: if the target unit is the execution unit, determining a target workgroup assigned to the target unit, and creating multiple threads corresponding to the target workgroup, wherein each of the threads corresponds to data in a tensor block corresponding to the target workgroup; the first instruction scheduling unit sends the target instruction to the second instruction scheduling unit in the execution unit; in the execution unit, determining the created threads as several thread bundles, wherein each thread bundle includes a second number of threads; in the execution unit, scheduling the target instruction to the threads in the several thread bundles respectively through the second instruction scheduling unit.

[0013] According to one aspect of the present disclosure, an instruction scheduling device is provided, the device comprising: an instruction acquisition unit, configured to acquire a target instruction through the first instruction scheduling unit; a first identification unit, configured to identify a target unit for executing the target instruction, the target unit being one of an execution unit or a first functional unit; a target workgroup determination unit, configured to determine, when the target unit is the first functional unit, a target tensor block corresponding to the target instruction and a target workgroup corresponding to the target tensor block, the target workgroup being configured to indicate a memory address range of the target tensor block; and a first scheduling unit, configured to schedule the target instruction to the target workgroup through the first instruction scheduling unit, and to schedule the target workgroup to the target unit, so that the target unit reads the target tensor block based on the memory address range and executes the target instruction on the target tensor block.

[0014] According to one aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing a connection and communication between the processor and the memory, wherein the program implements the instruction scheduling method described in any one of the above items when executed by the processor.

[0015] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the instruction scheduling method as described in any one of the above items.

[0016] According to one aspect of the present disclosure, an instruction scheduling method, an electronic device and a storage medium are provided, by directly integrating a first instruction scheduling unit and at least one first functional unit for executing a specific data processing task in a computing unit; thereafter, when the computing unit needs to be used to execute a data processing task, the target instruction to be executed is first obtained through the first instruction scheduling unit, and a target unit for executing the target instruction is identified; thereafter, if the target unit is a first functional unit, a target tensor block that needs to be used to execute the target instruction and a target workgroup corresponding to the target tensor block are determined; thereafter, the target instruction is scheduled to a target workgroup, and the target workgroup loaded with the target instruction is scheduled to a target unit, so that the target unit can determine the address of the target tensor block in the memory based on the target workgroup, thereby reading the target tensor block and executing the data processing task corresponding to the target instruction on the target tensor block. Based on this, in this embodiment, when the target instruction is an instruction executed by the first functional unit such as the tensor core and the copy engine, the instruction is directly scheduled by the first instruction scheduling unit in units of work groups, so that the target unit reads the target tensor block based on the memory address range indicated by the target work group and executes the target instruction on the target tensor block. In this way, the granularity of the computing unit as the underlying hardware when performing instruction scheduling is consistent with the task decomposition granularity set by the user in the kernel function, and the first instruction scheduling unit directly schedules the target instruction to the target work group instead of a thread or a thread bundle. At this time, there is no need to create multiple threads and thread bundles, and thus there is no need to set a synchronization mechanism to manage the operation of the thread bundle, nor is there any need to perform Warp specialization to indicate what instructions each thread bundle is used to execute. In this way, the instruction scheduling mode of the computing unit can be more suitable for kernels written in programming languages ​​such as triton.

[0017] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0019] Figure 1 is a system architecture diagram of a computing unit to which an embodiment of the present disclosure is applied;

[0020] Figure 2 is a main flow chart of an instruction scheduling method according to an embodiment of the present disclosure;

[0021] Figure 3 yes Figure 2 Flow chart of step S201 and its associated steps;

[0022] Figure 4 is a schematic diagram of the structure of a target kernel according to an embodiment of the present disclosure;

[0023] Figure 5 yes Figure 2 A sub-flowchart of step S202;

[0024] Figure 6 yes Figure 5 Flow chart of step S501 and its associated steps;

[0025] Figure 7 is a flowchart of creating a work group and dividing a tensor into tensor blocks corresponding to the work group according to an embodiment of the present disclosure;

[0026] Figure 8 is a flow chart of an instruction scheduling method according to another embodiment of the present disclosure;

[0027] Fig. 9 It is a system architecture diagram of an instruction scheduling device according to an embodiment of the present disclosure.

[0028] Fig.10 It is a system architecture diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0030] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0031] Thread: It is the smallest execution unit of the graphics processing unit when performing data processing tasks. Each thread can independently perform the same processing mode on different data.

[0032] Work Group: A subtask entity corresponding to a tensor block of a user-specified size in a block-based programming language such as triton. Specifically, in programming languages ​​such as triton, users can divide the tensor to be processed into multiple tensor blocks by setting meta parameters such as block_size when writing kernel functions. Block_size is used to indicate the number of elements contained in a single tensor block, and each tensor block corresponds to a work group. For example, the tensor to be processed is a three-dimensional tensor with a shape of (32, 256, 128), and block_size is (2, 32, 128). At this time, the tensor to be processed is divided into 128 tensor blocks with a shape of (2, 32, 128), each tensor block contains 2*32*128 elements, and 128 work groups are created at the same time. Each work group corresponds to a tensor block. Specifically, each work group itself can carry information indicating the address of the corresponding tensor block in memory, such as the address index or address offset of the tensor block in memory.

[0033] Triton: A programming language for writing kernel functions for graphics processing units. Unlike CUDA, in the Triton language, users can more flexibly and proactively decompose the tensors to be processed into tensor blocks of specified sizes, decompose the data processing tasks to be performed into subtasks of user-specified sizes, and assign each tensor block to each computing unit, so as to assign each subtask to each computing unit for execution.

[0034] Warp: It is the basic unit for data processing by the execution unit. Each warp generally includes 32 fixed threads. Of course, in some graphics processing units, the number of threads in a single warp can also be 16, 64 or 128. But for any given graphics processing unit, the number of threads contained in a single warp is fixed. The same thread is often executed in the single instruction multiple thread (SIMT) mode, that is, the threads in the same warp are often used to execute the same instruction in parallel on different data.

[0035] Synchronization mechanism: A mechanism for managing multi-threaded operations. In a multi-threaded environment, multiple threads are often executed independently, and some threads may be executed by the hardware unit before other threads. In the instruction scheduling mode in the relevant technology, the resources occupied by the thread that is executed first will be released in advance and used to execute the next instruction, which will lead to unpredictable situations such as conditional competition. Therefore, in a multi-threaded environment, it is often necessary to manage multi-threaded operations through a synchronization mechanism. Specifically, the synchronization mechanism is to set a synchronization node for multiple threads. When some of the threads execute to the synchronization node first, these threads will be suspended and wait for other threads to execute to the synchronization node. When all threads with the synchronization mechanism are executed to the synchronization node, they will continue to execute subsequent processing. In the graphics processing unit of the related technology, since instruction scheduling is performed in units of thread bundles through the instruction scheduler in the execution unit, when multiple thread bundles are used to collaboratively complete data processing tasks, the user needs to set up synchronization mechanisms at the level of different threads in the same thread bundle and at the level of different thread bundles during programming. The synchronization mechanism becomes extremely complicated. If the synchronization nodes of the synchronization mechanism are not properly set, some threads need to wait for a long time for other threads in the same thread bundle or other thread bundles to execute to the synchronization nodes, thereby greatly reducing computing efficiency, which greatly increases the setting of the synchronization mechanism.

[0036] Race Condition: It means that multiple threads try to modify the data at the same location at the same time. When multiple thread bundles are used to complete the same data processing task, because each thread bundle cannot access the memory occupied by other thread bundles, these thread bundles need to cooperate through shared memory. However, when scheduling instructions in thread bundles, because the process of scheduling instructions to different thread bundles is independent of each other, and the execution process of each thread bundle is also independent of each other, this may cause some thread bundles to be executed before other thread bundles. The instruction scheduler will directly schedule the next instruction to the thread bundle that has executed the previous instruction. When executing the next instruction through this part of the thread bundle, it may be necessary to access the shared memory to modify the data generated by the other thread bundle when executing the previous instruction. However, at this time, the other thread bundle has not actually executed the previous instruction. This may cause the threads of the two thread bundles to write data to the same location at the same time, causing race conditions.

[0037] Description of the system architecture of the present disclosure embodiment

[0038] Figure 1 It is a system architecture diagram of a computing unit applied by the instruction scheduling method of an embodiment of the present disclosure, which includes: a first instruction scheduling unit 110, at least one execution unit 120 and at least one first functional unit.

[0039] The first functional unit may include at least one of the tensor core 131, the replication engine 132, and the scalar processing unit 133, and each of the first functional units is used to execute a certain type of instruction. It can be understood that the execution unit 120, as a general hardware unit, can execute various instructions by itself, and the difference between the execution unit 120 and the replication engine 132, the tensor core 131, and the scalar processing unit 133 lies in the efficiency of executing a specific type of task.

[0040] In this embodiment, the first functional unit integrated in the computing unit may include only one of the tensor core 131, the copy engine 132 or the scalar processing unit 133. For example, the computing unit only integrates the tensor core 131 as the first functional unit, without integrating the scalar processing unit 132 and the copy engine 133, and the instructions for data copying and scalar operations are executed by the execution unit 120. Similarly, the computing unit may also integrate only the copy engine 132 as the first functional unit, without integrating the tensor core 131 and the scalar processing unit 133. Of course, the computing unit may also integrate the tensor core 131, the copy engine 132 and the scalar processing unit 133 at the same time, respectively as the first functional unit for executing different instructions. In addition, the computing unit may also integrate a dedicated hardware module for performing certain specific data processing tasks in addition to the tensor core 131, the copy engine 132 and the scalar processing unit 133, and the dedicated hardware module is also used as the first functional unit. Specifically, the first functional unit in the computing unit needs to be determined according to the actual hardware architecture of the computing unit.

[0041] The first instruction scheduling unit 110 is used to obtain a target instruction to be executed and identify a target unit for executing the target instruction, wherein the target unit may be one of the execution unit 120 or the first functional unit.

[0042] The first instruction scheduling unit 110 is also used to determine the target tensor block corresponding to the target instruction and the target work group corresponding to the target tensor block when the target unit is the first functional unit, and schedule the target instruction to the target work group, wherein the target tensor block is a tensor block composed of data for executing the target instruction. For example, the target instruction is a matrix multiplication instruction, and the target tensor block is a matrix block in two matrices required for matrix multiplication operations. In this way, when the target unit is other dedicated hardware other than the execution unit, the instruction scheduling can be performed in units of work groups through the first instruction scheduling unit integrated outside the execution unit, without the need to perform instruction scheduling in units of thread bundles through the instruction scheduler in the execution unit.

[0043] Of course, refer to Figure 1The computing unit may also be integrated with a local memory, and the computing unit may be coupled with an external on-chip memory and an off-chip memory. When the computing unit is integrated with a copy engine 132, the copy engine 132 is integrated at a position closer to the local memory, the on-chip memory, and the off-chip memory than the execution unit, so as to shorten the physical path length when the data stored in one memory space is copied to another memory space through the copy engine, thereby improving the efficiency of executing corresponding data copy instructions using the copy engine.

[0044] Overall implementation of the instruction scheduling method of the disclosed embodiment

[0045] The present disclosure provides an instruction scheduling method for use in Figure 1 In the calculation unit shown, refer to Figure 2 , the instruction scheduling method includes:

[0046] Step S201, obtaining a target instruction through a first instruction scheduling unit;

[0047] Step S202, identifying a target unit for executing a target instruction, where the target unit is one of an execution unit or a first functional unit;

[0048] Step S203, when the target unit is the first functional unit, determining a target tensor block corresponding to the target instruction and a target work group corresponding to the target tensor block, where the target work group is used to indicate a memory address range of the target tensor block;

[0049] Step S204, scheduling the target instruction to the target work group through the first instruction scheduling unit, and scheduling the target work group to the target unit, so that the target unit reads the target tensor block based on the memory address range and executes the target instruction on the target tensor block.

[0050] In this embodiment, the tensor core is a dedicated hardware module in the computing unit dedicated to performing tensor operations. In some architectures, it is also called a tensor engine, which is used to process tensors of two dimensions and above. That is, for the instructions executed by the tensor core, at least one of the data involved in the operation is operated in the form of a two-dimensional or higher-dimensional tensor. It should be noted that in this embodiment, two dimensions and above refer to the dimensions of the data when it is used for calculations, rather than the dimensions in which the data is organized when it is stored. For example, when normalizing each matrix row of a matrix separately, although the matrix itself is in two-dimensional form, each matrix row is normalized separately during the actual operation, that is, the normalization process is actually performed in the form of a one-dimensional vector. Accordingly, the instruction for normalizing each matrix row of the matrix is ​​not an instruction executed by the tensor core. For another example, when adding two matrices, the elements at the same position in the two matrices are actually added separately. Each addition operation actually only involves two elements at the same position in the two matrices. The process is actually performed in the form of a zero-dimensional scalar. Accordingly, the instruction for summing the two matrices is not an instruction executed by the tensor core.

[0051] Specifically, in this embodiment, the tensor core can be used to execute at least matrix multiplication instructions and matrix multiply accumulate (MMA) instructions. The tensor core may include a general matrix multiplication unit (GEMM) and an accumulation buffer, the general matrix multiplication unit is used to perform matrix multiplication on the input matrix, and the accumulation buffer can be used to accumulate the matrix multiplication results of the general matrix multiplication unit. Exemplarily, when executing a matrix multiplication and accumulation instruction, the two matrices that need to be matrix multiplied will be divided into multiple matrix blocks according to preset rules, and then these matrix blocks will be input into the tensor core in a certain order, the general matrix multiplication unit is used to perform matrix multiplication on the input matrix block, and the accumulation buffer accumulates the matrix multiplication results of the matrix blocks output by the general matrix multiplication unit according to a certain period to perform matrix multiplication and accumulation operations.

[0052] The Copy Engine is a hardware module in the computing unit that is dedicated to data copying and moving. Graphics processing units are often equipped with three levels of memory: global memory, local memory, and registers, and their memory structure is relatively complex. The global memory is set outside the computing unit and can be accessed by all computing units, but the access speed of the global memory is slower than that of the local memory and registers; the register is set inside the execution unit in the computing unit and can only be accessed by the threads in the corresponding execution unit, and the access speed of the register is faster than that of the global memory and local memory. Due to this complex memory structure, when the graphics processing unit is training and reasoning the neural network model, in order to improve efficiency, it will perform a large number of data copy operations to copy the data required by the hardware modules in each computing unit when performing the corresponding computing tasks to the memory space that can be quickly accessed. For example, the data required by the execution unit when performing the corresponding computing task is copied from the global memory to the register in the corresponding execution unit. The Copy Engine is a hardware unit dedicated to executing such data copy instructions.

[0053] The Scalar Processing Unit (Scalar Unit) is a dedicated hardware module in the graphics processing unit for scalar operations. It can be used to perform basic arithmetic and logical operation instructions on data. For example, relatively simple basic arithmetic operation instructions such as addition, subtraction, multiplication and division of elements in two matrices, exponentialization and logarithmization of each element in a single matrix, and basic logical operation instructions such as logical AND, logical OR, bitwise AND, bitwise OR, and XOR. The Scalar Processing Unit can be an Arithmetic and Logical Unit (ALU).

[0054] The execution unit is a general data processing module in the computing unit. The hardware structure of the execution unit itself is not optimized for certain specific data processing tasks. The efficiency of the execution unit in executing these specific tasks is lower than that of special hardware such as tensor cores and replication engines. However, as a general data processing module, the execution unit can execute most instructions in the neural network model. For some instructions that are not suitable for execution by special hardware such as tensor cores and replication engines, they will be assigned to the execution unit for execution.

[0055] In step S201, the target instruction is an instruction that the computing unit needs to execute subsequently, which indicates that the computing unit needs to perform a data processing task, such as performing arithmetic or logical operations on corresponding data, accessing specific memory, etc.

[0056] It is understandable that when the graphics processing unit is performing training or reasoning tasks of a neural network model, each hardware module in the computing unit is driven by instructions, and the computing unit will control each execution unit or first functional unit therein to perform data processing tasks corresponding to these instructions according to these instructions, so as to complete the reasoning or calculation tasks of the neural network model. For example, the graphics processing unit needs to perform convolution processing on the intermediate data generated by a neural network layer. At this time, the graphics processing unit needs to distribute the corresponding tasks to each computing unit therein, so that each computing unit is used to perform convolution processing on a part of the data respectively, and each computing unit will obtain instructions for instructing the execution unit or a first functional unit in the computing unit to perform convolution processing on a part of the data assigned to the computing unit.

[0057] Specifically, the first instruction scheduling unit often maintains an instruction queue, which includes multiple instructions to be executed by the computing unit, and the target instruction is actually an instruction in the instruction queue. When the first instruction scheduling unit detects that the execution condition of an instruction in the instruction queue is met, it will obtain the instruction from the instruction scheduling as the target instruction to perform subsequent instruction scheduling operations. Specifically, the first instruction scheduling unit can monitor whether the hardware unit used to execute each instruction in the instruction queue is idle, and whether the data required to execute the instruction is ready. When the corresponding hardware unit is idle and the corresponding data is ready, the first instruction scheduling unit obtains the corresponding instruction from the instruction queue as the target instruction. For example, the first instruction scheduling unit can determine whether the data required for each instruction in the instruction queue is ready and whether the hardware unit executing the instruction is idle through the scoreboard mechanism.

[0058] In step S202, the target instruction indicates the data processing task that the computing unit needs to perform on the data later, for example, it indicates that the data needs to be processed by convolution, matrix multiplication, normalization, etc. In this embodiment, each first functional unit is used to perform a specific type of data processing task, for example, the tensor core is used to operate on tensors of two dimensions and above; the copy engine is used to perform data copy operations. Data processing tasks other than the specific types of data processing tasks performed by these first functional units are performed by the execution unit as a general processing unit. The target instruction is used to drive the hardware module in the graphics processing unit. It is actually one of a plurality of preset instructions, and these preset instructions are used to instruct the hardware in the graphics processing unit to perform different data processing tasks. Based on this, in this embodiment, the mapping relationship between each preset instruction and the execution unit or the first functional unit can be pre-set, and when the target instruction is obtained, the target unit for executing the target instruction can be identified based on the mapping relationship. The details of step S202 will be described later, and will not be repeated here.

[0059] In step S203, the target tensor block is a tensor block that needs to be used to execute the data processing task corresponding to the target instruction, the target workgroup is a workgroup corresponding to the target tensor block, and the target workgroup itself will carry a memory address range for characterizing the target tensor block. Specifically, in a graphics processing unit, the elements in a tensor will be stored in consecutive addresses, each address corresponding to an element in the tensor, and a tensor block often contains several consecutive elements in the tensor, that is, each tensor block is actually composed of multiple elements stored in a continuous address in the memory. Based on this, in this embodiment, the target workgroup can indicate the memory address of the target tensor block by indicating the interval composed of a continuous address corresponding to the multiple elements constituting the tensor block.

[0060] Specifically, when using a block-based programming language such as triton to write a kernel function, the user can set the relevant parameters of the work group through specific code statements in the kernel function to instruct the graphics processing unit how to divide the tensor to be processed into multiple tensor blocks, so as to decompose the computing task to be performed by the graphics processing unit into multiple subtasks and distribute them to the computing units for parallel execution. For example, in triton, the size of a single tensor block, that is, the number of elements in a single tensor block, can be set through parameters such as block_size and grid_size; at this time, when the graphics processing unit starts a kernel function written in a programming language such as triton, it will create multiple work groups based on the grid_size parameter set by the user. These work groups will be organized into an N-dimensional grid, and each work group will be assigned corresponding N-dimensional coordinates. The block_size parameter indicates the step size of the work group in each dimension. Based on the coordinates of the work group, the block_size parameter, and the base address corresponding to the first element, the address of the element in the tensor block corresponding to the work group can be determined, thereby dividing the tensor to be processed into tensor blocks corresponding to each work group. For example, the base address is A, the coordinates corresponding to a certain work group are (4,0), and the step size of the two dimensions is determined to be (128,1) based on the block_size parameter. Then the elements stored at addresses A+512 to A+639 constitute the tensor block corresponding to the work group.

[0061] It should be noted that the workgroup itself is only used to indicate the address range of a tensor block. The workgroup itself may not contain any threads or only contain one thread, and the workgroup works in SPMD (Single-Program, Multiple-Data) mode, that is, each workgroup can be used to execute the same program (i.e., instructions) on multiple elements contained in the corresponding tensor block.

[0062] It can be understood that the number of workgroups is determined by the kernel function written in a programming language such as triton, while the number of computing units in a single image processing unit is fixed. Based on this, when the workgroups created based on the kernel function are assigned to computing units, each computing unit can correspond to multiple workgroups, that is, each computing unit can be assigned multiple tensor blocks.

[0063] In one embodiment, after the tensor to be processed is divided into multiple tensor blocks and multiple work groups are created, these work groups can be directly assigned to various computing units, so that each computing unit is used to process a fixed number of tensor blocks.

[0064] In addition, referring to step S201, it can be known that the first instruction scheduling unit itself maintains an instruction queue, and the first instruction scheduling unit may obtain the target instruction from the instruction queue for instruction scheduling after detecting that the hardware unit for executing an instruction in the instruction queue is idle and the data required to execute the instruction is ready. That is, in this embodiment, the first instruction scheduling unit may take out the target instruction from the instruction queue for instruction scheduling after determining the target tensor block and the target unit.

[0065] In step S204, the first instruction scheduling unit is integrated outside the execution unit. After determining the target work group corresponding to the target unit, the first instruction scheduling unit can directly perform instruction scheduling in units of work groups, and schedule the target instruction to the corresponding target work group. Specifically, in this embodiment, after creating multiple work groups, each work group will be provided with a corresponding work group identifier, i.e., block id. Based on this, after determining the target work group assigned to the target unit for execution, the first instruction scheduling unit can directly schedule the target instruction to the target work group based on the work group identifier corresponding to the target work group. In this way, instruction scheduling can be performed in units of work groups by the first instruction scheduling unit. In this instruction scheduling mode, referring to step S203, it can be known that a single work group itself corresponds to at most one thread and the work group works in a single program and multiple data mode, i.e., in this embodiment, at most one thread is used to indicate all elements of the tensor block corresponding to a work group to the target unit, without involving the situation where multiple threads collaborate to complete the same computing task, so there is no need to manage the operation of multiple threads by setting a synchronization mechanism.

[0066] In addition, during the operation of the neural network model, the graphics processing unit often needs to execute multiple instructions at the same time. For example, under flash attention, each computing unit needs to perform data copying tasks and matrix multiplication tasks on different data at the same time. In the related art, when the instruction scheduler in the execution unit performs instruction scheduling in units of thread bundles, since each thread bundle only includes 32 threads, a data processing task may require multiple thread bundles to complete. When the computing unit needs to execute multiple data processing tasks at the same time, the instruction scheduler of the same execution unit needs to schedule different instructions to different thread bundles, and the number of thread bundles to which each instruction needs to be scheduled varies with the number of threads executing a single data processing task. The instruction scheduler itself cannot intelligently identify how many thread bundles the instruction corresponding to each data processing task needs to be scheduled to execute. This requires the user to perform Warp specialization on each thread bundle during programming, that is, to additionally assign a thread bundle identifier to each thread bundle, and preset the instructions executed by each thread bundle. When performing instruction scheduling, the instruction scheduler determines which data processing task each thread bundle is used to complete by detecting the thread bundle identifier, thereby determining which instruction needs to be scheduled to the thread bundle. When performing Warp specialization, it is often necessary to combine some characteristics of the underlying hardware of the computing unit to maintain the efficiency of instruction scheduling and thread bundle execution, which greatly increases the complexity of user programming. In the instruction scheduling method proposed in this embodiment, the first instruction scheduling unit performs instruction scheduling in units of work groups. When programming, the user does not need to perform additional Warp specialization to guide the first instruction scheduling unit to schedule each instruction to which thread warps, thereby greatly reducing the complexity of user programming.

[0067] In addition, in the related art, since instruction scheduling is performed in units of thread bundles, thread bundles need to follow the SIMT mode for execution, that is, threads in the same thread bundle can only be used to execute the same instruction on different data. When users use thread block-based programming frameworks such as triton for programming, they will specify the size of a single work group by setting block_size in the kernel function. This size represents the number of elements corresponding to a single work group. If the value of block_size is not an integer multiple of the number of threads in the thread bundle, when the data corresponding to the work group is allocated to multiple thread bundles, the number of elements allocated to at least one thread bundle cannot be evenly distributed to all threads in the thread bundle, that is, at least one thread in the thread bundle will not be allocated the data to be processed. However, since a single thread bundle needs to follow the SIMT mode for execution, that is, threads in a thread bundle can only execute the same instruction on different data at the same time, these threads that are not allocated corresponding elements cannot be used to execute other instructions. At this time, some threads in a part of the thread bundle will be wasted. For example, block_size is set to 158, that is, a work group contains 158 units of data. When scheduling instructions in units of thread bundles, the instruction scheduler in the execution unit will distribute these 158 units of data to different threads for execution, divide them into 5 thread bundles and schedule the instructions to these 5 thread bundles respectively. However, there are actually only 30 threads in the 5th thread bundle that belong to these 158 threads. However, due to the limitation of SIMT, the remaining two threads in the thread bundle cannot be used to execute other instructions and are idle. This part of the computing resources will be wasted, and due to the idleness of these two threads, the synchronization process between threads within the thread bundle will become more complicated. Therefore, when writing kernel functions, users need to ensure that the data corresponding to a single workgroup can be evenly divided into several thread bundles to ensure the utilization of the computing resources of the graphics processing unit. This limits the flexibility of users in setting block_size to a certain extent. In addition, when users write operators based on programming frameworks such as triton, they often control the allocation of computing resources of the graphics processing unit in units of workgroups. However, in the computing unit, this instruction scheduling mode based on thread bundles is not friendly to thread block-based programming frameworks such as triton.In the method disclosed in this embodiment, instruction scheduling is performed in units of work groups through a first instruction scheduling unit integrated outside the execution unit. When performing instruction scheduling, the target instruction is directly scheduled to the work group corresponding to the tensor block that needs to be used to execute the target instruction, and the work group is scheduled to the corresponding first functional unit for execution, without creating threads through the instruction scheduler in the execution unit, and then scheduling instructions in units of thread bundles. Even if the data contained in a single work group is not an integer multiple of 32, some threads will not be idle, thereby improving the user's flexibility in setting block_size, and being more friendly to thread block-based programming frameworks such as triton.

[0068] In the embodiment disclosed in steps S201 to S204, by integrating the first instruction scheduling unit outside the execution unit, thereafter, when performing instruction scheduling, first identify the target unit for executing the target instruction, and when the target unit is a dedicated hardware such as a tensor core, a copy engine or a scalar processing unit, determine the target tensor block corresponding to the target instruction and the target workgroup corresponding to the target tensor block, the target workgroup indicating the memory address range of the target tensor block, and then, the first instruction scheduling unit performs instruction scheduling in units of workgroups, directly schedules the target instruction to the target workgroup, and schedules the target workgroup loaded with the target instruction to the target unit, so that the target unit is based on the target workgroup indicated by the target tensor block. The target tensor block is read from the memory address range indicated and the target instruction is executed on the target tensor block. In this way, the granularity of the computing unit as the underlying hardware during instruction scheduling is consistent with the task decomposition granularity set by the user in the kernel function, and the first instruction scheduling unit directly schedules the target instruction to the target work group instead of the thread or thread warp. At this time, there is no need to create multi-threading and thread warps, so there is no need to set up a synchronization mechanism to manage the operation of the thread warp, and there is no need to perform Warpspecialization to indicate what instructions each thread warp is used to execute. In this way, the instruction scheduling mode of the computing unit can be more suitable for kernels written in programming languages ​​such as triton.

[0069] Detailed description of step S201

[0070] In one embodiment, before step S201, the method further includes:

[0071] Step S301, obtaining a target command, where the target command is used to instruct a computing unit to start a target kernel corresponding to the target command, where the target kernel is composed of multiple instructions;

[0072] Step S302, parsing the target command through the command processor to determine the target kernel to be started, determining a plurality of instructions to be executed based on the target kernel, and writing the instructions to be executed into the instruction cache area of ​​the first instruction scheduling unit;

[0073] Step S301 includes:

[0074] Step S303: Obtain the target instruction from the corresponding instruction cache through the first instruction scheduling unit.

[0075] In step S301, the target command is a command stream generated based on the program code and sent to the graphics unit when the host executes the program code of the neural network model. The host can be a central processing unit. It can be understood that the graphics processing unit itself is composed of a large number of relatively simple computing units. It is suitable for tasks that require large-scale parallel computing, but it cannot handle some relatively complex control logic and computing tasks. In the relevant technology, the operation process of the neural network model is completed by the central processing unit and the graphics processing unit in collaboration. The central processing unit is used to run the program code of the neural network model and execute the relatively complex logic processing therein, while some relatively simple tasks that require large-scale parallel execution will be scheduled to the graphics processing unit for execution. At this time, the central processing unit will generate a corresponding command stream based on the program code of the executed neural network model and send it to the computing unit. The command stream contains multiple commands, and the target command is one of them. The target command indicates that the graphics processing unit needs to perform a data processing task.

[0076] Specifically, the program code of the neural network model is often composed of multiple operators that are used to complete specific computing tasks. Each operator can actually be regarded as a packaged kernel function. The kernel functions of these operators can be converted into a series of instructions that need to be executed in order to complete the computing tasks corresponding to the operator. When the central processing unit executes these operators in the program code of the neural network model, it will generate corresponding target commands and send them to the graphics processing unit to indicate which operator's kernel function the graphics processing unit needs to start.

[0077] Exemplarily, the neural network model is provided with an average pooling layer. When the central processing unit executes the program code of the average pooling layer, it sends a command to the graphics processing unit to perform average pooling on a matrix. The essence of average pooling is to find the average value of each region in the corresponding matrix according to a given pooling window. Based on this, after the command parser parses the command, it generates a data copy instruction for copying the data of each region of the matrix stored in the global memory to the local memory of each computing unit according to the pooling window, and a mean calculation instruction for instructing the computing unit to calculate the mean of the data of each region in the matrix.

[0078] In step S302, the command processor is a module used to parse the target command issued by the central processing unit into instructions that can be understood and executed by the hardware module in the graphics processing unit. It is understandable that the underlying hardware in the graphics processing unit can only understand machine language, and cannot directly understand the program code of the neural network model written by the user. Therefore, the graphics processing unit needs to parse the commands in the command stream issued by the central processing unit through the command processor, and convert these commands into instructions that can be understood and executed by the hardware in the graphics processing unit. Specifically, the command processor parses the target command to determine the target kernel corresponding to the target command, that is, the kernel function that needs to be started when executing the target command. Referring to the detailed description of step S301, it can be seen that the kernel function is composed of a series of instructions required to complete the computing task of the target command. After determining the kernel function that needs to be started, multiple instructions that need to be executed by the computing unit subsequently are determined, that is, the instructions to be executed. It can be understood that these instructions often need to be executed in order, and some of them need to wait for other instructions to be executed before they can start executing. Based on this, after obtaining multiple instructions to be executed, these instructions to be executed are written into the instruction cache area corresponding to the first instruction scheduling unit to form an instruction queue, so that the first instruction scheduling unit can subsequently read each instruction to be executed in order from the corresponding instruction cache area as the target instruction for instruction scheduling.

[0079] In step S303, it can be understood that, referring to the relevant description of step S201, the first instruction scheduling unit monitors in real time whether the execution conditions of each instruction to be executed in the instruction cache are met, that is, whether there is an available idle working group and whether the data required to execute each instruction is ready. When the execution condition of an instruction in the instruction cache is met, the first instruction scheduling unit reads the corresponding instruction to be executed from the instruction cache as the target instruction to be scheduled.

[0080] In the embodiment disclosed in step S301 to step S303, the target command issued by the central processing unit is obtained, and the target command is parsed by the command processor to determine the target kernel to be started, and then the target kernel corresponding to the target command is started, and each instruction contained in the target kernel is written as an instruction to be executed into the instruction cache corresponding to the first instruction scheduling unit, so as to convert the target command issued by the central processing unit into a machine code instruction that can be understood and executed by the hardware module in the computing unit. In this way, the first instruction scheduling unit can perform instruction scheduling by sequentially reading each instruction to be executed from the instruction cache as a target instruction and scheduling it to the corresponding work group.

[0081] In one embodiment, the target kernel includes at least a main function kernel part, a scalar kernel part and a vector kernel part, the scalar kernel part includes a plurality of preset scalar operation kernels, the vector kernel part includes a plurality of preset vector operation kernels, and the main function kernel part includes a call instruction for calling at least one of the scalar operation kernel and the vector operation kernel. Each scalar operation kernel is composed of at least one scalar operation instruction, and the vector operation kernel is composed of at least one vector operation instruction.

[0082] For example, refer to Figure 4 , Figure 4 is a schematic diagram of an operator kernel of an embodiment of the present disclosure, the operator kernel includes a main function kernel part, a scalar kernel part and a vector kernel part, wherein in the scalar kernel part, lines 0 to 12 constitute the first scalar operation kernel of the scalar kernel part, namely kernel0, wherein the ret instruction in line 12 indicates that kernel0 has been executed and returns to the main function kernel part, and so on. Specifically, as Figure 4 As shown in the figure, the instruction with serial number 38 in the main function kernel, namely launch vector 100, is a call instruction for calling the instruction with serial number 100 in the vector kernel part. When the main function kernel part executes the instruction with serial number 38, it will jump to execute the instruction with serial number 100 in the vector kernel part, and execute the subsequent instructions in the vector kernel part in sequence until the ret instruction is executed, indicating that kernel1 in the vector kernel part has been executed and returned to the main function kernel; similarly, when the instruction with serial number 70 in the main function kernel is executed, it will jump to execute the instruction with serial number 13 in the scalar kernel part (namely Figure 4 13 kernel1 in the main function kernel) to call kernel1 of the scalar kernel part, and execute the subsequent instructions in the scalar kernel part in sequence until the ret instruction is executed. At this time, it indicates that kernel1 of the scalar kernel part has been executed, and returns to the main function kernel part to execute the instruction with sequence number 71 in the main function kernel.

[0083] Based on this, by setting the scalar kernel and vector kernel parts in the target kernel, and adding the preset scalar operation kernel and vector operation kernel to the corresponding scalar kernel part and vector kernel part respectively, when the user writes the kernel function of the operator, he only needs to write the main function kernel part, and when the operator needs to perform scalar operation and vector operation, he can directly jump to the corresponding position in the scalar kernel part or the vector kernel part through the call instruction.

[0084] At the same time, when repeated scalar operations or vector operations need to be performed on multiple nodes, it is only necessary to add call instructions for jumping to the corresponding positions in the scalar kernel part or the vector kernel part at the corresponding positions in the main function kernel, without repeatedly writing instructions for performing the same type of operation in the main function kernel. Figure 5 In the example shown, the 38th instruction in the main function kernel is used to call the vector operation kernel kernel1. If in the operator, after executing the 48th instruction of the main function kernel, it is necessary to execute the vector operation task corresponding to the corresponding operation kernel kernel1 again, the 49th instruction of the main function kernel can be written as launch vector 100, so that the vector operation kernel kernel1 is called again when the 49th instruction of the main function kernel is executed. In this way, the complexity of the kernel function of the operator can be further reduced, so as to reduce the difficulty of users writing the kernel function of the operator.

[0085] In addition, it can be understood that since the graphics processing unit itself is often used to perform some relatively simple but large-scale parallel computing tasks, and the operation of tensors above two dimensions is relatively complex, and the operation of tensor data above two dimensions in the neural network model is also mainly matrix multiplication, therefore, the hardware structure of the tensor core itself is optimized and designed with the goal of accelerating matrix multiplication. For example, referring to the above embodiment, the tensor core is composed of a general matrix multiplication unit and an accumulation buffer. At this time, the tensor core itself is often only used to perform matrix multiplication and matrix multiplication and accumulation computing tasks, that is, the instructions executed by the tensor core actually often only include matrix multiplication instructions and matrix multiplication and accumulation instructions, and the instructions executed by the tensor core are actually relatively simple. Based on this, in this embodiment, there is no need to specially set up a part for filling in the tensor operation kernel in the target kernel; similarly, the copy engine itself is used to perform data copy instructions, and the type of instructions executed by itself is also relatively simple. Based on this, in this embodiment, there is no need to set up the kernel function part corresponding to the copy engine in the target kernel.

[0086] Detailed description of step S202

[0087] In one embodiment, referring to Figure 5 , step S202 includes:

[0088] Step S501, identifying the instruction type corresponding to the target instruction;

[0089] Step S502 , determining a target unit for executing a target instruction based on the instruction type.

[0090] In step S501, the instruction type is used to characterize the type of data processing task corresponding to the target instruction. It can be understood that each first functional unit is actually a hardware structure that is designed for certain specific types of data processing tasks. The underlying hardware, for example, the tensor core is provided with a general matrix multiplication unit and an accumulation buffer, and the tensor core is designed to perform matrix multiplication and matrix multiplication and accumulation calculation tasks; each first functional unit actually only executes instructions corresponding to one or more specific types of data processing tasks, for example, the scalar processing unit is used to execute basic arithmetic operation instructions such as addition, subtraction, multiplication and division, and basic logical operation instructions such as AND, OR, and NOT. Based on this, in this embodiment, multiple preset instructions can be divided into different instruction types based on the instructions executed by each first functional unit, and each instruction belongs to one of the instruction types.

[0091] Specifically, in one embodiment, the instruction type includes at least one of: tensor operation instructions, scalar operation instructions, vector operation instructions and data copy instructions. Tensor operation instructions are instructions that need to be scheduled to the tensor core for execution. Tensor operation instructions refer to instructions used when operating on tensors of two or more dimensions. In this embodiment, tensor operation instructions at least include matrix multiplication instructions or matrix accumulation multiplication instructions. Scalar operation instructions are instructions that need to be scheduled to the scalar processing unit for execution. They are instructions used when operating on zero-dimensional scalar data, such as basic arithmetic instructions such as addition, subtraction, multiplication, division, exponentiation, and averaging, and basic logical operation instructions such as AND or NOT. Data copy instructions are instructions that need to be scheduled to the copy engine, which are used to instruct the copy engine to copy data stored in one memory address to another memory address. For example, the data that the computing unit needs to process later is copied from the global memory to the local register or local memory of the computing unit, or the data generated by the local calculation of the computing unit is copied to the global memory for access by other computing units in the graphics processing unit. Vector operation instructions are instructions that need to be scheduled to the execution unit for execution, and are used to perform operations on one-dimensional vector data, such as normalization, vector dot product, etc. Based on this, according to the type of the first functional unit integrated in the computing unit, the preset instructions are divided into a plurality of instruction types corresponding to the execution unit and each first functional unit, thereby, the target unit for executing the target instruction can be determined based on the instruction type of the target instruction.

[0092] In a possible embodiment, it can be understood that each first functional unit can only be used to execute a specific type of instruction, and the first functional unit in the computing unit includes at least one of a tensor core, a copy engine, and a scalar processing unit, that is, according to the different hardware structures of the computing unit itself, the first functional unit integrated in the computing unit is different. Based on this, in this embodiment, the division of instruction types is determined based on the first functional unit integrated in the computing unit. According to the different hardware structures of the computing unit itself, the instruction type can only include one or more of the above-mentioned tensor operation instructions, scalar operation instructions, vector operation instructions, and data copy instructions. For example, in addition to the execution unit, the computing unit only integrates the tensor core as the first functional unit, and does not integrate the scalar processing unit and the copy engine. At this time, the instruction type only includes tensor operation instructions and vector operation instructions. When and only when the instruction type of the target instruction is a tensor operation instruction, the tensor core is determined as the corresponding computing unit, and if the target instruction is a vector operation instruction or other instructions that are not divided into instruction types, the execution unit is determined as the corresponding target unit. Similarly, when the computing unit integrates tensor cores, scalar processing units and replication engines at the same time, the instruction types may include tensor operation instructions, scalar operation instructions, vector operation instructions and data replication instructions. Of course, if the computing unit integrates other dedicated hardware in addition to tensor cores, replication engines and scalar processing units, the instruction type corresponding to the dedicated hardware can also be determined based on the data processing tasks performed by the dedicated hardware, and the instructions that need to be scheduled to be executed by the dedicated hardware can be divided into the instruction type corresponding to the dedicated hardware.

[0093] In another possible embodiment, when setting the instruction type, the type of the first functional unit integrated in the computing unit may not be considered, but the preset instructions may be directly divided into multiple instruction types such as tensor operation instructions, scalar operation instructions, vector operation instructions, and data copy instructions. When the first instruction scheduling unit performs instruction scheduling, after identifying the instruction type of the target instruction, the target unit corresponding to the target instruction is determined according to the situation of the first functional unit integrated in the computing unit itself, that is, when the computing unit is integrated with the first functional unit corresponding to the instruction type, the first functional unit is determined as the corresponding target unit, and if the computing unit is not integrated with the first functional unit corresponding to the instruction type, the execution unit is determined as the corresponding target unit. Exemplarily, the computing unit is only integrated with a tensor core and a copy engine as the first functional unit, and the preset instruction types include tensor operation instructions, scalar operation instructions, vector operation instructions, and data copy instructions. At this time, if the instruction type corresponding to the target instruction is a scalar operation instruction, since the corresponding first functional unit is not integrated in the computing unit, the execution unit is determined as the corresponding target unit.

[0094] In step S502, referring to step S501, it can be known that the execution unit and each first functional unit correspond to different instruction types respectively. After determining the instruction type corresponding to the target instruction, the corresponding target unit can be determined based on the instruction type.

[0095] Exemplarily, the target instruction is a basic arithmetic operation instruction for adding two data, and its instruction type is a scalar operation instruction. When a scalar operation unit is integrated in the computing unit, the scalar operation instruction will be scheduled to the scalar processing unit for processing, that is, the target unit corresponding to the target instruction is the scalar processing unit.

[0096] In the embodiment disclosed in step S501 to step S502, by presetting the instruction types corresponding to the execution unit and each first functional unit, when the first instruction scheduling unit obtains the target instruction to be scheduled to be executed, it identifies the instruction type corresponding to the target instruction, and determines the target unit for executing the target instruction based on the instruction type. Based on this, after obtaining the target instruction, the first instruction scheduling unit can automatically determine which hardware module in the computing unit needs to schedule the target instruction to execute.

[0097] In one embodiment, referring to Figure 6 Before step S501, the method further includes:

[0098] Step S601, creating a first instruction set;

[0099] Step S602, dividing the preset instructions into a plurality of instruction subsets, each instruction subset including at least one preset instruction, and the preset instructions in a single instruction subset correspond to the same instruction type;

[0100] Step S501 includes:

[0101] Step S603: Identify the instruction subset corresponding to the target instruction to determine the instruction type corresponding to the target instruction.

[0102] In step S601, the first instruction set is a preset instruction set, and the first instruction set includes a plurality of preset instructions; the preset instructions are machine code instructions preset by the user and can be understood and executed by the underlying hardware of the graphics processing unit. It can be understood that each hardware module in the computing unit can only understand and execute machine code instructions, but cannot directly understand the program code written by the user. Based on this, in this embodiment, it is necessary to pre-create a first instruction set composed of machine code instructions that can be understood and executed by each hardware module. In this way, by converting the program code written by the user into a series of corresponding instructions and dispatching them to the corresponding hardware modules for execution, the corresponding data processing tasks can be completed by each hardware module in the computing unit.

[0103] In step S602, the preset instructions include various data processing instructions that can be executed by the computing unit, and referring to the description of the above step S501, it can be known that when the computing unit is integrated with the first functional unit, it is necessary to identify the instruction type corresponding to the instruction to determine which hardware module in the computing unit each instruction needs to be scheduled to. Based on this, in this embodiment, after creating the first instruction set, the preset instructions in the first instruction set can be divided based on different instruction types to obtain instruction subsets corresponding to each instruction type.

[0104] In step S603, since the first instruction set is divided into instruction subsets corresponding to each instruction type, each preset instruction will be divided into an instruction subset. Based on the instruction subset to which the target instruction belongs and the correspondence between each instruction subset and the instruction type, the instruction type corresponding to the target instruction can be determined.

[0105] In the embodiment disclosed in steps S601 to S603, after creating the first instruction set, the preset instructions in the first instruction set are divided into instruction subsets corresponding to different instruction types. In this way, when the target instruction needs to be subsequently scheduled, the instruction type corresponding to the target instruction can be determined based on the instruction subset to which the target instruction belongs.

[0106] Detailed description of step S204

[0107] In one embodiment, referring to Figure 7 , before step S204, the method further includes:

[0108] Step S701, obtaining work group creation parameters, the work group creation parameters including work group size and work group number parameters, the work group size represents the size of the tensor block corresponding to a single work group in each dimension;

[0109] Step S702, creating multiple workgroups based on the workgroup number parameter and assigning a workgroup identifier to each workgroup, and dividing the tensor to be processed into multiple tensor blocks based on the workgroup size, wherein the workgroups and tensor blocks correspond one to one.

[0110] In step S701, the workgroup creation parameter is used to instruct the graphics processing unit how to divide the tensor to be processed and create workgroups corresponding to each tensor block, so as to allocate the computing resources of the graphics processing unit to each workgroup, so as to execute the same instruction or different instructions in parallel on different data. In this embodiment, the workgroup creation parameter includes at least a workgroup size and a workgroup number parameter, wherein the workgroup size is used to indicate the size of the tensor block corresponding to a single workgroup in each dimension, that is, the number of elements contained in a single workgroup in each dimension, and the number of elements contained in a single workgroup can be determined based on the workgroup size, and the workgroup number parameter is used to indicate the number of workgroups. In one embodiment, the workgroup number parameter can be a value that directly represents the number of workgroups to be created, or it can also be an array of the size of the N-dimensional grid composed of the workgroups. For example, the workgroup number parameter can be (M, N, K), which means that M*N*K workgroups need to be created, and these workgroups are organized into a three-dimensional grid with a shape of (M, N, K). It should be noted that, in this embodiment, the product of the workgroup size and the workgroup number parameter may correspond to the total number of elements in the tensor to be processed, thereby ensuring that each tensor block obtained by dividing the tensor to be processed corresponds to a workgroup.

[0111] In another embodiment, the workgroup number parameter may be a set of parameters for characterizing the shape of an N-dimensional grid formed by the workgroups. Specifically, referring to step S203, the user may control the shape of the grid formed by the workgroups by setting grid_size in the kernel function to organize the workgroups into a workgroup grid. Exemplarily, the user may set gridsize to (a2, b2, c2) in the kernel function. In this case, it indicates that the workgroup grid includes a2 workgroups along the x dimension, b2 workgroups along the y dimension, and c2 workgroups along the z dimension. In this case, the number of workgroups to be created is a2*b2*c2.

[0112] In step S702, the workgroup number parameter is used to characterize the number of workgroups to be created, and the workgroup size characterizes the number of elements contained in each dimension of the tensor block corresponding to a single workgroup. By creating a workgroup based on the workgroup number parameter and processing the tensor based on the workgroup size, multiple workgroups and the tensor blocks corresponding to each workgroup can be obtained.

[0113] It is understandable that after creating multiple workgroups, each workgroup corresponds to a different tensor block, and a corresponding workgroup identifier needs to be assigned to each workgroup to distinguish each workgroup, and the identifier corresponding to each workgroup is different. For example, a serial number can be assigned to each workgroup as the identifier corresponding to the workgroup.

[0114] In another possible implementation, multiple workgroups are organized into an N-dimensional workgroup grid, and each workgroup is a grid point in the workgroup grid. At this time, each workgroup has corresponding coordinates to indicate the position of the workgroup in the grid. Since the coordinates of each workgroup in the workgroup grid are different, based on this, the coordinates corresponding to each workgroup can be the workgroup ID corresponding to the workgroup. For example, the workgroups constitute a three-dimensional grid, and the workgroup ID corresponding to each workgroup can be expressed as (x1, y1, z1).

[0115] At this time, referring to the relevant description of the above step S203, the address range of the elements corresponding to the tensor block corresponding to the work group in the memory can be indicated based on the identifier corresponding to the work group, the address of the first element in the tensor to be processed, and the size of the work group.

[0116] It should be noted that, referring to the above step S301, the target kernel often includes multiple instructions that need to be executed in order, and each time a kernel function corresponding to an operator is executed, only one workgroup needs to be created, and each workgroup corresponds to a tensor block in the original input tensor of the operator and the intermediate data generated by the tensor block after executing each instruction. These workgroups will be assigned to execute multiple instructions in the kernel function. Specifically, each workgroup will be assigned to the corresponding hardware module for execution after being scheduled. When the corresponding hardware module completes the execution of the workgroup, the workgroup will enter an idle state until the next instruction to be executed is scheduled to the workgroup. At this time, if instruction scheduling is performed in units of thread bundles through the relevant technology, the corresponding data in a work group will be divided into multiple thread bundles for instruction scheduling. At this time, once a thread bundle executes the previous instruction before other thread bundles, the computing resources of the thread bundle will be released in advance, and the instruction scheduler will schedule the next instruction to the thread bundle in advance so that the thread bundle is used to execute the next instruction. This will cause the threads in a work group to execute different instructions at the same time, causing conditional competition or other unpredictable situations. Therefore, when instruction scheduling is performed in units of thread bundles through the methods in the relevant technology, it is necessary to set up a complex synchronization mechanism between thread bundles to constrain each thread bundle to prevent some thread bundles from being used to execute the next instruction in advance. However, when the instruction scheduling method disclosed in the present invention is used to schedule instructions in units of work groups using the first instruction scheduling unit, there is no need to create multiple threads and thread bundles. Instead, the instructions are directly scheduled to the work group, and then the work group is scheduled to the target unit for execution. Before the work group is executed, the work group will not enter an idle state and the occupied computing resources will not be released. The first instruction scheduling unit will not schedule the next instruction to the work group. When and only when the work group is executed, that is, all elements in the corresponding tensor block have been executed The data processing task corresponding to the target instruction, at this time, the first instruction scheduling unit will schedule the next instruction to the work group. The entire process does not involve multi-threaded processing, and there is no need to consider the situation of conditional competition caused by scheduling the next instruction in advance to part of the thread bundle that has been executed first. Therefore, there is no need to constrain the behavior of each thread by setting a relatively complex synchronization mechanism.

[0117] In the embodiment disclosed in steps S701 to S703, workgroup creation parameters are obtained and multiple workgroups are created accordingly, and then the tensors to be processed are divided into tensor blocks corresponding to each workgroup based on the workgroup size, and the workgroup identifier corresponding to each workgroup is determined. In this way, the memory address range of the elements in the tensor blocks corresponding to each workgroup can be determined subsequently according to the workgroup identifier.

[0118] Detailed description of instruction scheduling by thread warp in compatible execution units

[0119] In one embodiment, each execution unit is further provided with a second instruction scheduling unit, referring to Figure 8 , the method further comprises:

[0120] Step S801, if the target unit is an execution unit, determine a target workgroup assigned to the target unit, and create multiple threads corresponding to the target workgroup, wherein each thread corresponds to data in a tensor block corresponding to the target workgroup;

[0121] Step S802, the first instruction scheduling unit sends the target instruction to the second instruction scheduling unit in the execution unit;

[0122] Step S803, in the execution unit, determining the created threads into a plurality of thread warps, wherein each thread warp includes a second number of threads;

[0123] Step S804: In the execution unit, the target instruction is respectively scheduled to threads in a plurality of warps by the second instruction scheduling unit.

[0124] In step S801, if the target unit is an execution unit, the hardware structure of the execution unit is designed based on the mode that the execution unit performs instruction scheduling and concurrent execution of thread bundles in units of thread bundles, that is, the hardware structure of the execution unit itself is more suitable for instruction scheduling and concurrent execution of threads in units of thread bundles. At the same time, the memory resources and computing resources of a single execution unit are relatively limited, and the size of the work group is set by the user when writing the kernel function. The size of a single work group is variable. When the size of a single work group is large, that is, there are many elements in the tensor block corresponding to the work group, the computing resources and memory resources of the execution unit itself may not be sufficient to concurrently execute the target instruction for all elements in the tensor block corresponding to the work group, that is, the execution unit itself may not be able to support instruction scheduling and execution in units of work groups. Based on this, in this embodiment, if the target unit is an execution unit, instruction scheduling is not performed through the first instruction scheduling unit, but multiple threads are created in the execution unit, so that instruction scheduling is performed in units of thread bundles through the second instruction scheduling unit in the execution unit later. Specifically, the number of threads created may be determined based on the number of elements in the target tensor block, for example, each thread corresponds to one element in the target tensor block.

[0125] In step S802, referring to step S801, the execution unit itself is more suitable for executing data processing tasks in units of thread bundles, while the first instruction scheduling unit performs instruction scheduling in units of work groups. If the target instruction is directly scheduled to the work group composed of threads assigned to the execution unit through the first instruction scheduling unit, and then the work group loaded with the target instruction is scheduled to the execution unit for execution, this may make it difficult for the execution unit to execute and manage these threads in units of thread bundles. Based on this, in this embodiment, when the target unit is the execution unit, the first instruction scheduling unit can directly send the target instruction to the second instruction scheduling unit in the execution unit, and the second instruction scheduling unit will schedule the target instruction to each thread in units of thread bundles, so that it can better adapt to the mode in which the execution unit itself manages and executes threads in units of threads.

[0126] In step S803, the second number refers to the number of threads in a single thread bundle, and the second number can be 16, 32, 64, etc. Specifically, the second number is determined by the graphics processing unit itself, and under the premise of being determined by the graphics processing unit, the second number is fixed. As described in steps S801 and S802, the execution unit itself is more suitable for managing and executing threads in units of thread bundles. Therefore, in this embodiment, after scheduling multiple threads to the execution unit, the threads assigned to the execution unit are divided into multiple thread bundles, and then the second instruction scheduling unit in the execution unit can perform instruction scheduling in units of thread bundles.

[0127] In step S804, the second instruction scheduling unit may be a warp scheduler, which performs instruction scheduling in units of thread bundles, that is, schedules one instruction to 32 threads of the same thread bundle each time. Specifically, after the threads assigned to the execution unit are divided into multiple thread bundles, the process of the second instruction scheduling unit performing instruction scheduling in units of thread bundles can refer to the process of performing instruction scheduling in units of thread bundles in the related art, which will not be repeated here. It should be emphasized that in the embodiment of the present disclosure, when and only when the target instruction is an instruction that needs to be assigned to the execution unit to be completed, the second instruction scheduling unit in the execution unit will be used to schedule the instruction in units of thread bundles, and when the target instruction is an instruction executed by a tensor core, a copy engine or a scalar processing unit, the instruction will not be scheduled in units of thread bundles through the second instruction scheduling unit, but the instruction will be scheduled in units of work groups through the first instruction scheduling unit integrated outside the execution unit.

[0128] In the embodiment disclosed in steps S801 to S804, when the target unit is an execution unit, the first instruction scheduling unit is not used to perform instruction scheduling in units of work groups. Instead, the threads assigned to the target unit are directly scheduled to the execution unit, and instruction scheduling is performed in units of thread bundles through the second instruction scheduling unit in the execution unit, thereby being more adapted to the data processing mode of the execution unit itself, so that the instruction scheduling method disclosed in the present invention can be better compatible with the hardware characteristics of the execution unit itself.

[0129] Description of the apparatus and device of the present disclosure

[0130] Reference Fig. 9 , Fig. 9 is a structural diagram of an instruction scheduling device 900 proposed in the present disclosure, the device comprising:

[0131] An instruction acquisition unit 910, configured to acquire a target instruction through a first instruction scheduling unit;

[0132] A first identification unit 920, used to identify a target unit for executing a target instruction, the target unit being one of an execution unit or a first functional unit;

[0133] A target workgroup determining unit 930 is used to determine a target tensor block corresponding to a target instruction and a target workgroup corresponding to the target tensor block when the target unit is a first functional unit, wherein the target workgroup is used to indicate a memory address range of the target tensor block;

[0134] The first scheduling unit 940 is used to schedule the target instruction to the target work group through the first instruction scheduling unit, and schedule the target work group to the target unit, so that the target unit reads the target tensor block based on the memory address range and executes the target instruction on the target tensor block.

[0135] The instruction scheduling device disclosed in the present invention is used to execute the instruction scheduling method as in the above embodiment. Its specific processing process is the same as the instruction scheduling method in the above embodiment, and will not be repeated here.

[0136] The present disclosure also provides an electronic device 1000, including:

[0137] at least one processor, and

[0138] a memory communicatively connected to at least one processor; wherein,

[0139] The memory stores instructions, and the instructions are executed by at least one processor so that when the at least one processor executes the instructions, a method as described in any one of the above embodiments of the present application is implemented.

[0140] Combine the following Fig.10The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 1010 , a memory 1020 , an input / output interface 1030 , a communication interface 1040 and a bus 1050 .

[0141] The processor 1010 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0142] The memory 1020 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1020 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1020, and the processor 1010 calls and executes the instruction scheduling method of the embodiments of this disclosure;

[0143] Input / output interface 1030, used to implement information input and output;

[0144] Communication interface 1040, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0145] Bus 1050 , which transmits information between various components of the device (e.g., processor 1010 , memory 1020 , input / output interface 1030 , and communication interface 1040 );

[0146] The processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 are connected to each other in communication within the device via a bus 1050 .

[0147] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the instruction scheduling method of the above embodiment, which will not be repeated here.

[0148] The computing unit disclosed in the present invention is used to execute the instruction scheduling method as in the above embodiment, and its specific processing process is the same as the instruction scheduling method in the above embodiment, which will not be repeated here.

[0149] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0151] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.

[0152] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0156] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. An instruction scheduling method, characterized in that: Applied to a computing unit, the computing unit includes a first instruction scheduling unit, at least one execution unit, and at least one first functional unit, the first functional unit includes at least one of a copy engine, a tensor core, and a scalar processing unit, the method includes: Acquire a target instruction through the first instruction scheduling unit; identifying a target unit for executing the target instruction, the target unit being one of the execution unit or the first functional unit; In a case where the target unit is the first functional unit, determining a target tensor block corresponding to the target instruction and a target workgroup corresponding to the target tensor block, wherein the target workgroup is used to indicate a memory address range of the target tensor block; The target instruction is scheduled to the target workgroup through the first instruction scheduling unit, and the target workgroup is scheduled to the target unit, so that the target unit reads the target tensor block based on the memory address range and executes the target instruction on the target tensor block.

2. The instruction scheduling method according to claim 1, characterized in that: The first instruction scheduling unit is provided with a corresponding instruction cache area, the computing unit further includes a command processor, and before acquiring the target instruction through the first instruction scheduling unit, the method further includes: Obtaining a target command, wherein the target command is used to instruct the computing unit to start a target kernel corresponding to the target command, wherein the target kernel is composed of a plurality of instructions; Parsing the target command by the command processor to determine the target kernel to be started, determining a plurality of instructions to be executed based on the target kernel, and writing the instructions to be executed into the instruction cache of the first instruction scheduling unit; The acquiring the target instruction through the first instruction scheduling unit includes: The target instruction is obtained from the corresponding instruction cache through the first instruction scheduling unit.

3. The instruction scheduling method according to claim 2, characterized in that: The target kernel includes at least a main function kernel part, a scalar kernel part and a vector kernel part. The scalar kernel part includes multiple preset scalar operation kernels, and the vector kernel part includes multiple preset vector operation kernels. The main function kernel part calls the scalar operation kernel and the vector operation kernel through a calling instruction.

4. The instruction scheduling method according to claim 1, characterized in that: The identifying a target unit for executing the target instruction comprises: Identify an instruction type corresponding to the target instruction, where the instruction type is used to characterize a type of data processing task corresponding to the target instruction, and the execution unit and each of the first functional units correspond to different instruction types; A target unit for executing the target instruction is determined based on the instruction type.

5. The instruction scheduling method according to claim 4, characterized in that: Prior to identifying a target unit for executing the target instruction, the method further includes: Creating a first instruction set, wherein the first instruction set includes a plurality of preset instructions, and the target instruction is one of the preset instructions; Dividing the preset instructions into a plurality of instruction subsets, each of the instruction subsets includes at least one of the preset instructions, and the preset instructions in a single instruction subset correspond to the same instruction type; The identifying the instruction type corresponding to the target instruction includes: An instruction subset corresponding to the target instruction is identified to determine an instruction type corresponding to the target instruction.

6. The instruction scheduling method according to claim 4, characterized in that: The instruction type includes at least one of the following: A tensor operation instruction, the tensor operation instruction indicating an instruction to be executed by the tensor core; a scalar operation instruction, the scalar operation instruction indicating an instruction to be executed by the scalar processing unit; a vector operation instruction, the vector operation instruction indicating an instruction to be executed by the execution unit; A data replication instruction is an instruction executed by the replication engine.

7. The instruction scheduling method according to claim 1, characterized in that: Before determining the target tensor block corresponding to the target instruction and the target working group corresponding to the target tensor block, the method further includes: Obtaining workgroup creation parameters, wherein the workgroup creation parameters include workgroup size and workgroup number parameters, wherein the workgroup size represents the size of a tensor block corresponding to a single workgroup in each dimension; Based on the workgroup number parameter, a plurality of workgroups are created and a workgroup identifier is assigned to each of the workgroups, and based on the workgroup size, the tensor to be processed is divided into a plurality of tensor blocks, wherein the workgroups and the tensor blocks correspond one to one.

8. The instruction scheduling method according to claim 1, characterized in that: Each of the execution units includes a second instruction scheduling unit, and the method further includes: If the target unit is the execution unit, determine a target workgroup assigned to the target unit, and create a plurality of threads corresponding to the target workgroup, wherein each of the threads corresponds to data in a tensor block corresponding to the target workgroup; The first instruction scheduling unit sends the target instruction to the second instruction scheduling unit in the execution unit; In the execution unit, the created threads are determined to be a plurality of thread warps, wherein each thread warp includes a second number of threads; In the execution unit, the target instructions are respectively scheduled to threads in the plurality of thread warps through the second instruction scheduling unit.

9. An instruction scheduling device, characterized in that: The device comprises: an instruction acquisition unit, configured to acquire a target instruction through the first instruction scheduling unit; A first identification unit, configured to identify a target unit for executing the target instruction, wherein the target unit is one of an execution unit or a first functional unit; a target workgroup determining unit, configured to determine, when the target unit is the first functional unit, a target tensor block corresponding to the target instruction and a target workgroup corresponding to the target tensor block, wherein the target workgroup is used to indicate a memory address range of the target tensor block; A first scheduling unit is used to schedule the target instruction to the target workgroup through the first instruction scheduling unit, and schedule the target workgroup to the target unit, so that the target unit reads the target tensor block based on the memory address range and executes the target instruction on the target tensor block.

10. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the instruction scheduling method according to any one of claims 1 to 8 is realized.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the instruction scheduling method according to any one of claims 1 to 8.

Images