Data scheduling method and device, electronic equipment and medium

Abstract

The invention provides a data scheduling method and device, electronic equipment and a medium. The method comprises the steps of obtaining a first input tensor and a target working group grid corresponding to the first input tensor; based on the working group coordinates corresponding to the working groups, dividing the target working group grid into working group sets respectively corresponding to the block processing clusters, and scheduling the working groups in the working group sets to the corresponding block processing clusters; in the block processing cluster, respectively mapping the working groups in the corresponding working group set to the computing units in the block processing cluster; the first input tensor is divided into tensor blocks corresponding to each work group, and each tensor block is stored in a local memory of a computing unit for executing the corresponding work group. Based on the data scheduling method disclosed by the invention, the efficiency of accessing the storage unit by the computing unit for data access can be improved, the access power consumption is reduced, and meanwhile, the situation that the computing unit cannot read the required data in time due to the bandwidth limitation of the routing module is avoided.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a data scheduling method and device, an electronic device, and a medium. Background Art

[0002] In the prior art, multiple compute units within a graphics processing unit (GPU) use a unified memory access (UMA) approach to store and retrieve data. This approach randomly distributes data input to the GPU across its various memory units for storage. The GPU then allocates several threads to each compute unit during computation, assigning the corresponding data processing tasks to each unit. These compute units then access the memory units storing the data they need to execute their threads, using routing modules such as crossbars or on-chip networks.

[0003] In the prior art, when a computing unit accesses a storage unit, it needs to go through a routing module for routing before it can access the storage unit storing the required data. This results in a longer communication link when the computing unit accesses the storage unit, a low memory access rate, and high memory access power consumption. At the same time, these routing modules are often shared by multiple computing units. When multiple computing units simultaneously call the same routing module to access the storage unit, they are easily limited by the bandwidth of the routing module, resulting in the computing unit being unable to access the storage unit for data access in a timely manner. Summary of the Invention

[0004] The embodiments of the present disclosure provide a data scheduling method and device, an electronic device, and a medium, which can improve the efficiency of a computing unit accessing a storage unit for data access, reduce memory access power consumption, and avoid the computing unit being unable to read the required data in a timely manner due to bandwidth limitations of a routing module.

[0005] According to one aspect of the present disclosure, a data scheduling method is provided for a processing device, the processing device including at least one block processing cluster, the block processing cluster including multiple computing units, and the computing units having corresponding local memories. The method includes: obtaining a first input tensor and a target workgroup grid corresponding to the first input tensor, the target workgroup grid including multiple workgroups, the workgroup including multiple threads, the threads corresponding to elements in the first input tensor, the threads being used to cause the computing units to process the elements corresponding to the threads, each workgroup being provided with corresponding workgroup coordinates, the workgroup coordinates being the coordinates of the corresponding workgroup in the target workgroup grid; dividing the target workgroup grid into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates corresponding to each workgroup, and scheduling the workgroups in each workgroup set to the corresponding block processing cluster; in the block processing cluster, mapping the workgroups in the corresponding workgroup set to the computing units in the block processing cluster; dividing the first input tensor into tensor blocks corresponding to each workgroup, and storing each tensor block in the local memory of the computing unit used to execute the corresponding workgroup, so that the computing unit reads the tensor block corresponding to the workgroup from the corresponding local memory when executing the corresponding workgroup.

[0006] Optionally, based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters, including: based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is converted into a one-dimensional workgroup sequence, wherein each of the workgroups is ordered in the workgroup sequence, and the first serial number of each of the workgroups in the workgroup sequence is determined according to the workgroup coordinates corresponding to the workgroup; based on the first serial number corresponding to each of the workgroups, the multiple workgroups are divided into workgroup sets corresponding to the block processing clusters, wherein the first serial numbers corresponding to the workgroups in each of the workgroup sets are continuous.

[0007] Optionally, the number of dimensions of the target workgroup grid is N, and the target workgroup grid is an N-dimensional grid. The converting of the target workgroup grid into a one-dimensional workgroup sequence based on the workgroup coordinates corresponding to each of the workgroups includes: obtaining the first dimension step corresponding to the N dimensions of the target workgroup grid, the first dimension step representing the maximum number of workgroups of the target workgroup grid in the corresponding dimension; determining the first serial number corresponding to each of the workgroups based on the workgroup coordinates of each of the workgroups in the target workgroup grid and the first dimension step of each dimension; and converting the target workgroup grid into the workgroup sequence according to the first serial number corresponding to each of the workgroups.

[0008] Optionally, after the multiple workgroups are divided into multiple workgroup sets corresponding to the block processing clusters based on the first serial numbers corresponding to each of the workgroups, the method further includes: determining a first number of computing units in the block processing cluster; dividing the workgroups in the workgroup set corresponding to the block processing cluster into a plurality of ordered workgroup clusters based on the first number, wherein the number of workgroups in a single workgroup cluster is an integer multiple of the first number, the workgroups in a single workgroup cluster are sequentially mapped to the computing units in the block processing cluster, the multiple workgroups in a single workgroup cluster are executed in parallel in the first number of computing units in the block processing cluster, and the ordered plurality of workgroup clusters are executed serially in the first number of computing units in the block processing cluster.

[0009] Optionally, after dividing the multiple workgroups into workgroup sets corresponding to the block processing clusters based on the first serial numbers corresponding to each of the workgroups, the method further includes: in the workgroup set, determining the second serial numbers of the workgroups in the corresponding workgroup set according to the first serial numbers corresponding to each of the workgroups; mapping the workgroups in the corresponding workgroup set to the computing units in the block processing cluster includes: for each of the workgroups, mapping the multiple workgroups to the computing units in the block processing cluster based on the first remainder obtained by taking the modulus of the first number based on the corresponding second serial number.

[0010] Optionally, the target workgroup grid is an N-dimensional grid, and the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates corresponding to each of the workgroups, including: obtaining the preset dimension width corresponding to the N dimensions of the target workgroup grid for a single workgroup cluster, the preset dimension width representing the coordinate span of the workgroup contained in the corresponding dimension of the target workgroup grid for a single workgroup cluster; determining the workgroup cluster coordinates corresponding to the workgroup based on the workgroup coordinates corresponding to each of the workgroups and the preset dimension width corresponding to the N dimensions, and determining the multiple workgroups corresponding to the same workgroup cluster coordinates as a single workgroup cluster; and dividing the multiple workgroup clusters into the workgroup sets corresponding to the block processing clusters based on the workgroup cluster coordinates.

[0011] Optionally, the workgroup cluster coordinates corresponding to the workgroup are determined based on the workgroup coordinates corresponding to each of the workgroups and the preset dimension widths corresponding to the N dimensions, including: for each of the workgroups, determining the coordinate values ​​of the workgroup in each dimension according to the corresponding workgroup coordinates, and determining a first ratio of the coordinate value of the workgroup in each dimension to the corresponding preset dimension width, rounding the first ratio of the workgroup in each dimension to obtain the value of the workgroup cluster coordinates of the workgroup in each dimension, so as to determine the workgroup cluster coordinates corresponding to each of the workgroups.

[0012] Optionally, based on the workgroup cluster coordinates, the multiple workgroup clusters are divided into the workgroup sets corresponding to the block processing clusters respectively, including: determining the second dimension step corresponding to the N dimensions of the target workgroup grid, the second dimension step representing the number of the workgroup clusters contained in the target workgroup grid in the corresponding dimension; based on the workgroup cluster coordinates corresponding to each of the workgroup clusters and the second dimension step of each dimension, determining the third serial number corresponding to each of the workgroup clusters; based on the third serial number, mapping each of the workgroup clusters to the block processing cluster respectively, and determining the multiple workgroup clusters corresponding to a single block processing cluster as the workgroup set corresponding to the block processing cluster.

[0013] Optionally, the number of block processing clusters is a second number, and mapping each of the workgroup clusters to the block processing cluster based on the third serial number includes: taking the modulus of the second number based on the third serial number corresponding to each of the workgroup clusters to obtain a second remainder corresponding to the workgroup cluster; mapping the corresponding workgroup clusters to the block processing clusters in the processing device based on the second remainder, wherein the workgroup clusters with equal corresponding second remainders are mapped to the same block processing cluster, and the workgroup clusters with unequal corresponding second remainders are mapped to different block processing clusters in the processing device.

[0014] According to one aspect of the present disclosure, a data scheduling device is provided, wherein the data scheduling device is integrated in a processing device, wherein the processing device includes at least one block processing cluster, wherein the block processing cluster includes multiple computing units, and the computing units are provided with corresponding local memories. The data scheduling device includes: an acquisition unit for acquiring a first input tensor and a target workgroup grid corresponding to the first input tensor, wherein the target workgroup grid includes multiple workgroups, wherein the workgroup includes multiple threads, wherein the threads correspond to elements in the first input tensor, and wherein the threads are used to enable the computing unit to process the elements corresponding to the threads, and each of the workgroups is provided with corresponding workgroup coordinates, wherein the workgroup coordinates are the coordinates of the corresponding workgroup in the target workgroup grid. ; a first dividing unit, for dividing the target workgroup grid into workgroup sets corresponding to the block processing clusters respectively based on the workgroup coordinates corresponding to each of the workgroups, and scheduling the workgroups in each of the workgroup sets to the corresponding block processing clusters; a first mapping unit, for mapping the workgroups in the corresponding workgroup sets to the computing units in the block processing cluster in the block processing cluster; a data scheduling unit, for dividing the first input tensor into tensor blocks corresponding to each workgroup, and storing each tensor block into the local memory of the computing unit for executing the corresponding workgroup, so that the computing unit reads the tensor blocks corresponding to the workgroup from the corresponding local memory when executing the corresponding workgroup.

[0015] According to one aspect of the present disclosure, an electronic device is provided, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection and communication between the processor and the memory, wherein the program is executed by the processor to implement the data scheduling method described in any one of the above items.

[0016] According to one aspect of the present disclosure, a computer-readable storage medium is provided, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the data scheduling method as described in any one of the above items.

[0017] In the data scheduling method proposed in the embodiment of the present disclosure, after obtaining the first input tensor to be processed and the target workgroup grid composed of threads for instructing each computing unit in the processing device to process the elements in the first input tensor, based on the workgroup coordinates of each workgroup in the target workgroup grid, multiple workgroups are divided into multiple workgroup sets corresponding to the block processing clusters in the processing device, and each workgroup set is scheduled to the corresponding block processing cluster; then, in each block processing cluster, each workgroup contained in the corresponding workgroup set is mapped to each computing unit in the block cluster, so that the computing unit for executing each workgroup is determined; then, since each thread is used to instruct the computing unit to process the element corresponding to the thread in the first input tensor, and each workgroup A work group contains multiple threads, and the multiple threads contained in a single work group often correspond to elements in a continuous region in the first input tensor, that is, each work group actually corresponds to an element in a region of the first input tensor. Based on this, in this embodiment, the first input tensor is divided into tensor blocks corresponding to each work group, and each tensor block is separately scheduled to the local memory of the computing unit used to execute the corresponding work group. In this way, when the computing unit subsequently executes each corresponding functional work group, it can directly read the tensor block consisting of the data required to execute the work group from the corresponding local memory, and the latency and power consumption of the computing unit when accessing the corresponding local memory are relatively low. In this way, the efficiency of each computing unit in reading the data required to execute the corresponding work group can be improved, and the memory access power consumption can be reduced. At the same time, the computing unit can directly access the corresponding local memory, so the computing unit does not need to access the global memory through a routing module such as a crossbar to obtain the data required to execute the work group, thereby preventing the computing unit from being unable to read the data required to execute the corresponding work group in a timely manner due to bandwidth limitations of the routing module.

[0018] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation to the technical solution of the present disclosure.

[0020] Figure 1 is an architectural diagram of a processing device to which the data scheduling method according to an embodiment of the present disclosure is applied;

[0021] Figure 2is a main flow chart of a data scheduling method according to an embodiment of the present disclosure;

[0022] Figure 3 is a schematic diagram of converting a target work group grid into a work group sequence according to an embodiment of the present disclosure;

[0023] Figure 4 is a schematic diagram of dividing a set of workgroups corresponding to each block processing cluster into multiple workgroup clusters according to an embodiment of the present disclosure;

[0024] Figure 5 is another schematic diagram of dividing a set of workgroups corresponding to each block processing cluster into a plurality of workgroup clusters according to an embodiment of the present disclosure;

[0025] Figure 6 is a schematic diagram of directly dividing a target workgroup table into a plurality of workgroup clusters based on preset dimension widths of respective dimensions according to an embodiment of the present disclosure;

[0026] Figure 7 is a schematic diagram of cyclically mapping a work group cluster to each block processing cluster based on a third sequence number according to an embodiment of the present disclosure;

[0027] Figure 8 is a system architecture diagram of a data scheduling device according to an embodiment of the present disclosure;

[0028] Figure 9 FIG. 4 is a system architecture diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not intended to limit the present disclosure.

[0030] Before further explaining the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are explained. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0031] Thread: It is the smallest execution unit of the graphics processing unit when performing data processing tasks. Each thread can independently perform the same mode of processing on different data. For example, the graphics processing unit needs to normalize the elements of each matrix row in an M*N matrix. At this time, M threads can be called in the graphics processing unit, and each thread is responsible for normalizing the elements of one matrix row in the matrix, so that the elements of the M matrix rows of the matrix can be processed in parallel by M threads. Alternatively, multiple threads perform different modes of processing on different data at the same time. For example, when the graphics processing unit normalizes an M*N matrix, it needs to copy the matrix data from the global memory to the register of the graphics processing unit's computing unit, and then the computing unit performs the normalization. At this time, some threads can be used to copy the matrix data from the global memory to the register of the computing unit, and the other threads are used to instruct the computing unit to normalize the data copied to the register.

[0032] Work Group: A group of threads consisting of multiple threads, also called a thread block or work item. Threads in a single work group are generally used to perform the same processing pattern on different data.

[0033] Grid: Consists of multiple workgroups. Each workgroup acts as a grid point in the grid, and each workgroup is assigned corresponding grid point coordinates. The grid can be one-dimensional, two-dimensional, or three-dimensional. It is understood that the shape of the grid is set by the user during programming.

[0034] A Compute Unit (CU) is a processing module in electronic devices such as graphics processing units. In some architectures, a CU is also called a stream multiprocessor. A graphics processing unit can include multiple CUs. A CU can be considered a processing core within the graphics processing unit. Each CU can independently execute instructions, and the graphics processing unit can process data in parallel using multiple CUs.

[0035] Block Process Cluster: A cluster consisting of multiple computing units in a single graphics processing unit. In some graphics processing unit architectures, a block processing cluster may also be called a graphics processing cluster (GPC). In a single graphics processing unit, each block processing cluster includes an equal number of computing units. For example, a graphics processing unit includes 32 computing units, and every 4 computing units are integrated into a block processing cluster, then the graphics processing unit includes 8 block processing clusters. It can be understood that the communication distance between computing units in the same block processing cluster is relatively close, while the communication distance between computing units in different block processing clusters is relatively far.

[0036] Non-Uniform Memory Access (NUMA): A memory architecture in which each node has its own local memory. Each node can access its own local memory at high speed, and different nodes can communicate across nodes to access the local memory of other nodes.

[0037] Description of the system architecture used in the embodiments of the present disclosure

[0038] Figure 1 It is an architecture diagram of a processing device 100 applied to the data scheduling method of an embodiment of the present disclosure, which includes: at least one block processing cluster 110, each block processing cluster includes multiple computing units 111, and each computing unit 111 is provided with a corresponding local memory 112.

[0039] Each computing unit 111 can independently execute different workgroups, thereby processing various elements in the first input tensor in parallel. Specifically, each computing unit may include multiple execution units, tensor cores, and other modules for performing arithmetic operations or logical operations on the elements in the first input tensor. When the computing unit executes a workgroup, the corresponding execution unit or tensor core and other modules execute each thread in the workgroup to process the elements in the first input tensor corresponding to each thread.

[0040] The block processing cluster 110 is a cluster composed of several computing units 111 in a processing device. In a possible implementation, a block processing cluster may be composed of several computing units integrated on a single silicon chip.

[0041] The local memory 112 is a storage space that the computing unit can directly access without routing through the routing module. It can be understood that the physical distance between the local memory and the corresponding computing unit is relatively close. Based on this, the communication link between the computing unit and the corresponding local memory can also be relatively short. The access speed of the computing unit when accessing the corresponding local memory will be higher than the access speed when the computing unit accesses other storage spaces in the processing device.

[0042] In addition, considering that when running a neural network model, when performing certain types of computing tasks, each computing unit may need to use the data stored in the local memory of other computing units when executing the corresponding work group. For example, in the attention mechanism, it is necessary to calculate the attention weight between each word and all words. Figure 1 In the processing device shown, different computing units in a single block processing cluster can be connected through an interconnection module, and different block processing clusters in the processing device can also be communicatively connected through the interconnection module. Each computing unit can access the corresponding local memory at high speed, and can also access the local memory corresponding to other computing units in the same block processing cluster and the local memory corresponding to computing units in other block processing clusters through the interconnection module to read the data stored in the local memory corresponding to other computing units. It should be noted that when each computing unit accesses the local memory corresponding to other computing units, it needs to access it through the interconnection module, and the distance between each computing unit and the local memory corresponding to other computing units will be farther than the distance between each computing unit and the local memory corresponding to the computing unit itself. Therefore, the access speed of each computing unit when accessing the local memory corresponding to other computing units will be slightly slower than the access speed when accessing the local memory corresponding to the computing unit itself. The local memories corresponding to multiple computing units in the processing device can constitute a non-uniform memory access architecture.

[0043] Overall implementation of the data scheduling method in the disclosed embodiment

[0044] The present disclosure provides a data scheduling method for use in Figure 1 The processing device shown in the figure includes at least one block processing cluster, each block processing cluster includes multiple computing units, each computing unit is provided with a corresponding local memory, and the processing device includes at least one block processing cluster, each block processing cluster includes a plurality of ... Figure 2 , the data scheduling method includes:

[0045] Step S201: Obtain a first input tensor and a target workgroup grid corresponding to the first input tensor. The target workgroup grid includes multiple workgroups. Each workgroup includes multiple threads. A thread corresponds to an element in the first input tensor. A thread is used to indicate that a computing unit processes an element corresponding to the thread. Each workgroup is provided with corresponding workgroup coordinates. The workgroup coordinates are the coordinates of the corresponding workgroup in the target workgroup grid.

[0046] Step S202 , based on the workgroup coordinates corresponding to each workgroup, divide the target workgroup grid into workgroup sets corresponding to the block processing clusters, and schedule the workgroups in each workgroup set to the corresponding block processing cluster;

[0047] Step S203 , in the block processing cluster, mapping the workgroups in the corresponding workgroup set to computing units in the block processing cluster respectively;

[0048] Step S204: divide the first input tensor into tensor blocks corresponding to each workgroup, and store each tensor block in the local memory of the computing unit used to execute the corresponding workgroup, so that the computing unit reads the tensor block corresponding to the workgroup from the corresponding local memory when executing the corresponding workgroup.

[0049] It will be understood that in this embodiment, the processing device may be a device for performing arithmetic or logical operations on the first input tensor. For example, the processing device may be a graphics processing unit (GPU). A computing unit is an independent computing core in a processing device. A processing device may include multiple computing units, and different computing units may independently execute the same or different instructions on different data.

[0050] A block processing cluster is a cluster composed of computing units in a graphics processing unit. A graphics processing unit may include at least one block processing cluster, and each block processing cluster includes multiple computing units. For example, a graphics processing unit includes 32 computing units, and these 32 computing units can be integrated into 8 block processing clusters, each of which includes 4 computing units. Alternatively, a graphics processing unit includes only 8 computing units, and these 8 computing units are integrated into a single block processing cluster. Specifically, the number of block processing clusters in the processing device and the number of convolution computing units in a single block processing cluster are determined based on the hardware architecture of the processing device, and are not limited in this embodiment. It should be noted that in one possible implementation, all computing units in the processing device may not be clearly divided into different blocks in the hardware architecture. In this case, the entire processing device can be regarded as a single block processing cluster.

[0051] In step S201, the first input tensor is a tensor that the processing device needs to process when running the neural network model. It can be a set of intermediate data organized into a tensor form generated according to the input data of the neural network model. For example, the input of the neural network model. The dimension of the first input tensor can be one-dimensional or multi-dimensional. For example, the first input tensor can be a two-dimensional tensor with a shape of (H, W), which is composed of H W-dimensional vectors; or the first input tensor can be a three-dimensional tensor with a shape of (N, H, W), which is composed of N H*W matrices, each matrix is ​​composed of H W-dimensional vectors.

[0052] The target workgroup grid is a grid composed of multiple workgroups. Each workgroup includes multiple threads that instruct a computational unit to process each element in the first input tensor. Each thread corresponds to a different element in the first input tensor, and the entire target workgroup grid contains threads that correspond to each element in the first input tensor. That is, the target workgroup grid contains all threads that instruct a computational unit to process each element in the first input tensor. It is understood that in neural network models, input data is often converted into a form suitable for processing by the neural network model before subsequent processing. In other words, in neural network models, input data is actually in the form of multiple feature vectors, and the first input tensor is actually a tensor composed of multiple feature vectors. In this case, each thread can correspond to one or more feature vectors that constitute the first input tensor, and all threads that constitute the target workgroup grid correspond to all feature vectors that constitute the first input tensor. For example, if the shape of the first input tensor is (N, H, W), indicating that the first input tensor contains N*H W-dimensional feature vectors, the target workgroup grid can be composed of N*H threads, each thread corresponding to a W-dimensional feature vector in the first input tensor. In one possible implementation, in order to simplify thread management, a single workgroup can be composed of multiple consecutive threads, and the multiple consecutive threads can correspond to multiple consecutive feature vectors in the first input tensor, that is, a workgroup can be regarded as corresponding to a continuous group of elements in the first input tensor.

[0053] In one embodiment, the shape of the target workgroup grid and the shape of a single workgroup can be preset by the user. For example, in CUDA, the shape of the target workgroup grid and the shape of the workgroup can be preset respectively by the statement dim3 gridDim(a1, b1, c1) and the statement dim3 blockDim(a2, b2, c2). The statement indicates that the target workgroup grid contains a1 workgroups in the x dimension, b1 workgroups in the y dimension, and c1 workgroups in the z dimension; a single workgroup contains a2 threads in the x dimension, b2 threads in the y dimension, and c2 threads in the z dimension. At this time, the target workgroup grid contains a1*b1*c1 workgroups, and each workgroup contains a2*b2*c2 threads. In addition, it can be understood that the shape of the target workgroup grid and the shape of the workgroup set by the user are used to instruct the processing device how to allocate multiple computing units for parallel processing of the first input tensor. In order to maximize the utilization of computing resources in the processing device, the shape of the target workgroup grid and the shape of the workgroup are often determined based on the shape of the first input tensor. The number of workgroups contained in the target workgroup grid is often an integer multiple of the number of computing units in the processing device. This ensures that multiple workgroups can be evenly allocated to the computing units in the processing device for parallel processing, and the number of workgroups required to be processed by each computing unit is equal.

[0054] The workgroup coordinates are coordinates used to indicate the position of the corresponding workgroup in the target workgroup grid. These coordinates can be one-dimensional or multi-dimensional, and the dimensions of the workgroup coordinates can be determined based on the dimensions of the target workgroup grid. For example, if the target workgroup grid is a three-dimensional grid, then the workgroup coordinates are also three-dimensional coordinates. It is understood that in a single target workgroup grid, the workgroup coordinates of each workgroup are unique, and the workgroup coordinates corresponding to different workgroups are different.

[0055] Specifically, the target workgroup grid can be created by the processing device after the central processing unit (CPU) issues an instruction to the processing device. Specifically, the user will set a corresponding kernel function in the CPU. The kernel function indicates the shape of a single workgroup and the shape of the target workgroup grid, that is, it indicates the number of threads contained in a single workgroup in each dimension and the number of workgroups contained in the target workgroup grid in each dimension. When the CPU executes the kernel function, it issues a corresponding instruction to the processing device. In response to the instruction, the processing device creates multiple threads for instructing the computing unit to process each element of the first input tensor respectively, and organizes the created threads into multiple workgroups according to the shape indicated by the kernel function, and then organizes the multiple workgroups into the target workgroup grid.

[0056] In step S202, the workgroup set refers to a set consisting of multiple workgroups in the target workgroup grid. It can be understood that the workgroup coordinates indicate the position of the workgroup in the target workgroup grid. In the target workgroup grid, the workgroup corresponding to each workgroup coordinate is unique, and each workgroup coordinate can serve as a unique identifier of the corresponding workgroup. Based on this, in this embodiment, the workgroups in the target workgroup grid can be divided into several workgroup sets corresponding to the block processing clusters in the processing device based on the workgroup coordinates corresponding to each workgroup, thereby mapping each workgroup to a block processing cluster.

[0057] Specifically, referring to the description of step S201 above, it can be seen that the number of workgroups in the target workgroup grid is often an integer multiple of the total number of computing units in the processing device, and the number of computing units in each block processing cluster is equal, that is, the total number of workgroups is actually an integer multiple of the number of block processing clusters, and the computing units in a single processing device are often homogenized, that is, the computing power of each computing unit in a single processing device is often the same. At this time, dividing multiple workgroups into multiple block processing clusters can make more full use of the computing units in the processing device. Based on this, in this embodiment, when dividing the target workgroup grid, it can be based on the number of block processing clusters, and according to preset rules, the target workgroup grid can be divided into multiple workgroup sets corresponding to each block processing cluster.

[0058] It is understandable that in a processing device, when multiple computing units perform parallel computations on various portions of data in a single input tensor, the computations on the various portions of data in the first input tensor are often relatively independent, and computing the various portions of data in a single input tensor in a different order does not affect the computational results. Based on this, in this embodiment, the method for evenly dividing the target workgroup grid into workgroup sets corresponding to each block processing cluster can be based on actual settings and is not limited in this embodiment.

[0059] In one possible embodiment, the target workgroup grid can be divided into multiple equal-sized regions based on the number of block processing clusters. The workgroups contained in each region constitute a workgroup set corresponding to a block processing cluster. For example, if there are eight block processing clusters in a processing device and the target workgroup grid is a (32, 16, 2) grid, the target workgroup grid can be divided into eight equal-sized regions of (8, 8, 2) shape. The workgroups contained in each region constitute the workgroups contained in the workgroup set corresponding to a block processing cluster.

[0060] In another possible embodiment, the workgroup coordinates corresponding to each workgroup can be linearized according to preset rules, and the workgroup coordinates corresponding to each workgroup can be converted into a corresponding serial number. Then, according to the serial numbers of each workgroup, the multiple workgroups in the target workgroup grid can be divided into workgroup sets corresponding to each block processing cluster. For example, the target workgroup grid is a two-dimensional grid with a shape of (16, 8), that is, the target workgroup grid includes 16 workgroups along the x dimension and 8 workgroups along the y dimension. At this time, the coordinates of each workgroup are linearized along the x dimension of the target workgroup grid, and the workgroup with workgroup coordinates of (0, 0) corresponds to the serial number 0, and the workgroup with workgroup coordinates of (1, 0) corresponds to the serial number 1. Similarly, after the corresponding serial numbers of all workgroups with the y coordinate of 0, that is, the first row of the target workgroup grid, are determined, the linearization starts with the y coordinate of 1, that is, the data in the second row of the target workgroup grid. Since the target workgroup grid contains 16 workgroups along the x dimension, that is, the serial number corresponding to the last workgroup in the first row is 15, the workgroups with the y coordinate of 1 have a starting serial number of 16. Similarly, the workgroups with the y coordinate of 2 have a starting serial number of 32. And so on, the correspondence can be made according to each workgroup. After linearizing the workgroup coordinates of each workgroup and obtaining the serial numbers corresponding to each workgroup, the multiple workgroups can be divided into multiple workgroup sets corresponding to each block processing cluster according to the serial numbers. Specifically, several workgroups with consecutive serial numbers can be divided into a workgroup set. For example, for a two-dimensional target workgroup grid with a shape of (16, 8), after linearization, the serial numbers corresponding to each workgroup are 0 to 107. When the number of block processing clusters is 8, each block processing cluster corresponds to 16 workgroups. At this time, the workgroups with serial numbers 0 to 15 are used as the workgroup set corresponding to one block processing cluster, and the workgroups with serial numbers 16 to 31 are used as the workgroup set corresponding to another block processing cluster. And so on, the workgroup sets corresponding to each block processing cluster are obtained.

[0061] In another possible embodiment, after linearizing the workgroup coordinates corresponding to each workgroup, each workgroup can be mapped to the workgroup set corresponding to each block processing cluster according to the corresponding serial number in a cyclic manner. For example, the number of block processing clusters is 8, and each workgroup is cyclically mapped to the 1st block processing cluster to the 8th block processing cluster according to the serial number. For example, the workgroup with serial number 0 is mapped to the workgroup set corresponding to the 1st block processing cluster, the workgroup with serial number 1 is mapped to the workgroup set corresponding to the 2nd block processing cluster, and so on. After mapping the workgroup with serial number 7 to the workgroup set corresponding to the 8th block processing cluster, the workgroup with serial number 8 is remapped to the workgroup set corresponding to the 1st block processing cluster, and the next round of cyclic mapping begins. By analogy, the workgroup set corresponding to each block processing cluster can be obtained.

[0062] After dividing the target workgroup grid into workgroup sets corresponding to each block processing cluster, each workgroup set is scheduled to the corresponding block processing cluster. In this way, the computing units in each block processor cluster can be used to execute the workgroups in the corresponding workgroup set.

[0063] In step S203, it is understood that each workgroup actually needs to be executed by a computing unit in a block processing cluster. A block processing cluster contains multiple computing units, and a workgroup set contains multiple workgroups. Based on this, in this embodiment, after scheduling each workgroup set to the corresponding block processing cluster, each workgroup needs to be mapped to a computing unit in the block processing cluster to determine which computing unit will subsequently execute each workgroup.

[0064] Specifically, referring to the relevant description of step S202, it can be known that the number of workgroups in a single workgroup set is often several times the number of computing units in a single block processing cluster. Based on this, in this embodiment, the workgroups in the workgroup set corresponding to the block processing cluster can be evenly divided into the various computing units in the block processing cluster, and the method of mapping the workgroups in the workgroup set corresponding to the block processing cluster to the computing units in the block processing cluster can refer to the method of mapping the workgroups in the target workgroup grid to the block processing cluster in step S202. For example, determine the serial number of each workgroup in the corresponding workgroup set; then, cyclically map each workgroup to each computing unit in the block processing cluster based on the serial number of each workgroup in the corresponding workgroup set; or, after determining the serial number, divide the workgroups in a single workgroup set into workgroup clusters corresponding to each computing unit in the block processing cluster based on the number of computing units in the block processing cluster, so as to map each workgroup in the workgroup set to a computing unit in the block processing cluster.

[0065] In one embodiment, after determining the compute units corresponding to each workgroup, the compute shader in the processing device can color-code each workgroup. Multiple workgroups corresponding to the same compute unit are coded with the same color, while workgroups corresponding to different compute units are coded with different colors. This allows the scheduler in the processing device to determine which workgroups should be scheduled to the same compute unit for execution based solely on the colors corresponding to the workgroups, simplifying the scheduling logic used by the scheduler when scheduling workgroups.

[0066] In step S204, local memory refers to a storage unit that can be directly accessed by the computing unit. The local memory can be a memory area that is physically tightly coupled with the corresponding computing unit. Specifically, the local memory can be a register inside the computing unit; or a memory that is physically directly connected to the corresponding computing unit and the connection link distance is extremely short. Alternatively, in a possible implementation, the computing unit is directly integrated into a storage unit, forming a storage and computing integrated module with the storage unit. In this case, the local memory corresponding to the computing unit is the storage unit where the computing unit is located. It can be understood that since the local memory is often a storage space integrated into the computing unit, or a storage space integrated in a position that is physically very close to the computing unit, or even a storage space that directly integrates the computing unit, the communication link between the computing unit and the corresponding local memory is extremely short, which means that the latency when the computing unit accesses the corresponding local memory will become relatively low, the bandwidth will be relatively large, and the power consumption caused by the access will also be relatively low.

[0067] In one embodiment, the local memory is an off-chip memory that is physically tightly coupled with the corresponding computing unit. As an off-chip memory, it is integrated outside the chip where the computing unit is located and does not need to occupy the area in the chip where the computing unit is located. At this time, the local memory can use a memory with a larger memory space, such as a dynamic random access memory, to ensure that the memory space of the local memory is sufficient to store the data that the computing unit needs to process.

[0068] A tensor block refers to a data block consisting of elements in the first input tensor corresponding to threads in the corresponding workgroup. It can be understood that each workgroup includes several threads, and each thread corresponds to an element in the first input tensor, that is, a workgroup actually corresponds to a part of the elements in the first input tensor, and the data block consisting of this part of the elements is a tensor block corresponding to the workgroup. Exemplarily, the first input tensor is composed of feature vectors corresponding to 1024 word elements, and each feature vector corresponds to a thread in the target workgroup grid, wherein a workgroup contains 128 threads corresponding to the feature vectors of the first 128 word elements. At this time, in the first input tensor, a tensor block consisting of feature vectors of 128 word elements is the tensor block corresponding to the workgroup.

[0069] When executing each workgroup, the computing unit actually needs to perform corresponding arithmetic or logical operations on the elements corresponding to each thread in the workgroup through the computing unit. That is, the data required by the computing unit when executing the workgroup is the tensor block corresponding to the workgroup in the first input tensor. Based on this, in this embodiment, after the first input tensor is divided into tensor blocks corresponding to each workgroup, each tensor block is scheduled to the local memory of the computing unit used to execute the corresponding workgroup; then, when executing each corresponding workgroup, the computing unit can directly read the required tensor block from the corresponding local memory to perform the corresponding arithmetic or logical operation on the tensor block. Because the link between the computing unit and the corresponding local memory is short and the bandwidth is relatively high, this can improve the efficiency of the computing unit in reading the data required to execute the corresponding workgroup when executing the corresponding workgroup, while reducing memory access power consumption. In addition, because the computing unit and the corresponding local memory are directly connected, the computing unit does not need to be routed through a routing module such as a crossbar or mesh on-chip network when accessing data from the corresponding local memory, thereby avoiding the limitation of the maximum bandwidth of the routing module and the inability to access data in a timely manner.

[0070] In the embodiment disclosed in steps S201 to S204, the target workgroup grid is first divided into workgroup sets corresponding to each block processing cluster, and each workgroup set is respectively scheduled to the corresponding block processing cluster; then, in each block processing cluster, the workgroups contained in the corresponding workgroup set are respectively mapped to each computing unit of the block processing cluster to determine which computing unit will execute each workgroup subsequently; then, the first input tensor is divided into tensor blocks corresponding to each workgroup, and each tensor block is respectively scheduled to the local memory of the computing unit for executing the workgroup corresponding to the tensor block; in this way, when the computing unit subsequently executes each workgroup, it can directly read the corresponding tensor block from the corresponding local memory , thereby reducing the latency when the computing unit reads the data required for the execution work group, increasing the bandwidth when the computing unit reads the required data, thereby improving the efficiency of the computing unit when reading the required data, and reducing the memory access power consumption of the computing unit; at the same time, since the computing unit can directly access the corresponding local memory without the need for routing addressing through the routing module, this can avoid the situation where the computing unit cannot read the required data in time due to the bandwidth limitation of the routing module when multiple computing units share the routing module. At the same time, since the computing unit does not need to address through routing modules such as crossbar when accessing memory, there is no need to integrate high-cost routing modules such as crossbar in the graphics processing unit, thereby reducing the cost of the graphics processing unit.

[0071] Implementation method of linearizing the work group coordinates to divide the target work group grid into multiple work group sets

[0072] It is understandable that when dividing the target workgroup grid into workgroup sets corresponding to each block processing cluster, if the target workgroup grid is divided into workgroup sets based on relatively random or relatively complex logic, and each workgroup is then mapped to the corresponding computing unit, then the order in which the computing unit executes the workgroups in the target workgroup grid will become relatively complex or even relatively random, which makes the mapping relationship between the workgroup and the computing unit relatively complex, which will cause the subsequent process of scheduling each workgroup to the computing unit in each block processing cluster for execution to become relatively complex, and it is more difficult to accurately schedule each workgroup to the corresponding computing unit. And the process of scheduling the tensor blocks corresponding to each workgroup to the local memory of the corresponding computing unit also becomes relatively complex and prone to errors. Based on this, in one embodiment, step S202 includes:

[0073] Step S301 , based on the workgroup coordinates corresponding to each workgroup, convert the target workgroup grid into a one-dimensional workgroup sequence;

[0074] Step S302 : Based on the first serial number corresponding to each work group, the multiple work groups are divided into work group sets corresponding to the block processing clusters.

[0075] In step S301, the workgroup sequence is a one-dimensional sequence of workgroups in the target workgroup grid obtained by linearizing the target workgroup grid and converting it into a one-dimensional form. Each workgroup can be regarded as a point in the workgroup sequence. It is understood that in the workgroup sequence, each workgroup is assigned a corresponding first sequence number to indicate its position in the workgroup sequence. The first sequence number corresponding to each workgroup is determined based on the grid point coordinates of the workgroup in the target workgroup grid, and the first sequence numbers obtained based on different workgroup coordinates are different.

[0076] In this embodiment, the target work group grid can be linearized into a one-dimensional work group sequence along the direction of the preset dimension, and the first sequence number corresponding to each work group in the work group sequence is determined, for example, linearization is performed along the direction of the x dimension of the target work group grid. Figure 3 The target workgroup grid is a three-dimensional grid with a shape of (8, 4, 2). In this case, the target workgroup grid needs to be linearized along the x-dimension. First, the 8 workgroups with both y and z coordinates of 0 are used as the first 8 workgroups in the sequence. Then, the 8 workgroups with y coordinates + 1 are used as the 9th to 16th workgroups in the sequence. This continues until the y coordinate increases to a maximum value of 3. At this point, the 32 workgroups with z coordinates of 0 are used as the first 32 workgroups in the sequence. Then, the 8 workgroups with z coordinates of 1 and y coordinates of 0 are used as the 33rd to 40th workgroups in the sequence. This continues until the 8 workgroups with z coordinates of 1 and y coordinates of 3 are used as the last 8 workgroups. In other words, the target workgroup grid can be linearized along the x-dimension into a 64-workgroup sequence, with the corresponding workgroups being numbered 0 to 63. In this way, in the resulting workgroup sequence, the workgroups corresponding to consecutive first numbers have workgroup coordinates in the target workgroup grid that are also continuous along the dimension selected during linearization. Specifically, when linearizing the target workgroup grid, the corresponding first serial number can be calculated based on the workgroup coordinates corresponding to each workgroup, and then the workgroups in the target workgroup grid can be mapped to the workgroup sequence in order from small to large according to the first serial number, so as to linearize the target workgroup grid into a workgroup sequence. The specific method of step S301 will be explained later and will not be repeated here.

[0077] In step S302, after converting the target workgroup grid into a one-dimensional workgroup sequence, the workgroup sequence can be divided into multiple subsequences corresponding to each block processing cluster based on the number of block processing clusters. The multiple workgroups contained in each subsequence constitute a workgroup set corresponding to a block processing cluster. Specifically, the number of workgroups that need to be assigned to a single block processing cluster can be determined based on the total number of workgroups in the workgroup sequence and the number of block processing clusters. Then, according to the first sequence number of each workgroup, the multiple workgroups in the first sequence of the workgroup sequence are divided into a workgroup set corresponding to a block processing cluster. For example, if the number of workgroups in each workgroup set is 24, then the workgroups with corresponding first sequence numbers of 0 to 23 are divided into the first workgroup set, and the workgroups with corresponding sequence numbers of 24 to 47 are divided into the second workgroup set, and so on. When calculating the first sequence number based on the workgroup coordinates, the multiple consecutive first sequence numbers actually correspond to multiple workgroups that are continuous along a certain dimension of the corresponding workgroup coordinates. Based on this, after grouping multiple consecutive workgroups into a workgroup set, the workgroups within the same workgroup set are actually multiple workgroups that are continuous along a certain dimension in the target workgroup grid. At this point, the workgroup set corresponding to each block processing cluster is composed of multiple workgroups that are continuous along a certain dimension in the target workgroup grid. This allows the computing units in the block processing cluster to subsequently schedule each workgroup to the corresponding computing unit for execution in a relatively simple order, reducing the difficulty of workgroup scheduling.

[0078] At the same time, by treating multiple workgroups with consecutive first serial numbers as a workgroup set, when the tensor blocks corresponding to each workgroup are subsequently scheduled to the local memory of the computing unit used to execute the workgroup, the tensor blocks stored in the local memory of each computing unit in a single block processing cluster need to be scheduled to correspond to multiple consecutive workgroups, rather than being completely disordered. This can simplify the complexity of scheduling each tensor block to the local memory of the corresponding computing unit and reduce the probability of error.

[0079] In the embodiment disclosed in steps S301 to S302, the target workgroup grid is linearized along a preset dimension, thereby converting the target workgroup grid into a one-dimensional workgroup sequence, and determining the first serial number corresponding to each workgroup in the sequence, so that the workgroups corresponding to the consecutive first serial numbers in the workgroup sequence are multiple workgroups that are continuous along a certain dimension in the target workgroup grid. Thereafter, based on the first serial number, the workgroups in the workgroup sequence are divided into multiple workgroup sets corresponding to each block processing cluster, and the first serial numbers corresponding to the workgroups in each workgroup set are continuous, that is, the workgroups that need to be processed by the computing units in a single block processing cluster are continuous. In this way, the difficulty of the processing device in managing the execution status of each workgroup and the difficulty of scheduling each tensor block to the local memory in the corresponding computing unit can be reduced.

[0080] In one embodiment, step S301 includes:

[0081] Step S401: Obtain the first dimension step lengths corresponding to the N dimensions of the target work group grid;

[0082] Step S402 , determining a first serial number corresponding to each work group based on the work group coordinates of each work group in the target work group grid and the first dimension step length of each dimension;

[0083] Step S403 : converting the target work group grid into a work group sequence according to the first serial number corresponding to each work group.

[0084] In step S401, the first dimension step size represents the maximum number of workgroups in the target workgroup grid in the corresponding dimension, that is, the number of workgroups in each dimension in the target workgroup grid. The first dimension step size corresponding to each dimension of the target workgroup grid is preset. For example, the user sets the shape of the target workgroup grid through the statement dim3 gridDim(x1, y1, z1). At this time, the target workgroup grid is a three-dimensional grid with a shape of (x1, y1, z1), and its first dimension step sizes corresponding to the x, y, and z dimensions are x1, y1, and z1, respectively.

[0085] It can be understood that when the value of z1 is 1, the target work group grid can be regarded as a two-dimensional grid; when the values ​​of y1 and z1 are both 1, the target work group grid can be regarded as a one-dimensional grid.

[0086] In step S402, after obtaining the first dimension step size corresponding to each dimension of the target work group grid, each first dimension step size represents the number of work groups included in the target work group grid along the corresponding dimension. Based on this, for each work group, the first sequence number corresponding to each work group can be determined based on the corresponding work group coordinates and the first dimension step size corresponding to each dimension.

[0087] Specifically, in one possible implementation, the function for calculating the corresponding first serial number based on the workgroup coordinates can be determined according to the dimension of the target workgroup grid, and then the corresponding first serial number can be calculated according to the corresponding function and the workgroup coordinates corresponding to each workgroup.

[0088] In one embodiment, when the target workgroup grid is a one-dimensional grid, the workgroup coordinates are also one-dimensional coordinates along the x-dimension, which can be expressed as (3d_id.x). At this time, for each workgroup, the first serial number corresponding to the workgroup can be determined by referring to the formula rank = 3d_id.x, where rank is the first serial number and 3d_id.x is the coordinate corresponding to the workgroup in the x-dimension. At this time, the values ​​of the workgroup coordinates corresponding to each workgroup are directly determined as the corresponding first serial number.

[0089] In another embodiment, the target workgroup grid is a two-dimensional grid, and the workgroup coordinates of each workgroup can be expressed as (3d_id.x, 3d_id.y). In this case, the target workgroup grid is linearized into a workgroup sequence along the x-dimension. At this time, the first serial number corresponding to each workgroup can be calculated according to the formula rank=3d_id.x+3d_id.y*stride.x, where stride.x represents the first dimension step corresponding to the x-dimension of the target workgroup table, that is, the number of workgroups contained in the target workgroup grid along the x-dimension. It is not difficult to see that the first serial number corresponding to the workgroup with workgroup coordinates of (0, y1+1) is the first serial number corresponding to the workgroup with workgroup coordinates of (stride.x, y1) plus 1. In this way, the first serial numbers corresponding to workgroups with different y-coordinates can be kept continuous, thereby linearizing the target workgroup grid into a continuous one-dimensional workgroup sequence.

[0090] In another embodiment, the target workgroup table is a three-dimensional grid, and the workgroup coordinates of each workgroup can be expressed as (3d_id.x, 3d_id.y, 3d_id.z). At this time, the function for calculating the first serial number of each workgroup in the three-dimensional grid is rank=3d_id.x+3d_id.y*stride.x+3d_id.z*stride.x*stride.y; where stride.y represents the first dimension step corresponding to the y dimension of the target workgroup grid, that is, the number of workgroups included along the y dimension in the target workgroup grid. At this time, stride.x*stride.y can actually represent the number of workgroups corresponding to the same z coordinate, that is, the number of workgroups included in an xoy plane in the three-dimensional target workgroup grid. When linearizing the target workgroup grid based on the above formula, the workgroups with the same z coordinate can be regarded as a two-dimensional grid, and the two-dimensional grid can be linearized into a continuous one-dimensional sequence according to the method in the above embodiment. At the same time, it is not difficult to see that the first serial number corresponding to the workgroup with workgroup coordinates (0, 0, z1+1) is the first serial number corresponding to the workgroup with workgroup coordinates (stride.x, stride.y, z1) plus 1. In this way, the serial numbers between the workgroups corresponding to different z coordinates can be kept continuous, thereby linearizing the three-dimensional target workgroup grid into a continuous one-dimensional workgroup sequence.

[0091] In a possible implementation, the first sequence number corresponding to each work group may be directly calculated based on the following function:

[0092] rank =3d_id.x +3d_id.y*stride.x + 3d_id.z*stride.x * stride.y; At this time, if the target workgroup grid is a one-dimensional grid, it can be regarded as a three-dimensional grid with a dimensional step of 1 along the y-dimension and z-dimension directions, and the y-coordinate and z-coordinate of each workgroup are 0. In this way, the workgroup coordinates of each workgroup can be converted from the one-dimensional coordinate (x1) to the corresponding three-dimensional coordinate (x1, 0, 0); similarly, when the target workgroup grid is a two-dimensional grid, it can be regarded as a three-dimensional grid with a dimensional step of 1 along the z-dimension direction, and the z-coordinate of each workgroup is 0. The workgroup coordinates of the workgroup are converted from the two-dimensional coordinate (x1, y1) to the corresponding three-dimensional coordinate (x1, y1, 0). In this way, when calculating the corresponding first serial number based on the workgroup coordinates corresponding to the workgroup, there is no need to consider the dimension of the target workgroup grid.

[0093] In step S403, after determining the first serial number corresponding to each work group, the work groups corresponding to the first serial numbers are arranged in ascending or descending order, so as to convert the target work group grid into a one-dimensional work group sequence.

[0094] In the embodiment disclosed in steps S401 to S403, the first serial number corresponding to each workgroup is determined based on the workgroup coordinates corresponding to each workgroup and the first dimensional step size corresponding to each dimension of the target workgroup grid, and the workgroups in the target workgroup table are grouped into corresponding workgroup sequences in the order of the first serial numbers. In this way, the target workgroup table can be linearized into a one-dimensional workgroup sequence along a preset dimension, and then, based on the first serial number corresponding to each workgroup in the workgroup sequence, multiple workgroups can be divided into workgroup sets corresponding to each block processing cluster, and the workgroups in each workgroup set are continuous along a certain dimension in the target workgroup grid, so as to reduce the difficulty of managing and scheduling the workgroups of the processing equipment, and scheduling the tensor blocks corresponding to the workgroups to the local memory of the corresponding computing units.

[0095] In one embodiment, step S302 includes:

[0096] Step S501, determining a first number of computing units in a block processing cluster;

[0097] Step S502: Based on the first number, the workgroups in the workgroup set corresponding to the block processing cluster are divided into a plurality of ordered workgroup clusters, wherein the number of workgroups in a single workgroup cluster is an integer multiple of the first number, the workgroups in the single workgroup cluster are sequentially mapped to the computing units in the block processing cluster, the plurality of workgroups in the single workgroup cluster are executed in parallel in the first number of computing units in the block processing cluster, and the plurality of ordered workgroup clusters are executed serially in the first number of computing units in the block processing cluster.

[0098] In step S501, the first number is the number of computing units in a single block processing cluster. The first number is determined based on the hardware architecture of the processing device. For example, in a processing device, four computing units are integrated on a single silicon chip as a block processing cluster. In this case, the first number is 4.

[0099] In step S502, a workgroup cluster refers to a cluster consisting of the first number of consecutive workgroups in a single workgroup set. It can be understood that, referring to the relevant description of step S201, the total number of workgroups in the target workgroup grid is an integer multiple of the number of computing units in the processing device, that is, when executing the workgroups in the target workgroup grid through the computing units in the processing device, each computing unit needs to execute multiple workgroups in sequence. Based on this, after scheduling each workgroup set to the corresponding block processing cluster, it is necessary to determine the number of workgroups that can be executed in parallel by the block processing cluster based on the number of computing units in the block processing cluster, and thereby divide the workgroup set into multiple workgroup clusters that are executed serially in the block processing cluster, so as to determine the order in which the various workgroups contained in the workgroup set are scheduled to the corresponding computing units for execution.

[0100] In this embodiment, the number of workgroups contained in a single workgroup cluster can be determined based on the first number, for example, the first number is determined as the number of workgroups contained in a single workgroup cluster. Thereafter, according to the determined number of workgroups in the single workgroup cluster, the multiple consecutive workgroups in the workgroup set are divided into multiple workgroup clusters. For example, based on the first number, the number of workgroups in a single workgroup cluster is determined to be 8. At this time, according to the first serial number corresponding to each workgroup, the first 8 workgroups in the corresponding workgroup set are determined as the first workgroup cluster, the 9th to 16th workgroups are determined as the second workgroup cluster, and so on. In this way, the workgroup set corresponding to the block processing cluster can be divided into a plurality of ordered workgroup clusters. Subsequently, in each time period, the workgroups in the workgroup set can be scheduled to the first number of computing units of the block processing cluster for parallel execution in units of workgroup clusters. The first number of computing units can execute the workgroups in the ordered multiple workgroup clusters in series, that is, after the first number of computing units have executed all the workgroups in one workgroup cluster, they will execute the workgroups in the next workgroup cluster.

[0101] For example, referring to Figure 4After dividing the workgroups in the target workgroup table into workgroup sets corresponding to each block processing cluster, 8 workgroup sets are obtained, namely workgroup set 0 to workgroup set 7, and each workgroup set includes 24 workgroups. The number of computing units in a single block processing cluster is 4. At this time, the first number can be directly determined as the number of workgroups contained in a single workgroup cluster; then, every 4 consecutive workgroups in the workgroup set are determined as a workgroup cluster, so that the workgroup set corresponding to each block processing cluster is divided into 6 workgroup clusters. For example, the first 4 workgroups in the workgroup set are determined as the first workgroup cluster, the 5th to 8th workgroups are determined as the second workgroup cluster, and so on, to obtain multiple workgroup clusters. At the same time, referring to Figure 4 The multiple workgroups in each workgroup cluster are also ordered, and are recorded as workgroup 0 to workgroup 3. Afterwards, the four workgroups in the first workgroup cluster are first dispatched to the four computing units in the block processing cluster for parallel execution; after the computing units have executed the workgroups in the first workgroup cluster, the four workgroups in the second workgroup cluster are dispatched to these four computing units for parallel execution. Among them, workgroup 0 in the second workgroup cluster is dispatched to the computing unit used to execute workgroup 0 in the first workgroup cluster, and so on.

[0102] In a possible implementation, the number of workgroups in a single workgroup cluster may be several times the first number. In this case, the workgroups in the single workgroup cluster may be cyclically mapped to the computing units in the block processing cluster for execution in the order of the workgroups in the single workgroup cluster. Figure 5 A single block processing cluster includes 4 computing units, and the serial numbers of these 4 computing units are recorded from 0 to 3. At this time, every 8 workgroups in the workgroup set corresponding to the block processing cluster are divided into a workgroup cluster. At this time, the number of workgroups in a single workgroup cluster is twice the first number. At this time, for each workgroup cluster, workgroup 0 and workgroup 4 can be mapped to the computing unit with serial number 0, and workgroup 1 and workgroup 5 can be mapped to the computing unit with serial number 1, and so on.

[0103] In the embodiment disclosed in steps S501 to S502, the workgroups in the workgroup set corresponding to the block processing cluster are divided into a plurality of ordered workgroup clusters based on the number of computing units in a single block processing cluster. The workgroups in the same workgroup cluster can be executed in parallel by the first number of computing units in the block processing cluster, while the workgroups in different block processing clusters can only be executed serially. Based on this, the execution order of scheduling the various workgroups in the workgroup set to the first number of computing units in the block processing cluster for execution can be determined, so that the first number of computing units in the block processing cluster can accurately execute the first number of workgroups in the same workgroup cluster in parallel, and execute the workgroups in different workgroup clusters serially. This reduces the difficulty of mapping multiple workgroups in a single workgroup set to the first number of computing units in the corresponding block processing cluster.

[0104] In one embodiment, step S203 includes:

[0105] Step S601: In a work group set, determine the second serial number of each work group in the corresponding work group set according to the first serial number corresponding to each work group;

[0106] Map the workgroups in the corresponding workgroup set to the computing units in the block processing cluster, including:

[0107] Step S602 : for each work group, based on a first remainder obtained by performing a modulo operation on the first number using the corresponding second serial number, map the multiple work groups to computing units in the block processing cluster.

[0108] In step S601, the second sequence number refers to the sequence number of each work group in the corresponding work group set after being divided into the corresponding work group set. It can be understood that, referring to step S302, the first sequence numbers corresponding to the work groups in a single work group set are continuous, and the work groups in a single work group set are actually ordered.

[0109] Specifically, the first serial numbers corresponding to the various workgroups in a single workgroup set can be sorted in ascending or descending order, and the second serial numbers of the various workgroups in the corresponding workgroup set can be determined according to the sorting results. For example, a workgroup set contains 32 workgroups, and the first serial numbers corresponding to the workgroups in the workgroup set corresponding to a certain block processing cluster are 32 to 63, respectively. At this time, the workgroup with the first serial number 32 is the first workgroup in the workgroup set. Based on this, the second serial number corresponding to the workgroup with the first serial number 32 is 0.

[0110] In another possible implementation, referring to the relevant description of step S201, it can be seen that the number of workgroups in each block processing cluster is the same, and referring to the relevant description of step S302, it can be seen that the first serial numbers corresponding to the workgroups in each block processing cluster are continuous. In this case, for each workgroup, the serial number in the corresponding workgroup set can be obtained by subtracting several times the number of workgroups in a single workgroup set from the corresponding first serial number. Based on this, in one embodiment, by determining the second number of workgroups in a single workgroup set, and then for each workgroup, taking the corresponding first serial number modulo the second number, the second serial number corresponding to the workgroup can be obtained.

[0111] In step S602, for each block processing cluster, the workgroups in the corresponding workgroup set need to be assigned to the computing units therein for execution, and in order for the first number of computing units to accurately execute the workgroups in the workgroup set in parallel, it is necessary to determine the mapping relationship between each workgroup and the computing unit. In this embodiment, the first number is modulo the second serial number corresponding to each workgroup, and multiple workgroups with the same first remainder obtained by the modulo are mapped to the same computing unit, and multiple workgroups with different first remainders obtained by the modulo are respectively mapped to different computing units. In this way, the workgroups in the workgroup set can be evenly divided into the first number of computing units in the block processing cluster.

[0112] Specifically, in one embodiment, a first number of computing units in a single block processing cluster are sequentially assigned corresponding computing unit serial numbers. For example, each block processing cluster is composed of four workgroups, and the computing unit serial numbers corresponding to the four workgroups can be 0 to 3, respectively. At this time, after the first number corresponding to each workgroup is obtained by taking the modulo of the second serial number of each workgroup and the first remainder, each workgroup is mapped to a computing unit having the same computing unit serial number as the first remainder corresponding to the workgroup. For example, a workgroup with a first remainder of 0 is mapped to a computing unit with a computing unit serial number of 0.

[0113] In this way, each workgroup in a single workgroup set can be evenly mapped to the first number of computing units in the corresponding block processing cluster. The complexity of this mapping logic is relatively low, which can reduce the difficulty and error probability of the processing device allocating local computing resources (i.e., computing units) to execute each workgroup, so that the first number of computing units can accurately and orderly execute each workgroup in the workgroup set in parallel.

[0114] In the embodiment disclosed in steps S601 and S602, the first sequence number corresponding to each workgroup is converted into the second sequence number of the workgroup in the corresponding workgroup set, and then each workgroup is mapped to the first number of computing units in the block processing cluster based on the remainder of the second sequence number and the first number. In this way, each workgroup can be evenly divided into the first number of computing units in the block processing cluster using relatively simple mapping logic, so that the first number of computing units can accurately and orderly execute each workgroup in the workgroup set in parallel.

[0115] The implementation method is to first divide the workgroups in the target workgroup grid into multiple workgroup clusters, and then divide the multiple workgroup clusters into multiple workgroup sets.

[0116] In the above embodiment, a method is shown in which the first serial number corresponding to each workgroup is determined based on the workgroup coordinates corresponding to each workgroup, so as to linearize the target workgroup grid into a one-dimensional workgroup sequence along a preset dimension, and then determine multiple workgroups with consecutive first serial numbers as a single workgroup set, thereby dividing the target workgroup grid into workgroup sets corresponding to each block processing cluster. That is, in the above embodiment, multiple workgroups that are continuous along a single dimension of the target workgroup grid are divided into a workgroup set. In another implementation method, multiple workgroups that are continuous along multiple dimensions can also be divided into a workgroup set. Based on this, in one embodiment, step S202 includes:

[0117] Step S701: obtaining preset dimension widths corresponding to N dimensions of a single work group cluster in a target work group grid;

[0118] Step S702: determining the workgroup cluster coordinates corresponding to the workgroup based on the workgroup coordinates corresponding to each workgroup and the preset dimension widths corresponding to the N dimensions, and determining multiple workgroups corresponding to the same workgroup cluster coordinates as a single workgroup cluster;

[0119] Step S703 : Based on the work group cluster coordinates, the multiple work group clusters are divided into work group sets corresponding to the block processing clusters.

[0120] In step S701, the preset dimension width represents the coordinate span of the workgroups included in a single workgroup cluster along the corresponding dimension of the target workgroup grid. For example, if the preset dimension width of the y-dimension is 2, it means that in the target workgroup grid, each target workgroup cluster includes two workgroups along the y-dimension.

[0121] Specifically, the preset dimension widths of each dimension can be determined based on the number of computing units in a single block processing cluster, and the product of the preset dimension widths corresponding to each dimension is equal to the number of computing units in the single block processing cluster. For example, if the target workgroup grid is a three-dimensional grid and the number of computing units in a single block processing cluster is 8, the preset dimension widths corresponding to the three dimensions can be set to 4, 2, and 1, or to 2, 2, and 2, respectively, so that the number of workgroups in a single workgroup cluster is equal to the number of computing units in the single block processing cluster. In this way, the workgroups in a workgroup cluster can be assigned to the various computing units in the single block processing cluster for parallel execution.

[0122] In step S702, the workgroup cluster coordinates are the coordinates corresponding to each workgroup cluster in the target workgroup grid after the workgroups in the target workgroup grid are divided into multiple workgroup clusters. It can be understood that after the workgroups in the target workgroup grid are divided into different workgroup clusters based on the preset dimension widths corresponding to each dimension, each workgroup cluster actually corresponds to a matrix block of a specified shape in the target workgroup grid. At this time, the target workgroup grid can actually be regarded as a grid composed of workgroup clusters, and each workgroup cluster is a grid point in the grid. For example, referring to Figure 6 The target workgroup grid is a three-dimensional grid with a shape of (8, 4, 2), and the preset dimension widths corresponding to the x, y, and z dimensions are all 2. In this case, each workgroup in a three-dimensional region of the target workgroup grid with a shape of (2, 2, 2) is identified as a workgroup cluster. The target workgroup grid can be considered a three-dimensional grid with a shape of (4, 2, 1) with workgroup clusters as grid points. The process of determining the workgroup cluster coordinates corresponding to each workgroup in step S702 will be described later and is not repeated here.

[0123] In step S703, the process of dividing multiple workgroup clusters into multiple workgroup sets based on the workgroup cluster coordinates can refer to the process of dividing the target workgroup grid into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates in the above step S202, and will not be repeated here.

[0124] In the embodiment disclosed in steps S701 to S703, based on this, by presetting the preset dimension width corresponding to each dimension, the workgroups in the target workgroup grid can be more flexibly organized into multiple workgroup clusters; then, the workgroup set corresponding to each block processing cluster is determined based on the workgroup cluster. This allows the workgroups in the target workgroup grid to be more flexibly scheduled to the computing units in the processing device for parallel processing while ensuring that the scheduling logic complexity when scheduling the workgroups to the computing units for execution is low.

[0125] In one embodiment, in step S702, based on the workgroup coordinates corresponding to each workgroup and the preset dimension widths corresponding to the N dimensions, determining the workgroup cluster coordinates corresponding to the workgroup includes:

[0126] Step S801, for each workgroup, determine the coordinate values ​​of the workgroup in each dimension based on the corresponding workgroup coordinates, and determine the first ratio of the coordinate values ​​of the workgroup in each dimension to the corresponding preset dimension width, round the first ratio of the workgroup in each dimension, and obtain the values ​​of the workgroup cluster coordinates of the workgroup in each dimension to determine the workgroup cluster coordinates corresponding to each workgroup.

[0127] In step S801, the first ratio between the value of the workgroup coordinates of the target workgroup in each dimension and the corresponding preset dimension width is calculated, and the first ratio of each dimension is rounded to determine the coordinate value of the workgroup cluster coordinates corresponding to the workgroup in each dimension. In this way, the workgroup cluster coordinates corresponding to multiple workgroups whose values ​​of a certain dimension in the workgroup coordinates are in the same interval in this dimension are equal. For example, the preset dimension width of the x dimension is 4, then for workgroups with x coordinates of 0, 1, 2, and 3, their corresponding workgroup cluster coordinates are all equal in the x dimension. Afterwards, by determining multiple workgroups whose corresponding workgroup cluster coordinates have equal values ​​in each dimension as a workgroup cluster, the workgroups in an area of ​​a specified shape in the target workgroup grid can be divided into a workgroup cluster, and the workgroup cluster coordinates corresponding to each workgroup cluster can also be determined.

[0128] Specifically, the following formula can be used to determine the values ​​of the workgroup cluster coordinates of the workgroup in the corresponding dimension based on the coordinate values ​​of the workgroup in each dimension:

[0129] cwg_id.x= ;

[0130] cwg_id.y= ;

[0131] cwg_id.z= ;

[0132] Among them, cwg_id.x, cwg_id.y, and cwg_id.z are the values ​​of the workgroup cluster coordinates in the x, y, and z dimensions of the workgroup respectively; grid.x, grid.y, and grid.y are the values ​​of the workgroup coordinates in the x, y, and z dimensions of the workgroup respectively; Represents a floor function; 、 、 The preset dimension widths corresponding to the x, y, and z dimensions, respectively. 、 、 These are the number of workgroups contained in a single workgroup cluster in the x, y, and z dimensions, respectively. 、 、 These are the coordinate spans of a single workgroup in the x, y, and z dimensions, respectively. 、 、 The value of is usually 1.

[0133] It is understood that the above example shows the calculation method of the workgroup cluster coordinates when the target workgroup grid is a three-dimensional grid. If the target workgroup grid is a one-dimensional or two-dimensional grid, the calculation can be performed using the corresponding dimension according to the dimension set by the target workgroup grid. For example, when the target workgroup grid is a one-dimensional grid, the workgroup coordinates are also one-dimensional coordinates. In this case, it is only necessary to use the formula cwg_id.x= The corresponding workgroup cluster coordinates can be calculated.

[0134] For example, the target workgroup grid is a three-dimensional grid, including three dimensions of x, y, and z. The preset dimension widths corresponding to each dimension are 2, 2, and 1, respectively. The workgroup coordinates corresponding to a workgroup are (3, 1, 0), and the workgroup cluster coordinates corresponding to the workgroup can be expressed as ( , , ), that is, the workgroup cluster coordinates corresponding to this workgroup are (1,0,0). Similarly, the workgroup cluster coordinates corresponding to the three workgroups with workgroup coordinates (2,1,0), (2,0,0), and (3,0,0) are also (1,0,0). These four workgroups can then be divided into a workgroup cluster.

[0135] In addition, in the above embodiment, the values ​​of the workgroup cluster coordinates of each workgroup in each dimension are calculated by rounding down. In another embodiment, the calculation can also be performed by rounding up. However, it should be noted that if the workgroup cluster coordinates are calculated by rounding up, then when generating the target workgroup grid, the starting value of the coordinates of each dimension should be 1 instead of 0, that is, the coordinate value corresponding to the first workgroup in each dimension should be 1, and there cannot be a workgroup with a coordinate value of 0 in a certain dimension, so as to prevent the preset dimension width of a certain dimension from being greater than 1, and the workgroup with a coordinate value of 0 in this dimension from being divided into the same workgroup cluster with other workgroups with coordinate values ​​greater than 0 in this dimension.

[0136] In one embodiment, step S703 includes:

[0137] Step S901, determining the second dimension step lengths corresponding to the N dimensions of the target work group grid;

[0138] Step S902: determining a third sequence number corresponding to each workgroup cluster based on the workgroup cluster coordinates corresponding to each workgroup cluster and the second dimension step length of each dimension;

[0139] Step S903 : Based on the third sequence number, each workgroup cluster is mapped to a block processing cluster, and multiple workgroup clusters corresponding to a single block processing cluster are determined as a workgroup set corresponding to the block processing cluster.

[0140] In step S901, the second dimension step size represents the number of workgroup clusters contained in the target workgroup grid in the corresponding dimension. Specifically, for each dimension of the target workgroup grid, the corresponding second dimension step size can be determined based on the ratio of the corresponding first dimension step size to the preset dimension width. For example, if the first dimension step size corresponding to a certain dimension is 8 and the corresponding preset dimension width is 2, the corresponding second dimension step size is 8 / 2=4, which means that in the target workgroup grid, 8 workgroups are contained along this dimension, and every two workgroups are divided into one workgroup cluster. In this case, the workgroups can be divided into 4 workgroup clusters along this dimension.

[0141] In step S902, the corresponding third serial number may be determined based on the work group cluster coordinates according to the following formula:

[0142] cwg_flat_id=cwg_id.x;

[0143] cwg_flat_id=cwg_id.x+cwg_id.y*stride_in_cwg.x;

[0144] cwg_flat_id=cwg_id.x+cwg_id.y*stride_in_cwg.x+cwg_id.z*stride_in_cwg.x*stride_in_cwg.y;

[0145] Where cwg_flat_id represents the third ordinal number corresponding to the workgroup cluster; cwg_id.x, cwg_id.y, and cwg_id.z are the values ​​of the x, y, and z dimensions of the workgroup cluster coordinates, respectively; stride_in_cwg.x and stride_in_cwg.y are the second-dimension strides corresponding to the x and y dimensions, respectively. The detailed process of step S902 can be referred to above as step S402 and will not be repeated here.

[0146] In step S903, after determining the third serial number corresponding to each workgroup cluster, each workgroup cluster can be mapped to each block processing cluster in sequence according to the third serial number. In this way, multiple workgroup clusters mapped to the same block processing cluster constitute the workgroup set corresponding to the block processing cluster. Specifically, referring to the above step S302, based on the number of workgroup clusters and the number of block processing clusters, the number of workgroup clusters that need to be mapped to a single block processing cluster can be determined, and the workgroup clusters corresponding to multiple consecutive third serial numbers can be mapped to the same block processing cluster as the workgroup set corresponding to the block processing cluster. In this way, when scheduling the workgroup to execute in the computing unit, it is only necessary to schedule each workgroup cluster first.

[0147] In the embodiment disclosed in steps S901 to S903, after dividing multiple workgroups in the target workgroup grid into multiple workgroup clusters according to the preset dimensional width corresponding to each dimension, the second dimension step used to characterize the number of workgroup clusters contained in each dimension along the target workgroup grid is determined, and the target workgroup grid is linearized based on the workgroup cluster coordinates corresponding to each workgroup cluster and the second dimension step corresponding to each dimension, so as to determine the third serial number corresponding to each workgroup cluster; thereafter, each workgroup cluster is respectively mapped to each block processing cluster based on the third serial number, and the multiple workgroup clusters mapped to a single block processing cluster are determined as the workgroup set corresponding to the block processing cluster.

[0148] In one embodiment, step S903 includes:

[0149] Step S1001, performing a modulo operation on a second number based on a third serial number corresponding to each work group cluster to obtain a second remainder corresponding to the work group cluster;

[0150] In step S1002, corresponding workgroup clusters are respectively mapped to block processing clusters in the processing device based on the second remainder, wherein corresponding workgroup clusters with equal second remainders are mapped to the same block processing cluster, and corresponding workgroup clusters with unequal second remainders are mapped to different block processing clusters in the processing device.

[0151] The detailed description of step S1001 can refer to the above-mentioned step S601 and will not be repeated here.

[0152] In step S1002, specifically, each block processing cluster may be provided with a corresponding cluster number. After calculating the second remainder corresponding to each work group cluster, the work group cluster may be mapped to the block processing cluster whose cluster number is equal to the second remainder. In this way, each work group cluster may be cyclically mapped to each block processing cluster. For example, referring to Figure 7, every 4 workgroups in the target workgroup grid are regarded as a workgroup cluster, thereby dividing the workgroups in the target workgroup grid into 32 workgroup clusters. The third serial numbers corresponding to these 32 workgroup clusters are 0 to 31, and the processing device includes 8 block processing clusters. The serial numbers corresponding to the 8 block processing clusters are 0 to 7. At this time, the second remainders corresponding to the 4 workgroup clusters with third serial numbers of 0, 8, 16, and 24 are all 0. Based on this, the 4 workgroup clusters with third serial numbers of 0, 8, 16, and 24 are mapped to the block processing cluster with serial number 0.

[0153] It should be noted that, different from the embodiment disclosed in the above steps S301 to S302, in this embodiment, multiple workgroups are organized into multiple workgroup clusters, and then mapped to block processing clusters in units of workgroup clusters. Each workgroup cluster itself includes multiple workgroups that are continuous in various dimensions. Based on this, in this embodiment, each workgroup cluster is cyclically mapped to each block processing cluster based on the third sequence number, and then multiple computing units in the block processing cluster execute the workgroups in a single workgroup cluster in parallel. It can also ensure that the multiple workgroups executed in parallel by the computing units in a single block processing cluster at the same time are continuous in a certain dimension, and the complexity of scheduling the workgroups in the target workgroup grid to each computing unit for execution is still low.

[0154] Description of the apparatus and device of the present disclosure

[0155] Reference Figure 8 , Figure 8 1 is a structural diagram of a data scheduling device 800 proposed in the present disclosure. The data scheduling device 800 is integrated in a processing device. The processing device includes at least one block processing cluster. The block processing cluster includes multiple computing units. The computing units are provided with corresponding local memories. The data scheduling device 800 includes:

[0156] An acquisition unit 810 is configured to acquire a first input tensor and a target workgroup grid corresponding to the first input tensor, where the target workgroup grid includes multiple workgroups, each workgroup includes multiple threads, each thread corresponds to an element in the first input tensor, and each thread is configured to cause a computing unit to process the element corresponding to the thread. Each workgroup is provided with corresponding workgroup coordinates, which are coordinates of the corresponding workgroup in the target workgroup grid.

[0157] A first division unit 820 is configured to divide the target workgroup grid into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates corresponding to the workgroups, and schedule the workgroups in each workgroup set to the corresponding block processing cluster;

[0158] A first mapping unit 830 is configured to map, in the block processing cluster, the workgroups in the corresponding workgroup set to computing units in the block processing cluster;

[0159] The data scheduling unit 840 is used to divide the first input tensor into tensor blocks corresponding to each work group, and store each tensor block in the local memory of the computing unit used to execute the corresponding work group, so that the computing unit reads the tensor block corresponding to the work group from the corresponding local memory when executing the corresponding work group.

[0160] The data scheduling device disclosed in the present invention is used to execute the data scheduling method as described in the above embodiment. Its specific processing process is the same as the data scheduling method of the above embodiment and will not be repeated here.

[0161] The present disclosure also provides an electronic device 900, including:

[0162] at least one processor, and

[0163] a memory communicatively connected to at least one processor; wherein,

[0164] The memory stores instructions, and the instructions are executed by at least one processor so that the at least one processor implements a method as described in any one of the above embodiments of the present application when executing the instructions.

[0165] The following combination Figure 9 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 910 , a memory 920 , an input / output interface 930 , a communication interface 940 , and a bus 950 .

[0166] The processor 910 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0167] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called by the processor 910 to execute the data scheduling method of the embodiments of the present disclosure.

[0168] Input / output interface 930, used to implement information input and output;

[0169] Communication interface 940, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); and

[0170] bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0171] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0172] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the data scheduling method of the above embodiment, which will not be repeated here.

[0173] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present disclosure described herein, for example, can be implemented in orders other than those illustrated or described herein. In addition, the terms "comprises" and "comprising," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0174] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0175] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.

[0176] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0177] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0179] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0180] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A data scheduling method, characterized in that: Applied to a processing device, the processing device includes at least one block processing cluster, the block processing cluster includes a plurality of computing units, the computing units are provided with corresponding local memories, the method includes: Obtain a first input tensor and a target workgroup grid corresponding to the first input tensor, wherein the target workgroup grid includes a plurality of workgroups, the workgroup includes a plurality of threads, the threads correspond to elements in the first input tensor, the threads are used to enable the computing unit to process the elements corresponding to the threads, each of the workgroups is provided with a corresponding workgroup coordinate, and the workgroup coordinate is a coordinate of the corresponding workgroup in the target workgroup grid; Based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters, and the workgroups in each of the workgroup sets are scheduled to the corresponding block processing clusters; In the block processing cluster, mapping the corresponding workgroups in the workgroup set to computing units in the block processing cluster respectively; The first input tensor is divided into tensor blocks corresponding to each work group, and each tensor block is stored in the local memory of the computing unit used to execute the corresponding work group, so that the computing unit reads the tensor block corresponding to the work group from the corresponding local memory when executing the corresponding work group.

2. The data scheduling method according to claim 1, characterized in that: The step of dividing the target workgroup grid into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates corresponding to each of the workgroups includes: Based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is converted into a one-dimensional workgroup sequence, wherein each of the workgroups is ordered in the workgroup sequence, and the first sequence number of each of the workgroups in the workgroup sequence is determined according to the workgroup coordinates corresponding to the workgroup; Based on the first serial numbers corresponding to the respective working groups, the plurality of working groups are divided into working group sets corresponding to the block processing clusters respectively, wherein the first serial numbers corresponding to the working groups in the respective working group sets are continuous.

3. The data scheduling method according to claim 2, characterized in that: The dimension of the target working group grid is N, and the target working group grid is an N-dimensional grid. The target working group grid is converted into a one-dimensional working group sequence based on the working group coordinates corresponding to each working group, including: Obtaining first dimension step lengths corresponding to N dimensions of the target working group grid, respectively, where the first dimension step lengths represent the maximum number of working groups of the target working group grid in the corresponding dimension; Determine a first serial number corresponding to each of the working groups based on the working group coordinates of each of the working groups in the target working group grid and the first dimension step length of each of the dimensions; According to the first serial numbers corresponding to the respective working groups, the target working group grid is converted into the working group sequence.

4. The data scheduling method according to claim 2, characterized in that: After the plurality of workgroups are divided into a plurality of workgroup sets corresponding to the block processing clusters based on the first sequence numbers corresponding to the workgroups, the method further includes: determining a first number of computing units in the block processing cluster; Based on the first number, the workgroups in the workgroup set corresponding to the block processing cluster are divided into a plurality of ordered workgroup clusters, wherein the number of workgroups in a single workgroup cluster is an integer multiple of the first number, the workgroups in a single workgroup cluster are sequentially mapped to the computing units in the block processing cluster, the plurality of workgroups in a single workgroup cluster are executed in parallel in the first number of computing units in the block processing cluster, and the ordered plurality of workgroup clusters are executed serially in the first number of computing units in the block processing cluster.

5. The data scheduling method according to claim 4, characterized in that: After the plurality of work groups are divided into work group sets corresponding to the block processing clusters based on the first sequence numbers corresponding to the work groups, the method further includes: In the work group set, according to the first serial number corresponding to each work group, determine the second serial number of the work group in the corresponding work group set; The mapping of the corresponding workgroups in the workgroup set to computing units in the block processing cluster respectively includes: For each of the working groups, based on a first remainder obtained by taking the first number modulo the corresponding second sequence number, the plurality of working groups are respectively mapped to the computing units in the block processing cluster.

6. The data scheduling method according to claim 1, characterized in that: The target workgroup grid is an N-dimensional grid, and the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates corresponding to each of the workgroups, including: Obtaining preset dimension widths corresponding to N dimensions of a single workgroup cluster in the target workgroup grid, wherein the preset dimension width represents a coordinate span of the workgroups included in the corresponding dimension of a single workgroup cluster in the target workgroup grid; Based on the workgroup coordinates corresponding to each of the workgroups and the preset dimension widths corresponding to the N dimensions, determine the workgroup cluster coordinates corresponding to the workgroup, and determine the multiple workgroups corresponding to the same workgroup cluster coordinates as a single workgroup cluster; Based on the work group cluster coordinates, the plurality of work group clusters are divided into the work group sets respectively corresponding to the block processing clusters.

7. The data scheduling method according to claim 6, characterized in that: The determining the workgroup cluster coordinates corresponding to the workgroup based on the workgroup coordinates corresponding to each of the workgroups and the preset dimension widths corresponding to the N dimensions includes: For each of the working groups, the coordinate values ​​of the working group in each dimension are determined according to the corresponding working group coordinates, and a first ratio of the coordinate values ​​of the working group in each dimension to the corresponding preset dimension width is determined. The first ratio of the working group in each dimension is rounded to obtain the values ​​of the working group cluster coordinates of the working group in each dimension, so as to determine the working group cluster coordinates corresponding to each of the working groups.

8. The data scheduling method according to claim 6, characterized in that: The dividing the plurality of workgroup clusters into the workgroup sets respectively corresponding to the block processing clusters based on the workgroup cluster coordinates includes: Determine the second dimension step lengths corresponding to the N dimensions of the target work group grid, respectively, where the second dimension step lengths represent the number of the work group clusters included in the target work group grid in the corresponding dimension; Determine a third serial number corresponding to each of the working group clusters based on the working group cluster coordinates corresponding to each of the working group clusters and the second dimension step length of each dimension; Based on the third sequence number, each of the workgroup clusters is mapped to the block processing cluster, and the multiple workgroup clusters corresponding to a single block processing cluster are determined as a workgroup set corresponding to the block processing cluster.

9. The data scheduling method according to claim 8, characterized in that: The number of the block processing clusters is a second number, and mapping each of the workgroup clusters to the block processing clusters based on the third sequence number includes: Taking a modulus of the second number based on the third sequence number corresponding to each of the working group clusters, to obtain a second remainder corresponding to the working group cluster; Based on the second remainder, the corresponding workgroup clusters are respectively mapped to the block processing clusters in the processing device, wherein the corresponding workgroup clusters with equal second remainders are mapped to the same block processing cluster, and the corresponding workgroup clusters with unequal second remainders are mapped to different block processing clusters in the processing device.

10. A data scheduling device, characterized in that: The data scheduling device is integrated in a processing device, the processing device includes at least one block processing cluster, the block processing cluster includes a plurality of computing units, the computing units are provided with corresponding local memories, and the data scheduling device includes: an acquisition unit, configured to acquire a first input tensor and a target workgroup grid corresponding to the first input tensor, wherein the target workgroup grid includes a plurality of workgroups, the workgroup includes a plurality of threads, the threads correspond to elements in the first input tensor, the threads are configured to enable the computing unit to process the elements corresponding to the threads, each of the workgroups is provided with a corresponding workgroup coordinate, and the workgroup coordinate is a coordinate of the corresponding workgroup in the target workgroup grid; A first division unit, configured to divide the target work group grid into work group sets corresponding to the block processing clusters respectively based on work group coordinates corresponding to each of the work groups, and schedule the work groups in each of the work group sets to the corresponding block processing clusters; A first mapping unit, configured to map, in the block processing cluster, the corresponding workgroups in the workgroup set to computing units in the block processing cluster respectively; A data scheduling unit is used to divide the first input tensor into tensor blocks corresponding to each work group, and store each tensor block in the local memory of the computing unit used to execute the corresponding work group, so that the computing unit reads the tensor block corresponding to the work group from the corresponding local memory when executing the corresponding work group.

11. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the data scheduling method according to any one of claims 1 to 9 is realized.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data scheduling method according to any one of claims 1 to 9.

Images