Data scheduling method and device, electronic device and medium

Abstract

The present disclosure provides a data scheduling method and device, an electronic device, and a medium, the method comprising: obtaining a first input tensor and a target workgroup grid corresponding to the first input tensor; based on the workgroup coordinates corresponding to each workgroup, dividing the target workgroup grid into workgroup sets corresponding to block processing clusters, and scheduling the workgroups in each workgroup set to the corresponding block processing cluster; in the block processing cluster, mapping the workgroups in the corresponding workgroup set to the computing units in the block processing cluster respectively; dividing the first input tensor into tensor blocks corresponding to each workgroup, and storing each tensor block in the local memory of the computing unit for executing the corresponding workgroup. Based on the data scheduling method disclosed in the present disclosure, the efficiency of the computing unit accessing the storage unit for data access can be improved, the power consumption of memory access can be reduced, and at the same time, the computing unit can be prevented from being unable to read the required data in time due to the bandwidth limitation of the routing module.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a data scheduling method and device, an electronic device, and a medium. Background Art

[0002] In the prior art, multiple computing units in a graphics processing unit use a uniform memory access (UMA) method to store and read data. This method randomly distributes the data input to the graphics processing unit to the various storage units of the graphics processing unit for storage. Afterwards, when calculating, the graphics processing unit will allocate several threads to each computing unit to assign corresponding data processing tasks to these computing units. The computing unit is routed through a routing module such as a Crossbar or an on-chip network to access the storage unit that stores the data required by the computing unit when executing the corresponding thread.

[0003] In the prior art, when a computing unit accesses a storage unit, it needs to be routed through a routing module in order to access the storage unit storing the required data. This results in a longer communication link when the computing unit accesses the storage unit, a low memory access rate, and high memory access power consumption. At the same time, these routing modules are often shared by multiple computing units. When multiple computing units simultaneously call the same routing module to access the storage unit, it is easy to be limited by the bandwidth of the routing module, resulting in the computing unit being unable to access the storage unit in a timely manner for data access. Summary of the invention

[0004] The embodiments of the present disclosure provide a data scheduling method and device, an electronic device and a medium, which can improve the efficiency of a computing unit accessing a storage unit for data access, reduce the power consumption of memory access, and avoid the computing unit being unable to read the required data in a timely manner due to the bandwidth limitation of the routing module.

[0005] According to one aspect of the present disclosure, a data scheduling method is provided, which is applied to a processing device, wherein the processing device includes at least one block processing cluster, the block processing cluster includes multiple computing units, and the computing units are provided with corresponding local memories. The method includes: obtaining a first input tensor and a target workgroup grid corresponding to the first input tensor, the target workgroup grid includes multiple workgroups, the workgroup includes multiple threads, the threads correspond to elements in the first input tensor, and the threads are used to enable the computing unit to process the elements corresponding to the threads, each of the workgroups is provided with corresponding workgroup coordinates, and the workgroup coordinates are the coordinates of the corresponding workgroup in the target workgroup grid; based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters, and the workgroups in each of the workgroup sets are scheduled to the corresponding block processing clusters; in the block processing cluster, the workgroups in the corresponding workgroup sets are respectively mapped to the computing units in the block processing cluster; the first input tensor is divided into tensor blocks corresponding to each of the workgroups, and each tensor block is stored in the local memory of the computing unit for executing the corresponding workgroup, so that the computing unit reads the tensor blocks corresponding to the workgroup from the corresponding local memory when executing the corresponding workgroup.

[0006] Optionally, based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters, including: based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is converted into a one-dimensional workgroup sequence, wherein each of the workgroups is ordered in the workgroup sequence, and the first serial number of each of the workgroups in the workgroup sequence is determined according to the workgroup coordinates corresponding to the workgroup; based on the first serial number corresponding to each of the workgroups, the multiple workgroups are divided into workgroup sets corresponding to the block processing clusters, wherein the first serial numbers corresponding to the workgroups in each of the workgroup sets are continuous.

[0007] Optionally, the target workgroup grid has N dimensions, and the target workgroup grid is an N-dimensional grid. Based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is converted into a one-dimensional workgroup sequence, including: obtaining the first dimension step corresponding to the N dimensions of the target workgroup grid, the first dimension step representing the maximum number of workgroups of the target workgroup grid in the corresponding dimension; determining the first serial number corresponding to each of the workgroups based on the workgroup coordinates of each of the workgroups in the target workgroup grid and the first dimension step of each dimension; and converting the target workgroup grid into the workgroup sequence according to the first serial number corresponding to each of the workgroups.

[0008] Optionally, after dividing the multiple workgroups into multiple workgroup sets corresponding to the block processing clusters respectively based on the first serial numbers corresponding to each of the workgroups, the method further includes: determining a first number of computing units in the block processing cluster; dividing the workgroups in the workgroup set corresponding to the block processing cluster into a plurality of ordered workgroup clusters based on the first number, wherein the number of workgroups in a single workgroup cluster is an integer multiple of the first number, the workgroups in a single workgroup cluster are sequentially mapped to the computing units in the block processing cluster, the multiple workgroups in a single workgroup cluster are executed in parallel in the first number of computing units in the block processing cluster, and the ordered plurality of workgroup clusters are executed serially in the first number of computing units in the block processing cluster.

[0009] Optionally, after dividing the multiple workgroups into workgroup sets corresponding to the block processing clusters respectively based on the first serial numbers corresponding to each of the workgroups, the method further includes: in the workgroup set, determining the second serial number of the workgroup in the corresponding workgroup set according to the first serial numbers corresponding to each of the workgroups; and mapping the workgroups in the corresponding workgroup set to the computing units in the block processing cluster respectively includes: for each of the workgroups, based on the first remainder obtained by taking the first number modulo the corresponding second serial number, mapping the multiple workgroups to the computing units in the block processing cluster respectively.

[0010] Optionally, the target workgroup grid is an N-dimensional grid, and the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates corresponding to each of the workgroups, including: obtaining a preset dimension width corresponding to the N dimensions of the target workgroup grid for a single workgroup cluster, the preset dimension width representing the coordinate span of the workgroups contained in the corresponding dimension of the target workgroup grid for a single workgroup cluster; determining the workgroup cluster coordinates corresponding to the workgroup based on the workgroup coordinates corresponding to each of the workgroups and the preset dimension widths corresponding to the N dimensions, and determining multiple workgroups corresponding to the same workgroup cluster coordinates as a single workgroup cluster; and dividing multiple workgroup clusters into the workgroup sets corresponding to the block processing clusters based on the workgroup cluster coordinates.

[0011] Optionally, the working group cluster coordinates corresponding to the working group are determined based on the working group coordinates corresponding to each of the working groups and the preset dimensional widths corresponding to the N dimensions, including: for each of the working groups, determining the coordinate values ​​of the working group in each dimension according to the corresponding working group coordinates, and determining a first ratio of the coordinate value of the working group in each dimension to the corresponding preset dimensional width, rounding the first ratio of the working group in each dimension to obtain the value of the working group cluster coordinates of the working group in each dimension, so as to determine the working group cluster coordinates corresponding to each of the working groups.

[0012] Optionally, based on the workgroup cluster coordinates, dividing the multiple workgroup clusters into the workgroup sets corresponding to the block processing clusters respectively includes: determining the second dimension step sizes corresponding to the N dimensions of the target workgroup grid, the second dimension step size representing the number of the workgroup clusters contained in the target workgroup grid in the corresponding dimension; determining the third serial number corresponding to each workgroup cluster based on the workgroup cluster coordinates corresponding to each of the workgroup clusters and the second dimension step size of each dimension; based on the third serial number, mapping each of the workgroup clusters to the block processing cluster respectively, and determining the multiple workgroup clusters corresponding to a single block processing cluster as the workgroup set corresponding to the block processing cluster.

[0013] Optionally, the number of block processing clusters is a second number, and mapping each of the workgroup clusters to the block processing cluster based on the third serial number includes: taking the modulus of the second number based on the third serial number corresponding to each of the workgroup clusters to obtain a second remainder corresponding to the workgroup cluster; and mapping the corresponding workgroup clusters to the block processing clusters in the processing device based on the second remainder, wherein the corresponding workgroup clusters with equal second remainders are mapped to the same block processing cluster, and the corresponding workgroup clusters with unequal second remainders are mapped to different block processing clusters in the processing device.

[0014] According to one aspect of the present disclosure, a data scheduling device is provided, wherein the data scheduling device is integrated in a processing device, wherein the processing device includes at least one block processing cluster, wherein the block processing cluster includes multiple computing units, and wherein the computing units are provided with corresponding local memories. The data scheduling device includes: an acquisition unit, configured to acquire a first input tensor and a target workgroup grid corresponding to the first input tensor, wherein the target workgroup grid includes multiple workgroups, wherein the workgroup includes multiple threads, wherein the threads correspond to elements in the first input tensor, and wherein the threads are configured to enable the computing unit to process the elements corresponding to the threads, wherein each of the workgroups is provided with corresponding workgroup coordinates, and wherein the workgroup coordinates are the coordinates of the corresponding workgroup in the target workgroup grid. ; a first division unit, for dividing the target workgroup grid into workgroup sets corresponding to the block processing clusters respectively based on the workgroup coordinates corresponding to each of the workgroups, and scheduling the workgroups in each of the workgroup sets to the corresponding block processing clusters; a first mapping unit, for mapping the workgroups in the corresponding workgroup sets to the computing units in the block processing cluster in the block processing cluster; a data scheduling unit, for dividing the first input tensor into tensor blocks corresponding to each of the workgroups, and storing each tensor block into the local memory of the computing unit for executing the corresponding workgroup, so that the computing unit reads the tensor blocks corresponding to the workgroup from the corresponding local memory when executing the corresponding workgroup.

[0015] According to one aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing a connection and communication between the processor and the memory, wherein the program, when executed by the processor, implements the data scheduling method as described in any one of the above items.

[0016] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data scheduling method as described in any one of the above items.

[0017] In the data scheduling method proposed in the embodiment of the present disclosure, after obtaining the first input tensor to be processed and the target workgroup grid composed of threads for instructing each computing unit in the processing device to process the elements in the first input tensor, based on the workgroup coordinates of each workgroup in the target workgroup grid, multiple workgroups are divided into multiple workgroup sets corresponding to the block processing clusters in the processing device, and each workgroup set is scheduled to the corresponding block processing cluster; then, in each block processing cluster, each workgroup contained in the corresponding workgroup set is mapped to each computing unit in the block cluster, so that the computing unit used to execute each workgroup is determined; then, since each thread is used to instruct the computing unit to process the element corresponding to the thread in the first input tensor, and each workgroup A work group contains multiple threads, and the multiple threads contained in a single work group often correspond to the elements in a continuous area in the first input tensor, that is, each work group actually corresponds to the elements in an area in the first input tensor. Based on this, in this embodiment, the first input tensor is divided into tensor blocks corresponding to each work group, and each tensor block is respectively scheduled to the local memory of the computing unit used to execute the corresponding work group. In this way, when the computing unit subsequently executes each corresponding functional work group, it can directly read the tensor block consisting of the data required to execute the work group from the corresponding local memory, and the latency and power consumption of the computing unit when accessing the corresponding local memory are relatively low, so that the efficiency of each computing unit in reading the data required to execute the corresponding work group can be improved, and the power consumption of memory access can be reduced. At the same time, the computing unit can directly access the corresponding local memory, so the computing unit does not need to access the global memory through routing modules such as crossbar to obtain the data required to execute the work group, thereby preventing the computing unit from being unable to read the data required to execute the corresponding work group in time due to the bandwidth limitation of the routing module.

[0018] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0020] Figure 1 is an architectural diagram of a processing device to which the data scheduling method according to an embodiment of the present disclosure is applied;

[0021] Figure 2is a main flow chart of a data scheduling method according to an embodiment of the present disclosure;

[0022] Figure 3 is a schematic diagram of converting a target work group grid into a work group sequence according to an embodiment of the present disclosure;

[0023] Figure 4 is a schematic diagram of dividing a workgroup set corresponding to each block processing cluster into a plurality of workgroup clusters according to an embodiment of the present disclosure;

[0024] Figure 5 is another schematic diagram of dividing a set of work groups corresponding to each block processing cluster into a plurality of work group clusters according to an embodiment of the present disclosure;

[0025] Figure 6 is a schematic diagram of directly dividing a target workgroup table into a plurality of workgroup clusters based on preset dimension widths of each dimension according to an embodiment of the present disclosure;

[0026] Figure 7 is a schematic diagram of cyclically mapping a work group cluster to each block processing cluster based on a third sequence number according to an embodiment of the present disclosure;

[0027] Figure 8 is a system architecture diagram of a data scheduling device according to an embodiment of the present disclosure;

[0028] Fig. 9 It is a system architecture diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0030] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0031] Thread: It is the smallest execution unit of the graphics processing unit when performing data processing tasks. Each thread can independently perform the same mode of processing on different data. For example, the graphics processing unit needs to normalize the elements of each matrix row in an M*N matrix. At this time, M threads can be called in the graphics processing unit, and each thread is responsible for normalizing the elements of a matrix row in the matrix, so that the elements of the M matrix rows of the matrix can be processed in parallel by M threads. Alternatively, multiple threads perform different modes of processing on different data at the same time. For example, when the graphics processing unit normalizes the M*N matrix, it needs to copy the matrix data from the global memory to the register of the graphics processing unit's computing unit and then the computing unit performs normalization processing. At this time, some threads can be used to copy the matrix data from the global memory to the register of the computing unit, and the other threads are used to instruct the computing unit to normalize the data copied to the register.

[0032] Work Group: A group of threads consisting of multiple threads, also called a thread block or work item. Threads in a single work group are generally used to perform the same pattern of processing on different data.

[0033] Grid: It is composed of multiple workgroups. Each workgroup is a grid point in the Grid. Each workgroup is set with corresponding grid point coordinates. The Grid can be one-dimensional, two-dimensional or three-dimensional. It is understood that the shape of the Grid is set by the user during the programming process.

[0034] Compute Unit (CU): A processing module in electronic devices such as a graphics processing unit. In some architectures, a compute unit can also be called a stream multiprocessor. A graphics processing unit can include multiple compute units. A compute unit can be regarded as a processing core in a graphics processing unit. Each compute unit can execute instructions independently. A graphics processing unit can process data in parallel through multiple compute units.

[0035] Block Process Cluster: A cluster composed of multiple computing units in a single graphics processing unit. In some graphics processing unit architectures, a block processing cluster can also be called a graphics processing cluster (GPC). In a single graphics processing unit, each block processing cluster includes an equal number of computing units. For example, a graphics processing unit includes 32 computing units, and every 4 computing units are integrated into a block processing cluster, then the graphics processing unit includes 8 block processing clusters. It can be understood that the communication distance between computing units in the same block processing cluster is relatively close, while the communication distance between computing units in different block processing clusters is relatively far.

[0036] Non-Uniform Memory Access (NUMA): A memory architecture under multiple processing nodes. Under this memory architecture, each node is equipped with corresponding local memory. Each node can access the corresponding local memory at high speed, and different nodes can communicate across nodes to access the local memory of other nodes.

[0037] Description of the system architecture of the present disclosure embodiment

[0038] Figure 1 It is a system architecture diagram of a processing device 100 applied to the data scheduling method of an embodiment of the present disclosure, which includes: at least one block processing cluster 110, each block processing cluster includes a plurality of computing units 111, and each computing unit 111 is provided with a corresponding local memory 112.

[0039] Each computing unit 111 can independently execute different work groups, so as to process the various elements in the first input tensor in parallel. Specifically, each computing unit can include multiple execution units, tensor cores and other modules for performing arithmetic operations or logical operations on the elements in the first input tensor. When the computing unit executes the work group, the corresponding execution unit or tensor core and other modules execute each thread in the work group to process the elements in the first input tensor corresponding to each thread.

[0040] The block processing cluster 110 is a cluster composed of a plurality of computing units 111 in a processing device. In a possible implementation, a block processing cluster may be composed of a plurality of computing units integrated on a single silicon chip.

[0041] The local memory 112 is a storage space that the computing unit can directly access without routing through the routing module. It can be understood that the physical distance between the local memory and the corresponding computing unit is relatively close. Based on this, the communication link between the computing unit and the corresponding local memory can also be relatively short. The access speed of the computing unit when accessing the corresponding local memory will be higher than the access speed of the computing unit when accessing other storage spaces in the processing device.

[0042] In addition, when running a neural network model, when performing certain types of computing tasks, each computing unit may need to use the data stored in the local memory of other computing units when executing the corresponding work group. For example, in the attention mechanism, it is necessary to calculate the attention weight between each word unit and all word units. Figure 1 In the processing device shown, different computing units in a single block processing cluster can be connected through an interconnection module, and different block processing clusters in the processing device can also be communicatively connected through the interconnection module. Each computing unit can access the corresponding local memory at high speed, and can also access the local memory corresponding to other computing units in the same block processing cluster and the local memory corresponding to computing units in other block processing clusters through the interconnection module to read the data stored in the local memory corresponding to other computing units. It should be noted that when each computing unit accesses the local memory corresponding to other computing units, it needs to access it through the interconnection module, and the distance between each computing unit and the local memory corresponding to other computing units will be farther than the distance between the local memory corresponding to the computing unit itself. Therefore, the access speed of each computing unit when accessing the local memory corresponding to other computing units will be slightly slower than the access speed when accessing the local memory corresponding to the computing unit itself. The local memories corresponding to multiple computing units in the processing device can constitute a non-uniform memory access architecture.

[0043] Overall implementation of the data scheduling method of the disclosed embodiment

[0044] The present disclosure provides a data scheduling method, which is applied to Figure 1 The processing device shown in the figure includes at least one block processing cluster, each block processing cluster includes multiple computing units, each computing unit is provided with a corresponding local memory, and the processing device includes at least one block processing cluster, each block processing cluster includes a plurality of ... Figure 2 , the data scheduling method includes:

[0045] Step S201, obtaining a first input tensor and a target workgroup grid corresponding to the first input tensor, the target workgroup grid including a plurality of workgroups, the workgroup including a plurality of threads, the threads corresponding to the elements in the first input tensor, the threads being used to indicate that the computing unit processes the elements corresponding to the threads, each workgroup being provided with a corresponding workgroup coordinate, the workgroup coordinate being the coordinate of the corresponding workgroup in the target workgroup grid;

[0046] Step S202, based on the workgroup coordinates corresponding to each workgroup, divide the target workgroup grid into workgroup sets corresponding to the block processing clusters, and schedule the workgroups in each workgroup set to the corresponding block processing cluster;

[0047] Step S203, in the block processing cluster, mapping the workgroups in the corresponding workgroup set to computing units in the block processing cluster respectively;

[0048] Step S204, divide the first input tensor into tensor blocks corresponding to each work group, and store each tensor block in the local memory of the computing unit used to execute the corresponding work group, so that the computing unit reads the tensor block corresponding to the work group from the corresponding local memory when executing the corresponding work group.

[0049] It can be understood that in this embodiment, the processing device can be a device for performing arithmetic operations or logical operations on the first input tensor, for example, the processing device can be a graphics processing unit (GPU). The computing unit is an independent computing core in the processing device, and multiple computing units can be provided in the processing device, and different computing units can independently execute the same or different instructions on different data.

[0050] A block processing cluster is a cluster composed of computing units in a graphics processing unit. A graphics processing unit may include at least one block processing cluster, and each block processing cluster includes multiple computing units. Exemplarily, a graphics processing unit includes 32 computing units, and these 32 computing units can be integrated into 8 block processing clusters, each of which includes 4 computing units, or a graphics processing unit includes only 8 computing units, and these 8 computing units are integrated into a single block processing cluster. Specifically, the number of block processing clusters in the processing device and the number of convolution computing units in a single block processing cluster are determined based on the hardware architecture of the processing device, and are not limited in this embodiment. It should be noted that in a possible implementation, all computing units in the processing device may not be explicitly divided into different blocks in the hardware architecture. In this case, the entire processing device can be regarded as a single block processing cluster.

[0051] In step S201, the first input tensor is a tensor that the processing device needs to process when running the neural network model. It can be a set of intermediate data organized into a tensor form generated according to the input data of the neural network model. For example, the input of the neural network model. The dimension of the first input tensor can be one-dimensional or multi-dimensional. For example, the first input tensor can be a two-dimensional tensor with a shape of (H, W), which is composed of H W-dimensional vectors; or the first input tensor can be a three-dimensional tensor with a shape of (N, H, W), which is composed of N H*W matrices, each of which is composed of H W-dimensional vectors.

[0052] The target workgroup grid is a grid composed of multiple workgroups, each of which includes multiple threads for causing the computing unit to process each element in the first input tensor respectively, each thread corresponding to a different element in the first input tensor, and the threads contained in the entire target workgroup grid correspond to each element in the first input tensor respectively, that is, the target workgroup grid contains all threads for instructing the computing unit to process each element in the first input tensor. It can be understood that in the neural network model, the input data is often converted into a form suitable for processing by the neural network model for subsequent processing, that is, in the neural network model, the input data actually exists in the form of multiple feature vectors, and the first input tensor is actually a tensor composed of multiple feature vectors. At this time, each thread can correspond to one or more feature vectors constituting the first input tensor, and all threads constituting the target workgroup grid correspond to all feature vectors constituting the first input tensor respectively. For example, the shape of the first input tensor is (N, H, W), which indicates that the first input tensor contains N*H W-dimensional feature vectors. At this time, the target workgroup grid can be composed of N*H threads, each thread corresponding to a W-dimensional feature vector in the first input tensor. In one possible implementation, in order to simplify thread management, a single work group may be composed of multiple consecutive threads, and the multiple consecutive threads may correspond to multiple consecutive feature vectors in the first input tensor, that is, a work group may be regarded as corresponding to a continuous group of elements in the first input tensor.

[0053] In one embodiment, the shape of the target workgroup grid and the shape of a single workgroup can be preset by the user. For example, in CUDA, the shape of the target workgroup grid and the shape of the workgroup can be preset respectively by the statement dim3 gridDim(a1, b1, c1) and the statement dim3 blockDim(a2, b2, c2), which indicates that the target workgroup grid contains a1 workgroups in the x dimension, b1 workgroups in the y dimension, and c1 workgroups in the z dimension; a single workgroup contains a2 threads in the x dimension, b2 threads in the y dimension, and c2 threads in the z dimension. At this time, the target workgroup grid contains a1*b1*c1 workgroups, and each workgroup contains a2*b2*c2 threads. In addition, it can be understood that the shape of the target workgroup grid and the shape of the workgroup set by the user are used to instruct the processing device how to allocate multiple computing units for parallel processing of the first input tensor. In order to maximize the utilization of computing resources in the processing device, the shape of the target workgroup grid and the shape of the workgroup are often determined based on the shape of the first input tensor. The number of workgroups contained in the target workgroup grid is often an integer multiple of the number of computing units in the processing device. This ensures that multiple workgroups can be evenly allocated to the computing units in the processing device for parallel processing, and the number of workgroups required to be processed by each computing unit is equal.

[0054] The work group coordinates are coordinates for indicating the position of the corresponding work group in the target work group grid, and the coordinates can be one-dimensional or multi-dimensional, and the dimensions of the work group coordinates can be determined according to the dimensions of the target work group grid. For example, if the target work group grid is a three-dimensional grid, the work group coordinates are also three-dimensional coordinates. It is understandable that in a single target work group grid, the work group coordinates of each work group are unique, and the work group coordinates corresponding to different work groups are different.

[0055] Specifically, the target workgroup grid can be created by the processing device after the central processing unit (CPU) issues instructions to the processing device. Specifically, the user will set a corresponding kernel function in the central processing unit, and the kernel function indicates the shape of a single workgroup and the shape of the target workgroup grid, that is, it indicates the number of threads contained in a single workgroup in each dimension and the number of workgroups contained in the target workgroup grid in each dimension. When the central processing unit executes the kernel function, it issues a corresponding instruction to the processing device. In response to the instruction, the processing device creates multiple threads for instructing the computing unit to process each element of the first input tensor respectively, and organizes the created threads into multiple workgroups according to the shape indicated by the kernel function, and then organizes the multiple workgroups into the target workgroup grid.

[0056] In step S202, the work group set refers to a set consisting of multiple work groups in the target work group grid. It can be understood that the work group coordinates indicate the position of the work group in the target work group grid. In the target work group grid, the work group corresponding to each work group coordinate is unique, and each work group coordinate can be used as a unique identification of the corresponding work group. Based on this, in this embodiment, the work groups in the target work group grid can be divided into a number of work group sets corresponding to the block processing clusters in the processing device based on the work group coordinates corresponding to each work group, so that each work group is mapped to a block processing cluster respectively.

[0057] Specifically, referring to the description of step S201 above, it can be known that the number of workgroups in the target workgroup grid is often an integer multiple of the total number of computing units in the processing device, and the number of computing units in each block processing cluster is equal, that is, the total number of workgroups is actually an integer multiple of the number of block processing clusters, and the computing units in a single processing device are often homogeneous, that is, the computing power of each computing unit in a single processing device is often the same. At this time, dividing multiple workgroups into multiple block processing clusters can make more full use of the computing units in the processing device. Based on this, in this embodiment, when dividing the target workgroup grid, it can be based on the number of block processing clusters, and the target workgroup grid can be divided into multiple workgroup sets corresponding to each block processing cluster according to preset rules.

[0058] It is understandable that in a processing device, when multiple computing units perform parallel computing on various parts of data in a single input tensor, the process of computing various parts of data in the first input tensor is often relatively independent, and computing various parts of data in a single input tensor in different orders does not affect the computing result. Based on this, in this embodiment, the method of dividing the target workgroup grid into workgroup sets corresponding to each block processing cluster can be based on actual settings, which is not limited in this embodiment.

[0059] In a possible embodiment, the target workgroup grid can be divided into multiple regions of the same size based on the number of block processing clusters, and the workgroups contained in each region constitute a workgroup set corresponding to a block processing cluster. For example, the number of block processing clusters in the processing device is 8, and the target workgroup grid is a (32, 16, 2) grid. Then, the target workgroup grid is evenly divided into 8 regions of the shape of (8, 8, 2), and the workgroups contained in each region are the workgroups contained in the workgroup set corresponding to a block processing cluster.

[0060] In another possible embodiment, the workgroup coordinates corresponding to each workgroup can be linearized according to preset rules, and the workgroup coordinates corresponding to each workgroup can be converted into a corresponding serial number. Then, according to the serial numbers of each workgroup, multiple workgroups in the target workgroup grid can be divided into workgroup sets corresponding to each block processing cluster. For example, the target work group grid is a two-dimensional grid with a shape of (16, 8), that is, the target work group grid includes 16 work groups along the x dimension and 8 work groups along the y dimension. At this time, the coordinates of each work group are linearized along the x dimension of the target work group grid, and the work group with work group coordinates (0, 0) corresponds to the serial number 0, and the work group with work group coordinates (1, 0) corresponds to the serial number 1. By analogy, directly determine the corresponding serial numbers of all work groups with a y coordinate of 0, that is, the first row of the target work group grid, and then start linearizing the data with a y coordinate of 1, that is, the second row of the target work group grid. Among them, since the target work group grid contains 16 work groups along the x dimension, that is, the serial number corresponding to the last work group in the first row is 15, then each work group with a y coordinate of 1 has a starting serial number of 16. Similarly, each work group with a y coordinate of 2 has a starting serial number of 32. By analogy, it can be corresponded according to each work group. After linearizing the workgroup coordinates of each workgroup and obtaining the serial numbers corresponding to each workgroup, multiple workgroups can be divided into multiple workgroup sets corresponding to each block processing cluster according to the serial numbers. Specifically, several workgroups with consecutive serial numbers can be divided into a workgroup set. For example, for a two-dimensional target workgroup grid with a shape of (16, 8), after linearization, the serial numbers corresponding to each workgroup are 0 to 107. When the number of block processing clusters is 8, each block processing cluster corresponds to 16 workgroups. At this time, the workgroups with serial numbers 0 to 15 are used as a workgroup set corresponding to a block processing cluster, and the workgroups with serial numbers 16 to 31 are used as a workgroup set corresponding to another block processing cluster. By analogy, the workgroup sets corresponding to each block processing cluster are obtained.

[0061] In another possible embodiment, after linearizing the workgroup coordinates corresponding to each workgroup, each workgroup can be mapped to the workgroup set corresponding to each block processing cluster according to the corresponding serial number in a cyclic manner. For example, the number of block processing clusters is 8, and each workgroup is cyclically mapped to the 1st block processing cluster to the 8th block processing cluster according to the serial number. For example, the workgroup with serial number 0 is mapped to the workgroup set corresponding to the 1st block processing cluster, and the workgroup with serial number 1 is mapped to the workgroup set corresponding to the 2nd block processing cluster, and so on. After mapping the workgroup with serial number 7 to the workgroup set corresponding to the 8th block processing cluster, the workgroup with serial number 8 is remapped to the workgroup set corresponding to the 1st block processing cluster, and the next round of cyclic mapping begins. By analogy, the workgroup set corresponding to each block processing cluster can be obtained.

[0062] After dividing the target workgroup grid into workgroup sets corresponding to each block processing cluster, each workgroup set is scheduled to the corresponding block processing cluster, so that the computing units in each subsequent block processor cluster can be used to execute the workgroups in the corresponding workgroup set.

[0063] In step S203, it can be understood that each work group actually needs to be executed by a computing unit in a block processing cluster, and the block processing cluster includes multiple computing units, and the work group set includes multiple work groups. Based on this, in this embodiment, after scheduling each work group set to the corresponding block processing cluster, each work group needs to be mapped to each computing unit in the block processing cluster to determine which computing unit will execute each work group later.

[0064] Specifically, referring to the relevant description of step S202, it can be known that the number of workgroups in a single workgroup set is often several times the number of computing units in a single block processing cluster. Based on this, in this embodiment, the workgroups in the workgroup set corresponding to the block processing cluster can be evenly divided into each computing unit in the block processing cluster, and the way of mapping the workgroups in the workgroup set corresponding to the block processing cluster to the computing unit in the block processing cluster can refer to the way of mapping the workgroups in the target workgroup grid to the block processing cluster in step S202. For example, determine the serial number of each workgroup in the corresponding workgroup set; then, based on the serial number of each workgroup in the corresponding workgroup set, cyclically map each workgroup to each computing unit in the block processing cluster; or, after determining the serial number, based on the number of computing units in the block processing cluster, divide the workgroups in a single workgroup set into workgroup clusters corresponding to each computing unit in the block processing cluster, so as to map each workgroup in the workgroup set to a computing unit in the block processing cluster.

[0065] In one embodiment, after determining the computing units corresponding to each work group, each work group can be color-coded through the compute shader in the processing device, and multiple work groups corresponding to the same computing unit are coded with the same color, while work groups corresponding to different computing units are coded with different colors. In this way, the scheduler in the processing device only needs to determine which work groups need to be scheduled to the same computing unit for execution based on the colors corresponding to each work group, thereby simplifying the scheduling logic of the scheduler when scheduling work groups.

[0066] In step S204, the local memory refers to a storage unit that can be directly accessed by the computing unit, and the local memory can be a memory area that is physically tightly coupled with the corresponding computing unit. Specifically, the local memory can be a register inside the computing unit; or a memory that is physically directly connected to the corresponding computing unit and the link distance of the connection is extremely short. Alternatively, in a possible implementation, the computing unit is directly integrated in a storage unit, and constitutes a storage and computing module with the storage unit. At this time, the local memory corresponding to the computing unit is the storage unit where the computing unit is located. It can be understood that since the local memory is often a storage space integrated inside the computing unit, or a storage space integrated in a position that is physically very close to the computing unit, or even a storage space that directly integrates the computing unit inside, the communication link between the computing unit and the corresponding local memory is extremely short, which indicates that the latency of the computing unit when accessing the corresponding local memory will become relatively low, the bandwidth will be relatively large, and the power consumption caused by access will also be relatively low.

[0067] In one embodiment, the local memory is an off-chip memory that is physically tightly coupled with the corresponding computing unit. As an off-chip memory, it is integrated outside the chip where the computing unit is located, and does not need to occupy the area in the chip where the computing unit is located. At this time, the local memory can use a memory with a larger memory space, such as a dynamic random access memory, to ensure that the memory space of the local memory is sufficient to store the data that the computing unit needs to process.

[0068] A tensor block refers to a data block consisting of elements in the first input tensor corresponding to threads in the corresponding work group. It can be understood that each work group includes several threads, and each thread corresponds to an element in the first input tensor, that is, a work group actually corresponds to a part of the elements in the first input tensor, and the data block consisting of this part of elements is a tensor block corresponding to the work group. Exemplarily, the first input tensor is composed of feature vectors corresponding to 1024 word elements, and each feature vector corresponds to a thread in the target work group grid, wherein a work group contains 128 threads corresponding to the feature vectors of the first 128 word elements, respectively. At this time, in the first input tensor, a tensor block consisting of feature vectors of 128 word elements is the tensor block corresponding to the work group.

[0069] When the computing unit executes each work group, it actually needs to perform corresponding arithmetic operations or logical operations on the elements corresponding to each thread in the work group through the computing unit, that is, the data required by the computing unit when executing the work group is the tensor block corresponding to the work group in the first input tensor. Based on this, in this embodiment, the first input tensor is divided into tensor blocks corresponding to each work group, and each tensor block is scheduled to the local memory of the computing unit used to execute the corresponding work group; thereafter, when the computing unit executes each corresponding work group, it can directly read the required tensor block from the corresponding local memory to perform the corresponding arithmetic operations or logical operations on the tensor block. Since the link between the computing unit and the corresponding local memory is short and the bandwidth is relatively high, the efficiency of the computing unit in reading the data required to execute the work group when executing the corresponding work group can be improved, and the power consumption of memory access can be reduced. In addition, since the computing unit and the corresponding local memory are directly connected, the computing unit does not need to be routed through routing modules such as crossbar and mesh on-chip networks when accessing data from the corresponding local memory, thereby avoiding the inability to access data in a timely manner due to the maximum bandwidth of the routing module.

[0070] In the embodiment disclosed in steps S201 to S204, the target workgroup grid is first divided into workgroup sets corresponding to each block processing cluster, and each workgroup set is respectively scheduled to the corresponding block processing cluster; then, in each block processing cluster, the workgroups contained in the corresponding workgroup set are respectively mapped to each computing unit of the block processing cluster to determine which computing unit will execute each workgroup subsequently; then, the first input tensor is divided into tensor blocks corresponding to each workgroup, and each tensor block is respectively scheduled to the local memory of the computing unit for executing the workgroup corresponding to the tensor block; in this way, when the computing unit executes each workgroup subsequently, the corresponding tensor block can be directly read from the corresponding local memory , thereby reducing the latency when the computing unit reads the data required for the execution work group, increasing the bandwidth when the computing unit reads the required data, so as to improve the efficiency of the computing unit when reading the required data, and reducing the memory access power consumption of the computing unit; at the same time, since the computing unit can directly access the corresponding local memory without the need for routing addressing through the routing module, this can avoid the situation where the computing unit cannot read the required data in time due to the bandwidth limitation of the routing module when multiple computing units share the routing module. At the same time, since the computing unit does not need to address through routing modules such as crossbar when accessing memory, there is no need to integrate high-cost routing modules such as crossbar in the graphics processing unit, thereby reducing the cost of the graphics processing unit.

[0071] Implementation method of linearizing the work group coordinates to divide the target work group grid into multiple work group sets

[0072] It is understandable that when the target workgroup grid is divided into workgroup sets corresponding to each block processing cluster, if the target workgroup grid is divided into workgroup sets by relatively random or relatively complex logic, and each workgroup is then mapped to the corresponding computing unit, then the order in which the computing unit executes the workgroups in the target workgroup grid will become relatively complex or even relatively random, which makes the mapping relationship between the workgroup and the computing unit relatively complex, which will cause the subsequent scheduling of each workgroup to the computing unit in each block processing cluster for execution to become relatively complex, and it is difficult to accurately schedule each workgroup to the corresponding computing unit. And the process of scheduling the tensor blocks corresponding to each workgroup to the local memory of the corresponding computing unit also becomes relatively complex and prone to errors. Based on this, in one embodiment, step S202 includes:

[0073] Step S301, based on the workgroup coordinates corresponding to each workgroup, convert the target workgroup grid into a one-dimensional workgroup sequence;

[0074] Step S302 : Based on the first serial number corresponding to each working group, the multiple working groups are divided into working group sets corresponding to the block processing clusters respectively.

[0075] In step S301, the work group sequence is a one-dimensional sequence consisting of work groups in the target work group grid obtained by linearizing the target work group grid and converting it into a one-dimensional form. Each work group can be regarded as a point in the work group sequence. It can be understood that in the work group sequence, each work group is provided with a corresponding first serial number to indicate the position of the work group in the work group sequence. The first serial number corresponding to each work group is determined based on the grid point coordinates of the work group in the target work group grid, and the first serial numbers obtained based on different work group coordinates are different.

[0076] In this embodiment, the target work group grid can be linearized into a one-dimensional work group sequence along a preset dimension, and the first sequence number corresponding to each work group in the work group sequence is determined, for example, linearization is performed along the x-dimension of the target work group grid. Figure 3 , the target workgroup grid is a three-dimensional grid with a shape of (8,4,2). At this time, the target workgroup grid needs to be linearized along the x dimension. First, the 8 workgroups with both y and z coordinates of 0 are taken as the first 8 workgroups in the sequence, and then the 8 workgroups after y coordinate +1 are taken as the 9th to 16th workgroups in the sequence, and so on, until the y coordinate increases to the maximum value of 3. At this time, the 32 workgroups with z coordinates of 0 are taken as the first 32 workgroups in the sequence, and then the 8 workgroups with z coordinates of 1 and y coordinates of 0 are taken as the 33rd to 40th workgroups in the sequence, and so on, until the 8 workgroups with z coordinates of 1 and y coordinates of 3 are taken as the last 8 workgroups. That is, the target workgroup grid can be linearized along the x dimension into a workgroup sequence of length 64, and the serial numbers corresponding to each workgroup are 0 to 63. In this way, in the obtained workgroup sequence, the workgroups corresponding to the multiple consecutive first serial numbers have their workgroup coordinates in the target workgroup grid continuous along the direction of the dimension selected during linearization. Specifically, when linearizing the target work group grid, the corresponding first serial number can be calculated based on the work group coordinates corresponding to each work group, and then the work groups in the target work group grid can be mapped to the work group sequence in order from small to large according to the first serial number, so as to linearize the target work group grid into a work group sequence. The specific method of step S301 will be explained later and will not be repeated here.

[0077] In step S302, after the target work group grid is converted into a one-dimensional work group sequence, the work group sequence can be divided into multiple subsequences corresponding to each block processing cluster based on the number of block processing clusters, and the multiple work groups contained in each subsequence constitute a work group set corresponding to a block processing cluster. Specifically, the number of work groups that need to be allocated to a single block processing cluster can be determined based on the total number of work groups in the work group sequence and the number of block processing clusters, and then the multiple work groups in the first sequence in the work group sequence are divided into a work group set corresponding to a block processing cluster according to the first sequence number of each work group. For example, the number of work groups in each work group set is 24. At this time, the corresponding work groups with first sequence numbers of 0 to 23 are divided into the first work group set, and the corresponding work groups with sequence numbers of 24 to 47 are divided into the second work group set, and so on. When the first sequence number is calculated based on the work group coordinates, the multiple consecutive first sequence numbers actually correspond to the multiple work groups that are continuous along a certain dimension of the corresponding work group coordinates. Based on this, after dividing multiple continuous workgroups into a workgroup set, the workgroups in the same workgroup set are actually multiple workgroups that are continuous along a certain dimension in the target workgroup grid. At this time, the workgroup set corresponding to each block processing cluster is composed of multiple workgroups that are continuous along a certain dimension in the target workgroup grid. In this way, the computing units in the block processing cluster can subsequently schedule each workgroup to the corresponding computing unit for execution in a relatively simple order, reducing the difficulty of workgroup scheduling.

[0078] At the same time, by treating multiple workgroups with consecutive first serial numbers as a workgroup set, when the tensor blocks corresponding to each workgroup are subsequently scheduled to the local memory of the computing unit used to execute the workgroup, the tensor blocks stored in the local memory of each computing unit in a single block processing cluster need to be scheduled to correspond to multiple consecutive workgroups, rather than being completely disordered. This can simplify the complexity of scheduling each tensor block to the local memory of the corresponding computing unit and reduce the probability of error.

[0079] In the embodiment disclosed in steps S301 to S302, the target workgroup grid is linearized along a preset dimension, thereby converting the target workgroup grid into a one-dimensional workgroup sequence, and determining the first serial number corresponding to each workgroup in the sequence, so that the workgroups corresponding to the consecutive first serial numbers in the workgroup sequence are multiple workgroups that are continuous along a certain dimension in the target workgroup grid. Then, based on the first serial number, the workgroups in the workgroup sequence are divided into multiple workgroup sets corresponding to each block processing cluster, and the first serial numbers corresponding to the workgroups in each workgroup set are continuous, that is, the workgroups that need to be processed by the computing units in a single block processing cluster are continuous, so that the difficulty of the processing device in managing the execution status of each workgroup can be reduced, and the difficulty of scheduling each tensor block to the local memory in the corresponding computing unit can be reduced.

[0080] In one embodiment, step S301 includes:

[0081] Step S401, obtaining the first dimension step lengths corresponding to the N dimensions of the target work group grid respectively;

[0082] Step S402, determining a first serial number corresponding to each work group based on the work group coordinates of each work group in the target work group grid and the first dimension step length of each dimension;

[0083] Step S403: convert the target work group grid into a work group sequence according to the first serial number corresponding to each work group.

[0084] In step S401, the first dimension step size represents the maximum number of workgroups in the target workgroup grid in the corresponding dimension, that is, the number of workgroups in each dimension in the target workgroup grid. The first dimension step size corresponding to each dimension of the target workgroup grid is preset. For example, the user sets the shape of the target workgroup grid through the statement dim3 gridDim(x1, y1, z1). At this time, the target workgroup grid is a three-dimensional grid with a shape of (x1, y1, z1), and its first dimension step sizes corresponding to the x, y, and z dimensions are x1, y1, and z1, respectively.

[0085] It can be understood that when the value of z1 is 1, the target work group grid can be regarded as a two-dimensional grid; when the values ​​of y1 and z1 are both 1, the target work group grid can be regarded as a one-dimensional grid.

[0086] In step S402, after obtaining the first dimension step lengths corresponding to the target work group grid in each dimension, and each first dimension step length represents the number of work groups included in the target work group grid along the corresponding dimension. Based on this, for each work group, the first serial number corresponding to each work group can be determined based on the corresponding work group coordinates and the first dimension step lengths corresponding to each dimension.

[0087] Specifically, in a possible implementation, the function for calculating the corresponding first serial number based on the workgroup coordinates can be determined according to the dimension of the target workgroup grid, and then the corresponding first serial number is calculated according to the corresponding function and the workgroup coordinates corresponding to each workgroup.

[0088] In one embodiment, when the target workgroup grid is a one-dimensional grid, the workgroup coordinates are also one-dimensional coordinates along the x-dimension, which can be expressed as (3d_id.x). At this time, for each workgroup, the first serial number corresponding to the workgroup can be determined by referring to the formula rank = 3d_id.x, where rank is the first serial number, and 3d_id.x is the coordinate corresponding to the workgroup in the x-dimension. At this time, the value of the workgroup coordinates corresponding to each workgroup is directly determined as the corresponding first serial number.

[0089] In another embodiment, the target workgroup grid is a two-dimensional grid, and the workgroup coordinates of each workgroup can be expressed as (3d_id.x, 3d_id.y). At this time, the target workgroup grid is linearized into a workgroup sequence along the x-dimension. At this time, the first serial number corresponding to each workgroup can be calculated according to the formula rank=3d_id.x+3d_id.y*stride.x, where stride.x represents the first dimension step corresponding to the x-dimension of the target workgroup table, that is, the number of workgroups contained in the target workgroup grid along the x-dimension. It is not difficult to see that the first serial number corresponding to the workgroup with workgroup coordinates (0, y1+1) is the first serial number corresponding to the workgroup with workgroup coordinates (stride.x, y1) plus 1. In this way, the first serial numbers corresponding to workgroups with different y coordinates can be kept continuous, thereby linearizing the target workgroup grid into a continuous one-dimensional workgroup sequence.

[0090] In another embodiment, the target workgroup table is a three-dimensional grid, and the workgroup coordinates of each workgroup can be expressed as (3d_id.x, 3d_id.y, 3d_id.z). At this time, the function for calculating the first serial number of each workgroup in the three-dimensional grid is rank=3d_id.x+3d_id.y*stride.x+3d_id.z*stride.x*stride.y; wherein stride.y represents the first dimension step corresponding to the y dimension of the target workgroup grid, that is, the number of workgroups included along the y dimension in the target workgroup grid. At this time, stride.x*stride.y can actually represent the number of workgroups corresponding to the same z coordinate, that is, the number of workgroups included in an xoy plane in the three-dimensional target workgroup grid. When the target workgroup grid is linearized based on the above formula, the workgroup with the same z coordinate can be regarded as a two-dimensional grid, and the two-dimensional grid can be linearized into a continuous one-dimensional sequence according to the method in the above embodiment. At the same time, it is not difficult to see that the first serial number corresponding to the workgroup with workgroup coordinates (0, 0, z1+1) is the first serial number corresponding to the workgroup with workgroup coordinates (stride.x, stride.y, z1) plus 1. In this way, the serial numbers between the workgroups corresponding to different z coordinates can be kept continuous, thereby linearizing the three-dimensional target workgroup grid into a continuous one-dimensional workgroup sequence.

[0091] In a possible implementation, the first sequence number corresponding to each working group may also be directly calculated based on the following function:

[0092] rank =3d_id.x +3d_id.y*stride.x + 3d_id.z*stride.x * stride.y; At this time, if the target work group grid is a one-dimensional grid, it can be regarded as a three-dimensional grid with a dimension step of 1 along the y dimension and the z dimension, and the y coordinate and z coordinate of each work group are 0. In this way, the work group coordinates of each work group can be converted from the one-dimensional coordinate (x1) to the corresponding three-dimensional coordinate (x1, 0, 0); similarly, when the target work group grid is a two-dimensional grid, it can be regarded as a three-dimensional grid with a dimension step of 1 along the z dimension, and the z coordinate of each work group is 0. The work group coordinates of the work group are converted from the two-dimensional coordinates (x1, y1) to the corresponding three-dimensional coordinates (x1, y1, 0). In this way, when calculating the corresponding first serial number based on the work group coordinates corresponding to the work group, there is no need to consider the dimension of the target work group grid.

[0093] In step S403, after determining the first serial number corresponding to each work group, the work groups corresponding to the first serial numbers are arranged in ascending or descending order, so that the target work group grid can be converted into a one-dimensional work group sequence.

[0094] In the embodiment disclosed in steps S401 to S403, the first serial number corresponding to each workgroup is determined based on the workgroup coordinates corresponding to each workgroup and the first dimensional step length corresponding to each dimension of the target workgroup grid, and each workgroup in the target workgroup table is grouped into a corresponding workgroup sequence in the order of the first serial number. In this way, the target workgroup table can be linearized into a one-dimensional workgroup sequence along a preset dimension, and then, based on the first serial number corresponding to each workgroup in the workgroup sequence, multiple workgroups can be divided into workgroup sets corresponding to each block processing cluster, and the workgroups in each workgroup set are continuous along a certain dimension in the target workgroup grid, so as to reduce the difficulty of managing and scheduling the workgroups of the processing equipment, and scheduling the tensor blocks corresponding to the workgroups to the local memory of the corresponding computing units.

[0095] In one embodiment, step S302 includes:

[0096] Step S501, determining a first number of computing units in a block processing cluster;

[0097] Step S502, based on the first number, divide the workgroups in the workgroup set corresponding to the block processing cluster into multiple ordered workgroup clusters, wherein the number of workgroups in a single workgroup cluster is an integer multiple of the first number, the workgroups in the single workgroup cluster are sequentially mapped to the computing units in the block processing cluster, the multiple workgroups in the single workgroup cluster are executed in parallel in the first number of computing units in the block processing cluster, and the ordered multiple workgroup clusters are executed serially in the first number of computing units in the block processing cluster.

[0098] In step S501, the first number is the number of computing units in a single block processing cluster, and the first number is determined according to the hardware architecture of the processing device. For example, in the processing device, every 4 computing units are integrated on a single silicon chip as a block processing cluster, and in this case, the first number is 4.

[0099] In step S502, a workgroup cluster refers to a cluster composed of the first number of continuous workgroups in a single workgroup set. It can be understood that, referring to the relevant description of step S201, the total number of workgroups in the target workgroup grid is an integer multiple of the number of computing units in the processing device, that is, when the workgroups in the target workgroup grid are executed by the computing units in the processing device, each computing unit needs to execute multiple workgroups in sequence. Based on this, after scheduling each workgroup set to the corresponding block processing cluster, it is necessary to determine the number of workgroups that can be executed in parallel by the block processing cluster based on the number of computing units in the block processing cluster, and thereby divide the workgroup set into multiple workgroup clusters that are executed serially in the block processing cluster, so as to determine the order in which each workgroup contained in the workgroup set is scheduled to the corresponding computing unit for execution.

[0100] In this embodiment, the number of workgroups included in a single workgroup cluster can be determined based on the first number, for example, the first number is determined as the number of workgroups included in a single workgroup cluster. Afterwards, according to the determined number of workgroups in the single workgroup cluster, the multiple consecutive workgroups in the workgroup set are divided into multiple workgroup clusters. For example, based on the first number, the number of workgroups in a single workgroup cluster is determined to be 8. At this time, according to the first sequence number corresponding to each workgroup, the first 8 workgroups in the corresponding workgroup set are determined as the first workgroup cluster, and the 9th to 16th workgroups are determined as the second workgroup cluster, and so on. In this way, the workgroup set corresponding to the block processing cluster can be divided into multiple ordered workgroup clusters. Subsequently, in each time period, the workgroups in the workgroup set can be scheduled to the first number of computing units of the block processing cluster for parallel execution in units of workgroup clusters. The first number of computing units can serially execute the workgroups in the ordered multiple workgroup clusters, that is, after the first number of computing units execute all the workgroups in one workgroup cluster, the workgroups in the next workgroup cluster are executed.

[0101] For example, refer to Figure 4After the workgroups in the target workgroup table are divided into workgroup sets corresponding to each block processing cluster, 8 workgroup sets are obtained, namely workgroup set 0 to workgroup set 7, and each workgroup set includes 24 workgroups. The number of computing units in a single block processing cluster is 4. At this time, the first number can be directly determined as the number of workgroups contained in a single workgroup cluster; then, every 4 consecutive workgroups in the workgroup set are determined as a workgroup cluster, so that the workgroup set corresponding to each block processing cluster is divided into 6 workgroup clusters. For example, the first 4 workgroups in the workgroup set are determined as the first workgroup cluster, and the 5th to 8th workgroups are determined as the second workgroup cluster, and so on. Multiple workgroup clusters can be obtained. At the same time, referring to Figure 4 , the multiple workgroups in each workgroup cluster are also ordered, and are recorded as workgroup 0 to workgroup 3. After that, the four workgroups in the first workgroup cluster are first scheduled to the four computing units in the block processing cluster for parallel execution; when the computing unit finishes executing the workgroups in the first workgroup cluster, the four workgroups in the second workgroup cluster are then scheduled to the four computing units for parallel execution, among which workgroup 0 in the second workgroup cluster is scheduled to the computing unit used to execute workgroup 0 in the first workgroup cluster, and so on.

[0102] In a possible implementation, the number of workgroups in a single workgroup cluster may also be several times the first number. In this case, the workgroups in the single workgroup cluster may be cyclically mapped to the computing units in the block processing cluster in sequence according to the order of the workgroups in the single workgroup cluster for execution. Figure 5 , a single block processing cluster includes 4 computing units, and the serial numbers of these 4 computing units are recorded as 0 to 3 respectively. At this time, every 8 working groups in the working group set corresponding to the block processing cluster are divided into a working group cluster. At this time, the number of working groups in a single working group cluster is twice the first number. At this time, for each working group cluster, working group 0 and working group 4 can be mapped to the computing unit with serial number 0, and working group 1 and working group 5 can be mapped to the computing unit with serial number 1, and so on.

[0103] In the embodiment disclosed in steps S501 to S502, by dividing the workgroups in the workgroup set corresponding to the block processing cluster into a plurality of ordered workgroup clusters based on the number of computing units in a single block processing cluster, the workgroups in the same workgroup cluster can be executed in parallel by the first number of computing units in the block processing cluster, while the workgroups in different block processing clusters can only be executed serially. Based on this, the execution order of scheduling each workgroup in the workgroup set to the first number of computing units in the block processing cluster for execution can be determined, so that the first number of computing units in the block processing cluster can accurately execute the first number of workgroups in the same workgroup cluster in parallel, and execute the workgroups in different workgroup clusters serially. This reduces the difficulty of mapping multiple workgroups in a single workgroup set to the first number of computing units in the corresponding block processing cluster.

[0104] In one embodiment, step S203 includes:

[0105] Step S601, in a work group set, according to the first serial number corresponding to each work group, determine the second serial number of the work group in the corresponding work group set;

[0106] Mapping the workgroups in the corresponding workgroup set to the computing units in the block processing cluster respectively includes:

[0107] Step S602 : for each work group, based on a first remainder obtained by taking a modulus of a first number with respect to a corresponding second sequence number, map the plurality of work groups to computing units in the block processing cluster respectively.

[0108] In step S601, the second serial number refers to the serial number of each work group in the corresponding work group set after being divided into the corresponding work group set. It can be understood that, referring to step S302, the first serial numbers corresponding to the work groups in a single work group set are continuous, and the work groups in a single work group set are actually ordered.

[0109] Specifically, the first serial numbers corresponding to the various workgroups in a single workgroup set may be sorted in ascending order or descending order, and the second serial numbers of the various workgroups in the corresponding workgroup set may be determined according to the sorting result. For example, a workgroup set contains 32 workgroups, and the first serial numbers corresponding to the workgroups in the workgroup set corresponding to a certain block processing cluster are 32 to 63, respectively. At this time, the workgroup with the first serial number 32 is the first workgroup in the workgroup set. Based on this, the second serial number corresponding to the workgroup with the first serial number 32 is 0.

[0110] In another possible implementation, referring to the relevant description of step S201, it can be known that the number of workgroups in each block processing cluster is the same, and referring to the relevant description of step S302, it can be known that the first serial numbers corresponding to the workgroups in each block processing cluster are continuous. At this time, for each workgroup, the serial number in the corresponding workgroup set can be obtained by subtracting several times the number of workgroups in a single workgroup set from the corresponding first serial number. Based on this, in one embodiment, by determining the second number of workgroups in a single workgroup set, and then for each workgroup, taking the corresponding first serial number modulo the second number, the second serial number corresponding to the workgroup can be obtained.

[0111] In step S602, for each block processing cluster, the workgroups in the corresponding workgroup set need to be assigned to the computing units therein for execution, and in order for the first number of computing units to accurately and in parallel execute the workgroups in the workgroup set, it is necessary to determine the mapping relationship between each workgroup and the computing unit. In this embodiment, the first number is modulo the second serial number corresponding to each workgroup, and multiple workgroups with the same first remainder obtained by the modulo are mapped to the same computing unit, and multiple workgroups with different first remainders obtained by the modulo are respectively mapped to different computing units. In this way, the workgroups in the workgroup set can be evenly distributed to the first number of computing units in the block processing cluster.

[0112] Specifically, in one embodiment, the first number of computing units in a single block processing cluster are sequentially assigned corresponding computing unit numbers. For example, each block processing cluster is composed of 4 working groups, and the computing unit numbers corresponding to the 4 working groups can be 0 to 3 in sequence. At this time, based on the second number of each working group, the first number is modulo, and after obtaining the first remainder corresponding to each working group, each working group is mapped to a computing unit with the same computing unit number as the first remainder corresponding to the working group. For example, a working group with a first remainder of 0 is mapped to a computing unit with a computing unit number of 0.

[0113] In this way, each workgroup in a single workgroup set can be evenly mapped to the first number of computing units in the corresponding block processing cluster. The complexity of this mapping logic is relatively low, which can reduce the difficulty and error probability of the processing device allocating local computing resources (i.e., computing units) to execute each workgroup, so that the first number of computing units can accurately and orderly execute each workgroup in the workgroup set in parallel.

[0114] In the embodiment disclosed in step S601 to step S602, by converting the first serial number corresponding to each work group into the second serial number of the work group in the corresponding work group set, and then mapping each work group to the first number of computing units in the block processing cluster based on the second serial number and the remainder of the first number. In this way, each work group can be evenly divided into the first number of computing units in the block processing cluster by a relatively simple mapping logic, so that the first number of computing units can accurately and orderly execute each work group in the work group set in parallel.

[0115] An implementation method of first dividing the workgroups in the target workgroup grid into multiple workgroup clusters, and then dividing the multiple workgroup clusters into multiple workgroup sets

[0116] In the above embodiment, a method is shown for determining the first serial number corresponding to each workgroup based on the workgroup coordinates corresponding to each workgroup, so as to linearize the target workgroup grid into a one-dimensional workgroup sequence along a preset dimension, and then determine multiple workgroups with consecutive first serial numbers as a single workgroup set, thereby dividing the target workgroup grid into workgroup sets corresponding to each block processing cluster. That is, in the above embodiment, multiple workgroups that are continuous along a single dimension of the target workgroup grid are divided into a workgroup set. In another implementation method, multiple workgroups that are continuous along multiple dimensions can also be divided into a workgroup set. Based on this, in one embodiment, step S202 includes:

[0117] Step S701, obtaining preset dimension widths corresponding to N dimensions of a single work group cluster in a target work group grid;

[0118] Step S702, based on the workgroup coordinates corresponding to each workgroup and the preset dimension widths corresponding to the N dimensions, determine the workgroup cluster coordinates corresponding to the workgroup, and determine multiple workgroups corresponding to the same workgroup cluster coordinates as a single workgroup cluster;

[0119] Step S703 : based on the work group cluster coordinates, multiple work group clusters are divided into work group sets corresponding to the block processing clusters respectively.

[0120] In step S701, the preset dimension width represents the coordinate span of the workgroups included in the corresponding dimension of the target workgroup grid by a single workgroup cluster. For example, if the preset dimension width of the y dimension is 2, it means that in the target workgroup grid, each target workgroup cluster includes two workgroups along the y dimension.

[0121] Specifically, the preset dimension width of each dimension can be determined based on the number of computing units in a single block processing cluster, and the product of the preset dimension widths corresponding to each dimension is equal to the number of computing units in a single block processing cluster. Exemplarily, the target workgroup grid is a three-dimensional grid, and the number of computing units in a single block processing cluster is 8. At this time, the preset dimension widths corresponding to the three dimensions can be set to 4, 2, 1 or 2, 2, 2 respectively, so that the number of workgroups in a single workgroup cluster is equal to the number of computing units in a single block processing cluster. In this way, the workgroups in a workgroup cluster can be allocated to the various computing units in a single block processing cluster for parallel execution.

[0122] In step S702, the workgroup cluster coordinates are the coordinates corresponding to each workgroup cluster in the target workgroup grid after the workgroups in the target workgroup grid are divided into multiple workgroup clusters. It can be understood that after the workgroups in the target workgroup grid are divided into different workgroup clusters based on the preset dimension widths corresponding to each dimension, each workgroup cluster actually corresponds to a matrix block of a specified shape in the target workgroup grid. At this time, the target workgroup grid can actually be regarded as a grid composed of workgroup clusters, and each workgroup cluster is a grid point in the grid. For example, referring to Figure 6 , the target work group grid is a three-dimensional grid with a shape of (8, 4, 2), and the preset dimension widths corresponding to the x-dimension, y-dimension, and z-dimension are all 2. At this time, the work group in each three-dimensional area with a shape of (2, 2, 2) in the target work group grid is determined as a work group cluster. At this time, the target work group grid can be regarded as a three-dimensional grid with a shape of (4, 2, 1) with the work group cluster as the grid point. In step S702, the process of determining the work group cluster coordinates corresponding to each work group will be described later and will not be repeated here.

[0123] In step S703, based on the work group cluster coordinates, the process of dividing multiple work group clusters into multiple work group sets can refer to the process of dividing the target work group grid into work group sets corresponding to the block processing clusters based on the work group coordinates in the above step S202, which will not be repeated here.

[0124] In the embodiment disclosed in step S701 to step S703, based on this, by presetting the preset dimension width corresponding to each dimension, the workgroups in the target workgroup grid can be more flexibly organized into multiple workgroup clusters; then, the workgroup set corresponding to each block processing cluster is determined in units of workgroup clusters. In this way, the workgroups in the target workgroup grid can be more flexibly scheduled to the computing units in the processing device for parallel processing while ensuring that the complexity of the scheduling logic when scheduling the workgroups to the computing units for execution is low.

[0125] In one embodiment, in step S702, based on the workgroup coordinates corresponding to each workgroup and the preset dimension widths corresponding to the N dimensions, determining the workgroup cluster coordinates corresponding to the workgroup includes:

[0126] Step S801, for each workgroup, determine the coordinate value of the workgroup in each dimension according to the corresponding workgroup coordinates, and determine the first ratio of the coordinate value of the workgroup in each dimension to the corresponding preset dimension width, round the first ratio of the workgroup in each dimension, and obtain the value of the workgroup cluster coordinates of the workgroup in each dimension to determine the workgroup cluster coordinates corresponding to each workgroup.

[0127] In step S801, the first ratio between the value of the workgroup coordinates of the target workgroup in each dimension and the corresponding preset dimension width is calculated, and the first ratio of each dimension is rounded to determine the coordinate value of the workgroup cluster coordinates corresponding to the workgroup in each dimension. In this way, the workgroup cluster coordinates corresponding to multiple workgroups whose values ​​of a certain dimension in the workgroup coordinates are in the same interval in this dimension can be equal. For example, the preset dimension width of the x dimension is 4, then for workgroups with x coordinates of 0, 1, 2, and 3, their corresponding workgroup cluster coordinates in the x dimension are all equal. Afterwards, by determining multiple workgroups whose corresponding workgroup cluster coordinates have equal values ​​in each dimension as a workgroup cluster, the workgroups in an area of ​​a specified shape in the target workgroup grid can be divided into a workgroup cluster, and the workgroup cluster coordinates corresponding to each workgroup cluster can also be determined.

[0128] Specifically, the following formula can be used to determine the value of the workgroup cluster coordinate of the workgroup in the corresponding dimension based on the coordinate value of the workgroup in each dimension:

[0129] cwg_id.x= ;

[0130] cwg_id.y= ;

[0131] cwg_id.z= ;

[0132] Among them, cwg_id.x, cwg_id.y, cwg_id.z are the values ​​of the workgroup cluster coordinates of the workgroup in the x, y, and z dimensions respectively; grid.x, grid.y, grid.y are the values ​​of the workgroup coordinates of the workgroup in the x, y, and z dimensions respectively; represents the function of floor rounding; , , These are the preset dimension widths corresponding to the x, y, and z dimensions, respectively. , , These are the number of workgroups contained in a single workgroup cluster in the x, y, and z dimensions, respectively. , , These are the coordinate spans of a single workgroup in the x, y, and z dimensions, respectively. , , The value of is usually 1.

[0133] It is understandable that the above example shows the calculation method of the workgroup cluster coordinates when the target workgroup grid is a three-dimensional grid. If the target workgroup grid is a one-dimensional or two-dimensional grid, the calculation can be performed using the corresponding dimension according to the dimension set by the target workgroup grid. For example, when the target workgroup grid is a one-dimensional grid, the workgroup coordinates are also one-dimensional coordinates. At this time, it is only necessary to use the formula cwg_id.x= The corresponding workgroup cluster coordinates can be calculated.

[0134] For example, the target workgroup grid is a three-dimensional grid, including three dimensions of x, y, and z. The preset dimension widths corresponding to each dimension are 2, 2, and 1, respectively. The workgroup coordinates corresponding to a workgroup are (3, 1, 0), and the workgroup cluster coordinates corresponding to the workgroup can be expressed as ( , , ), that is, the workgroup cluster coordinates corresponding to this workgroup are (1,0,0). Similarly, the workgroup cluster coordinates corresponding to the three workgroups with workgroup coordinates (2,1,0), (2,0,0), and (3,0,0) are also (1,0,0). These four workgroups can be subsequently divided into a workgroup cluster.

[0135] In addition, in the above embodiment, the values ​​of the workgroup cluster coordinates of each workgroup in each dimension are calculated by rounding down. In another implementation, the calculation can also be performed by rounding up. However, it should be noted that if the workgroup cluster coordinates are calculated by rounding up, then when generating the target workgroup grid, the starting value of the coordinates of each dimension should be 1 instead of 0, that is, the first workgroup in each dimension should have a coordinate value of 1 corresponding to the dimension, and there cannot be a workgroup with a coordinate value of 0 in a certain dimension, so as to prevent the workgroup with a coordinate value of 0 in a certain dimension from being divided into the same workgroup cluster with other workgroups with coordinate values ​​greater than 0 in the dimension when the preset dimensional width of the dimension is greater than 1.

[0136] In one embodiment, step S703 includes:

[0137] Step S901, determining the second dimension step lengths corresponding to the N dimensions of the target work group grid respectively;

[0138] Step S902, determining the third serial number corresponding to each work group cluster based on the work group cluster coordinates corresponding to each work group cluster and the second dimension step length of each dimension;

[0139] Step S903 : Based on the third sequence number, each work group cluster is mapped to a block processing cluster respectively, and multiple work group clusters corresponding to a single block processing cluster are determined as a work group set corresponding to the block processing cluster.

[0140] In step S901, the second dimension step size represents the number of workgroup clusters contained in the target workgroup grid in the corresponding dimension. Specifically, for the target workgroup grid in each dimension, the corresponding second dimension step size can be determined based on the ratio of the corresponding first dimension step size and the preset dimension width. Exemplarily, the first dimension step size corresponding to a certain dimension is 8, the corresponding preset dimension width is 2, and the corresponding second dimension step size is 8 / 2=4, which indicates that in the target workgroup grid, 8 workgroups are contained along this dimension, and every two workgroups are divided into one workgroup cluster. At this time, the workgroups can be divided into 4 workgroup clusters along this dimension.

[0141] In step S902, the corresponding third serial number may be determined based on the work group cluster coordinates according to the following formula:

[0142] cwg_flat_id=cwg_id.x;

[0143] cwg_flat_id=cwg_id.x+cwg_id.y*stride_in_cwg.x;

[0144] cwg_flat_id=cwg_id.x+cwg_id.y*stride_in_cwg.x+cwg_id.z*stride_in_cwg.x*stride_in_cwg.y;

[0145] Among them, cwg_flat_id represents the third serial number corresponding to the work group cluster; cwg_id.x, cwg_id.y, cwg_id.z are the values ​​of the x, y, and z dimensions of the work group cluster coordinates respectively, and stride_in_cwg.x and stride_in_cwg.y are the second dimension steps corresponding to the x and y dimensions respectively. The detailed process of step S902 can refer to the above step S402, which will not be repeated here.

[0146] In step S903, after determining the third serial number corresponding to each workgroup cluster, each workgroup cluster can be mapped to each block processing cluster in sequence according to the third serial number, so that multiple workgroup clusters mapped to the same block processing cluster constitute the workgroup set corresponding to the block processing cluster. Specifically, referring to the above step S302, based on the number of workgroup clusters and the number of block processing clusters, the number of workgroup clusters that need to be mapped to a single block processing cluster can be determined, and the workgroup clusters corresponding to multiple consecutive third serial numbers can be mapped to the same block processing cluster as the workgroup set corresponding to the block processing cluster. In this way, when scheduling the workgroup to execute in the computing unit, it is only necessary to schedule each workgroup cluster first.

[0147] In the embodiment disclosed in steps S901 to S903, after dividing multiple workgroups in the target workgroup grid into multiple workgroup clusters according to the preset dimensional width corresponding to each dimension, the second dimensional step length used to characterize the number of workgroup clusters included in each dimension along the target workgroup grid is determined, and the target workgroup grid is linearized based on the workgroup cluster coordinates corresponding to each workgroup cluster and the second dimensional step length corresponding to each dimension, so as to determine the third serial number corresponding to each workgroup cluster; thereafter, each workgroup cluster is respectively mapped to each block processing cluster based on the third serial number, and the multiple workgroup clusters mapped to a single block processing cluster are determined as the workgroup set corresponding to the block processing cluster.

[0148] In one embodiment, step S903 includes:

[0149] Step S1001, taking a modulus of a second number based on a third sequence number corresponding to each working group cluster to obtain a second remainder corresponding to the working group cluster;

[0150] Step S1002, based on the second remainder, the corresponding workgroup clusters are respectively mapped to the block processing clusters in the processing device, wherein the corresponding workgroup clusters with equal second remainders are mapped to the same block processing cluster, and the corresponding workgroup clusters with unequal second remainders are mapped to different block processing clusters in the processing device.

[0151] The detailed description of step S1001 can refer to the above step S601 and will not be repeated here.

[0152] In step S1002, specifically, each block processing cluster may be respectively provided with a corresponding cluster number. After calculating the second remainder corresponding to each work group cluster, the work group cluster may be mapped to a block processing cluster having a cluster number equal to the second remainder. In this way, each work group cluster may be cyclically mapped to each block processing cluster. For example, referring to Figure 7, every 4 work groups in the target work group grid are regarded as a work group cluster, so that the work groups in the target work group grid are divided into 32 work group clusters. The third serial numbers corresponding to these 32 work group clusters are 0 to 31 respectively. The processing device includes 8 block processing clusters. The serial numbers corresponding to the 8 block processing clusters are 0 to 7 respectively. At this time, the second remainders corresponding to the 4 work group clusters with third serial numbers of 0, 8, 16, and 24 are all 0. Based on this, the 4 work group clusters with third serial numbers of 0, 8, 16, and 24 are mapped to the block processing cluster with serial number 0.

[0153] It should be noted that, different from the embodiment disclosed in the above steps S301 to S302, in this embodiment, multiple workgroups are organized into multiple workgroup clusters, and then mapped to block processing clusters in units of workgroup clusters. Each workgroup cluster itself includes multiple workgroups that are continuous in various dimensions. Based on this, in this embodiment, each workgroup cluster is cyclically mapped to each block processing cluster based on the third sequence number, and then multiple computing units in the block processing cluster are made to execute the workgroups in a single workgroup cluster in parallel. It can also ensure that the multiple workgroups executed in parallel by the computing units in a single block processing cluster at the same time are continuous in a certain dimension, and the complexity of scheduling the workgroups in the target workgroup grid to each computing unit for execution is still low.

[0154] Description of the apparatus and device of the present disclosure

[0155] Reference Figure 8 , Figure 8 1 is a structural diagram of a data scheduling device 800 proposed in the present disclosure. The data scheduling device 800 is integrated in a processing device. The processing device includes at least one block processing cluster. The block processing cluster includes multiple computing units. The computing units are provided with corresponding local memories. The data scheduling device 800 includes:

[0156] An acquisition unit 810 is used to acquire a first input tensor and a target workgroup grid corresponding to the first input tensor, where the target workgroup grid includes a plurality of workgroups, where the workgroup includes a plurality of threads, where the threads correspond to elements in the first input tensor, where the threads are used to enable a computing unit to process elements corresponding to the threads, where each workgroup is provided with corresponding workgroup coordinates, where the workgroup coordinates are coordinates of the corresponding workgroup in the target workgroup grid;

[0157] A first division unit 820 is used to divide the target work group grid into work group sets corresponding to the block processing clusters based on the work group coordinates corresponding to each work group, and schedule the work groups in each work group set to the corresponding block processing cluster;

[0158] A first mapping unit 830 is used to map the workgroups in the corresponding workgroup set to the computing units in the block processing cluster in the block processing cluster;

[0159] The data scheduling unit 840 is used to divide the first input tensor into tensor blocks corresponding to each work group, and store each tensor block in the local memory of the computing unit used to execute the corresponding work group, so that the computing unit reads the tensor block corresponding to the work group from the corresponding local memory when executing the corresponding work group.

[0160] The data scheduling device disclosed in the present invention is used to execute the data scheduling method as in the above embodiment. Its specific processing process is the same as the data scheduling method in the above embodiment, and will not be repeated here.

[0161] The present disclosure also provides an electronic device 900, including:

[0162] at least one processor, and

[0163] a memory communicatively connected to at least one processor; wherein,

[0164] The memory stores instructions, and the instructions are executed by at least one processor so that when the at least one processor executes the instructions, a method as described in any one of the above embodiments of the present application is implemented.

[0165] Combine the following Fig. 9 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 910 , a memory 920 , an input / output interface 930 , a communication interface 940 and a bus 950 .

[0166] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0167] The memory 920 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the data scheduling method of the embodiments of the present disclosure;

[0168] Input / output interface 930, used to implement information input and output;

[0169] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0170] bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0171] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0172] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the data scheduling method of the above embodiment, which will not be repeated here.

[0173] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0174] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0175] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0176] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0179] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0180] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A data scheduling method, characterized in that: Applied to a processing device, the processing device includes at least one block processing cluster, the block processing cluster includes a plurality of computing units, the computing units are provided with corresponding local memories, the method includes: Obtain a first input tensor and a target workgroup grid corresponding to the first input tensor, wherein the target workgroup grid includes a plurality of workgroups, the workgroup includes a plurality of threads, the threads correspond to elements in the first input tensor, the threads are used to enable the computing unit to process the elements corresponding to the threads, each of the workgroups is provided with a corresponding workgroup coordinate, and the workgroup coordinate is a coordinate of the corresponding workgroup in the target workgroup grid; Based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters, and the workgroups in each of the workgroup sets are scheduled to the corresponding block processing clusters; In the block processing cluster, mapping the corresponding workgroups in the workgroup set to computing units in the block processing cluster respectively; The first input tensor is divided into tensor blocks corresponding to each work group, and each tensor block is stored in the local memory of the computing unit used to execute the corresponding work group, so that the computing unit reads the tensor block corresponding to the work group from the corresponding local memory when executing the corresponding work group.

2. The data scheduling method according to claim 1, characterized in that: The step of dividing the target workgroup grid into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates corresponding to each of the workgroups includes: Based on the workgroup coordinates corresponding to each of the workgroups, the target workgroup grid is converted into a one-dimensional workgroup sequence, wherein each of the workgroups is ordered in the workgroup sequence, and the first sequence number of each of the workgroups in the workgroup sequence is determined according to the workgroup coordinates corresponding to the workgroup; Based on the first serial numbers corresponding to the respective working groups, the plurality of working groups are divided into working group sets corresponding to the block processing clusters respectively, wherein the first serial numbers corresponding to the working groups in the respective working group sets are continuous.

3. The data scheduling method according to claim 2, characterized in that: The dimension of the target working group grid is N, and the target working group grid is an N-dimensional grid. The target working group grid is converted into a one-dimensional working group sequence based on the working group coordinates corresponding to each working group, including: Obtaining first dimension step lengths corresponding to N dimensions of the target working group grid, respectively, where the first dimension step lengths represent the maximum number of working groups of the target working group grid in the corresponding dimension; Determine a first serial number corresponding to each of the working groups based on the working group coordinates of each of the working groups in the target working group grid and the first dimension step length of each of the dimensions; According to the first serial numbers corresponding to the respective working groups, the target working group grid is converted into the working group sequence.

4. The data scheduling method according to claim 2, characterized in that: After the plurality of workgroups are divided into a plurality of workgroup sets corresponding to the block processing clusters based on the first sequence numbers corresponding to the workgroups, the method further includes: determining a first number of computing units in the block processing cluster; Based on the first number, the workgroups in the workgroup set corresponding to the block processing cluster are divided into a plurality of ordered workgroup clusters, wherein the number of workgroups in a single workgroup cluster is an integer multiple of the first number, the workgroups in a single workgroup cluster are sequentially mapped to the computing units in the block processing cluster, the plurality of workgroups in a single workgroup cluster are executed in parallel in the first number of computing units in the block processing cluster, and the ordered plurality of workgroup clusters are executed serially in the first number of computing units in the block processing cluster.

5. The data scheduling method according to claim 4, characterized in that: After the plurality of work groups are divided into work group sets corresponding to the block processing clusters based on the first sequence numbers corresponding to the work groups, the method further includes: In the work group set, according to the first serial number corresponding to each work group, determine the second serial number of the work group in the corresponding work group set; The mapping of the corresponding workgroups in the workgroup set to computing units in the block processing cluster respectively includes: For each of the working groups, based on a first remainder obtained by taking the first number modulo the corresponding second sequence number, the plurality of working groups are respectively mapped to the computing units in the block processing cluster.

6. The data scheduling method according to claim 1, characterized in that: The target workgroup grid is an N-dimensional grid, and the target workgroup grid is divided into workgroup sets corresponding to the block processing clusters based on the workgroup coordinates corresponding to each of the workgroups, including: Obtaining preset dimension widths corresponding to N dimensions of a single workgroup cluster in the target workgroup grid, wherein the preset dimension width represents a coordinate span of the workgroups included in the corresponding dimension of a single workgroup cluster in the target workgroup grid; Based on the workgroup coordinates corresponding to each of the workgroups and the preset dimension widths corresponding to the N dimensions, determine the workgroup cluster coordinates corresponding to the workgroup, and determine the multiple workgroups corresponding to the same workgroup cluster coordinates as a single workgroup cluster; Based on the work group cluster coordinates, the plurality of work group clusters are divided into the work group sets respectively corresponding to the block processing clusters.

7. The data scheduling method according to claim 6, characterized in that: The determining the workgroup cluster coordinates corresponding to the workgroup based on the workgroup coordinates corresponding to each of the workgroups and the preset dimension widths corresponding to the N dimensions includes: For each of the working groups, the coordinate values ​​of the working group in each dimension are determined according to the corresponding working group coordinates, and a first ratio of the coordinate values ​​of the working group in each dimension to the corresponding preset dimension width is determined. The first ratio of the working group in each dimension is rounded to obtain the values ​​of the working group cluster coordinates of the working group in each dimension, so as to determine the working group cluster coordinates corresponding to each of the working groups.

8. The data scheduling method according to claim 6, characterized in that: The dividing the plurality of workgroup clusters into the workgroup sets respectively corresponding to the block processing clusters based on the workgroup cluster coordinates includes: Determine the second dimension step lengths corresponding to the N dimensions of the target work group grid, respectively, where the second dimension step lengths represent the number of the work group clusters included in the target work group grid in the corresponding dimension; Determine a third serial number corresponding to each of the working group clusters based on the working group cluster coordinates corresponding to each of the working group clusters and the second dimension step length of each dimension; Based on the third sequence number, each of the workgroup clusters is mapped to the block processing cluster, and the multiple workgroup clusters corresponding to a single block processing cluster are determined as a workgroup set corresponding to the block processing cluster.

9. The data scheduling method according to claim 8, characterized in that: The number of the block processing clusters is a second number, and mapping each of the workgroup clusters to the block processing clusters based on the third sequence number includes: Taking a modulus of the second number based on the third sequence number corresponding to each of the working group clusters, to obtain a second remainder corresponding to the working group cluster; Based on the second remainder, the corresponding workgroup clusters are respectively mapped to the block processing clusters in the processing device, wherein the corresponding workgroup clusters with equal second remainders are mapped to the same block processing cluster, and the corresponding workgroup clusters with unequal second remainders are mapped to different block processing clusters in the processing device.

10. A data scheduling device, characterized in that: The data scheduling device is integrated in a processing device, the processing device includes at least one block processing cluster, the block processing cluster includes a plurality of computing units, the computing units are provided with corresponding local memories, and the data scheduling device includes: an acquisition unit, configured to acquire a first input tensor and a target workgroup grid corresponding to the first input tensor, wherein the target workgroup grid includes a plurality of workgroups, the workgroup includes a plurality of threads, the threads correspond to elements in the first input tensor, the threads are configured to enable the computing unit to process the elements corresponding to the threads, each of the workgroups is provided with a corresponding workgroup coordinate, and the workgroup coordinate is a coordinate of the corresponding workgroup in the target workgroup grid; A first division unit, configured to divide the target work group grid into work group sets corresponding to the block processing clusters respectively based on work group coordinates corresponding to each of the work groups, and schedule the work groups in each of the work group sets to the corresponding block processing clusters; A first mapping unit, configured to map, in the block processing cluster, the corresponding workgroups in the workgroup set to computing units in the block processing cluster respectively; A data scheduling unit is used to divide the first input tensor into tensor blocks corresponding to each work group, and store each tensor block in the local memory of the computing unit used to execute the corresponding work group, so that the computing unit reads the tensor block corresponding to the work group from the corresponding local memory when executing the corresponding work group.

11. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the data scheduling method according to any one of claims 1 to 9 is realized.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data scheduling method according to any one of claims 1 to 9.

Images