Hardware computing resource allocation method, hardware computing resource, and electronic device

Abstract

A method for allocating hardware computing resources, hardware computing resources, and electronic equipment. The method for allocating hardware computing resources includes: obtaining the data shape of an operator's input tensor for a model computing task; obtaining hardware parameters of the hardware computing resources; determining a resource allocation scheme for the operator based on the hardware parameters and the data shape, wherein the resource allocation scheme indicates a way of splitting the input tensor in multiple dimensions of the input tensor, and a way of allocating the input tensor to hardware computing resources for processing based on the splitting method; splitting the input tensor based on the resource allocation scheme, and sending each of the split tensors to a corresponding computing resource to perform the model computing task; wherein elements belonging to the same row in the input tensor have data dependency when performing model computing, and in the resource allocation scheme, elements in the same row perform model computing tasks in the same computing unit.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a hardware computing resource allocation method, hardware computing resources, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] A tensor is a multilinear mapping defined on the Cartesian product of some vector spaces and some dual spaces. For example, a scalar can be regarded as a 0-dimensional tensor, a vector can be regarded as a one-dimensional tensor, and a matrix can be regarded as a two-dimensional tensor. Tensor operations are widely used in processors such as parallel processors. In current hardware devices, data processing integrated circuits such as central processing units (CPUs), graphics processing units (GPUs), and general-purpose computing on graphics processing units (GPGPUs) can execute programs to complete various tensor-related computing tasks. With the increase in computing needs, hardware devices have higher and higher performance requirements for operators. Better performance means faster completion of calculations and reduced computing overhead. Evenly mapping computing tasks to hardware devices and maximizing the parallel execution efficiency of hardware devices have become the key to performance improvement. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a method for allocating hardware computing resources, wherein the hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, each execution unit is configured to run one or more thread warps, A, B, and C are all positive integers greater than 1, and the method for allocating hardware computing resources includes: obtaining a data shape of an input tensor of an operator for a model computing task, wherein the input tensor is a three-dimensional tensor including an N dimension, an H dimension, and a W dimension, and the data shape of the input tensor is [n, h, w], wherein the N dimension represents the number of batches n of the input tensor, the H dimension represents the height h of each batch, and the W dimension represents the width w of each batch, and n, h, and w are positive integers; obtaining hardware parameters of the hardware computing resources, wherein the hardware parameters include the The number A of programmable multiprocessors and the number L of execution units included in each programmable multiprocessor, L being a positive integer and equal to B×C; adaptively determining a resource allocation scheme for the operator according to the hardware parameters and the data shape, wherein the resource allocation scheme indicates a way of splitting the input tensor in multiple dimensions of the input tensor, and an allocation method of allocating the input tensor to the hardware computing resources for processing according to the splitting method; splitting the input tensor according to the resource allocation scheme, and sending the split tensors to corresponding computing resources to perform the model computing task; wherein elements belonging to the same row in the input tensor have data dependency when performing the model computing task, and in the resource allocation scheme, the elements in the same row perform the model computing task in the same computing unit.

[0004] For example, in the hardware computing resource allocation method provided in at least one embodiment of the present disclosure, the resource allocation scheme of the operator is adaptively determined according to the hardware parameters and the data shape, including: according to the size relationship between h and L, and the size relationship between n and A×L, the resource allocation scheme of the operator is adaptively determined.

[0005] For example, in the hardware computing resource allocation method provided by at least one embodiment of the present disclosure, the resource allocation scheme of the operator is adaptively determined according to the size relationship between h and L, and the size relationship between n and A×L, including: in response to h being less than L and n being greater than or equal to A×L, determining that the operator adopts a first resource allocation scheme, wherein the first resource allocation scheme indicates that the input tensor is first split in the N dimensions, and the multiple tensors split in the N dimensions are allocated to A×L execution units for separate execution, and each execution unit traverses the allocated tensors row by row in the order of row direction first and column direction later.

[0006] For example, in the hardware computing resource allocation method provided in at least one embodiment of the present disclosure, the first resource allocation scheme indicates that the input tensor is evenly divided into A×L first tensors in the N dimensions, and the A×L first tensors are allocated to the A×L execution units for separate execution, with each execution unit processing 1 first tensor.

[0007] For example, in the hardware computing resource allocation method provided by at least one embodiment of the present disclosure, the input tensor is split according to the resource allocation scheme, and the split tensors are sent to corresponding computing resources to execute the model computing task, including: splitting the input tensor into A×L first tensors in the N dimensions, and sending the A×L first tensors to the A×L execution units respectively, wherein each first tensor includes n / (A×L) batches; setting each execution unit to traverse the received first tensor row by row in the order of row direction first and column direction later, taking the thread warp as a unit, to execute the model computing task.

[0008] For example, in the hardware computing resource allocation method provided in at least one embodiment of the present disclosure, the resource allocation scheme of the operator is adaptively determined according to the size relationship between h and L, and the size relationship between n and A×L, including: in response to h being less than L and n being less than A×L, determining that the operator adopts a second resource allocation scheme; wherein the second resource allocation scheme indicates that each batch in the input tensor is split into m parts in the W dimension and allocated to m execution units for execution respectively, wherein m= min ( A×L / n ,C), Represents the rounding function, min( ) represents a minimum function, and the m execution units belong to the same computing unit.

[0009] For example, in the hardware computing resource allocation method provided in at least one embodiment of the present disclosure, model computing tasks of the same batch in the input tensor are executed in the same computing unit.

[0010] For example, in the hardware computing resource allocation method provided by at least one embodiment of the present disclosure, the input tensor is split according to the resource allocation scheme, and the split tensors are sent to corresponding computing resources to execute the model computing task, including: splitting the input tensor into multiple second tensors in the N dimension, and sending the multiple second tensors to multiple computing units, wherein each computing unit receives a second tensor, and each second tensor includes at least 1 batch; splitting each batch of the second tensor into m third tensors in the W dimension, and sending the multiple third tensors obtained by splitting the second tensor to multiple execution units in the computing unit that receives the second tensor, wherein each third tensor includes w / m rows and h columns of elements; setting each execution unit to traverse the received third tensor row by row in the order of row direction first and column direction later, taking the thread bundle as a unit, to execute the model computing task.

[0011] For example, in the hardware computing resource allocation method provided by at least one embodiment of the present disclosure, the input tensor is split into multiple second tensors in the N dimension, including: in response to n being greater than or equal to A×B, the input tensor is split into A×B second tensors in the N dimension, wherein each second tensor includes n / (A×B) batches; in response to n being less than A×B, the input tensor is split into n second tensors in the N dimension, wherein each second tensor includes 1 batch.

[0012] For example, in the hardware computing resource allocation method provided by at least one embodiment of the present disclosure, the resource allocation scheme of the operator is adaptively determined according to the size relationship between h and L, and the size relationship between n and A×L, including: in response to h being greater than or equal to L, determining that the operator adopts a third resource allocation scheme, wherein the third resource allocation scheme indicates that the input tensor is first split in the N dimensions, and the multiple tensors after the split in the N dimensions are allocated to the A programmable multiprocessors for execution, and then the tensors allocated to each programmable processor are split in the H dimension, and the multiple tensors after the split in the H dimension are allocated to B×C execution units in the programmable processor for execution, and each execution unit traverses the tensors allocated by the execution unit row by row in the order of row direction first and column direction later.

[0013] For example, in the hardware computing resource allocation method provided by at least one embodiment of the present disclosure, the input tensor is split according to the resource allocation scheme, and the split tensors are sent to corresponding computing resources to execute the model computing task, including: splitting the input tensor into A fourth tensors in the N dimension, and sending the A fourth tensors to the A programmable multiprocessors respectively, wherein each fourth tensor includes n / A batches; splitting the fourth tensor received by each programmable multiprocessor into multiple fifth tensors in the H dimension, and sending the multiple fifth tensors to the corresponding execution units, wherein each fifth tensor includes a row of elements or multiple consecutive rows of elements in the fourth tensor; setting each execution unit to traverse the received fifth tensor row by row in the order of row direction first and column direction later, taking the thread warp as a unit, to execute the model computing task.

[0014] For example, in the hardware computing resource allocation method provided by at least one embodiment of the present disclosure, elements belonging to the same row have data dependency representation when performing the model calculation task, and for one element among the elements belonging to the same row, the inference result corresponding to the one element is related to at least two elements in the same row, and each computing unit includes a shared memory, and the C execution units in the computing unit exchange data through the shared memory.

[0015] At least one embodiment of the present disclosure provides a hardware computing resource, including A programmable multiprocessors, each programmable multiprocessor including B computing units, each computing unit including C execution units, each execution unit configured to run one or more thread warps, A, B, and C are all positive integers greater than 1, and the hardware computing resource also includes: a first acquisition module, configured to acquire the data shape of an input tensor of an operator for a model computing task, wherein the input tensor is a three-dimensional tensor, including an N dimension, an H dimension, and a W dimension, and the data shape of the input tensor is [n, h, w], the N dimension represents the number of batches n of the input tensor, the H dimension represents the height h of each batch, and the W dimension represents the width w of each batch, and n, h, and w are positive integers; a second acquisition module, configured to acquire hardware parameters of the hardware computing resource, wherein the hardware parameters include the A number A, and the number L of execution units included in each programmable multiprocessor, L is a positive integer and equal to B×C; a scheme determination module is configured to adaptively determine the resource allocation scheme of the operator according to the hardware parameters and the data shape, wherein the resource allocation scheme indicates the partitioning method of the input tensor in multiple dimensions of the input tensor, and the allocation method of allocating the input tensor to the hardware computing resources for processing according to the partitioning method; a resource allocation module is configured to partition the input tensor according to the resource allocation scheme, and send the partitioned tensor to the corresponding computing resources to perform the model computing task; wherein the elements belonging to the same row in the input tensor have data dependency when performing the model computing task, and in the resource allocation scheme, the elements in the same row perform the model computing task in the same computing unit.

[0016] For example, in the hardware computing resources provided in at least one embodiment of the present disclosure, the hardware computing resources are a graphics processor or a general-purpose graphics processor.

[0017] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transitorily stores computer-executable instructions; and a processor, which is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the hardware computing resource allocation method according to any embodiment of the present disclosure.

[0018] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the hardware computing resource allocation method according to any embodiment of the present disclosure or the hardware computing resource allocation method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0020] Figure 1 A schematic diagram of the structure of a general-purpose graphics processing unit (GPGPU);

[0021] Figure 2 It is a schematic diagram of the structure of tensor data;

[0022] Figure 3 A schematic flow chart of a method for allocating hardware computing resources provided in at least one embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a third resource mapping solution provided for at least one embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a first resource mapping solution provided for at least one embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of a second resource mapping solution provided for at least one embodiment of the present disclosure;

[0026] Figure 7 A schematic structural diagram of hardware computing resources provided for at least one embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure;

[0028] Fig. 9 A schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0031] Figure 1 A schematic structural diagram of a general-purpose graphics processing unit (GPGPU) is shown.

[0032] like Figure 1 As shown, the general-purpose graphics processor is actually an array of programmable multiprocessors. For example, the programmable multiprocessor can be a streaming processor cluster (SPC), for example, including Figure 1 The stream processor clusters 1, ..., and stream processor cluster M are shown, where M is a positive integer greater than 1. In a general-purpose graphics processor, one stream processor cluster processes one computing task, or multiple stream processor clusters process one computing task. Multiple stream processor clusters share data through a global cache or a global memory.

[0033] like Figure 1 As shown, taking stream processor cluster 1 as an example, one stream processor cluster includes multiple computing units, such as Figure 1 In the figure, there are computing units 1, 2, ..., N, where N is a positive integer. Each computing unit (CU) is used to perform arithmetic and logical operations, such as accumulation, reduction, conventional addition, subtraction, multiplication, and division. A computing unit includes multiple cores (also called computing cores or computing cores), each of which includes an arithmetic logic unit (ALU), a floating-point computing unit, etc. The computing core is used to perform specific computing tasks. In addition, the computing unit also includes registers (e.g. Figure 1 The register file in the computing unit and the shared memory are used to hierarchically store source data and destination data related to computing tasks. The shared memory in a computing unit is used to share data between the cores of the computing unit.

[0034] In parallel computing, computing tasks are generally performed by multiple threads. These threads are divided into multiple thread blocks before being executed in a general-purpose graphics processor (or parallel computing processor), and then distributed through the thread block distribution module ( Figure 1 The CPU (not shown) distributes multiple thread blocks to each computing unit. All threads in a thread block must be assigned to the same computing unit for execution. At the same time, the thread block is split into minimum execution thread bundles (or simply thread bundles, warps), each of which contains a fixed number (or less than this fixed number) of threads, for example, 32 threads. Thread bundles run on execution units (EUs), and threads run on thread hardware, for example, 1 thread bundle runs on 1 execution unit, and 1 thread runs on 1 thread hardware. Multiple thread blocks can be executed in the same computing unit, or in different computing units.

[0035] In each compute unit, the warp scheduling / dispatching module ( Figure 1 The thread warps are scheduled and allocated (not shown) so that multiple computing cores of the computing unit can run the thread warps. Depending on the number of computing cores in the computing unit, multiple thread warps in a thread block can be executed simultaneously or in time-sharing. Multiple threads in each thread warp will execute the same instruction. The memory execution instruction will be emitted to the shared memory in the computing unit or further emitted to the intermediate level cache or global cache or global memory for read and write operations, etc.

[0036] A graphics processor or a general-purpose graphics processor can be used for reasoning of a network model, for example, the network model can be a large language model, a visual language model, a multimodal model, etc.

[0037] The tensor data used in the network model inference process is a three-dimensional tensor, including N dimension, H dimension and W dimension. The shape of the tensor data is specifically expressed as [n,h,w], where n, h, and w are positive integers. Here, the N dimension represents the batch size n of the tensor data, that is, the number of data samples captured in one training. The H dimension represents the height h of each batch, and the W dimension represents the width w of each batch.

[0038] For example, if the tensor data is an image, the specific value n of the N dimension is 1, the value h of the H dimension represents the height of the image, and the value w of the W dimension represents the width of the image.

[0039] Figure 2 is a schematic diagram of the structure of tensor data. Figure 2 As shown, there are n batches of data in the network model reasoning process, such as Figure 2In the figure, there are multiple large rectangular boxes batch0, batch1, ... batch(n-1). A batch consists of h×w elements. Figure 2 A small grid in a batch represents an element, for example, an element can have a data width of 8 bits, 16 bits, etc. Here, n, h, and w are positive integers.

[0040] The data in the network model reasoning process usually uses a unified resource mapping logic for resource segmentation to distribute the data reasoning tasks to different levels of computing resources for execution.

[0041] For example, currently, for a three-dimensional tensor, a programmable multiprocessor is usually used to split the tensor into N dimensions, an execution unit is used to split the tensor into H dimensions, and each execution unit traverses the W dimension.

[0042] However, the three-dimensional tensor sizes involved in the large model reasoning process vary, and a unified resource mapping logic will cause some computing resources to be idle under certain special tensor sizes, thereby reducing overall utilization. For example, in the above scenario, if h is less than the total number of execution units in a programmable multiprocessor, some execution units will inevitably be idle, affecting global performance.

[0043] At least one embodiment of the present disclosure provides a method for allocating hardware computing resources, hardware computing resources, electronic devices, and non-transitory computer-readable storage media. The method for allocating hardware computing resources includes: obtaining the data shape of the input tensor of the operator for the model computing task, wherein the input tensor is a three-dimensional tensor, including N dimensions, H dimensions, and W dimensions, and the data shape of the input tensor is [n, h, w], the N dimension represents the number of batches n of the input tensor, the H dimension represents the height h of each batch, and the W dimension represents the width w of each batch, and n, h, and w are positive integers; obtaining hardware parameters of the hardware computing resources, wherein the hardware parameters include the number A of programmable multiprocessors and the number L of execution units included in each programmable multiprocessor, L is a positive integer and equal to B×C; according to the hardware parameters and the data shape, the resource allocation scheme of the operator is determined, wherein the resource allocation scheme indicates the way in which the input tensor is split in multiple dimensions of the input tensor, and the way in which the input tensor is allocated to the hardware computing resources for processing according to the splitting method; the input tensor is split according to the resource allocation scheme, and the split tensor is sent to the corresponding computing resources to perform the model computing task; wherein, the elements belonging to the same row in the input tensor have data dependency when performing model computing, and in the resource allocation scheme, the elements in the same row perform the model computing task in the same computing unit.

[0044] The hardware computing resource allocation method can adaptively map computing resources, automatically determine which resource mapping logic the operator uses based on the hardware parameters of the computing resources and the data shape of the operator's input tensor, and each operator can adaptively determine the most appropriate resource allocation scheme, so that computing resource allocation is more reasonable, and some computing resources are avoided from being idle as much as possible, and computing resource waste is reduced as much as possible, thereby improving system performance. In addition, when elements belonging to the same row in the input tensor have data dependencies when performing model computing tasks, the resource allocation scheme obtained by the resource allocation method provided in the embodiment of the present disclosure sets the elements of the same row to perform model computing tasks in the same computing unit when splitting, so as to make full use of computing resources, avoid resource waste, and minimize the additional memory access overhead caused by data dependencies, thereby improving system performance.

[0045] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0046] Figure 3 A schematic flowchart of a method for allocating hardware computing resources provided in at least one embodiment of the present disclosure.

[0047] like Figure 3 As shown, the hardware computing resource allocation method provided by at least one embodiment of the present disclosure includes at least steps S10-S40.

[0048] For example, the hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, each execution unit is configured to run one or more thread warps, and A, B, and C are positive integers greater than 1.

[0049] For example, the hardware computing resource may be a graphics processor or a general-purpose graphics processor. For example, the programmable multiprocessor, computing unit, and execution unit in the hardware computing resource constitute a multi-level computing resource. For a detailed description of the programmable multiprocessor, computing unit, and execution unit and their relationship, please refer to the aforementioned Figure 1 The relevant content will not be repeated here.

[0050] like Figure 3 As shown, in step S10, the data shape of the input tensor of the operator for the model calculation task is obtained.

[0051] For example, the data format of the input tensor is [N, H, W], where the N dimension represents the number of batches of the input tensor, the H dimension represents the height of each batch, and the W dimension represents the width of each batch. The input tensor is represented as [n, h, w], where n, h, and w are positive integers. Here, n represents the specific number of batches of the input tensor, h represents the specific height of each batch in the input tensor, and w represents the specific width of each batch in the input tensor. For example, the input tensor can include n batches, each of which includes h rows and w columns of elements.

[0052] For example, model computing tasks include model reasoning tasks, etc.

[0053] In step S20, hardware parameters of hardware computing resources are obtained.

[0054] For example, the hardware parameters include the number A of programmable multiprocessors and the number L of execution units included in each programmable multiprocessor, where L is a positive integer and equal to B×C.

[0055] In step S30, a resource allocation scheme for operators is determined according to hardware parameters and data shape.

[0056] For example, the resource allocation scheme indicates how the input tensor is split along multiple dimensions of the input tensor, and how the input tensor is allocated to hardware computing resources for processing according to the splitting scheme.

[0057] In step S40, the input tensor is segmented according to the resource allocation scheme, and each of the segmented tensors is sent to a corresponding computing resource to execute the model computing task.

[0058] For example, the corresponding computing resources here may include a programmable multiprocessor, a computing unit, an execution unit, etc.

[0059] For example, elements belonging to the same row in the input tensor have data dependency when performing model calculations, and in the resource allocation scheme, elements in the same row perform model calculation tasks in the same computing unit.

[0060] For example, when the elements in the same row are performing model calculation, there is a data dependency representation, and for one element in the elements in the same row, the inference result corresponding to the element is related to at least two elements in the elements in the same row. For example, this data dependency can be formed by operations such as reduction and accumulation, that is, the inference result corresponding to one element requires the interaction of multiple elements in a row.

[0061] As mentioned above, each computing unit includes a shared memory, and the C execution units in the computing unit exchange data through the shared memory.

[0062] Since elements in the same row have data dependencies when performing model calculations, when elements in the same row perform model calculation tasks in the same computing unit, they can exchange data with each other through shared memory. This eliminates the need to exchange data from the main memory or the secondary cache, reduces memory access overhead, improves computing efficiency, and improves system performance.

[0063] The processes of step S30 and step S40 in the method for allocating hardware computing resources provided by at least one embodiment of the present disclosure are described in detail below with reference to the accompanying drawings.

[0064] For example, step S30 may include: determining a resource allocation scheme for the operator according to a size relationship between h and L, and a size relationship between n and A×L.

[0065] For example, if h is greater than L, it is reasonable for the execution unit to split the H dimension. For example, in response to h being greater than or equal to L, it is determined that the operator adopts the third resource allocation scheme. The third resource allocation scheme indicates that the input tensor is first split in N dimensions, and the multiple tensors after being split in N dimensions are allocated to A programmable multiprocessors for execution, and then the tensors allocated to each programmable processor are split in H dimensions, and the multiple tensors after being split in H dimensions are allocated to B×C execution units in the programmable processor for execution, and each execution unit traverses the allocated tensors row by row in the order of row direction first and column direction second, in units of thread warps.

[0066] Figure 4 A schematic diagram of a third resource mapping solution provided for at least one embodiment of the present disclosure.

[0067] like Figure 4 As shown, the input tensor includes batch0, batch1, ... batch(n-1), a total of n batches, each batch includes h rows and w columns of elements.

[0068] For example, in the third resource mapping scheme, n batches are evenly distributed to A programmable multiprocessors, and each programmable multiprocessor processes n / A batches, which serve as a fourth tensor. If it is not possible to evenly distribute, the number of batches processed by each programmable multiprocessor is balanced as much as possible to achieve the purpose of load balancing.

[0069] Divide one batch of the fourth tensor received by each programmable multiprocessor into multiple fifth tensors in the H dimension, or use a row of elements in the fourth tensor as a fifth tensor, and each execution unit executes one fifth tensor. If there are multiple batches, the multiple batches are executed in sequence. For example, Figure 4A row of shaded elements in batch0 in FIG. 1 can be regarded as a fifth tensor. For example, each fifth tensor is traversed and executed in the execution unit. For an execution unit, the fifth tensor is traversed from the W dimension in units of thread warps, for example, traversed and executed row by row in the order of row direction first and column direction second, first processing all elements of the first row, then processing all elements of the second row, and so on, until the last row.

[0070] In this embodiment, the processing of the input tensor includes three layers of loops: the outermost loop is a plurality of fourth tensors obtained by dividing the fourth tensor into N dimensions according to the number of programmable multiprocessors; the middle loop is a plurality of fifth tensors obtained by dividing the fourth tensor into H dimensions according to the number of execution units of each programmable processor; and the innermost loop is a traversal of the fifth tensor from the W dimension in units of thread warps.

[0071] For example, in this embodiment, step S40 may include: splitting the input tensor into A fourth tensors in N dimensions, and sending the A fourth tensors to A programmable multiprocessors respectively, wherein each fourth tensor includes n / A batches; splitting the fourth tensor received by each programmable multiprocessor into multiple fifth tensors in H dimensions, and sending the multiple fifth tensors to corresponding execution units, wherein each fifth tensor includes a row of elements or multiple consecutive rows of elements in the fourth tensor; setting each execution unit to traverse the received fifth tensor row by row in the order of row direction first and column direction later to perform the model calculation task.

[0072] For example, if h is less than L, then using the third resource allocation scheme will cause some execution units to be idle. For example, for the decoding stage of the inference process, h=1. At this time, if the previous scheme is still used for segmentation, only one execution unit will work. Therefore, different segmentation methods and resource allocation methods can be set according to the size relationship between n and A×L.

[0073] For example, in some embodiments, determining the resource allocation scheme of the operator based on the size relationship between h and L, and the size relationship between n and A×L may include: in response to h being less than L and n being greater than or equal to A×L, determining that the operator adopts a first resource allocation scheme, wherein the first resource allocation scheme indicates that the input tensor is first split in N dimensions, and the multiple tensors split in N dimensions are allocated to A×L execution units for separate execution, and each execution unit traverses the allocated tensors row by row in the order of row direction first and column direction later.

[0074] For example, the first resource allocation scheme indicates that the input tensor is evenly divided into A×L first tensors in N dimensions, and the A×L first tensors are allocated one by one to A×L execution units for execution respectively.

[0075] For example, in this embodiment, although the input tensor is also split in N dimensions first, the number of programmable multiprocessors is no longer used as the basis for splitting. Instead, all execution units in the entire hardware computing resources are regarded as a whole, and the input tensor is split according to the number of execution units without distinguishing between programmable multiprocessors.

[0076] Figure 5 A schematic diagram of a first resource mapping solution provided for at least one embodiment of the present disclosure.

[0077] like Figure 5 As shown, the input tensor includes N batches such as batch0, and each batch includes h rows and w columns of elements.

[0078] refer to Figure 5 In the first resource mapping scheme, the input tensor is evenly divided into A×L first tensors in N dimensions, each of which includes n / (A×L) batches, that is, n batches are evenly divided into A×L execution units. If it is not possible to divide them evenly, the number of batches processed by each execution unit is balanced as much as possible to achieve the purpose of load balancing.

[0079] In the first resource allocation scheme, each execution unit no longer executes one or more rows in a batch, but executes one batch or multiple batches. Specifically, each execution unit traverses the allocated first tensor row by row in the order of first row direction and then column direction in the unit of thread warp. When the first tensor has multiple batches, traverse them one by one. For example, after completing the traversal of one batch, traverse the next batch. Each batch is still traversed row by row in the order of first row direction and then column direction.

[0080] In this embodiment, the processing of the input tensor includes two layers of loops: the outermost loop is to obtain multiple first tensors by dividing the tensors into N dimensions according to the total number of execution units in the hardware computing resources; the innermost loop is to traverse the first tensors from the W dimension in units of thread warps.

[0081] For example, in this embodiment, step S40 may include: splitting the input tensor into A×L first tensors in N dimensions, and sending the A×L first tensors to A×L execution units respectively, wherein each first tensor includes n / (A×L) batches; setting each execution unit to traverse the received first tensor row by row in the order of row direction first and column direction later in units of thread warps to execute the model calculation task.

[0082] If h is smaller than L and n is smaller than A×L, since the number of execution units is greater than n, some execution units will still be idle if the first resource allocation scheme is still adopted.

[0083] For example, in some other embodiments, determining the resource allocation scheme of the operator according to the size relationship between h and L, and the size relationship between n and A×L may include: in response to h being less than L and n being less than A×L, determining that the operator adopts a second resource allocation scheme; wherein the second resource allocation scheme indicates that each batch in the input tensor is divided into m parts in the W dimension and allocated to m execution units for execution respectively, wherein m=min( A×L / n ,C),min( ) represents the minimum function, which is used to select A×L / n and the smaller one in C as m, Represents a ceiling function, where m execution units belong to the same computational unit.

[0084] In order to make full use of the execution units, in the second resource allocation scheme, one batch is split into multiple parts along the W dimension and allocated to multiple execution units for execution.

[0085] For example, the model calculation tasks of the same batch in the input tensor are executed in the same computing unit. That is to say, in this embodiment, the same batch will not be executed across computing units, and the C execution units in one computing unit can make one batch or multiple batches. This is because a row of elements is assigned to m execution units for separate execution, and in the execution process of the model calculation tasks for the input tensor in at least one embodiment of the present disclosure, there is data dependency between the elements belonging to the same row in the input tensor, so the task calculation of one element needs to use elements in other positions. When a row of elements is executed in different execution units of the same computing unit, the execution units can exchange data through the shared memory of the computing unit. For example, the required elements can be obtained from the shared memory without having to read from the memory or the secondary cache, thereby reducing memory access overhead, improving execution efficiency, and improving system performance.

[0086] For example, in this embodiment, although the input tensor is also split in N dimensions first, the splitting is no longer based on the number of programmable multiprocessors or execution units. Instead, all computing units in the entire hardware computing resources are considered as a whole for splitting.

[0087] Figure 6 A schematic diagram of a second resource mapping solution provided for at least one embodiment of the present disclosure.

[0088] like Figure 6 As shown, the input tensor includes batch0, batch1, ... batch(n-1), a total of n batches, each batch includes h rows and w columns of elements.

[0089] For example, refer to Figure 6 , in the second resource mapping scheme, referring to the number A×B of computing units in the programmable multiprocessor, the input tensor is evenly divided into multiple second tensors in N dimensions.

[0090] Specifically, splitting the input tensor in N dimensions into multiple second tensors may include: in response to n being greater than or equal to A×B, splitting the input tensor in N dimensions into A×B second tensors, wherein each second tensor includes n / (A×B) batches; in response to n being less than A×B, splitting the input tensor in N dimensions into n second tensors, wherein each second tensor includes 1 batch.

[0091] In this embodiment, one batch needs one computing unit to perform the model calculation task. Therefore, if n is greater than the total number of computing units in the hardware computing resources, the n batches are divided into A×B second tensors, each of which includes n / (A×B) batches. If n is less than the total number of computing units in the hardware computing unit, each computing unit is still kept to do one batch, that is, the input tensor is divided into n second tensors in N dimensions, each second tensor includes one batch, and each computing unit executes one second tensor.

[0092] The second resource allocation scheme also indicates that each batch in the second tensor is divided into m parts in the W dimension and allocated to m execution units for execution respectively. Specifically, in the second resource allocation scheme, the total number of execution units in the hardware computing resources and the result of rounding up n are used as the number of execution units that execute 1 batch. However, 1 batch is executed by at most C execution units to perform the corresponding model calculation tasks, so if the rounding result is greater than C, m=C is still set.

[0093] For example, refer to Figure 6 , m=4, that is, batch0 is divided into 4 parts in the W dimension, and 4 third tensors are obtained. Each execution unit executes 1 third tensor. Each execution unit still uses the thread warp as a unit, and traverses the received third tensor row by row in the order of row direction first and column direction later to perform the model calculation task.

[0094] In this embodiment, the processing of the input tensor includes two layers of loops: the outermost loop is to obtain multiple second tensors by dividing the tensors into N dimensions in combination with the total number of computing units in the hardware computing resources, and each batch of each second tensor is divided into m third tensors in the W dimension; the innermost loop is to traverse the third tensor from the W dimension in units of thread warps.

[0095] For example, in this embodiment, step S40 may include: splitting the input tensor into multiple second tensors in N dimensions, and sending the multiple second tensors to multiple computing units, wherein each computing unit receives a second tensor and each second tensor includes at least 1 batch; splitting each batch in the second tensor into m third tensors in W dimensions, and sending the multiple third tensors obtained by splitting the second tensor to multiple execution units in the computing unit that receives the second tensor, respectively, wherein each third tensor includes w / m rows and h columns of elements; setting each execution unit to traverse the received third tensor row by row in the order of row direction first and column direction later in a thread warp to execute the model calculation task.

[0096] For example, assuming A×L=256, h=1 and less than L, A×B=64, C=4, if n=64, then m=4, in which case the second tensor includes 1 batch, and 1 computing unit performs 1 batch of model computing tasks. In addition, each batch is divided into 4 parts in the W dimension to obtain 4 third tensors, each of which includes 1 row w / 4 columns, and the 4 execution units in the computing unit respectively perform 1 third tensor model computing task, for example, the execution unit traverses the third tensor in the W dimension in units of thread warps.

[0097] If n=128, then in this embodiment, m=2, and the second tensor includes 2 batches, and 1 computing unit performs the model calculation tasks of 2 batches. In addition, each batch is divided into 2 parts in the W dimension to obtain 2 third tensors, each of which includes 1 row w / 2 columns. The 2 execution units in the computing unit perform the model calculation tasks of two third tensors of 1 batch, and the other 2 execution units perform the model calculation tasks of two third tensors of another batch.

[0098] If n=32, then m=4. In this case, the second tensor includes 1 batch, and 1 computing unit performs 1 batch of model calculation tasks. The process is the same as that when n=64, and will not be repeated here.

[0099] In the hardware resource allocation method provided in at least one embodiment of the present disclosure, for input tensors of any data shape, the corresponding resource allocation scheme can be adaptively and automatically matched, and all execution units can be used as evenly as possible under different data shapes, thereby avoiding computing resource idleness as much as possible and improving system performance.

[0100] Taking the inference task of a large language model as an example, actual tests were conducted for h=1, n=32, 256, 512, 1024, and 2048. The performance can be improved by 3-8 times by using the hardware computing resource allocation method provided by at least one embodiment of the present disclosure.

[0101] At least one embodiment of the present disclosure further provides a hardware computing resource. Figure 7 A schematic structural diagram of hardware computing resources provided for at least one embodiment of the present disclosure.

[0102] For example, the hardware computing resources include A programmable multiprocessors ( Figure 7 Not shown), each programmable multiprocessor includes B computing units ( Figure 7 Not shown), each computing unit includes C execution units ( Figure 7 (not shown), each execution unit is configured to run one or more thread warps, and A, B, and C are positive integers greater than 1.

[0103] For example, the hardware computing resource is a graphics processor or a general-purpose graphics processor. For the functions, structures and relationships of the programmable multiprocessor, computing unit and execution unit, please refer to the aforementioned Figure 1 The relevant description will not be repeated here.

[0104] For example, Figure 7 As shown, the hardware computing resource 100 also includes a first acquisition module 101, a second acquisition module 102, a solution determination module 103, and a resource allocation module 104. It should be noted that Figure 7 The components and structures of the hardware computing resource 100 shown are merely exemplary and non-limiting. The hardware computing resource 100 may also have other components and structures as required.

[0105] For example, these modules can be implemented by hardware (e.g., circuit) modules, software modules, or any combination of the two, etc. The following embodiments are the same and will not be described in detail. For example, these units can be implemented by a data processor (GPU), a general purpose data processor (GPGPU), a tensor processor (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities and corresponding computer instructions.

[0106] For example, the first acquisition module 101 is configured to obtain the data shape of the input tensor of the operator for the model calculation task, wherein the data format of the input tensor is [N, H, W], the N dimension represents the number of batches of the input tensor, the H dimension represents the height of each batch, and the W dimension represents the width of each batch. The input tensor is expressed as [n, h, w], where n, h, and w are positive integers.

[0107] The second acquisition module 102 is configured to acquire hardware parameters of the hardware computing resources, wherein the hardware parameters include the number A of programmable multiprocessors and the number L of execution units included in each programmable multiprocessor, where L is a positive integer and equal to B×C.

[0108] The scheme determination module 103 is configured to determine the resource allocation scheme of the operator according to the hardware parameters and the data shape, wherein the resource allocation scheme indicates the way in which the input tensor is split in multiple dimensions of the input tensor, and the way in which the input tensor is allocated to the hardware computing resources for processing according to the splitting method.

[0109] The resource allocation module 104 is configured to split the input tensor according to the resource allocation scheme, and send the split tensor to the corresponding computing resources to execute the model computing task.

[0110] For example, elements belonging to the same row in the input tensor have data dependency when performing model calculations, and in the resource allocation scheme, elements in the same row perform model calculation tasks in the same computing unit.

[0111] For example, the first acquisition module 101, the second acquisition module 102, the solution determination module 103, and the resource allocation module 104 may include codes and programs stored in a memory; the processor may execute the codes and programs to implement some or all functions of the first acquisition module 101, the second acquisition module 102, the solution determination module 103, and the resource allocation module 104 as described above. For example, the first acquisition module 101, the second acquisition module 102, the solution determination module 103, and the resource allocation module 104 may be dedicated hardware devices, used to implement some or all functions of the first acquisition module 101, the second acquisition module 102, the solution determination module 103, and the resource allocation module 104 as described above. For example, the first acquisition module 101, the second acquisition module 102, the solution determination module 103, and the resource allocation module 104 may be a circuit board or a combination of multiple circuit boards, used to implement the functions as described above. In an embodiment of the present application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) firmware stored in the memory that can be executed by the processor.

[0112] It should be noted that the first acquisition module 101 can be used to implement Figure 3 In step S10 shown in FIG. 1 , the second acquisition module 102 may be used to implement Figure 3 In step S20, the solution determination module 103 can be used to implement Figure 3 In step S30, the resource allocation module 104 may be used to implement Figure 3Step S40 shown. Therefore, for the specific description of the functions that can be implemented by the first acquisition module 101, reference can be made to the relevant description of step S10 in the embodiment of the above-mentioned hardware computing resource allocation method. For the specific description of the functions that can be implemented by the second acquisition module 102, reference can be made to the relevant description of step S20 in the embodiment of the above-mentioned hardware computing resource allocation method. For the specific description of the functions that can be implemented by the solution determination module 103, reference can be made to the relevant description of step S30 in the embodiment of the above-mentioned hardware computing resource allocation method. For the specific description of the functions that can be implemented by the resource allocation module 104, reference can be made to the relevant description of step S40 in the embodiment of the above-mentioned hardware computing resource allocation method. The repeated parts will not be repeated here.

[0113] The hardware computing resources provided by at least one embodiment of the present disclosure can adaptively and automatically match the corresponding resource allocation scheme for input tensors of any data shape, and all execution units can be used as evenly as possible under different data shapes, thereby avoiding idle computing resources as much as possible and improving system performance.

[0114] The hardware computing resources 100 can achieve technical effects similar to those of the aforementioned hardware computing resource allocation method, which will not be described in detail here.

[0115] It should be noted that in the embodiments of the present disclosure, the hardware computing resource 100 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or can be composed in other applicable ways.

[0116] Figure 8 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 8 As shown, the storage medium 200 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 201 may be non-transitory stored on the storage medium 200. For example, when the computer-readable instructions 201 are executed by a processor, one or more steps in the hardware computing resource allocation method described above may be executed.

[0117] For example, the storage medium 200 may be applied in an electronic device 300 , for example, the storage medium 200 may include a memory in the electronic device.

[0118] For example, the storage device may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the data processor may run the computer-readable instructions to implement various functions of the data processor. Various applications and various data may also be stored in the storage medium.

[0119] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0120] Fig. 9 The following is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Fig. 9 As shown, the electronic device 300 is suitable for implementing the hardware computing resource allocation method provided in the embodiment of the present disclosure. It should be noted that Fig. 9 The components of the electronic device 300 shown are merely exemplary and non-limiting. The electronic device 300 may also have other components according to actual application requirements.

[0121] For example, Fig. 9 As shown, in some examples, the electronic device 300 includes a processing device (e.g., a graphics processor, a general-purpose graphics processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303 to achieve various functions, such as executing the hardware computing resource allocation method provided in at least one embodiment of the present disclosure.

[0122] For example, when the computer readable instructions are executed by the processing device 301, one or more steps in the hardware computing resource allocation method described in any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the hardware computing resource allocation method, reference may be made to the relevant description in the embodiment of the hardware computing resource allocation method, and the repeated parts will not be repeated.

[0123] Various programs and data required for the operation of the computer system are also stored in the RAM 303 . The processing device 301 , the ROM 302 , and the RAM 303 are connected to each other via a bus 304 . An input / output (I / O) interface 305 is also connected to the bus 304 .

[0124] For example, the memory may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory (cache), etc. For example, computer-readable instructions may be loaded from storage device 308 into random access memory (RAM) 303 to run computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by applications, etc., may also be stored in the computer-readable storage medium.

[0125] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Fig. 9 An electronic device 300 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown, and the electronic device 300 may be implemented or have more or fewer devices instead. For example, the processor 301 can control other components in the electronic device 300 to perform the desired functions. The processing device 301 can be a device with data processing capabilities and / or program execution capabilities such as a central processing unit (CPU), a tensor processing unit (TPU) or a graphics processing unit (GPU). The central processing unit can be an X86 or ARM architecture, etc. The graphics processor can be directly integrated into the motherboard separately, or built into the north bridge chip of the motherboard. The graphics processor can also be built into the central processing unit.

[0126] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure), etc. The peripheral interface may be various types of interfaces, such as a USB interface, a lightning interface, etc. The communication device 309 may communicate with a network and other devices through wireless communication, such as the Internet, an intranet and / or a wireless network such as a cellular phone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN). Wireless communication may use any of a variety of communication standards, protocols, and techniques, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0127] For example, the electronic device 300 may be any device such as a mobile phone, a tablet computer, a laptop computer, an e-book, a game console, a television, a digital photo frame, a navigator, etc., or may be any combination of electronic devices and hardware, which is not limited in the embodiments of the present disclosure.

[0128] For example, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned hardware computing resource allocation method defined in the method of the embodiment of the present disclosure is executed.

[0129] It should be noted that the computer-readable medium of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0130] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0133] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0134] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0135] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0136] There are a few points to note about this disclosure:

[0137] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to the general design.

[0138] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0139] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A method for allocating hardware computing resources, wherein the hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, each execution unit is configured to run one or more thread warps, A, B, and C are all positive integers greater than 1, The hardware computing resource allocation method comprises: Get the data shape of the input tensor of the operator for the model calculation task, wherein the input tensor is a three-dimensional tensor, including N dimension, H dimension and W dimension, and the data shape of the input tensor is [n,h,w], where the N dimension represents the number of batches n of the input tensor, the H dimension represents the height h of each batch, and the W dimension represents the width w of each batch, and n, h, and w are positive integers; Acquire hardware parameters of the hardware computing resources, wherein the hardware parameters include the number A of the programmable multiprocessors and the number L of the execution units included in each programmable multiprocessor, where L is a positive integer and equal to B×C; Determine a resource allocation scheme for the operator according to the hardware parameters and the data shape, wherein the resource allocation scheme indicates a segmentation method of the input tensor in multiple dimensions of the input tensor, and an allocation method of allocating the input tensor to the hardware computing resource for processing according to the segmentation method; Splitting the input tensor according to the resource allocation scheme, and sending each tensor obtained by the splitting to a corresponding computing resource to execute the model computing task; Among them, the elements belonging to the same row in the input tensor have data dependency when performing the model calculation task, and in the resource allocation scheme, the elements in the same row perform the model calculation task in the same computing unit.

2. The method for allocating hardware computing resources according to claim 1, wherein: Determining a resource allocation scheme for the operator according to the hardware parameters and the data shape includes: The resource allocation scheme of the operator is determined according to the size relationship between h and L, and the size relationship between n and A×L.

3. The method for allocating hardware computing resources according to claim 2, wherein: According to the size relationship between h and L, and the size relationship between n and A×L, a resource allocation scheme for the operator is determined, including: In response to h being less than L and n being greater than or equal to A×L, determining that the operator adopts a first resource allocation scheme, Among them, the first resource allocation scheme indicates that the input tensor is first split in the N dimensions, and the multiple first tensors obtained by splitting in the N dimensions are allocated to A×L execution units for separate execution, and each execution unit traverses the allocated first tensor row by row in the order of row direction first and column direction later.

4. The method for allocating hardware computing resources according to claim 3, wherein: The first resource allocation scheme indicates that the input tensor is evenly divided into A×L first tensors in the N dimensions, and the A×L first tensors are allocated to the A×L execution units for separate execution, with each execution unit processing one first tensor.

5. The method for allocating hardware computing resources according to claim 3, wherein: The input tensor is split according to the resource allocation scheme, and each of the split tensors is sent to a corresponding computing resource to execute the model computing task, including: Splitting the input tensor into A×L first tensors in the N dimensions, and sending the A×L first tensors to the A×L execution units respectively, wherein each first tensor includes n / (A×L) batches; Each execution unit is set to traverse the received first tensor row by row in the order of row direction first and column direction later, taking the thread warp as a unit, so as to execute the model calculation task.

6. The method for allocating hardware computing resources according to claim 2, wherein: According to the size relationship between h and L, and the size relationship between n and A×L, a resource allocation scheme for the operator is determined, including: In response to h being less than L and n being less than A×L, determining that the operator adopts a second resource allocation scheme; The second resource allocation scheme indicates that each batch in the input tensor is divided into m parts in the W dimension and allocated to m execution units for execution respectively, where m= min ( A×L / n ,C),min( ) represents the minimum function, represents a ceiling function, and the m execution units belong to the same computing unit.

7. The method for allocating hardware computing resources according to claim 6, wherein: The model computing tasks of the same batch in the input tensor are executed in the same computing unit.

8. The method for allocating hardware computing resources according to claim 6, wherein: The input tensor is split according to the resource allocation scheme, and each of the split tensors is sent to a corresponding computing resource to execute the model computing task, including: Splitting the input tensor into a plurality of second tensors in the N dimensions, and sending the plurality of second tensors to a plurality of computing units, wherein each computing unit receives a second tensor, and each second tensor includes at least one batch; Splitting each batch in the second tensor into m third tensors in the W dimension, and sending the plurality of third tensors obtained by splitting the second tensor to a plurality of execution units in the computing unit receiving the second tensor, respectively, wherein each third tensor includes w / m rows and h columns of elements; Each execution unit is set to traverse the received third tensor row by row in the order of row direction first and column direction later, taking the thread warp as a unit, to execute the model calculation task.

9. The method for allocating hardware computing resources according to claim 8, wherein: Splitting the input tensor in the N dimension into a plurality of second tensors includes: In response to n being greater than or equal to A×B, splitting the input tensor into A×B second tensors in the N dimension, wherein each second tensor includes n / (A×B) batches; In response to n being less than A×B, the input tensor is split into n second tensors in the N dimension, wherein each second tensor includes 1 batch.

10. The method for allocating hardware computing resources according to claim 2, wherein: According to the size relationship between h and L, and the size relationship between n and A×L, a resource allocation scheme for the operator is determined, including: In response to h being greater than or equal to L, determining that the operator adopts a third resource allocation scheme, Among them, the third resource allocation scheme indicates that the input tensor is first split in the N dimensions, and the multiple tensors obtained by splitting in the N dimensions are allocated to the A programmable multiprocessors for separate execution, and then the tensor allocated to each programmable processor is split in the H dimensions, and the multiple tensors obtained by splitting in the H dimensions are allocated to B×C execution units in the programmable processor for separate execution, and each execution unit traverses the tensors allocated by the execution unit row by row in the order of row direction first and column direction later.

11. The method for allocating hardware computing resources according to claim 10, wherein: The input tensor is split according to the resource allocation scheme, and the split tensors are sent to corresponding computing resources to execute the model computing task, including: Splitting the input tensor into A fourth tensors in the N dimensions, and sending the A fourth tensors to the A programmable multiprocessors respectively, wherein each fourth tensor includes n / A batches; Splitting the fourth tensor received by each programmable multiprocessor into a plurality of fifth tensors in the H dimension, and sending the plurality of fifth tensors to corresponding execution units, wherein each fifth tensor includes a row of elements or a plurality of consecutive rows of elements in the fourth tensor; Each execution unit is set to traverse the received fifth tensor row by row in the order of row direction first and column direction later, taking the thread warp as a unit, to execute the model calculation task.

12. The method for allocating hardware computing resources according to any one of claims 1 to 11, wherein: The elements belonging to the same row have data dependency when performing the model calculation task, and for one element in the elements belonging to the same row, the inference result corresponding to the one element is related to at least two elements in the elements in the same row, Each computing unit includes a shared memory, and the C execution units in the computing unit exchange data through the shared memory.

13. A hardware computing resource, comprising A programmable multiprocessors, each programmable multiprocessor comprising B computing units, each computing unit comprising C execution units, each execution unit being configured to run one or more thread warps, A, B, and C being positive integers greater than 1, The hardware computing resources also include: A first acquisition module is configured to acquire a data shape of an input tensor of an operator for a model calculation task, wherein the input tensor is a three-dimensional tensor including an N dimension, an H dimension, and a W dimension, and the data shape of the input tensor is [n, h, w], where the N dimension represents the number of batches n of the input tensor, the H dimension represents the height h of each batch, and the W dimension represents the width w of each batch, and n, h, and w are positive integers; A second acquisition module is configured to acquire hardware parameters of the hardware computing resources, wherein the hardware parameters include the number A of the programmable multiprocessors and the number L of execution units included in each programmable multiprocessor, where L is a positive integer and equal to B×C; a scheme determination module, configured to determine a resource allocation scheme of the operator according to the hardware parameters and the data shape, wherein the resource allocation scheme indicates a segmentation method of the input tensor in multiple dimensions of the input tensor, and an allocation method of allocating the input tensor to the hardware computing resources for processing according to the segmentation method; A resource allocation module, configured to split the input tensor according to the resource allocation scheme, and send each tensor obtained by splitting to a corresponding computing resource to execute the model computing task; Among them, the elements belonging to the same row in the input tensor have data dependency when performing the model calculation task, and in the resource allocation scheme, the elements in the same row perform the model calculation task in the same computing unit.

14. An electronic device comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the hardware computing resource allocation method according to any one of claims 1-12 is implemented.

15. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer executable instructions are executed by a processor, the hardware computing resource allocation method according to any one of claims 1 to 12 is implemented.

Images