Hardware computing resource allocation method, hardware computing resource, and electronic device

Abstract

A hardware computing resource allocation method, a hardware computing resource, an electronic device, and a non-transitory computer-readable storage medium. The hardware computing resource allocation method comprises: acquiring the data shape of an input tensor, used for a model computing task, of an operator; acquiring hardware parameters of hardware computing resources; determining a resource allocation scheme of the operator on the basis of the hardware parameters and the data shape, wherein the resource allocation scheme indicates a partitioning approach of the input tensor in a plurality of dimensions of the input tensor, and an allocation approach for allocating, on the basis of the partitioning approach, the input tensor to the hardware computing resources for processing; and partitioning the input tensor on the basis of the resource allocation scheme, and sending the partitioned tensors to corresponding computing resources so as to execute the model computing task, wherein elements belonging to the same row in the input tensor exhibit data dependency during model computing, and in the resource allocation scheme, the elements within the same row are processed within the same computing unit for the model computing task.

Description

Hardware computing resource allocation methods, hardware computing resources, electronic devices

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411455431.7, filed on October 18, 2024, entitled "Hardware Computing Resource Allocation Method, Hardware Computing Resources, Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to a hardware computing resource allocation method, hardware computing resources, electronic devices, and non-transitory computer-readable storage media. Background Technology

[0004] A tensor is a multilinear mapping defined on the Cartesian product of some vector spaces and some dual spaces. For example, a scalar can be viewed as a 0-dimensional tensor, a vector as a one-dimensional tensor, and a matrix as a two-dimensional tensor. Tensor operations are widely used in processors such as parallel processors. In current hardware devices, data processing integrated circuits such as central processing units (CPUs), graphics processing units (GPUs), and general-purpose computing on graphics processing units (GPGPUs) can execute programs to perform various tensor-related computational tasks. With increasing computational demands, hardware devices place increasingly higher performance requirements on operators; better performance means faster computation and reduced computational overhead. Mapping computational tasks evenly across hardware devices and maximizing the parallel execution efficiency of hardware devices has become key to performance improvement. Summary of the Invention

[0005] According to various embodiments disclosed in this application, a hardware computing resource allocation method, hardware computing resources, and electronic devices are provided.

[0006] A method for allocating hardware computing resources, wherein the hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, and each execution unit is configured to run one or more thread bundles, where A, B, and C are all positive integers greater than 1. The method includes: obtaining the data shape of an input tensor for a model computation task, wherein the input tensor is a three-dimensional tensor, including N, H, and W dimensions, and the data shape of the input tensor is [n, h, w], where N dimension represents the batch size n of the input tensor, H dimension represents the height h of each batch, and W dimension represents the width w of each batch, and n, h, and w are positive integers; and obtaining the hardware parameters of the hardware computing resources, wherein the hardware parameters include the programmable multiprocessors. The number A and the number L of execution units included in each programmable multiprocessor, where L is a positive integer and equal to B×C; Based on the hardware parameters and the data shape, an adaptive resource allocation scheme for the operator is determined, wherein the resource allocation scheme indicates the method of partitioning the input tensor in multiple dimensions of the input tensor, and the allocation method of allocating the input tensor to the hardware computing resources for processing according to the partitioning method; and the input tensor is partitioned according to the resource allocation scheme, and the partitioned tensor is sent to the corresponding computing resources to execute the model calculation task; wherein elements belonging to the same row in the input tensor have data dependencies when performing the model calculation task, and in the resource allocation scheme, the elements in the same row perform the model calculation task in the same computing unit.

[0007] A hardware computing resource includes A programmable multiprocessors, each programmable multiprocessor including B computing units, each computing unit including C execution units, and each execution unit configured to run one or more thread bundles, where A, B, and C are all positive integers greater than 1. The hardware computing resource further includes: a first acquisition module configured to acquire the data shape of an input tensor for a model computation task, wherein the input tensor is a three-dimensional tensor including N, H, and W dimensions, and the data shape of the input tensor is [n, h, w], where N dimension represents the batch size n of the input tensor, H dimension represents the height h of each batch, and W dimension represents the width w of each batch, and n, h, and w are positive integers; and a second acquisition module configured to acquire hardware parameters of the hardware computing resource, wherein the hardware parameters include the number A of the programmable multiprocessors, and... Each programmable multiprocessor includes the number of execution units L, where L is a positive integer and equal to B×C; a scheme determination module is configured to adaptively determine the resource allocation scheme of the operator based on the hardware parameters and the data shape, wherein the resource allocation scheme indicates the way the input tensor is segmented in multiple dimensions of the input tensor, and the allocation method for allocating the input tensor to the hardware computing resources for processing according to the segmentation method; and a resource allocation module is configured to segment the input tensor according to the resource allocation scheme and send the segmented tensor to the corresponding computing resources to execute the model calculation task; wherein elements belonging to the same row in the input tensor have data dependencies when performing the model calculation task, and in the resource allocation scheme, the elements in the same row perform the model calculation task in the same computing unit.

[0008] An electronic device includes: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to execute the computer-executable instructions.

[0009] The hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, and each execution unit is configured to run one or more thread bundles. A, B, and C are all positive integers greater than 1. The computer-executable instructions are executed by the processor to implement the following steps:

[0010] Obtain the data shape of the input tensor used for model computation tasks, where the input tensor is a three-dimensional tensor, including N, H, and W dimensions, and the data shape of the input tensor is [n, h, w]. N dimension represents the batch size n of the input tensor, H dimension represents the height h of each batch, and W dimension represents the width w of each batch. n, h, and w are positive integers. Obtain the hardware parameters of the hardware computing resources, including the number A of programmable multiprocessors and the number L of execution units included in each programmable multiprocessor, where L is a positive integer equal to B × C. Based on the hardware parameters and... The data shape determines the resource allocation scheme for the operator. The resource allocation scheme indicates how the input tensor is partitioned across multiple dimensions, and how the input tensor is allocated to hardware computing resources for processing according to the partitioning method. The input tensor is partitioned according to the resource allocation scheme, and each partitioned tensor is sent to the corresponding computing resources to perform model computation tasks. Elements in the same row of the input tensor have data dependencies when performing model computation tasks, and in the resource allocation scheme, elements in the same row perform model computation tasks in the same computing unit.

[0011] One or more non-transitory computer-readable storage media storing computer-executable instructions, wherein the non-transitory computer-readable storage media stores computer-executable instructions, and the hardware computing resources include A programmable multiprocessors, each programmable multiprocessor including B computing units, each computing unit including C execution units, each execution unit being configured to run one or more thread bundles, where A, B, and C are all positive integers greater than 1, and the computer-executable instructions, when executed by a processor, cause the one or more processors to perform the following steps:

[0012] Obtain the data shape of the input tensor used for model computation tasks, where the input tensor is a three-dimensional tensor, including N, H, and W dimensions, and the data shape of the input tensor is [n, h, w]. N dimension represents the batch size n of the input tensor, H dimension represents the height h of each batch, and W dimension represents the width w of each batch. n, h, and w are positive integers. Obtain the hardware parameters of the hardware computing resources, including the number A of programmable multiprocessors and the number L of execution units included in each programmable multiprocessor, where L is a positive integer equal to B × C. Based on the hardware parameters and... The data shape determines the resource allocation scheme for the operator. The resource allocation scheme indicates how the input tensor is partitioned across multiple dimensions, and how the input tensor is allocated to hardware computing resources for processing according to the partitioning method. The input tensor is partitioned according to the resource allocation scheme, and each partitioned tensor is sent to the corresponding computing resources to perform model computation tasks. Elements in the same row of the input tensor have data dependencies when performing model computation tasks, and in the resource allocation scheme, elements in the same row perform model computation tasks in the same computing unit.

[0013] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 is a schematic diagram of a general-purpose graphics processing unit (GPGPU).

[0016] Figure 2 is a schematic diagram of the structure of tensor data.

[0017] Figure 3 is a schematic flowchart of a hardware computing resource allocation method according to one or more embodiments.

[0018] Figure 4 is a schematic diagram of a third resource mapping scheme according to one or more embodiments.

[0019] Figure 5 is a schematic diagram of a first resource mapping scheme according to one or more embodiments.

[0020] Figure 6 is a schematic diagram of a second resource mapping scheme according to one or more embodiments.

[0021] Figure 7 is a schematic structural diagram of hardware computing resources according to one or more embodiments;

[0022] Figure 8 is a schematic diagram of a non-transitory computer-readable storage medium according to one or more embodiments;

[0023] Figure 9 is a schematic block diagram of an electronic device according to one or more embodiments. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0025] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0026] Figure 1 shows a schematic block diagram of a general-purpose graphics processing unit (GPGPU).

[0027] As shown in Figure 1, a general-purpose graphics processing unit (GPU) is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as streaming processor cluster 1, ..., streaming processor cluster M shown in Figure 1, where M is a positive integer greater than 1. In a general-purpose GPU, one streaming processor cluster processes one computational task, or multiple streaming processor clusters process one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0028] As shown in Figure 1, taking streaming processor cluster 1 as an example, a streaming processor cluster includes multiple computing units, such as computing unit 1, computing unit 2, ..., computing unit N in Figure 1, where N is a positive integer. Each computing unit (CU) is used to perform arithmetic and logical operations, such as accumulation, reduction, and conventional addition, subtraction, multiplication, and division. A computing unit includes multiple cores (also called computing kernels), each of which includes an arithmetic logic unit (ALU), a floating-point unit, etc., and is used to execute specific computing tasks. In addition, the computing unit also includes registers (such as the register file in Figure 1) and shared memory, which are used to hierarchically store source data and destination data related to computing tasks. The shared memory in a computing unit is used to share data among the cores of that computing unit.

[0029] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processing unit (or parallel computing processor). These thread blocks are then distributed to various computing units via a thread block distribution module (not shown in Figure 1). All threads within a thread block must be assigned to the same computing unit for execution. Simultaneously, thread blocks are further subdivided into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Thread bundles run on execution units (EUs), while threads run on thread hardware; for example, one thread bundle runs on one execution unit, and one thread runs on one thread hardware unit. Multiple thread blocks can execute within the same computing unit or in different computing units.

[0030] Within each computing unit, a thread bundle scheduling / distribution module (not shown in Figure 1) schedules and allocates thread bundles so that multiple computing cores within that unit can run them. Depending on the number of computing cores in the unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory for read / write operations, etc.

[0031] Graphics processors or general-purpose graphics processors can be used for inference in network models. For example, network models can be large language models, visual language models, multimodal models, etc.

[0032] The tensor data used in the network model inference process is a three-dimensional tensor, including N-dimensional, H-dimensional and W-dimensional tensors. The shape of the tensor data is specifically represented as [n,h,w], where n, h and w are positive integers. Here, the N-dimensional tensor data represents the batch size n, that is, the number of data samples captured in one training session. The H-dimensional tensor represents the height h of each batch, and the W-dimensional tensor represents the width w of each batch.

[0033] For example, if the tensor data is an image, the specific value n of the N dimension is 1, the value h of the H dimension represents the height of the image, and the value w of the W dimension represents the width of the image.

[0034] Figure 2 is a schematic diagram of a tensor data structure. As shown in Figure 2, the data in the network model inference process consists of n batches, represented by multiple large rectangles in Figure 2: batch0, batch1, ..., batch(n-1). Each batch contains h × w elements. In Figure 2, one small cell within a batch represents one element; for example, one element can have a data width of 8 bits, 16 bits, etc. Here, n, h, and w are positive integers.

[0035] The data in the network model inference process is usually partitioned using a unified resource mapping logic to allocate the data inference tasks to different levels of computing resources for execution.

[0036] For example, currently, for a three-dimensional tensor, it is usually divided into N dimensions by a programmable multiprocessor, H dimensions by an execution unit, and W dimensions by each execution unit.

[0037] However, the three-dimensional tensor sizes involved in large-scale model inference vary greatly. A unified resource mapping logic can lead to some computational resources being idle under certain specific tensor sizes, thereby reducing overall utilization. For example, in the scenario above, if h is less than the total number of execution units in a programmable multiprocessor, some execution units will inevitably be idle, affecting global performance.

[0038] This disclosure provides at least one embodiment of a hardware computing resource allocation method, hardware computing resources, an electronic device, and a non-transitory computer-readable storage medium. The hardware computing resource allocation method includes: acquiring the data shape of an input tensor for a model computation task, wherein the input tensor is a three-dimensional tensor including N-dimensional, H-dimensional, and W-dimensional dimensions, and the data shape of the input tensor is [n, h, w], where N-dimensionality represents the batch size n of the input tensor, H-dimensionality represents the height h of each batch, and W-dimensionality represents the width w of each batch, and n, h, and w are positive integers; and acquiring hardware parameters of the hardware computing resources, wherein the hardware parameters include the number A of programmable multiprocessors and the number L of execution units included in each programmable multiprocessor. The input tensor is a positive integer equal to B × C. Based on the hardware parameters and data shape, a resource allocation scheme for the operator is determined. This scheme indicates how the input tensor is partitioned across multiple dimensions, and how the input tensor is allocated to hardware computing resources for processing according to the partitioning method. The input tensor is partitioned according to the resource allocation scheme, and the partitioned tensors are sent to the corresponding computing resources to execute model computation tasks. Elements belonging to the same row in the input tensor have data dependencies during model computation, and in the resource allocation scheme, elements in the same row are computed within the same computing unit.

[0039] This hardware computing resource allocation method can adaptively map computing resources, automatically determining which resource mapping logic to use based on the hardware parameters of the computing resources and the data shape of the input tensor. This results in more rational allocation of computing resources, minimizing idle computing resources and reducing waste, thereby improving system performance. Furthermore, when elements in the same row of the input tensor have data dependencies during model computation, the resource allocation scheme provided in this embodiment sets elements in the same row to perform model computation within the same computing unit during partitioning. This fully utilizes computing resources, avoids waste, and minimizes additional memory access overhead due to data dependencies, further improving system performance.

[0040] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0041] Figure 3 is a schematic flowchart of a hardware computing resource allocation method provided in at least one embodiment of this disclosure.

[0042] As shown in Figure 3, the hardware computing resource allocation method provided in at least one embodiment of this disclosure includes at least steps S10-S40.

[0043] For example, the hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, and each execution unit is configured to run one or more thread bundles, where A, B, and C are positive integers greater than 1.

[0044] For example, hardware computing resources can be graphics processing units (GPUs) or general-purpose GPUs. For instance, programmable multiprocessors, computing units, and execution units within hardware computing resources constitute a multi-layered computing resource. For a detailed description of programmable multiprocessors, computing units, and execution units, as well as their relationships, please refer to the relevant content in Figure 1 above; further details will not be elaborated here.

[0045] As shown in Figure 3, in step S10, the data shape of the input tensor of the operator for the model calculation task is obtained.

[0046] For example, the data format of the input tensor is [N, H, W], where N represents the number of batches of the input tensor, H represents the height of each batch, and W represents the width of each batch. The input tensor is represented as [n, h, w], where n, h, and w are positive integers. Here, n represents the specific number of batches of the input tensor, h represents the specific height of each batch in the input tensor, and w represents the specific width of each batch in the input tensor. For example, the input tensor can include n batches, and each batch includes h rows and w columns of elements.

[0047] For example, model computation tasks include model inference tasks, etc.

[0048] In step S20, the hardware parameters of the hardware computing resources are obtained.

[0049] For example, hardware parameters include the number of programmable multiprocessors A and the number of execution units L included in each programmable multiprocessor, where L is a positive integer and equal to B×C.

[0050] In step S30, the resource allocation scheme of the operator is determined based on the hardware parameters and data shape.

[0051] For example, the resource allocation scheme indicates how the input tensor is partitioned across multiple dimensions of the input tensor, and how the input tensor is allocated to hardware computing resources for processing based on the partitioning scheme.

[0052] In step S40, the input tensor is segmented according to the resource allocation scheme, and each segmented tensor is sent to the corresponding computing resources to perform the model calculation task.

[0053] For example, the corresponding computing resources here may include programmable multiprocessors, computing units, execution units, etc.

[0054] For example, elements in the same row of the input tensor have data dependencies when performing model calculations, and in the resource allocation scheme, elements in the same row perform model calculation tasks in the same computing unit.

[0055] For example, elements belonging to the same row exhibit data dependency representation during model computation. For one element in the same row, the inference result corresponding to that element is related to at least two other elements in the same row. This data dependency can be formed through operations such as reduction and accumulation; that is, the inference result corresponding to one element requires the interaction of multiple elements in the same row.

[0056] As mentioned earlier, each computing unit includes shared memory, and the C execution units within the computing unit exchange data through the shared memory.

[0057] Since elements in the same row have data dependencies when performing model calculations, they can exchange data through shared memory when performing model calculation tasks in the same computing unit. This eliminates the need for data exchange from memory or secondary cache, reducing memory access overhead, improving computational efficiency, and enhancing system performance.

[0058] The process of steps S30 and S40 in the hardware computing resource allocation method provided in at least one embodiment of this disclosure is described in detail below with reference to the accompanying drawings.

[0059] For example, step S30 may include: determining the resource allocation scheme of the operator based on the relationship between h and L, and the relationship between n and A×L.

[0060] For example, if h is greater than L, it is reasonable for the execution units to partition the H dimension. For example, in response to h being greater than or equal to L, the operator is determined to adopt a third resource allocation scheme. The third resource allocation scheme instructs that the input tensor be first partitioned along the N dimension, and the multiple tensors partitioned along the N dimension be allocated to A programmable multiprocessors for execution. Then, the tensors allocated to each programmable processor are partitioned along the H dimension, and the multiple tensors partitioned along the H dimension are allocated to B×C execution units in the programmable processors for execution. Each execution unit traverses the allocated tensors row by row in the order of the lookahead direction and the next column direction, using thread bundles as units.

[0061] Figure 4 is a schematic diagram of a third resource mapping scheme provided in at least one embodiment of this disclosure.

[0062] As shown in Figure 4, the input tensor includes batch0, batch1, ..., batch(n-1) for a total of n batches, and each batch includes h rows and w columns of elements.

[0063] For example, in the third resource mapping scheme, n batches are evenly distributed among A programmable multiprocessors, with each programmable multiprocessor processing n / A batches, which is treated as a fourth tensor. If they cannot be evenly distributed, the number of batches processed by each programmable multiprocessor is balanced as much as possible to achieve load balancing.

[0064] Each batch of the fourth tensor received by each programmable multiprocessor is divided into multiple fifth tensors along the H dimension, or a row of elements in the fourth tensor is treated as a fifth tensor, and each execution unit executes one fifth tensor. If there are multiple batches, they are executed sequentially. For example, a row of shaded elements in batch 0 of Figure 4 can be considered a fifth tensor. For example, each fifth tensor is traversed and executed within an execution unit. For an execution unit, the fifth tensor is traversed along the W dimension in units of thread bundles, for example, traversing row by row in the row-ahead direction and column-ahead direction, processing all elements of the first row first, then all elements of the second row, and so on, until the last row.

[0065] In this embodiment, the processing of the input tensor includes three loops: the outermost loop is a plurality of fourth tensors obtained by dividing the fourth tensor into N dimensions according to the number of programmable multiprocessors; the middle loop is a plurality of fifth tensors obtained by dividing the fourth tensor into H dimensions according to the number of execution units of each programmable processor; and the innermost loop is a loop that traverses the fifth tensor from the W dimension in units of thread bundles.

[0066] For example, in this embodiment, step S40 may include: dividing the input tensor into A fourth tensors in the N-dimensional plane, and sending the A fourth tensors to A programmable multiprocessors respectively, wherein each fourth tensor includes n / A batches; dividing the fourth tensor received by each programmable multiprocessor into multiple fifth tensors in the H-dimensional plane, and sending the multiple fifth tensors to the corresponding execution units, wherein each fifth tensor includes a row of elements or multiple consecutive rows of elements from the fourth tensor; setting each execution unit to traverse the received fifth tensors row by row in the order of the lookahead direction and the next column direction to perform the model calculation task.

[0067] For example, if h is less than L, using the third resource allocation scheme will result in some execution units being idle. For instance, in the decoding stage of the inference process, h = 1, and continuing to divide according to the previous scheme will inevitably lead to only one execution unit working. Therefore, different division and resource allocation methods can be set according to the relationship between n and A*L.

[0068] For example, in some embodiments, determining the resource allocation scheme of the operator based on the relationship between h and L, and the relationship between n and A×L, may include: in response to h being less than L and n being greater than or equal to A×L, determining that the operator adopts a first resource allocation scheme, wherein the first resource allocation scheme indicates that the input tensor is first divided in the N-dimensional direction, and the multiple tensors after being divided in the N-dimensional direction are allocated to A×L execution units for execution respectively, and each execution unit traverses the allocated tensors row by row in the order of the lookahead direction and the next column direction.

[0069] For example, the first resource allocation scheme instructs to divide the input tensor into A×L first tensors in N dimensions, and then allocate the A×L first tensors to A×L execution units for execution.

[0070] For example, in this embodiment, although the input tensor is also first segmented in N dimensions, the number of programmable multiprocessors is no longer used as the segmentation basis. Instead, all execution units in the entire hardware computing resources are regarded as a whole, and the input tensor is segmented according to the number of execution units, without distinguishing between programmable multiprocessors.

[0071] Figure 5 is a schematic diagram of a first resource mapping scheme provided in at least one embodiment of this disclosure.

[0072] As shown in Figure 5, the input tensor includes N batches, such as batch0, and each batch includes h rows and w columns of elements.

[0073] Referring to Figure 5, in the first resource mapping scheme, the input tensor is evenly divided into A×L first tensors along the N dimensions. Each first tensor includes n / (A×L) batches, meaning n batches are evenly distributed among A×L execution units. If they cannot be evenly distributed, the number of batches processed by each execution unit is balanced as much as possible to achieve load balancing.

[0074] In the first resource allocation scheme, each execution unit no longer executes one or more lines within a batch, but rather executes one or more batches. Each execution unit, operating on a thread-by-thread basis, traverses the allocated first tensor line by line in a lookahead-to-column direction. When the first tensor has multiple batches, each batch is traversed sequentially; for example, one batch is traversed before moving on to the next, with each batch still traversed line by line in a lookahead-to-column direction.

[0075] In this embodiment, the processing of the input tensor includes two loops: the outermost loop is to divide the input tensor into multiple first tensors based on the total number of execution units in the hardware computing resources and N dimensions; the innermost loop is to traverse the first tensor from the W dimensions in units of thread bundles.

[0076] For example, in this embodiment, step S40 may include: dividing the input tensor into A×L first tensors in N dimensions, and sending the A×L first tensors to A×L execution units respectively, wherein each first tensor includes n / (A×L) batches; setting each execution unit to traverse the received first tensors row by row in the order of the forward direction and the backward direction, in order to perform the model calculation task.

[0077] If h is less than L and n is less than A×L, since the number of execution units is greater than n, some execution units will still be idle if the first resource allocation scheme is still used.

[0078] For example, in other embodiments, determining the resource allocation scheme of the operator based on the relationship between h and L, and the relationship between n and A×L, may include: in response to h being less than L and n being less than A×L, determining that the operator adopts a second resource allocation scheme; wherein the second resource allocation scheme indicates that each batch in the input tensor is divided into m parts in the W dimension and allocated to m execution units for execution respectively, where m = min(,C), representing the floor function, and the m execution units belong to the same computation unit.

[0079] To make the most of the execution units, in the second resource allocation scheme, one batch is divided into multiple parts along the W dimension and allocated to multiple execution units for execution.

[0080] For example, model computation tasks in the same batch within the input tensor are executed within the same computation unit. That is, in this embodiment, the same batch is not executed across computation units; the C execution units within one computation unit can perform one or more batches. This is because a row of elements is assigned to m execution units for execution. In at least one embodiment of this disclosure, during the execution of model computation tasks for the input tensor, elements belonging to the same row in the input tensor have data dependencies. Therefore, the computation of one element requires elements from other positions. When a row of elements is executed in different execution units within the same computation unit, the execution units can exchange data through the shared memory of the computation unit. For example, they can obtain the required elements from shared memory without having to read them from memory or the secondary cache, reducing memory access overhead, improving execution efficiency, and enhancing system performance.

[0081] For example, in this embodiment, although the input tensor is also first segmented in N dimensions, the number of programmable multiprocessors or execution units is no longer used as the basis for segmentation. Instead, all computing units in the entire hardware computing resources are treated as a whole for segmentation.

[0082] Figure 6 is a schematic diagram of a second resource mapping scheme provided in at least one embodiment of this disclosure.

[0083] As shown in Figure 6, the input tensor includes batch0, batch1, ..., batch(n-1) for a total of n batches, and each batch includes h rows and w columns of elements.

[0084] For example, referring to Figure 6, in the second resource mapping scheme, the input tensor is divided into multiple second tensors in the N-dimensional space, with reference to the number of computing units A×B in the programmable multiprocessor.

[0085] The process of splitting the input tensor into multiple second tensors along the N-dimensional plane can include: splitting the input tensor into A×B second tensors along the N-dimensional plane in response to n being greater than or equal to A×B, wherein each second tensor includes n / (A×B) batches; and splitting the input tensor into n second tensors along the N-dimensional plane in response to n being less than A×B, wherein each second tensor includes 1 batch.

[0086] In this embodiment, one batch requires one computing unit to perform the model calculation task. Therefore, if n is greater than the total number of computing units in the hardware computing resources, the n batches are divided into A×B second tensors, and each second tensor includes n / (A×B) batches. If n is less than the total number of computing units in the hardware computing units, each computing unit still performs one batch, that is, the input tensor is divided into n second tensors in N dimensions, each second tensor includes one batch, and each computing unit executes one second tensor.

[0087] The second resource allocation scheme further instructs that each batch in the second tensor be divided into m equal parts along the W dimension and allocated to m execution units for execution. In this scheme, the number of execution units executing one batch is the floor function of the total number of execution units in the hardware computing resources divided by n. However, since a batch can be executed by a maximum of C execution units for the corresponding model computation task, if the floor function is greater than C, then m = C is still set.

[0088] For example, referring to Figure 6, m=4, which means that batch0 is divided into 4 equal parts along the W dimension, resulting in 4 third tensors. Each execution unit executes one third tensor. Each execution unit still uses thread bundles as units, traversing the received third tensors row by row in the order of the lookahead direction and the next column direction to perform the model computation task.

[0089] In this embodiment, the processing of the input tensor includes two loops: the outermost loop divides the input tensor into multiple second tensors by N dimensions based on the total number of computing units in the hardware computing resources, and each batch of each second tensor is divided into m third tensors along the W dimension; the innermost loop traverses the third tensors along the W dimension in units of thread bundles.

[0090] For example, in this embodiment, step S40 may include: dividing the input tensor into multiple second tensors in the N-dimensional plane, and sending the multiple second tensors to multiple computing units, wherein each computing unit receives one second tensor, and each second tensor includes at least one batch; dividing each batch of the second tensor into m third tensors in the W-dimensional plane, and sending the multiple third tensors obtained from the division of the second tensor to multiple execution units in the computing unit that received the second tensor, wherein each third tensor includes w / m rows and h columns; setting each execution unit to traverse the received third tensors row by row in the order of the lookahead direction and the column direction, in order to perform the model calculation task.

[0091] For example, suppose A*L = 256, h = 1 and less than L, A*B = 64, C = 4, and if n = 64, then m = 4. In this case, the second tensor includes one batch, and one computation unit executes the model computation task for one batch. Furthermore, each batch is divided into 4 parts along the W dimension, resulting in 4 third tensors. Each third tensor includes 1 row and w / 4 columns. The 4 execution units in this computation unit execute the model computation task for one third tensor respectively. For example, the execution unit traverses the third tensor along the W dimension in units of thread bundles.

[0092] If n = 128, then in this embodiment, m = 2. In this case, the second tensor includes two batches, and one computing unit executes the model calculation tasks for two batches. Furthermore, each batch is divided into two parts along the W dimension, resulting in two third tensors. Each third tensor includes one row and two columns. Two execution units in this computing unit execute the model calculation tasks for the two third tensors in one batch, while the other two execution units execute the model calculation tasks for the two third tensors in the other batch.

[0093] If n = 32, then m = 4. In this case, the second tensor includes one batch, and one computing unit performs one batch of model calculation tasks. The process is the same as that for n = 64, so it will not be described again here.

[0094] In the hardware resource allocation method provided in at least one embodiment of this disclosure, for input tensors of any data shape, the corresponding resource allocation scheme can be automatically and adaptively matched, and all execution units are utilized as evenly as possible under different data shapes, so as to avoid idle computing resources and improve system performance.

[0095] Taking the inference task of a large language model as an example, actual tests were conducted for h=1, n=32, 256, 512, 1024, and 2048. The hardware computing resource allocation method provided in at least one embodiment of this disclosure can improve the performance by 3-8 times.

[0096] At least one embodiment of this disclosure also provides a hardware computing resource. Figure 7 is a schematic structural diagram of the hardware computing resource provided in at least one embodiment of this disclosure.

[0097] For example, the hardware computing resources include A programmable multiprocessors (not shown in Figure 7), each programmable multiprocessor includes B computing units (not shown in Figure 7), each computing unit includes C execution units (not shown in Figure 7), and each execution unit is configured to run one or more thread bundles, where A, B, and C are positive integers greater than 1.

[0098] For example, the hardware computing resources are graphics processors or general-purpose graphics processors. The functions, structures and relationships of programmable multiprocessors, computing units and execution units can be referred to the relevant description in Figure 1 above, and will not be repeated here.

[0099] For example, as shown in Figure 7, the hardware computing resource 100 also includes a first acquisition module 101, a second acquisition module 102, a scheme determination module 103, and a resource allocation module 104. It should be noted that the components and structure of the hardware computing resource 100 shown in Figure 7 are merely exemplary and not restrictive; the hardware computing resource 100 may also have other components and structures as needed.

[0100] For example, these modules can be implemented through hardware (e.g., circuit) modules, software modules, or any combination of both. The following embodiments are the same and will not be described again. For example, these units can be implemented through a data processor (GPU), a general-purpose data processor (GPGPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, as well as corresponding computer instructions.

[0101] For example, the first acquisition module 101 is configured to acquire the data shape of the input tensor of the operator for the model calculation task, wherein the data format of the input tensor is [N,H,W], where N dimension represents the number of batches of the input tensor, H dimension represents the height of each batch, W dimension represents the width of each batch, and the input tensor is represented as [n,h,w], where n,h,w are positive integers.

[0102] The second acquisition module 102 is configured to acquire hardware parameters of hardware computing resources, wherein the hardware parameters include the number of programmable multiprocessors A and the number of execution units L included in each programmable multiprocessor, where L is a positive integer and equal to B×C.

[0103] The scheme determination module 103 is configured to determine the resource allocation scheme of the operator based on hardware parameters and data shape. The resource allocation scheme indicates the way the input tensor is divided in multiple dimensions of the input tensor, and the allocation method of allocating the input tensor to hardware computing resources for processing according to the division method.

[0104] The resource allocation module 104 is configured to segment the input tensor according to the resource allocation scheme and send the segmented tensor to the corresponding computing resources to execute the model calculation task.

[0105] For example, elements in the same row of the input tensor have data dependencies when performing model calculations, and in the resource allocation scheme, elements in the same row perform model calculation tasks in the same computing unit.

[0106] For example, the first acquisition module 101, the second acquisition module 102, the scheme determination module 103, and the resource allocation module 104 may include code and programs stored in memory; the processor may execute the code and programs to implement some or all of the functions of the first acquisition module 101, the second acquisition module 102, the scheme determination module 103, and the resource allocation module 104 as described above. For example, the first acquisition module 101, the second acquisition module 102, the scheme determination module 103, and the resource allocation module 104 may be dedicated hardware devices used to implement some or all of the functions of the first acquisition module 101, the second acquisition module 102, the scheme determination module 103, and the resource allocation module 104 as described above. For example, the first acquisition module 101, the second acquisition module 102, the scheme determination module 103, and the resource allocation module 104 may be a circuit board or a combination of multiple circuit boards used to implement the functions described above. In the embodiments of this application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) firmware executable by the processor and stored in memory.

[0107] It should be noted that the first acquisition module 101 can be used to implement step S10 shown in Figure 3, the second acquisition module 102 can be used to implement step S20 shown in Figure 3, the scheme determination module 103 can be used to implement step S30 shown in Figure 3, and the resource allocation module 104 can be used to implement step S40 shown in Figure 3. Therefore, for a detailed description of the functions that the first acquisition module 101 can perform, please refer to the relevant description of step S10 in the embodiments of the above-described hardware computing resource allocation method; for a detailed description of the functions that the second acquisition module 102 can perform, please refer to the relevant description of step S20 in the embodiments of the above-described hardware computing resource allocation method; for a detailed description of the functions that the scheme determination module 103 can perform, please refer to the relevant description of step S30 in the embodiments of the above-described hardware computing resource allocation method; and for a detailed description of the functions that the resource allocation module 104 can refer to the relevant description of step S40 in the embodiments of the above-described hardware computing resource allocation method. Repeated descriptions will not be repeated.

[0108] The hardware computing resources provided in at least one embodiment of this disclosure can adaptively and automatically match the corresponding resource allocation scheme for input tensors of any data shape, and make all execution units as evenly utilized as possible under different data shapes, thereby minimizing the idle computing resources and improving system performance.

[0109] Hardware computing resources 100 can achieve similar technical effects to the aforementioned hardware computing resource allocation methods, and will not be elaborated further here.

[0110] It should be noted that, in the embodiments of this disclosure, the hardware computing resource 100 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited; it can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0111] Figure 8 is a schematic diagram of a non-transitory computer-readable storage medium provided in at least one embodiment of the present disclosure. For example, as shown in Figure 8, the storage medium 200 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 201 can be stored non-transitoryly. For example, when the computer-readable instructions 201 are executed by a processor, one or more steps in the hardware computing resource allocation method described above can be performed.

[0112] For example, the storage medium 200 can be used in an electronic device 300, such as the storage medium 200 including the memory in the electronic device.

[0113] For example, the storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the data processor may execute the computer-readable instructions to implement various functions of the data processor. Various application programs and various data may also be stored in the storage medium.

[0114] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0115] Figure 9 is a schematic block diagram of an electronic device according to an embodiment of this disclosure. As shown in Figure 9, the electronic device 300 is, for example, suitable for implementing the hardware computing resource allocation method provided in the embodiment of this disclosure. It should be noted that the components of the electronic device 300 shown in Figure 9 are exemplary and not limiting; the electronic device 300 may have other components depending on the actual application requirements.

[0116] For example, as shown in FIG9, in some examples, electronic device 300 includes a processing device (e.g., a graphics processor, a general-purpose graphics processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 302 or a program loaded from storage device 308 into random access memory (RAM) 303 to achieve various functions, such as executing the hardware computing resource allocation method provided in at least one embodiment of the present disclosure.

[0117] For example, when the computer-readable instruction processing device 301 executes, it can perform one or more steps of the hardware computing resource allocation method according to any of the above embodiments. It should be noted that a detailed description of the processing procedure of the hardware computing resource allocation method can be found in the relevant descriptions in the embodiments of the above hardware computing resource allocation method, and repeated details will not be repeated here.

[0118] RAM 303 also stores various programs and data required for the operation of the computer system. Processing device 301, ROM 302, and RAM 303 are connected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0119] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 308 into RAM 303 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0120] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 9 shows electronic device 300 with various devices, it should be understood that it is not required to implement or possess all the devices shown, and electronic device 300 may alternatively implement or possess more or fewer devices. For example, processor 301 can control other components in electronic device 300 to perform desired functions. Processing device 301 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), tensor processor (TPU), or graphics processing unit (GPU). The central processing unit may be an x86 or ARM architecture, etc. The graphics processing unit may be directly integrated onto the motherboard or built into the northbridge chip of the motherboard. The graphics processing unit can also be built into the central processing unit.

[0121] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure). This peripheral interface can be various types of interfaces, such as a USB interface, a Lightning interface, etc. The communication device 309 can communicate wirelessly with a network and other devices, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0122] For example, electronic device 300 can be any device such as mobile phone, tablet computer, laptop computer, e-book, game console, television, digital photo frame, navigator, etc., or any combination of electronic devices and hardware. The embodiments disclosed herein do not limit this.

[0123] For example, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, the hardware computing resource allocation method defined in the embodiments of this disclosure is performed.

[0124] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0125] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0128] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0129] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0130] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0131] The following points should be noted regarding this disclosure:

[0132] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0133] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0134] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for allocating hardware computing resources, wherein the hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, and each execution unit is configured to run one or more thread bundles, where A, B, and C are all positive integers greater than 1. The hardware computing resource allocation method includes: Obtain the data shape of the input tensor for the model calculation task of the operator, wherein the input tensor is a three-dimensional tensor, including N-dimensional, H-dimensional and W-dimensional, and the data shape of the input tensor is [n,h,w]. The N-dimensional represents the batch number n of the input tensor, the H-dimensional represents the height h of each batch, the W-dimensional represents the width w of each batch, and n, h, and w are positive integers. Obtain the hardware parameters of the hardware computing resources, wherein the hardware parameters include the number of programmable multiprocessors A and the number of execution units L included in each programmable multiprocessor, where L is a positive integer and equal to B×C; Based on the hardware parameters and the data shape, a resource allocation scheme for the operator is determined, wherein the resource allocation scheme indicates the method of partitioning the input tensor along multiple dimensions of the input tensor, and the method of allocating the input tensor to the hardware computing resources for processing according to the partitioning method; and The input tensor is segmented according to the resource allocation scheme, and each segmented tensor is sent to the corresponding computing resources to execute the model calculation task. In this context, elements belonging to the same row in the input tensor have data dependencies when performing the model calculation task, and in the resource allocation scheme, the elements in the same row perform the model calculation task in the same computing unit.

2. The method of claim 1, wherein, Determining the resource allocation scheme for the operator based on the hardware parameters and the data shape includes: Based on the relationship between h and L, and the relationship between n and A×L, the resource allocation scheme of the operator is determined.

3. The method of claim 2, wherein, Based on the relationship between h and L, and the relationship between n and A×L, the resource allocation scheme of the operator is determined, including: In response to h being less than L and n being greater than or equal to A×L, it is determined that the operator adopts the first resource allocation scheme. The first resource allocation scheme instructs that the input tensor be first divided along the N-dimensional plane, and that the multiple first tensors obtained by the division along the N-dimensional plane be allocated to A×L execution units for execution respectively. Each execution unit traverses the allocated first tensors row by row in the order of the lookahead direction and the next column direction.

4. The method of claim 3, wherein, The first resource allocation scheme instructs that the input tensor be divided into A×L first tensors in the N dimensions, and that the A×L first tensors be allocated to the A×L execution units for execution, with each execution unit processing one first tensor.

5. The method of claim 3, wherein, The input tensor is segmented according to the resource allocation scheme, and the segmented tensors are sent to the corresponding computing resources to execute the model calculation task, including: The input tensor is divided into A×L first tensors along the N dimensions, and the A×L first tensors are sent to the A×L execution units respectively, wherein each first tensor includes n / (A×L) batches; and Each execution unit is configured to traverse the received first tensor row by row in the order of the forward direction and the backward direction, using the thread bundle as the unit, in order to perform the model calculation task.

6. The method of claim 2, wherein, Based on the relationship between h and L, and the relationship between n and A×L, the resource allocation scheme of the operator is determined, including: In response to h being less than L and n being less than A×L, it is determined that the operator adopts the second resource allocation scheme; The second resource allocation scheme indicates that each batch in the input tensor is divided into m parts in the W dimension and allocated to m execution units for execution respectively, where m = min(,C), which represents the rounding up function, and the m execution units belong to the same computing unit.

7. The method of claim 6, wherein, The model computation tasks in the same batch of input tensors are executed within the same computation unit.

8. The method of claim 6, wherein, The input tensor is segmented according to the resource allocation scheme, and the segmented tensors are sent to the corresponding computing resources to execute the model calculation task, including: The input tensor is divided into multiple second tensors along the N-dimensional plane, and the multiple second tensors are sent to multiple computing units, wherein each computing unit receives one second tensor, and each second tensor includes at least one batch; Each batch in the second tensor is divided into m third tensors along the W dimension, and the multiple third tensors obtained from the division of the second tensor are sent to multiple execution units in the computing unit that receives the second tensor, wherein each third tensor includes w / m rows and h columns of elements; and Each execution unit is configured to traverse the received third tensor row by row in the order of the forward direction and the backward direction, using the thread bundle as the unit, in order to perform the model calculation task.

9. The method of claim 8, wherein, The input tensor is divided into multiple second tensors along the N-dimensional plane, including: In response to n being greater than or equal to A×B, the input tensor is split into A×B second tensors along the N dimensions, wherein each second tensor comprises n / (A×B) batches; and In response to n being less than A×B, the input tensor is divided into n second tensors along the N-dimensional plane, wherein each second tensor comprises one batch.

10. The method of claim 2, wherein, Based on the relationship between h and L, and the relationship between n and A×L, the resource allocation scheme of the operator is determined, including: In response to h being greater than or equal to L, it is determined that the operator adopts the third resource allocation scheme. The third resource allocation scheme instructs that the input tensor be first divided along the N-dimensional plane, and the multiple tensors obtained by the division along the N-dimensional plane be allocated to the A programmable multiprocessors for execution respectively. Then, the tensor allocated to each programmable processor is divided along the H-dimensional plane, and the multiple tensors obtained by the division along the H-dimensional plane are allocated to the B×C execution units in the programmable processor for execution respectively. Each execution unit traverses the tensors allocated to the execution unit row by row in the order of the lookahead direction and the next column direction.

11. The method of claim 10, wherein, The input tensor is segmented according to the resource allocation scheme, and the segmented tensors are sent to the corresponding computing resources to execute the model calculation task, including: The input tensor is divided into A fourth tensors along the N dimensions, and the A fourth tensors are sent to the A programmable multiprocessors respectively, wherein each fourth tensor includes n / A batches; The fourth tensor received by each programmable multiprocessor is divided into multiple fifth tensors along the H dimension, and the multiple fifth tensors are sent to the corresponding execution units, wherein each fifth tensor includes one row of elements or multiple consecutive rows of elements from the fourth tensor; and Each execution unit is configured to traverse the received fifth tensor row by row in the order of the forward direction and the backward direction, using the thread bundle as the unit, in order to perform the model calculation task.

12. The method of any one of claims 1-11, wherein, Elements belonging to the same row exhibit data dependency representation when performing the model computation task. For one element belonging to the same row, the inference result corresponding to that element is related to at least two elements in the same row. Each computing unit includes shared memory, through which the C execution units within the computing unit exchange data.

13. A hardware computing resource comprising A programmable multiprocessors, each programmable multiprocessor comprising B computing units, each computing unit comprising C execution units, each execution unit configured to run one or more thread bundles, wherein A, B, and C are all positive integers greater than 1. The hardware computing resources also include: The first acquisition module is configured to acquire the data shape of the input tensor of the operator for the model calculation task, wherein the input tensor is a three-dimensional tensor, including N-dimensional, H-dimensional and W-dimensional, and the data shape of the input tensor is [n,h,w]. The N-dimensional represents the batch number n of the input tensor, the H-dimensional represents the height h of each batch, the W-dimensional represents the width w of each batch, and n, h and w are positive integers. The second acquisition module is configured to acquire the hardware parameters of the hardware computing resources, wherein the hardware parameters include the number of programmable multiprocessors A and the number of execution units L included in each programmable multiprocessor, where L is a positive integer and equal to B×C. The scheme determination module is configured to determine a resource allocation scheme for the operator based on the hardware parameters and the data shape, wherein the resource allocation scheme indicates the method of partitioning the input tensor along multiple dimensions of the input tensor, and the allocation method of distributing the input tensor to the hardware computing resources for processing according to the partitioning method; and The resource allocation module is configured to segment the input tensor according to the resource allocation scheme, and send each segmented tensor to the corresponding computing resources to execute the model calculation task; In this context, elements belonging to the same row in the input tensor have data dependencies when performing the model calculation task, and in the resource allocation scheme, the elements in the same row perform the model calculation task in the same computing unit.

14. An electronic device comprising: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, and each execution unit is configured to run one or more thread bundles. A, B, and C are all positive integers greater than 1. The computer-executable instructions are executed by the processor to perform the following steps: Obtain the data shape of the input tensor for the model calculation task of the operator, wherein the input tensor is a three-dimensional tensor, including N-dimensional, H-dimensional and W-dimensional, and the data shape of the input tensor is [n,h,w]. The N-dimensional represents the batch number n of the input tensor, the H-dimensional represents the height h of each batch, the W-dimensional represents the width w of each batch, and n, h, and w are positive integers. Obtain the hardware parameters of the hardware computing resources, wherein the hardware parameters include the number of programmable multiprocessors A and the number of execution units L included in each programmable multiprocessor, where L is a positive integer and equal to B×C; Based on the hardware parameters and the data shape, a resource allocation scheme for the operator is determined, wherein the resource allocation scheme indicates the method of partitioning the input tensor along multiple dimensions of the input tensor, and the allocation method of distributing the input tensor to the hardware computing resources for processing according to the partitioning method; and The input tensor is segmented according to the resource allocation scheme, and each segmented tensor is sent to the corresponding computing resources to execute the model calculation task. In this context, elements belonging to the same row in the input tensor have data dependencies when performing the model calculation task, and in the resource allocation scheme, the elements in the same row perform the model calculation task in the same computing unit.

15. The electronic device of claim 14, wherein, When the computer-executable instructions are executed by the processor, a resource allocation scheme for the operator is determined based on the hardware parameters and the data shape, including: Based on the relationship between h and L, and the relationship between n and A×L, the resource allocation scheme of the operator is determined.

16. The electronic device of claim 15, wherein, When the computer-executable instructions are executed by the processor, the resource allocation scheme of the operator is determined based on the relationship between h and L, and the relationship between n and A×L, including: In response to h being less than L and n being greater than or equal to A×L, it is determined that the operator adopts the first resource allocation scheme. The first resource allocation scheme instructs that the input tensor be first divided along the N-dimensional plane, and that the multiple first tensors obtained by dividing along the N-dimensional plane be allocated to A×L execution units for execution respectively. Each execution unit traverses the allocated first tensors row by row in the order of the lookahead direction and the next column direction.

17. The electronic device of claim 16, wherein, The first resource allocation scheme instructs that the input tensor be divided into A×L first tensors in the N dimensions, and that the A×L first tensors be allocated to the A×L execution units for execution, with each execution unit processing one first tensor.

18. The electronic device of claim 16, wherein, When the computer-executable instructions are executed by the processor, the input tensor is segmented according to the resource allocation scheme, and the segmented tensors are sent to the corresponding computing resources to execute the model calculation task, including: The input tensor is divided into A×L first tensors along the N dimensions, and the A×L first tensors are sent to the A×L execution units respectively, wherein each first tensor includes n / (A×L) batches; Each execution unit is configured to traverse the received first tensor row by row in the order of the forward direction and the backward direction, using the thread bundle as the unit, in order to perform the model calculation task.

19. The electronic device of claim 15, wherein, When the computer-executable instructions are executed by the processor, the resource allocation scheme of the operator is determined based on the relationship between h and L, and the relationship between n and A×L, including: In response to h being less than L and n being less than A×L, it is determined that the operator adopts the second resource allocation scheme; The second resource allocation scheme indicates that each batch in the input tensor is divided into m parts in the W dimension and allocated to m execution units for execution respectively, where m = min(,C), which represents the rounding up function, and the m execution units belong to the same computing unit.

20. A non-transitory computer-readable storage medium, wherein, The non-transitory computer-readable storage medium stores computer-executable instructions. The hardware computing resources include A programmable multiprocessors, each programmable multiprocessor includes B computing units, each computing unit includes C execution units, and each execution unit is configured to run one or more thread bundles. A, B, and C are all positive integers greater than 1. When the computer-executable instructions are executed by the processor, the following steps are performed: Obtain the data shape of the input tensor for the model calculation task of the operator, wherein the input tensor is a three-dimensional tensor, including N-dimensional, H-dimensional and W-dimensional, and the data shape of the input tensor is [n,h,w]. The N-dimensional represents the batch number n of the input tensor, the H-dimensional represents the height h of each batch, the W-dimensional represents the width w of each batch, and n, h, and w are positive integers. Obtain the hardware parameters of the hardware computing resources, wherein the hardware parameters include the number of programmable multiprocessors A and the number of execution units L included in each programmable multiprocessor, where L is a positive integer and equal to B×C; Based on the hardware parameters and the data shape, a resource allocation scheme for the operator is determined, wherein the resource allocation scheme indicates the method of partitioning the input tensor along multiple dimensions of the input tensor, and the method of allocating the input tensor to the hardware computing resources for processing according to the partitioning method; and The input tensor is segmented according to the resource allocation scheme, and each segmented tensor is sent to the corresponding computing resources to execute the model calculation task. In this context, elements belonging to the same row in the input tensor have data dependencies when performing the model calculation task, and in the resource allocation scheme, the elements in the same row perform the model calculation task in the same computing unit.

Images