HBM distribution method and system, electronic equipment, storage medium and product

Abstract

The embodiment of the invention discloses an HBM allocation method and system, electronic equipment, a storage medium and a product, and belongs to the technical field of computer storage. The method comprises the following steps: determining the total memory access amount of a calculation task and the number of calculation units of the calculation task; based on the total memory access amount and the number, determining the memory access amount of each calculation unit; determining at least one storage module corresponding to each calculation unit in the HBM, wherein the storage amount of the at least one storage module is greater than the access amount of the corresponding calculation unit; and in the at least one storage module, distributing a storage space for storing the access amount of the corresponding computing unit. According to the invention, equal memory access pressure can be realized, memory access conflicts are avoided, the bandwidth utilization rate is improved, and the memory access efficiency is optimized.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and more specifically, to a method, system, electronic device, storage medium, and product for allocating High Bandwidth Memory (HBM). Background Art

[0002] HBM is a high-performance 3D stacked dynamic random access memory (DRAM), which can be used in applications that require a large amount of data transfer and high bandwidth requirements, such as high-performance computing, graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), artificial intelligence accelerators, and high-end servers, and so on.

[0003] HBM usually contains multiple storage modules (for example, HBM banks). When accessing HBM, it is expected that the memory access pressure on each storage module is equal. Summary of the Invention

[0004] The present invention provides a method, system, electronic device, storage medium, and product for allocating HBM, which helps to achieve equal memory access pressure.

[0005] The technical solutions of the embodiments of the present invention are as follows:

[0006] A method for allocating HBM includes:

[0007] Determine the total memory access amount of a computing task and the number of computing units of the computing task;

[0008] Based on the total memory access amount and the number, determine the memory access amount of each computing unit;

[0009] Determine at least one storage module corresponding to each computing unit in the high bandwidth memory, where the storage amount of the at least one storage module is greater than the memory access amount of the corresponding computing unit;

[0010] In the at least one storage module, allocate a storage space for storing the memory access amount of the corresponding computing unit.

[0011] In one embodiment, the determining at least one storage module corresponding to each computing unit in the HBM includes:

[0012] When the memory access amount of a computing unit is less than or equal to the storage amount of a single storage module, the single storage module directly connected to the computing unit is determined as the storage module corresponding to the computing unit;

[0013] When the memory access amount of a computing unit is greater than the storage amount of a single storage module, the multiple storage modules directly connected to the computing unit are determined as the storage modules corresponding to the computing unit.

[0014] In one embodiment, the computing task is a computing task of an artificial intelligence application program, and the total memory access amount is characterized in storage units;

[0015] The determining the memory access amount of each computing unit based on the total memory access amount and the number includes:

[0016] Evenly distributing the total memory access amount characterized in storage units to each computing unit.

[0017] In one embodiment, the computing task is a computing task of an artificial intelligence application program, and the total memory access amount is characterized in a tensor shape;

[0018] The determining the memory access amount of each computing unit based on the total memory access amount and the number includes:

[0019] Evenly distributing the total memory access amount characterized in a tensor shape to each computing unit.

[0020] In one embodiment, the evenly distributing the total memory access amount characterized in a tensor shape to each computing unit includes:

[0021] Performing slicing on one or more selected dimensions according to the slicing granularity of each dimension of the tensor to obtain the number of tensor blocks after slicing of the one or more selected dimensions;

[0022] In response to the number of tensor blocks being divisible by the number of computing units, determining that the slicing of the one or more selected dimensions is the best slicing strategy that conforms to the best-effort manner;

[0023] In response to the number of tensor blocks not being divisible by the number of computing units, selecting one or more other dimensions to determine whether the number of tensor blocks after slicing of the selected one or more other dimensions is divisible by the number of computing units;

[0024] In response to the number of tensor blocks after slicing of the selected one or more other dimensions being divisible by the number of computing units, determining that the slicing of the selected one or more other dimensions is the best slicing strategy that conforms to the best-effort manner.

[0025] In one embodiment, evenly distributing the total memory access amount characterized by a tensor shape and data type to each computing unit includes:

[0026] In response to the fact that all selections in one or more of the respective dimensions cannot make the number of tensor blocks divisible by the number of computing units, determine the one or more dimensions that make the remainder of the number of tensor blocks divided by the number of computing units the largest or the number of computing units required by the number of tensor blocks the largest, and perform slicing to obtain the best slicing strategy in a best-effort manner.

[0027] In one embodiment, in the at least one storage module, allocating a storage space for storing the memory access amount of the corresponding computing unit includes:

[0028] When the at least one storage module is a single storage module directly connected to the computing unit, allocate a continuous storage space for storing the memory access amount of the corresponding computing unit starting from the starting address of the single storage module, where the tail region of the single storage module contains a blank area without storing data;

[0029] When the at least one storage module is multiple storage modules directly connected to the computing unit, allocate a continuous storage space for storing the memory access amount of the corresponding computing unit starting from the starting address of the overall storage space formed by the multiple storage modules, where the tail region of the overall storage space contains a blank area without storing data.

[0030] In one embodiment, it includes:

[0031] Store data associated with the memory access amount of the corresponding computing unit in the storage space of the at least one storage module.

[0032] In one embodiment, it includes:

[0033] When the at least one storage module is a single storage module directly connected to the computing unit, use the starting address of the single storage module and the memory access amount of the computing unit to obtain the data from the single storage module, and use the computing unit to perform calculations on the data;

[0034] When the at least one storage module is multiple storage modules directly connected to the computing unit, use the starting address of the overall storage space and the memory access amount of the computing unit to obtain the data from the multiple storage modules, and use the computing unit to perform calculations on the data.

[0035] In one embodiment, it includes:

[0036] When the at least one storage module is a single storage module directly connected to the computing unit, the data is obtained from the single storage module by using the starting address of the single storage module and the memory access amount of the computing unit, and the data is stored in the continuous storage space of the central processing unit;

[0037] When the at least one storage module is multiple storage modules directly connected to the computing unit, the data is obtained from the multiple storage modules by using the starting address of the overall storage space and the memory access amount of the computing unit, and the data is stored in the continuous storage space of the central processing unit.

[0038] A system for allocating HBM includes:

[0039] A central processing unit (CPU) for determining the total memory access amount of a computing task and the number of computing units of the computing task; determining the memory access amount of each computing unit based on the total memory access amount and the number;

[0040] A graphics processing module including HBM;

[0041] The CPU is further configured to determine at least one storage module corresponding to each computing unit in the HBM, where the storage amount of the at least one storage module is greater than the memory access amount of the corresponding computing unit; and allocate a storage space for storing the memory access amount of the corresponding computing unit in the at least one storage module.

[0042] In one embodiment, the graphics processing module is configured to store data associated with the memory access amount of the corresponding computing unit in the storage space of the at least one storage module.

[0043] In one embodiment, the graphics processing module is configured to, when the at least one storage module is a single storage module directly connected to the computing unit, obtain the data from the single storage module by using the starting address of the single storage module and the memory access amount of the computing unit, and perform calculations on the data by using the computing unit; when the at least one storage module is multiple storage modules directly connected to the computing unit, obtain the data from the multiple storage modules by using the starting address of the overall storage space and the memory access amount of the computing unit, and perform calculations on the data by using the computing unit.

[0044] In one embodiment, the graphics processing module is configured to, when the at least one storage module is a single storage module directly connected to the computing unit, obtain the data from the single storage module by using the starting address of the single storage module and the memory access amount of the computing unit, and store the data in the continuous storage space of the CPU; when the at least one storage module is multiple storage modules directly connected to the computing unit, obtain the data from the multiple storage modules by using the starting address of the overall storage space and the memory access amount of the computing unit, and store the data in the continuous storage space of the CPU.

[0045] An electronic device includes:

[0046] A memory;

[0047] A processor;

[0048] Wherein an application program executable by the processor is stored in the memory, and is used to cause the processor to execute the method for allocating HBM as described in any one of the above.

[0049] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the processor is caused to execute the method for allocating HBM as described in any one of the above.

[0050] A program product includes a computer program, and when the computer program is executed by a processor, the method for allocating HBM as described in any one of the above is implemented.

[0051] As can be seen from the above technical solutions, in the embodiments of the present invention, the total memory access amount of the computing task and the number of computing units of the computing task are determined; based on the total memory access amount and the number of computing units, the memory access amount of each computing unit is determined; at least one storage module corresponding to each computing unit is determined in the HBM, where the storage amount of the at least one storage module is greater than the memory access amount of the corresponding computing unit; in the at least one storage module, a storage space for storing the memory access amount of the corresponding computing unit is allocated. It can be seen that in the embodiments of the present invention, the storage amount of at least one storage module corresponding to the computing unit is greater than the memory access amount of the computing unit. Therefore, the computing unit does not need to access data from the storage modules corresponding to other computing units, avoiding the memory access conflict problem, reducing the demand for routers, and achieving equal memory access pressure as much as possible, improving the bandwidth utilization rate.

[0052] Moreover, in the embodiments of the present invention, by determining the storage module directly connected to the computing unit as the storage module corresponding to the computing unit, the communication cost is reduced, the performance of the on-chip network is improved, and the memory access efficiency is optimized.

[0053] In addition, the embodiment of the present invention can support the slicing of any dimension of the sliced tensor for tensor dimension slicing to be processed in parallel by multiple computing units, which can evenly distribute the memory access amount of each computing unit as much as possible, facilitating the equalization of the memory access pressure among storage modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 FIG. is a schematic flowchart of a method for allocating HBM according to an embodiment of the present invention.

[0055] Figure 2 FIG. is a first exemplary schematic diagram of allocating HBM according to an embodiment of the present invention.

[0056] Figure 3 FIG. is a second exemplary schematic diagram of allocating HBM according to an embodiment of the present invention.

[0057] Figure 4 FIG. is a schematic structural diagram of a system for allocating HBM according to an embodiment of the present invention.

[0058] Figure 5 FIG. is an exemplary schematic diagram of executing a computing task based on HBM according to an embodiment of the present invention.

[0059] Figure 6 FIG. is a schematic structural diagram of a device for allocating HBM according to an embodiment of the present invention.

[0060] Figure 7 FIG. is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0062] For the sake of simplicity and intuitiveness in description, the solutions of the present invention will be elaborated below by describing several representative embodiments. A large number of details in the embodiments are only used to help understand the solutions of the present invention. However, it is obvious that the technical solutions of the present invention can be implemented without being limited to these details. To avoid unnecessarily obscuring the solutions of the present invention, some embodiments are not described in detail but only present the framework. Hereinafter, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to only according to...". Due to the language habits of Chinese, when the number of a component is not specifically indicated below, it means that the component can be one or more, or can be understood as at least one.

[0063] HBM generally includes multiple memory modules. For example, a memory module can be implemented as an HBM bank, or an HBM group that includes multiple HBM banks. During the gradual evolution of HBM, each generation of technology significantly improves bandwidth, transmission rate, and capacity. For example, HBM can include products of each generation such as HBM1, HBM2, HBM2E, HBM3, and HBM3E. In fact, HBM technology may continue to evolve and progress. The HBM in the embodiments of the present invention can cover existing HBM products and can also cover HBM products that may appear in the future. The embodiments of the present invention have no limitation in this regard. In the related art, when allocating the memory resources of HBM to a computing unit (for example, a Stream Processor Cluster (SPC)), the memory access pressure situation between each memory module is not considered, and the situation where multiple computing units access the same memory module often occurs. At this time, multiple computing units need to queue up to perform memory access operations on the same memory module, while the remaining memory modules in HBM are in an idle state, resulting in memory access conflicts.

[0064] For example, assume that each bank in HBM has a storage space of 1024 bytes (B). A user needs to execute a computing task with a total memory access amount of 2000B. According to the related art, 1024B (from the 0th B to the 1023rd B) of the computing data of the computing task is stored in bank0, and the remaining 976B (from the 1024th B to the 1999th B) of the data is stored in bank2. Assume that computing unit 1 needs to access the computing data from the 0th B to the 999th B, and computing unit 2 needs to access the computing data from the 1000th B to the 1999th B. Then, computing unit 2 not only needs to access the 976B (from the 1024th B to the 1999th B) of the data in bank2, but also needs to access the remaining data (that is, the data from the 1000th B to the 1023rd B stored in bank1) from bank1 via a router. And computing unit 1 also needs to access data (from the 0th B to the 999th B) from bank1, so a memory access conflict occurs.

[0065] The inventors found that: when each memory module in HBM is evenly accessed, the nominal maximum bandwidth of HBM can be achieved. Otherwise, when a memory access conflict occurs, the bandwidth utilization rate of HBM is low and the performance is poor. Among them: the nominal maximum bandwidth refers to the maximum data transmission rate that HBM can achieve under ideal conditions. For example, for the first-generation HBM product (HBM1), the nominal maximum bandwidth is 128 GB / s, and for the second-generation HBM product (HBM2), the nominal maximum bandwidth is 256 GB / s.

[0066] In the embodiments of the present invention, when multiple computing units access the HBM, the access pressure of each memory module in the HBM is made as equal as possible to avoid access conflicts and achieve equal access pressure, thereby improving the bandwidth utilization rate of the HBM. Moreover, the corresponding memory module assigned to the computing unit is physically as close as possible to the corresponding computing unit, thereby improving the performance of the on-chip network and optimizing the access efficiency.

[0067] The above disclosure details the technical defects existing in the related art, the reasons for these technical defects, and the thinking and analysis process for overcoming these technical defects. In fact, the recognition of the above technical defects is not common knowledge in the art, but a novel discovery by the inventor in the research. In addition, the tracing of the reasons for the technical defects and the thinking and analysis process for overcoming the technical defects are also the step-by-step analysis results of the inventor in the actual research process, and are not common knowledge in the art.

[0068] Figure 1 It is a schematic flowchart of a method for allocating the HBM according to an embodiment of the present invention. Figure 1 The method shown can be executed by a processor. The processor can be implemented as any one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Tensor Processing Unit (TPU), a Neural Network Processing Unit (NPU), a Deep Learning Processing Unit (DPU), an Accelerated Processing Unit (APU), and a General-Purpose Computing on Graphics Processing Unit (GPGPU).

[0069] As Figure 1 shown, the method includes:

[0070] Step 101: Determine the total memory access volume of the computing task and the number of computing units of the computing task.

[0071] Here, the computing tasks can involve multiple fields and applications. For example, the computing tasks can be those in the field of artificial intelligence. Examples: The computing tasks can include: (1) Machine Learning: Supervised learning: Learning through a labeled dataset to predict the output; Unsupervised learning: Finding patterns and structures in unlabeled data; Reinforcement learning: Learning the optimal policy through interaction with the environment. (2) Deep Learning: Image recognition: Identifying and classifying objects in images; Natural Language Processing (NLP): Understanding and generating human language; Speech recognition: Converting speech into text or executing speech commands. (3) Data Mining: Classification: Assigning data to different categories; Clustering: Grouping similar data points; Association rule learning: Discovering associations between data items. (4) Computer Vision: Object detection: Identifying objects in images and determining their locations; Facial recognition: Identifying facial features in images; Scene understanding: Understanding the content of scenes in images. (5) Natural Language Processing (NLP): Language translation: Translating one language into another; Text summarization: Generating a short version of the text; Sentiment analysis: Determining the sentiment tendency of the text. (6) Robotics: Path planning: Planning a path for a robot from one point to another; Object manipulation: Teaching a robot how to grasp and move objects; Environment perception: Enabling a robot to understand and navigate its environment; (7) Recommendation Systems: Collaborative filtering: Recommending items based on user behavior; Content-based recommendation: Recommending items based on item features, etc. Computing tasks in the field of artificial intelligence usually require a large amount of data processing, complex algorithms, and high-performance computing resources.

[0072] Based on the specific type and task attributes of the computing task, the total memory access volume of the computing task can be determined. Moreover, the number of computing units that perform specific computing processes for the computing task is usually multiple (which can be a predetermined number), and multiple computing units can perform parallel computing for the computing task (for example, parallel computing of tensor convolution operations or matrix multiplication operations).

[0073] In one embodiment, the total memory access volume of the computing task is characterized in storage units. Here, the storage units can be bits, bytes, kilobytes, or megabytes, etc. For example, the total memory access volume is 2000B.

[0074] In one embodiment, the total memory access volume is characterized by a tensor shape. For example, the tensor shape is: [1, 3, 100], where the data type of the tensor can be float32, etc.

[0075] Step 102: Determine the memory access amount of each computing unit based on the total memory access amount and the number.

[0076] In one embodiment, the computing task is a computing task of an artificial intelligence application program, and the total memory access amount is characterized in storage units; Step 102 includes: evenly distributing the total memory access amount characterized in storage units to each computing unit. For example: when the total memory access amount characterized in storage units can be divided evenly by the number of computing units, evenly distribute the total memory access amount characterized in storage units to each computing unit; when the total memory access amount characterized in storage units cannot be divided evenly by the number of computing units, try to evenly distribute the total memory access amount characterized in storage units to each computing unit.

[0077] Considering the possible characteristic scenario where it is theoretically impossible to evenly distribute the total memory access amount to each computing unit completely, the evenly distributing the total memory access amount characterized in storage units to each computing unit described here should be understood as evenly distributing the total memory access amount characterized in storage units to each computing unit in a Best-effort manner.

[0078] Specifically, in the Best-effort manner, it includes:

[0079] (1) When the total memory access amount can be evenly distributed to each computing unit theoretically, evenly distribute the total memory access amount to each computing unit. For example, if the total memory access amount is 2000B and the number of computing units is 2, then each computing unit is allocated 1000B.

[0080] (2) When the total memory access amount cannot be evenly distributed to each computing unit theoretically, evenly distribute the total memory access amount to each computing unit in an as-even-as-possible manner. For example, assume the number of computing units is N and the total memory access amount Q cannot be divided evenly by N. The specific allocation process includes: calculating the quotient of the total memory access amount Q divided by N; rounding up or down the quotient to obtain the average memory access amount; allocating the average memory access amount to (N - 1) computing units; allocating the remaining memory access amount of Q to the remaining 1 computing unit. Example: Assume the total memory access amount is 2500B and the number of computing units is 3. First, calculate the floor integer of 2500 / 3, which is 833. Then, 2 computing units are allocated 833B, and the remaining 1 computing unit is allocated (2500 - 833*2) = 834B.

[0081] Example 1: Assume the total memory access amount is 2000B and the number of computing units is 2, it can be determined that the memory access amount of each computing unit is 1000B.

[0082] Example 2: Assume that the total memory access amount is 2000B and the number of computing units is 3. It can be determined that the memory access amount of 2 computing units is 666B, and the memory access amount of 1 computing unit is 668B.

[0083] In one embodiment, the computing task is a computing task of an artificial intelligence application program, and the total memory access amount is characterized in a tensor shape; step 102 includes: distributing the total memory access amount characterized in a tensor shape evenly to each computing unit. For example, distributing the total memory access amount characterized in a tensor shape evenly to each computing unit includes: performing slicing on one or more selected dimensions according to the slicing granularity of each dimension of the tensor to obtain the number of tensor blocks after slicing the one or more selected dimensions; in response to the number of tensor blocks being divisible by the number of computing units, determining that the slicing of the one or more selected dimensions is the best slicing strategy in line with the best-effort manner; in response to the number of tensor blocks not being divisible by the number of computing units, selecting one or more other dimensions to determine whether the number of tensor blocks after slicing the one or more other selected dimensions is divisible by the number of computing units; in response to the number of tensor blocks after slicing the one or more other selected dimensions being divisible by the number of computing units, determining that the slicing of the one or more other selected dimensions is the best slicing strategy in line with the best-effort manner.

[0084] It can be seen that it is possible to more accurately determine whether the slicing of one or more selected dimensions is the best slicing strategy according to the number of tensor blocks obtained after slicing each dimension of the tensor and the number of computing units.

[0085] Perform slicing on one or more selected dimensions according to the slicing granularity of each dimension of the tensor to obtain the number of tensor blocks after slicing the one or more selected dimensions. Here, the slicing granularity refers to the slicing step length when slicing a dimension, that is, the size of each dimension slice after slicing. For tensors with different layouts, the slicing granularity is set for each dimension of the tensor.

[0086] Example: Assume a three-dimensional tensor [4 (batch size), 4096 (height), 8192 (width)], and the number of computing units is 8192. If we select the height and width dimensions of the above three-dimensional tensor, then the size of each selected dimension is divided by the partitioning granularity of that dimension to obtain the number of dimension partitions for that dimension. The number of dimension partitions for the height dimension is 4096 / 64 = 64, and the number of dimension partitions for the width dimension is 8192 / 64 = 128. Partitioning the tensor along the height and width dimensions of the three-dimensional tensor will result in 64 * 128 = 8192 tensor blocks, and the size of each tensor block is [4, 64, 64]. Assume the number of computing units is 16, and if we select only the height dimension of the above three-dimensional tensor, then the size of the selected height dimension is divided by the partitioning granularity of that dimension to obtain the number of dimension partitions for that dimension. The number of dimension partitions for the height dimension is 4096 / 64 = 64. Then, partitioning the tensor along the height dimension of the three-dimensional tensor will result in 64 tensor blocks, and the size of each tensor block is [4, 64, 8192].

[0087] In one embodiment, evenly distributing the total memory access amount characterized by the tensor shape and data type to each computing unit includes: in response to the fact that all selections in one or more dimensions cannot make the number of tensor blocks divisible by the number of computing units, determining one or more dimensions that make the remainder of the number of tensor blocks divided by the number of computing units the largest or the number of computing units required for the number of tensor blocks the largest, and performing partitioning on these dimensions to obtain the best partitioning strategy in a best-effort manner.

[0088] For example, in the divisible case, partitioning the tensor [4, 4096, 8192] along the height and width dimensions of the three-dimensional tensor results in 64 * 128 = 8192 tensor blocks. If the number of computing units is 32, then 8192 divided by 32 is divisible, and the quotient is 256. Then, it is determined that partitioning along the height and width dimensions of the three-dimensional tensor is the best partitioning strategy. Here, each of the 32 computing units can be used to perform operator operations on 256 tensor blocks. In this way, each computing unit can process the same number of tensor blocks simultaneously, and no computing unit will be idle at a certain moment.

[0089] For example, in the case of non - divisibility, the tensor [4, 4096, 8192] is sliced into 64 * 128 = 8192 tensor blocks according to the two dimensions of the height and width of the three - dimensional tensor. Suppose the number of computing units is 36, and 8192 divided by 36 is not divisible. Consider choosing one or more other dimensions to determine whether the number of tensor blocks after slicing the selected one or more other dimensions can be divisible by the number of computing units. If it can, it is determined that slicing the selected one or more other dimensions is the optimal slicing strategy. And so on, until all selections of one or more dimensions in each dimension cannot make the number of tensor blocks divisible by the number of computing units, then it is determined that slicing one or more dimensions that make the remainder of the number of tensor blocks divided by the number of computing units the largest or the number of computing units required for the number of tensor blocks the most is the optimal slicing strategy.

[0090] The above exemplary description shows typical examples of evenly distributing the total memory access amount to each computing unit. Those skilled in the art can realize that such a description is only exemplary and is not used to limit the protection scope of the embodiments of the present invention.

[0091] Step 103: Determine at least one storage module corresponding to each computing unit in the HBM, where the storage capacity of at least one storage module is greater than the memory access amount of the corresponding computing unit.

[0092] In one embodiment, determining at least one storage module corresponding to each computing unit in the HBM includes:

[0093] (1) When the memory access amount of the computing unit is less than or equal to the storage capacity of a single storage module, the single storage module directly connected to the computing unit is determined as the storage module corresponding to the computing unit.

[0094] Example: Suppose each bank in the HBM has a storage space of 1024B. A user needs to execute a computing task with a total memory access amount of 2000B, and the number of computing units is 2 (including computing unit 1 and computing unit 2). It can be determined that the memory access amount of each computing unit is 1000B. At this time, the single storage module directly connected to computing unit 1 (for example, bank 1) is determined as the storage module corresponding to computing unit 1; the single storage module directly connected to computing unit 2 (for example, bank 2) is determined as the storage module corresponding to computing unit 2.

[0095] Figure 2 The first exemplary schematic diagram for allocating the HBM according to the embodiments of the present invention. In Figure 2Among them, the four memory modules (bank0 to bank1) directly connected to the four computing units (SPC0 to SPC3) respectively are memory modules corresponding to their respective computing units. For example, bank0 corresponds to SPC0, bank0 can provide a data storage space for SPC0, and the storage space of bank0 is greater than or equal to the memory access amount of SPC0. Therefore, SPC0 does not need to access data from other banks; bank1 corresponds to SPC1, bank1 can provide a data storage space for SPC1, and the storage space of bank1 is greater than or equal to the memory access amount of SPC1. Therefore, SPC1 does not need to access data from other banks; bank2 corresponds to SPC2, bank2 can provide a data storage space for SPC2, and the storage space of bank2 is greater than or equal to the memory access amount of SPC2. Therefore, SPC2 does not need to access data from other banks; bank3 corresponds to SPC3, bank3 can provide a data storage space for SPC3, and the storage space of bank3 is greater than or equal to the memory access amount of SPC3. Therefore, SPC3 does not need to access data from other banks.

[0096] (2) When the memory access amount of the computing unit is greater than the storage amount of a single memory module, multiple memory modules directly connected to the computing unit are determined as the memory modules corresponding to the computing unit.

[0097] Example: Assume that each bank in HBM has a storage space of 1024B. The user needs to execute a computing task with a total memory access amount of 4000B, and the number of computing units is 2 (including computing unit 1 and computing unit 2). It can be determined that the memory access amount of each computing unit is 2000B. At this time, the two memory modules directly connected to computing unit 1 (for example, bank 1 and bank2) are determined as the memory modules corresponding to computing unit 1; the two memory modules directly connected to computing unit 2 (for example, bank 3 and bank4) are determined as the memory modules corresponding to computing unit 2.

[0098] Figure 3 Is the first exemplary diagram for allocating HBM according to the embodiments of the present invention. In Figure 3Among them, the five memory modules (bank0 to bank4) directly connected to each of the four computing units (SPC0 to SPC3) are memory modules corresponding to their respective computing units. For example, bank0 corresponds to SPC0. Bank0 can provide a data storage space for SPC0. The storage space of bank0 is greater than or equal to the memory access amount of SPC0. Therefore, SPC0 does not need to access data from other banks; bank1 corresponds to SPC1. Bank1 can provide a data storage space for SPC1. The storage space of bank1 is greater than or equal to the memory access amount of SPC1. Therefore, SPC1 does not need to access data from other banks; bank2 corresponds to SPC2. Bank2 can provide a data storage space for SPC2. The storage space of bank2 is greater than or equal to the memory access amount of SPC2. Therefore, SPC2 does not need to access data from other banks; bank3 and bank4 correspond to SPC3. Bank3 and bank4 can provide a data storage space for SPC3. The storage space of bank3 and bank4 is greater than or equal to the memory access amount of SPC3. Therefore, SPC3 does not need to access data from other banks.

[0099] It can be seen that in the embodiment of the present invention, the storage capacity of at least one memory module corresponding to the computing unit is greater than the memory access amount of the computing unit. Therefore, the computing unit does not need to access data from the memory modules corresponding to other computing units, avoiding the memory access conflict problem, reducing the demand for routers, and achieving equal memory access pressure as much as possible, improving the bandwidth utilization rate.

[0100] Step 104: In at least one memory module, allocate a storage space for storing the memory access amount of the corresponding computing unit.

[0101] In one embodiment, step 104 includes:

[0102] (1) When at least one memory module is a single memory module directly connected to the computing unit, allocate a continuous storage space for storing the memory access amount of the corresponding computing unit starting from the starting address of the single memory module, where the tail area of the single memory module contains a blank area (padding area) that does not store data.

[0103] Take Figure 2 SPC0 in as an example for illustration. Bank0 directly connected to SPC0 provides a data storage space for SPC0. The data of SPC0 can be continuously stored starting from the starting position of the storage space of bank0. Since the storage capacity of bank0 is greater than the memory access amount of SPC0, the storage space of bank0 will contain a blank area that does not store data.

[0104] (2) When at least one storage module is multiple storage modules directly connected to a computing unit, allocate a continuous storage space starting from the starting address of the overall storage space composed of the multiple storage modules for storing the access amount of the corresponding computing unit, where the tail area of the overall storage space contains blank areas that do not store data.

[0105] Take Figure 3 SPC3 in

[0106] as an example. Bank3 and bank4 directly connected to SPC3 provide the storage space for data of SPC3. The data of SPC3 can be continuously stored starting from the starting position of the overall storage space of bank3 and bank4. Since the storage capacity of the overall storage space of bank3 and bank4 is greater than the access amount of SPC3, there will be blank areas that do not store data in the overall storage space of bank3 and bank4. Figure 1 In one embodiment,

[0107] the method shown further includes: storing data associated with the access amount of the corresponding computing unit in the storage space of at least one storage module. Here, store the data associated with the access amount of the corresponding computing unit (that is, the data that the corresponding computing unit can access, and its storage size conforms to the access amount of the corresponding computing unit) into the allocated storage space of at least one storage module. Therefore, storing the data associated with the access amount of the corresponding computing unit in the allocated storage space of at least one storage module realizes a data storage method without access conflicts. Figure 1

[0108] In one embodiment, (1) When at least one storage module is a single storage module directly connected to a computing unit, use the starting address of the single storage module and the access amount of the computing unit to obtain data from the single storage module, and use the computing unit to perform calculations on the data.

[0109] For example: Figure 2 Take SPC0 in

[0110] as an example. Bank0 directly connected to SPC0 provides the storage space for data of SPC0. The data equivalent to the access amount of SPC0 can be continuously obtained starting from the starting position of the storage space of bank0, which is the data that SPC0 needs to access. Then, SPC0 performs calculations on this data.

[0111] For example: Figure 3Taking SPC3 in [[]] as an example for illustration. Bank3 and Bank4 directly connected to SPC3 provide the overall storage space for data of SPC3. Starting from the starting position of the overall storage space of Bank3 and Bank4, data equivalent to the access amount of SPC3 can be continuously obtained, which is the data that SPC3 needs to access. Then, SPC3 performs calculations on this data.

[0112] It can be seen that data can be obtained from the storage module based on the starting address of the storage module and the access amount of the computing unit, and the computing unit can perform calculations on the data. For example, the specific calculations performed by the computing unit can be implemented as: convolutional forward activation propagation, convolutional backward weight propagation, convolutional backward weight propagation, matrix multiplication, and so on.

[0113] In one embodiment, Figure 1 The method shown includes:

[0114] (1) When at least one storage module is a single storage module directly connected to the computing unit, use the starting address of the single storage module and the access amount of the computing unit to obtain data from the single storage module and store the data in the continuous storage space of the central processing unit.

[0115] For example: Taking Figure 2 SPC0 in [[]] as an example for illustration. Bank0 directly connected to SPC0 provides the storage space for data of SPC0. Starting from the starting position of the storage space of Bank0, data equivalent to the access amount of SPC0 can be continuously obtained, which is the data that SPC0 needs to access. Then, the data that SPC0 needs to access is stored in the continuous storage space of the CPU (for example, the memory of the CPU), thereby realizing the restoration of the data in the HBM to the CPU.

[0116] (2) When at least one storage module is multiple storage modules directly connected to the computing unit, use the starting address of the overall storage space and the access amount of the computing unit to obtain data from the multiple storage modules and store the data in the continuous storage space of the CPU.

[0117] For example: Taking Figure 3 SPC3 in [[]] as an example for illustration. Bank3 and Bank4 directly connected to SPC3 provide the overall storage space for data of SPC3. Starting from the starting position of the overall storage space of Bank3 and Bank4, data equivalent to the access amount of SPC3 can be continuously obtained, which is the data that SPC3 needs to access. Then, the data that SPC3 needs to access is stored in the continuous storage space of the CPU (for example, the memory of the CPU), thereby realizing the restoration of the data in the HBM to the CPU.

[0118] The embodiment of the present invention also proposes a system for allocating HBM.Figure 4 A schematic structural diagram of a system for allocating HBM according to an embodiment of the present invention. As Figure 4 shown, the system 400 for allocating HBM includes: a CPU 401, configured to determine the total memory access amount of a computing task and the number of computing units of the computing task; based on the total memory access amount and the number, determine the memory access amount of each computing unit; a graphics processing module 402, including HBM; the CPU 401 is further configured to determine at least one storage module corresponding to each computing unit in the HBM, where the storage amount of the at least one storage module is greater than the memory access amount of the corresponding computing unit; and allocate a storage space for storing the memory access amount of the corresponding computing unit in the at least one storage module.

[0119] The graphics processing module 402 may be implemented as a GPU or a GPGPU. In one embodiment, the graphics processing module 402 is configured to store data associated with the memory access amount of the corresponding computing unit in the storage space of the at least one storage module.

[0120] In one embodiment, the graphics processing module 402 is configured to, when the at least one storage module is a single storage module directly connected to the computing unit, use the starting address of the single storage module and the memory access amount of the computing unit to obtain data from the single storage module, and use the computing unit to perform calculations on the data; when the at least one storage module is multiple storage modules directly connected to the computing unit, use the starting address of the overall storage space and the memory access amount of the computing unit to obtain data from the multiple storage modules, and use the computing unit to perform calculations on the data.

[0121] In one embodiment, the graphics processing module 402 is configured to, when the at least one storage module is a single storage module directly connected to the computing unit, use the starting address of the single storage module and the memory access amount of the computing unit to obtain data from the single storage module, and store the data in the continuous storage space of the central processing unit; when the at least one storage module is multiple storage modules directly connected to the computing unit, use the starting address of the overall storage space and the memory access amount of the computing unit to obtain data from the multiple storage modules, and store the data in the continuous storage space of the CPU (for example, the memory of the CPU).

[0122] Specifically, the system 400 for allocating HBM may be implemented as an artificial intelligence chip.

[0123] Based on the coordinated cooperation of multiple application programming interfaces (APIs), it is possible to implement the execution of computing tasks based on HBM.

[0124] Figure 5 A schematic diagram of executing a computing task based on HBM according to an embodiment of the present invention. In Figure 5Among them, API1 determines the total memory access volume of a computing task (such as tensor data in CPU memory) and the number of computing units that specifically execute the computing task. Based on the total memory access volume and the number of computing units, it determines the memory access volume of each computing unit, and determines at least one storage module corresponding to each computing unit in HBM, where the storage capacity of at least one storage module is greater than the memory access volume of the corresponding computing unit. API2 is used to store the tensor data of each computing unit into the storage module corresponding to the computing unit in HBM respectively (for example, starting from the starting address of the storage module, and the storage capacity is the memory access volume of the computing unit). This process can be called rearrangement. API3 obtains the data of each computing unit based on the starting address of the storage module corresponding to each computing unit and the memory access volume of each computing unit, and performs various predetermined calculations on the data. For example, the calculations performed by API3 can be implemented as: convolutional forward activation propagation, convolutional backward weight propagation, convolutional backward weight propagation, matrix multiplication, and so on. API4 obtains the data of each computing unit based on the starting address of the storage module corresponding to each computing unit and the memory access volume of each computing unit, and stores the data back into the CPU memory again.

[0125] Taking the tensor shape [1, 3, 100] stored in CPU memory, the computing unit is SPC, and the number of SPCs is 3 as an example, the above process can specifically include:

[0126] Step 1: Pass in the tensor shape [1, 3, 100] through API1, where the data type of the tensor is float32. API1 distributes the tightly arranged tensor to each SPC as evenly as possible to balance the memory access during subsequent calculations. Therefore, each SPC should calculate 100 float32 data, that is, the memory access volume of each SPC is 100 float32 data.

[0127] Step 2: API1 calculates the HBM partitioning scheme corresponding to SPC based on the bank capacity of HBM (for example, 1024B) and the memory access volume allocated to each SPC. Among them: each bank should store 100 float32 data. The starting address and the total capacity requirement (that is, 2048 * 4B) of the bank corresponding to each SPC (preferably the bank directly connected to the SPC) are passed to the user through the return value of API1.

[0128] Step 3: The user allocates the corresponding HBM space according to the total capacity requirement returned by API1, and sends the starting address of the bank corresponding to each SPC to API2.

[0129] Step 4: API2 reorders the tensors in the CPU memory according to the starting address of the bank corresponding to each SPC, so as to store the tensors corresponding to each SPC in their respective corresponding banks. There is a blank area on the bank corresponding to each SPC, in which no valid data is stored.

[0130] Step 5: The user calls API3 and sends it the starting address of the bank corresponding to each SPC and the memory access amount of each SPC. Based on the starting address and memory access amount of the bank corresponding to each SPC, API3 can read the corresponding data of each SPC from the bank corresponding to each SPC and perform operations on the corresponding data using each SPC.

[0131] Step 6: The user calls API4 and sends it the starting address of the bank corresponding to each SPC and the memory access amount of each SPC. Based on the starting address and memory access amount of the bank corresponding to each SPC, API4 can read the corresponding data of each SPC from the bank corresponding to each SPC, remove the blank area, and restore the densely arranged tensors in the CPU memory.

[0132] The embodiment of the present invention also provides a device for allocating HBM. Figure 6 FIG. 1 is an exemplary structural diagram of an apparatus for allocating HBM according to an embodiment of the present invention. Figure 6 As shown, the device 600 for allocating HBM includes: a first determination module 601, used to determine the total memory access amount of the computing task and the number of computing units of the computing task; a second determination module 602, used to determine the memory access amount of each computing unit based on the total memory access amount and the number; a third determination module 603, used to determine at least one storage module corresponding to each computing unit in the HBM, wherein the storage amount of at least one storage module is greater than the memory access amount of the corresponding computing unit; an allocation module 604, used to allocate storage space for storing the memory access amount of the corresponding computing unit in at least one storage module.

[0133] In one embodiment, the third determination module 603 is used to determine a single storage module directly connected to the computing unit as the storage module corresponding to the computing unit when the memory access amount of the computing unit is less than or equal to the storage amount of a single storage module; when the memory access amount of the computing unit is greater than the storage amount of a single storage module, determine multiple storage modules directly connected to the computing unit as the storage modules corresponding to the computing unit.

[0134] In one embodiment, the computing task is a computing task of an artificial intelligence application, and the total memory access amount is represented by storage units; the second determination module 602 is used to evenly distribute the total memory access amount represented by storage units to each computing unit.

[0135] In one embodiment, the computing task is a computing task of an artificial intelligence application program, and the total memory access amount is characterized in a tensor shape; a second determination module 602, configured to evenly distribute the total memory access amount characterized in the tensor shape to each computing unit.

[0136] In one embodiment, the second determination module 602 is configured to perform slicing on one or more selected dimensions according to the slicing granularity of each dimension of the tensor to obtain the number of tensor blocks after slicing the one or more selected dimensions; in response to the number of tensor blocks being divisible by the number of computing units, determine that slicing the one or more selected dimensions is the best slicing strategy that conforms to the best-effort manner; in response to the number of tensor blocks not being divisible by the number of computing units, select one or more other dimensions to determine whether the number of tensor blocks after slicing the one or more other selected dimensions is divisible by the number of computing units; in response to the number of tensor blocks after slicing the one or more other selected dimensions being divisible by the number of computing units, determine that slicing the one or more other selected dimensions is the best slicing strategy that conforms to the best-effort manner.

[0137] In one embodiment, the second determination module 602 is configured to, in response to the fact that all selections of one or more dimensions among all dimensions cannot make the number of tensor blocks divisible by the number of computing units, determine one or more dimensions that make the remainder of the number of tensor blocks divided by the number of computing units the largest or the number of computing units required by the number of tensor blocks the largest to perform slicing as the best slicing strategy that conforms to the best-effort manner.

[0138] In one embodiment, the allocation module 604 is configured to, when at least one storage module is a single storage module directly connected to the computing unit, allocate a continuous storage space starting from the starting address of the single storage module for storing the memory access amount corresponding to the computing unit, where the tail region of the single storage module includes a blank region that does not store data; when at least one storage module is multiple storage modules directly connected to the computing unit, allocate a continuous storage space starting from the starting address of the overall storage space formed by the multiple storage modules for storing the memory access amount corresponding to the computing unit, where the tail region of the overall storage space includes a blank region that does not store data.

[0139] In one embodiment, the apparatus 600 includes a storage module ( Figure 6 not shown in the figure). The storage module is configured to store data associated with the memory access amount of the corresponding computing unit in the storage space of at least one storage module.

[0140] In one embodiment, the apparatus 600 includes a computing module ( Figure 6(not shown in the figure). A calculation module, configured to, when at least one storage module is a single storage module directly connected to a computing unit, obtain data from the single storage module by using the starting address of the single storage module and the memory access amount of the computing unit, and perform calculations on the data by using the computing unit; when at least one storage module is a plurality of storage modules directly connected to the computing unit, obtain data from the plurality of storage modules by using the starting address of the overall storage space and the memory access amount of the computing unit, and perform calculations on the data by using the computing unit.

[0141] In one embodiment, the apparatus 600 includes a reduction module ( Figure 6 (not shown in the figure). The reduction module is configured to, when at least one storage module is a single storage module directly connected to a computing unit, obtain data from the single storage module by using the starting address of the single storage module and the memory access amount of the computing unit, and store the data in the continuous storage space of the central processing unit; when at least one storage module is a plurality of storage modules directly connected to the computing unit, obtain data from the plurality of storage modules by using the starting address of the overall storage space and the memory access amount of the computing unit, and store the data in the continuous storage space of the central processing unit.

[0142] In summary, in the embodiments of the present invention, the total memory access amount of the calculation task and the number of computing units of the calculation task are determined; based on the total memory access amount and the number of computing units, the memory access amount of each computing unit is determined; at least one storage module corresponding to each computing unit is determined in the HBM, where the storage amount of at least one storage module is greater than the memory access amount of the corresponding computing unit; in at least one storage module, a storage space for storing the memory access amount of the corresponding computing unit is allocated. It can be seen that in the embodiments of the present invention, the storage amount of at least one storage module corresponding to the computing unit is greater than the memory access amount of the computing unit. Therefore, the computing unit does not need to access data from the storage modules corresponding to other computing units, avoiding the memory access conflict problem, reducing the demand for routers, and achieving equal memory access pressure as much as possible, and improving the bandwidth utilization rate.

[0143] Moreover, in the embodiments of the present invention, by determining the storage module directly connected to the computing unit as the storage module corresponding to the computing unit, the communication cost is reduced, the performance of the on-chip network is improved, and the memory access efficiency is optimized.

[0144] In addition, the embodiments of the present invention can support the tensor dimension slicing of any dimension of the sliced tensor to be processed in parallel by multiple computing units, and can evenly distribute the memory access amount of each computing unit as much as possible, which is beneficial to the equal memory access pressure among the storage modules.

[0145] The embodiments of the present invention also propose an electronic device having a processor-memory architecture. Figure 7 is a structural diagram of an electronic device according to an embodiment of the present invention. AsFigure 7 As shown, the electronic device includes a processor 701, a memory 702, and a computer program stored on the memory 702 and executable on the processor 701. When the computer program is executed by the processor 701, it implements the method of allocating HBM as described above. Among them, the memory 702 can be specifically implemented as various storage media such as electrically erasable programmable read-only memory (EEPROM), flash memory, programmable read-only memory (PROM), etc. The processor 701 can be implemented as including one or more central processing units or one or more field-programmable gate arrays, where the field-programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or central processing unit core can be implemented as a CPU, GPU, GPGPU, MCU, or DSP, etc.

[0146] It should be noted that not all steps and modules in the above processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted according to needs. The division of each module is only for the convenience of description in terms of function. In actual implementation, one module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0147] The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module can include a specially designed permanent circuit or logic device (such as a dedicated processor, such as an FPGA or ASIC) for performing specific operations. For example, specific operations can be completed in various types of chips (such as artificial intelligence chips). A hardware module can also include a programmable logic device or circuit temporarily configured by software (such as including a general-purpose processor or other programmable processors) for performing specific operations. As for whether to specifically use a mechanical method, a dedicated permanent circuit, or a temporarily configured circuit (such as configured by software) to implement the hardware module, it can be determined based on cost and time considerations.

[0148] The present invention also provides a machine-readable storage medium storing instructions for causing a machine to execute the method as described in the present application. Specifically, a system or device equipped with a storage medium can be provided, on which software program codes for implementing the functions of any one of the above-described embodiments are stored, and the computer (or CPU or MPU) of the system or device is caused to read and execute the program codes stored in the storage medium. In addition, based on the instructions of the program codes, the operating system or the like operating on the computer can be caused to complete part or all of the actual operations. The program codes read from the storage medium can also be written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program codes, the CPU or the like installed on the expansion board or the expansion unit is caused to execute part and all of the actual operations, thereby implementing the functions of any one of the above-described embodiments. Embodiments of the storage medium for providing program codes include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program codes can be downloaded from a server computer or the cloud via a communication network.

[0149] In this document, "schematic" means "serving as an example, instance, or illustration", and any illustration or embodiment described as "schematic" in this document should not be construed as a more preferred or more advantageous technical solution. To simplify the drawings, only the parts related to the present invention are schematically shown in each drawing, and do not represent the actual structure of the product. Additionally, to simplify the drawings and facilitate understanding, in some drawings, for components having the same structure or function, only one of them is schematically shown, or only one of them is labeled. In this document, "a" does not mean that the quantity of the relevant part of the present invention is limited to "only one", and "a" does not exclude the case where the quantity of the relevant part of the present invention is "more than one". In this document, "upper", "lower", "front", "rear", "left", "right", "inner", "outer", etc. are only used to represent the relative positional relationship between relevant parts, rather than defining the absolute positions of these relevant parts.

[0150] The above are only the preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for allocating high bandwidth memory, characterized in that: include: Determining the total amount of memory access of a computing task and the number of computing units of the computing task; Determining a memory access amount of each computing unit based on the total memory access amount and the number; Determine at least one storage module corresponding to each computing unit in the high bandwidth memory, wherein the storage capacity of the at least one storage module is greater than the memory access capacity of the corresponding computing unit; In the at least one storage module, a storage space is allocated for storing the memory access amount of the corresponding computing unit.

2. The method according to claim 1, characterized in that The determining, in the high bandwidth memory, at least one storage module corresponding to each computing unit comprises: When the memory access amount of the computing unit is less than or equal to the memory amount of a single storage module, determining the single storage module directly connected to the computing unit as the storage module corresponding to the computing unit; When the memory access amount of the computing unit is greater than the storage amount of a single storage module, a plurality of storage modules directly connected to the computing unit are determined as storage modules corresponding to the computing unit.

3. The method according to claim 1, characterized in that The computing task is a computing task of an artificial intelligence application program, and the total amount of memory access is represented by a storage unit; The determining the memory access amount of each computing unit based on the total memory access amount and the number includes: The total amount of memory access represented by storage units is evenly distributed to each computing unit.

4. The method according to claim 1, characterized in that: The computing task is a computing task of an artificial intelligence application program, and the total amount of memory access is represented in a tensor shape; The determining the memory access amount of each computing unit based on the total memory access amount and the number includes: Distribute the total amount of memory access represented by the tensor shape evenly to each compute unit.

5. The method according to claim 4, characterized in that The step of evenly distributing the total amount of memory access represented by the tensor shape to each computing unit comprises: Splitting the selected one or more dimensions according to the segmentation granularity of each dimension of the tensor, to obtain the number of tensor blocks after the selected one or more dimensions are segmented; In response to the number of tensor blocks being divisible by the number of computing units, determining an optimal splitting strategy for splitting the selected one or more dimensions in a best-effort manner; In response to the number of tensor blocks not being divisible by the number of computing units, selecting one or more other dimensions to determine whether the number of tensor blocks after the selected one or more other dimensions are divided is divisible by the number of computing units; In response to the number of tensor blocks after the selected one or more other dimensions are split being divisible by the number of the computing units, an optimal splitting strategy for splitting the selected one or more other dimensions in a best-effort manner is determined.

6. The method according to claim 5, characterized in that The method of evenly distributing the total amount of memory access represented by the tensor shape and data type to each computing unit includes: In response to the fact that all selections of one or more of the dimensions cannot make the number of tensor blocks divisible by the number of computing units, determine the one or more dimensions that have the largest remainder after dividing the number of tensor blocks by the number of computing units or the largest number of computing units required for the number of tensor blocks to perform splitting in a best-effort manner. An optimal splitting strategy is then performed.

7. The method according to any one of claims 1 to 6, characterized in that In the at least one storage module, allocating storage space for storing the memory access amount of the corresponding computing unit includes: When the at least one storage module is a single storage module directly connected to the computing unit, a continuous storage space for storing the access amount of the corresponding computing unit is allocated starting from the start address of the single storage module, wherein the tail area of ​​the single storage module includes a blank area where no data is stored; When the at least one storage module is a plurality of storage modules directly connected to the computing unit, a continuous storage space for storing the memory access amount of the corresponding computing unit is allocated starting from the starting address of the overall storage space formed by the plurality of storage modules, wherein the tail area of ​​the overall storage space includes a blank area that does not store data.

8. The method according to claim 7, characterized in that include: Data associated with the memory access amount of the corresponding computing unit is stored in the storage space of the at least one storage module.

9. The method according to claim 8, characterized in that include: When the at least one storage module is a single storage module directly connected to the computing unit, the data is obtained from the single storage module using the starting address of the single storage module and the memory access amount of the computing unit, and the computing unit is used to perform calculations on the data; When the at least one storage module is a plurality of storage modules directly connected to the computing unit, the data is obtained from the plurality of storage modules using the starting address of the overall storage space and the memory access amount of the computing unit, and the computing unit is used to perform calculations on the data.

10. The method according to claim 8, characterized in that include: When the at least one storage module is a single storage module directly connected to the computing unit, the data is obtained from the single storage module by using the starting address of the single storage module and the memory access amount of the computing unit, and the data is stored in the continuous storage space of the central processing unit; When the at least one storage module is a plurality of storage modules directly connected to the computing unit, the data is obtained from the plurality of storage modules using the starting address of the overall storage space and the memory access amount of the computing unit, and the data is stored in the continuous storage space of the central processing unit.

11. A system for allocating high bandwidth memory, characterized in that: include: A central processing unit, used to determine the total amount of memory access of a computing task and the number of computing units of the computing task; Determining a memory access amount of each computing unit based on the total memory access amount and the number; a graphics processing module including a high bandwidth memory; The central processor is further used to determine at least one storage module corresponding to each computing unit in the high bandwidth memory, wherein the storage capacity of the at least one storage module is greater than the memory access capacity of the corresponding computing unit; In the at least one storage module, a storage space is allocated for storing the memory access amount of the corresponding computing unit.

12. The system according to claim 11, characterized in that The graphics processing module is used to store data associated with the memory access amount of the corresponding computing unit in the storage space of the at least one storage module.

13. The system according to claim 12, characterized in that The graphics processing module is used to, when the at least one storage module is a single storage module directly connected to the computing unit, obtain the data from the single storage module using the starting address of the single storage module and the memory access amount of the computing unit, and use the computing unit to perform calculations on the data; when the at least one storage module is a plurality of storage modules directly connected to the computing unit, obtain the data from the plurality of storage modules using the starting address of the overall storage space and the memory access amount of the computing unit, and use the computing unit to perform calculations on the data.

14. The system according to claim 13, characterized in that The graphics processing module is configured to, when the at least one storage module is a single storage module directly connected to the computing unit, obtain the data from the single storage module using the starting address of the single storage module and the memory access amount of the computing unit, and store the data in the continuous storage space of the central processing unit; When the at least one storage module is a plurality of storage modules directly connected to the computing unit, the data is obtained from the plurality of storage modules using the starting address of the overall storage space and the memory access amount of the computing unit, and the data is stored in the continuous storage space of the central processing unit.

15. An electronic device, characterized in that: include: Memory; processor; The memory stores an application program executable by the processor, which is used to enable the processor to execute the method for allocating high bandwidth memory according to any one of claims 1 to 10.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the processor is caused to perform the method for allocating high bandwidth memory according to any one of claims 1 to 10.

17. A program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for allocating high bandwidth memory according to any one of claims 1 to 10 is implemented.

Images