Tensor processing method, device, storage medium and program product

Abstract

Embodiments of the present disclosure provide a tensor processing method, a tensor processing device, an electronic device, a storage medium, and a program product. The tensor processing method includes: loading multiple tensor parts obtained by splitting a tensor to be calculated into a tensor buffer, wherein the size of each tensor part in the multiple tensor parts is less than or equal to the size of the tensor buffer; reading the loaded tensor parts in the multiple tensor parts from the tensor buffer to perform one or more operations respectively to realize the calculation of the tensor to be calculated. In this way, the tensor buffer can be fully utilized and the speed of accessing data can be uniform, and the operation pipeline can be better arranged, which makes full use of the performance advantages of the tensor buffer and improves the operation efficiency of the tensor.

Description

Technical Field

[0001] The present application relates to processor and chip technologies, and more particularly, to a tensor processing method, a tensor processing device, an electronic device, a non-transitory computer-readable storage medium, and a computer program product. Background Art

[0002] When a standard graphics processing unit (GPU) uses tensor data to perform computations such as those in neural network models, it typically reads this data from high-bandwidth memory (HBM). HBM utilizes three-dimensional (3D) stacked dynamic random access memory (DRAM) technology and is suitable for computing-intensive applications such as artificial intelligence (AI) neural networks. Summary of the Invention

[0003] According to one aspect of the present application, a tensor processing method is provided, comprising: loading a plurality of tensor parts obtained by splitting a tensor to be calculated into a tensor buffer, respectively, wherein the size of each tensor part in the plurality of tensor parts is less than or equal to the size of the tensor buffer; reading the loaded tensor parts in the plurality of tensor parts from the tensor buffer to respectively perform one or more operations to realize the calculation of the tensor to be calculated.

[0004] According to another aspect of the present application, a tensor processing device is provided, comprising: a loading device configured to respectively load multiple tensor parts obtained by splitting a tensor to be calculated into a tensor buffer, wherein the size of each tensor part in the multiple tensor parts is less than or equal to the size of the tensor buffer; and a reading device configured to read the loaded tensor parts in the multiple tensor parts from the tensor buffer to respectively perform one or more operations to realize the calculation of the tensor to be calculated.

[0005] According to another aspect of the present application, an electronic device is provided, including: a memory for storing computer instructions; a processor for reading the computer instructions in the memory and executing a method according to an embodiment of the present application.

[0006] According to another aspect of the present application, a non-transitory computer-readable storage medium is provided, on which computer instructions are stored, wherein when the computer instructions are executed by a processor, the processor is caused to perform a method according to an embodiment of the present application.

[0007] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, the processor is caused to perform the method according to the embodiment of the present application.

[0008] According to another aspect of the present application, an electronic device is provided, comprising: one or more streaming processor clusters, wherein each streaming processor cluster comprises multiple computing units and a tensor buffer, wherein each computing unit of the multiple computing units executes a method according to an embodiment of the present application.

[0009] According to various aspects of the present application, the tensor buffer can be fully utilized and the speed of accessing data can be uniform, and the operation pipeline can be better arranged, which makes full use of the performance advantages of the tensor buffer and improves the operation efficiency of the tensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A schematic diagram of an electronic device according to an embodiment of the present application is shown.

[0012] Figure 2 A flowchart of a tensor processing method according to an embodiment of the present application is shown.

[0013] Figure 3 A schematic diagram showing allocating tensors larger than the tensor buffer size to both the tensor buffer and high bandwidth memory.

[0014] Figure 4 A schematic diagram of the splitting and loading process of the tensor processing method according to an embodiment of the present application is shown.

[0015] Figure 5 An example flowchart of a tensor processing and execution method using a kernel function according to at least one embodiment of the present application is shown.

[0016] Figure 6 A block diagram of a tensor processing device according to an embodiment of the present application is shown.

[0017] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown.

[0018] Figure 8A schematic diagram of a non-transitory computer-readable storage medium according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] Reference will now be made in detail to specific embodiments of the present application, examples of which are illustrated in the accompanying drawings. Although the present application will be described in conjunction with specific embodiments, it will be understood that the present application is not intended to be limited to the described embodiments. On the contrary, it is intended to cover variations, modifications, and equivalents within the spirit and scope of the present application. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of the two.

[0020] Typically, tensors are stored in High Bandwidth Memory (HBM). The streaming processor cluster reads tensors from HBM to perform operations on them. HBM is a readable and writable memory.

[0021] To speed up the reading and writing of tensors, a tensor buffer was designed. This tensor buffer is a memory with faster read and write speeds than high-bandwidth memory. It is used to buffer tensors so that the stream processor cluster (SPC) can read and write the buffered tensors at high speed, thereby accelerating the computational efficiency of scenarios such as artificial intelligence neural networks. When the stream processor cluster performs operations on tensors, it reads tensors from high-bandwidth memory into the tensor buffer. In this way, the processor unit can read tensors directly from the tensor buffer to perform operations. From a function perspective, reading and writing tensor buffers behaves the same as reading and writing high-bandwidth memory, which does not inconvenience function writing, but speeds up reading and writing.

[0022] Figure 1 FIG. 1 is a schematic diagram of an electronic device 100 according to an embodiment of the present application.

[0023] like Figure 1As shown, the electronic device 100 includes one or more stream processor clusters 110. Data is transmitted between the stream processor cluster 110 and other global memories 150 via a bus. Each stream processor cluster includes multiple computing units 120, a high-bandwidth memory 140, and a tensor buffer 130. The multiple computing units (Control Units) 120 in the stream processor cluster 110 issue various control commands or microinstructions through an internal bus to control the operation of various internal components (including high-bandwidth memory and tensor buffers) and receive various data from various internal components. The computing unit 120 can perform traditional operations or various operations of the tensor processing method according to various embodiments of the present application. Specifically, the computing unit 120 can load tensors from the high-bandwidth memory 140 to the tensor buffer 130, and then the computing unit 120 can read directly from the tensor buffer 130 when performing operations on the tensors to speed up the reading speed and thus speed up the overall operation speed.

[0024] When using the tensor buffer, there may be one or more of the following problems: 1. The way to store tensors in the tensor buffer is not flexible, and the tensor buffer cannot be fully utilized; 2. The tensor buffer itself has a size limit, which is generally limited to 4MB. When the size of the tensor to be calculated is larger than the size limit of the tensor buffer, it is impossible to make full use of the tensor buffer for all tensors to be calculated; 3. The tensor needs to be configured to bind the tensor to the tensor buffer accordingly in order to use the tensor buffer to buffer the tensor, and the configuration must start from the 0 address of the tensor, that is, any part of the tensor cannot be bound, and when the tensor exceeds the size limit of the tensor buffer, only part of the tensor can be bound to the tensor buffer. The size of the part of the tensor configured to be bound to the tensor buffer cannot exceed the size limit of the tensor buffer itself; 4. Before using the bound tensor part, the bound tensor part is loaded from the high-bandwidth memory to the tensor buffer. When reading the address of the bound part of the bound tensor, the data on the tensor buffer is directly read instead of the high-bandwidth memory, but when storing tensors, it is also directly stored in the tensor buffer. Only when write-through is configured (Write Direct) These limitations lead to various issues when storing tensors in and reading from tensor buffers.

[0025] Figure 2 A flowchart of a tensor processing method according to an embodiment of the present application is shown.

[0026] like Figure 2 As shown, the tensor processing method includes the following steps:

[0027] Step 210 , respectively loading multiple tensor parts obtained by splitting the tensor to be calculated into the tensor buffer, wherein the size of each tensor part in the multiple tensor parts is less than or equal to the size of the tensor buffer.

[0028] Step 220 , read the loaded tensor parts among the multiple tensor parts from the tensor buffer to respectively perform one or more operations to realize the calculation of the tensor to be calculated.

[0029] In this way, the tensor is split into multiple tensor parts so that they can be loaded into the tensor buffer separately, which can make full use of the tensor buffer. When the tensor is operated on, the tensor parts of the tensor are read separately from the tensor buffer, instead of reading data from the tensor buffer and the high-bandwidth memory at the same time, so that the speed of accessing data is uniform and the operation pipeline can be better arranged, which makes full use of the performance advantages of the tensor buffer and improves the operation efficiency of the tensor.

[0030] The tensor processing method according to the embodiment of the present application can also solve the problem caused when the tensor to be calculated is larger than the size of the tensor buffer.

[0031] Figure 3 A schematic diagram is shown of allocating a tensor larger than the tensor buffer size to both the tensor buffer and the high bandwidth memory when the tensor to be calculated is larger than the tensor buffer size.

[0032] like Figure 3 As shown, the host stores a tensor used by the device in HBM. The full size of the tensor is significantly larger than the size of the tensor buffer. In this case, a portion of the tensor (less than a portion of the tensor buffer size) is typically allocated (or bound) to the tensor buffer starting at address 0 (see the dark shading on the tensor buffer). Note that address 0 of the tensor is only a relative address of the tensor and may correspond to a specific memory address in HBM. The device loads the allocated portion of the tensor from HBM to the tensor buffer, while the remaining portion remains in HBM (see the dark shading on HBM). Note that in this article, the host typically includes a central processing unit (CPU) and memory, while the device typically includes a graphics processing unit (GPU) and video memory (i.e., HBM).

[0033] In this case, when allocating a tensor larger than the tensor buffer's upper limit to the tensor buffer, only a portion of the tensor, starting from 0 and up to the tensor buffer's upper limit, can be allocated. This results in the entire tensor data being stored in two different memories with different read and write speeds: the faster tensor buffer and the slower high-bandwidth memory. Before a neural network model function can operate on tensor data, it must preload a portion of the data allocated to the tensor buffer into the tensor buffer, while the remaining portion remains in the high-bandwidth memory. This prevents symmetric programming and increases the difficulty of building general functions. In actual neural network models, repeated access to the slower high-bandwidth memory can also slow the overall execution of the function. Furthermore, simultaneous access to a tensor within the same function requires accessing two different memory speeds (the tensor buffer and the high-bandwidth memory), resulting in uneven data access speeds at different stages of a program containing various functions, increasing the cost of detailed programming and tuning.

[0034] Figure 4 A schematic diagram of the splitting and loading process of the tensor processing method according to an embodiment of the present application is shown.

[0035] In such Figure 2 In step 210 shown, multiple tensor parts obtained by splitting the tensor to be calculated are loaded into the tensor buffer respectively, wherein the size of each tensor part in the multiple tensor parts is less than or equal to the size of the tensor buffer.

[0036] Before loading, the tensor to be calculated can be split into multiple tensor parts.

[0037] like Figure 4 As shown, the original tensor full size (assuming 448 bits, just an example) is Figure 4 As shown in the light shaded area. The tensor is split into multiple tensor parts: the first tensor part, the second tensor part, the third tensor part, and the fourth tensor part. You can set the correspondence between each tensor part and the corresponding part of the tensor buffer to be loaded, so that you can flexibly set the position of each tensor part loaded into the tensor buffer, that is, you can not always load the tensor from the address 0 of the tensor, but also flexibly load it to various positions in the tensor buffer. For example, in Figure 4 In an embodiment of the present invention, the first tensor portion, the second tensor portion, and the third tensor portion are of the same size and are identical to at least a portion of the tensor buffer (see Figure 4 ), and the fourth tensor portion corresponds to at least another portion of the tensor buffer (see the portion indicated by the dotted line connected to the fourth tensor portion). The size of the tensor buffer is assumed to be 192 bits, and the range is {0, 191 bits}.

[0038] Here, the size of the tensor is larger than the size of the tensor buffer, and the tensor larger than the tensor buffer is split so that each split tensor part is no larger than the size of the tensor buffer.

[0039] Here, the respective tensor parts can completely overlap, partially overlap, or not overlap with the corresponding portion in the tensor buffer, and can be set to correspond to any portion in the tensor buffer. The specific correlation configuration can be determined by the user.

[0040] For example, the first tensor portion has a starting address of 0 and a size of 128 bits, that is, an address range of {0, 127 bits}. The first tensor portion may correspond to a portion of the tensor buffer having a starting address of 0 and a size of 128 bits.

[0041] Of course, a tensor part can also correspond to multiple parts of a tensor buffer, as long as the multiple parts of the tensor buffer can store the tensor part in total. In other words, the correspondence between tensor parts and tensor buffers can be flexible. In some examples, for simplicity, the tensor can be split evenly so that the multiple tensor parts are the same size. Of course, the tensor can also be arbitrarily split into multiple tensor parts of any size, as long as the split tensor parts can be operated on separately, so that each calculation on the tensor part is meaningful.

[0042] The second tensor portion has a starting address of 128 bits and a size of 128 bits, meaning the address range is {128 bits, 255 different}. The second tensor portion can also correspond to a portion of the tensor buffer with a starting address of 0 and a size of 128 bits. In other words, the first and second tensor portions can completely overlap within the corresponding portions of the tensor buffer. This is because the first tensor portion can be loaded into the tensor buffer once, and the second tensor portion can be loaded into the tensor buffer the next time, as long as two or more overlapping tensor portions are not loaded simultaneously, to avoid data conflicts or overwrites. Of course, the two tensor portions can also partially overlap or not overlap within the corresponding portions of the tensor buffer. If two (or more) tensor portions do not overlap within the corresponding portions of the tensor buffer, for efficiency reasons, they can be loaded simultaneously into the tensor buffer to maximize the use of the tensor buffer, allowing operations on both (or more) tensor portions to be performed simultaneously and improving computational efficiency.

[0043] The third tensor portion has a starting address of 256 bits and a size of 128 bits, that is, an address range of {256 bits, 383 bits}. The third tensor portion may correspond to a portion of the tensor buffer having a starting address of 0 and a size of 128 bits.

[0044] The fourth tensor portion has a starting address of 384 bits and a size of 64 bits, that is, an address range of {384 bits, 447 bits}. The fourth tensor portion may also correspond to a portion of the tensor buffer having a starting address of 128 bits and a size of 64 bits.

[0045] When loading multiple tensor parts obtained by splitting a tensor to be calculated, if the size of the tensor to be calculated is larger than the size of the tensor buffer, the multiple tensor parts obtained by splitting the tensor to be calculated are loaded in batches.

[0046] In one embodiment, step 210 of respectively loading multiple tensor parts obtained by splitting the tensor to be calculated into the tensor buffer may include: loading one or more tensor parts of the multiple tensor parts into the tensor buffer.

[0047] Reading loaded tensor parts from a plurality of tensor parts from a tensor buffer to respectively perform one or more operations to implement step 220 of calculating the tensor to be calculated may include: reading one or more loaded tensor parts from a tensor buffer, and performing one or more operations on the one or more tensor parts.

[0048] Then, step 210 of loading multiple tensor parts obtained by splitting the tensor to be calculated into the tensor buffer respectively may include: after performing one or more operations on one or more tensor parts, loading another one or more tensor parts of the multiple tensor parts into the tensor buffer.

[0049] Step 220 of reading loaded tensor parts from a plurality of tensor parts from a tensor buffer to respectively perform one or more operations to implement calculation of the tensor to be calculated may include: reading one or more additional loaded tensor parts from the tensor buffer, and performing one or more operations on the additional one or more tensor parts.

[0050] And so on, gradually loading, reading, and executing all the multiple tensor parts.

[0051] When loading, you can consider maximizing the use of the tensor buffer. For example, knowing which part of the tensor buffer each tensor part will be loaded into, you can consider selecting which part or parts to load at a time to fully utilize the size limit of the tensor buffer, so that the tensor parts loaded at a time can fill the tensor buffer as much as possible without conflicting with each other or overwriting each other.

[0052] In one embodiment, step 210 of loading multiple tensor parts obtained by splitting the tensor to be calculated into the tensor buffer respectively may include: selecting a combination of one or more tensor parts from the multiple tensor parts to load into the tensor buffer according to the correspondence between each tensor part and the corresponding part of the tensor buffer.

[0053] In one embodiment, the correspondence relationship can be represented by the starting address and size of each tensor part in the tensor to be calculated and the starting address and size of at least a corresponding part of the tensor buffer. For example, a mapping table can be used to represent the mapping relationship between the starting address and size of each tensor part in the tensor to be calculated and the starting address and size of at least a corresponding part of the tensor buffer. Of course, other types of methods can also be used to represent this correspondence relationship. The above-mentioned correspondence relationship can be recorded by the host and used when loading the tensor part.

[0054] Note that the correspondence can be one-to-one, i.e., one tensor part corresponds to a continuous range of addresses on the tensor buffer (called a portion), but it can also be one-to-many, i.e., one tensor part corresponds to multiple parts on the tensor buffer, each of which is a continuous range of addresses. This allows for more flexibility in specifying one or more locations on the tensor buffer to which each tensor part is loaded, thus adapting to more possible application scenarios.

[0055] This correspondence also allows the host to predict which locations each tensor component will be loaded into the tensor buffer, or to formulate loading plans for various purposes. For example, this can include determining which tensor components to load into the tensor buffer at a time, or in what order, such as which tensor components to load this time and which to load next. This can ensure that the loaded tensor components fill the tensor buffer as fully as possible without conflicting with or overwriting each other (or for other purposes). The loading plan can also load tensor components in the order of their original tensor data (e.g., for matrix multiplication, the left matrix tensor takes data by row, while the right matrix tensor takes data by column, etc.), or it can load tensor components out of the order of their original tensor data for other purposes (e.g., directly using loading to implement tensor transformation operations, such as transposition, or for example, for matrix multiplication, the left matrix tensor (which should have been transposed first) takes data by column (i.e., the corresponding transposed left matrix tensor takes data by row), while the right matrix takes data by column, etc.). In other words, this correspondence and loading plan can be relatively flexible.

[0056] In one embodiment, based on the corresponding relationship, the corresponding portions of the tensor buffer corresponding to the combinations to be loaded into the tensor buffer at once do not overlap in the tensor buffer. Consider loading as many combinations of one or more tensor portions as possible at a time while ensuring that the portions loaded into the tensor buffer do not overlap, thereby maximizing the utilization of the tensor buffer. This allows the tensor portions loaded at a time to fill the tensor buffer as much as possible without conflicting with or being overwritten, thereby improving loading and execution efficiency.

[0057] Figure 4 The first to third tensor parts are shown as overlapping corresponding parts of the corresponding tensor buffers, so that any two (or more) of these three tensor parts are not loaded into the tensor buffer at the same time, otherwise the loading will cause data conflicts or overwrites. However, this is not a limitation, and at least two tensor parts that do not overlap can also be designed after the tensor buffer, so that the at least two non-overlapping tensor parts can be loaded at once for calculation to speed up the calculation efficiency.

[0058] That is, the first tensor portion corresponds to the first portion of the tensor buffer, and the second tensor portion corresponds to the second portion of the tensor buffer. The first and second portions overlap, and the first and second tensor portions are not loaded into the tensor buffer at the same time. This avoids conflicts when loading into the tensor buffer at the same time.

[0059] Furthermore, the first tensor portion corresponds to the first portion of the tensor buffer, and the fourth tensor portion corresponds to the fourth portion of the tensor buffer. The first and fourth portions do not overlap, and the first and fourth tensor portions are simultaneously loaded into the tensor buffer to perform operations on the first and fourth tensor portions. This ensures that the tensor portions loaded at once fill the tensor buffer as fully as possible without conflicting or overwriting each other, while also improving the efficiency of tensor operations.

[0060] The tensor parts obtained by the above split can also be called view tensors, which are regarded as virtual tensors defined by the starting address and size.

[0061] In one embodiment, the tensor to be calculated is split into multiple tensor parts, each of which can be independently operated on. This makes it meaningful when the multiple split tensor parts are loaded separately into the tensor buffer and then operated on by the function, and will not be completely incomplete. For example, the tensor part required for a matrix multiplication is generally a matrix size of N*M (where N and M are positive integers), rather than a non-matrix size of N*M-1.

[0062] Performing an operation on one or more tensor portions may also include performing an operation using a kernel function using (data from) the one or more tensor portions as parameters to the kernel function. Kernel functions are commonly used in neural network operations. The specific data of the tensor portion used as a parameter can be determined from the tensor using the starting address and size mentioned above.

[0063] After completing the loading of a portion of the tensor from the high-bandwidth memory to the tensor buffer, operations are performed on that portion of the tensor to "consume" (i.e., use) the data. After consuming this data, the remaining portions of the tensor are loaded from the high-bandwidth memory to the tensor buffer for operations and consumption, and so on, until all portions of the tensor in the high-bandwidth memory have been operated on and consumed.

[0064] In this way, the kernel function can automatically complete the calculation of multiple tensor parts. In the kernel function logic, when tensors that need to configure tensor buffers need to be read and written repeatedly, the performance improvement effect of fully utilizing the high-speed tensor buffer is more significant.

[0065] Note that all tensor parts can be used together as multiple parameters of the kernel function, and each time the kernel function uses one or more of these parameters to load one or more tensor parts into the tensor buffer in order to consume the corresponding one or more tensor parts for calculation. In this way, when all parameters have been used, it means that all tensor parts have been consumed and other operations of the kernel function can continue to be executed.

[0066] Splitting the tensor into multiple tensor parts so that they can be loaded into the tensor buffer separately can make full use of the tensor buffer. When the tensor is operated on, the tensor parts of the tensor are read separately from the tensor buffer instead of reading data from the tensor buffer and the high-bandwidth memory at the same time. This makes the data access speed uniform and can better arrange the operation pipeline. This makes full use of the performance advantages of the tensor buffer and improves the operation efficiency of the tensor.

[0067] Figure 5 An example flowchart of a tensor processing and execution method using a kernel function according to at least one embodiment of the present application is shown.

[0068] In step 501 , a tensor in a high bandwidth memory is split into a plurality of tensor parts.

[0069] In step 502, multiple view tensors are constructed using pointers to the starting addresses of the multiple split tensor parts. In this document, view tensors and tensor parts are interchangeably referred to.

[0070] In step 503 , all view tensors are mapped to at least a portion of the tensor buffer.

[0071] At step 504 , a kernel function is started using data of unused view tensors among all view tensors as parameters.

[0072] At step 505 , data of view tensors that are not used and do not overlap in the parameters of the kernel function are loaded from the high bandwidth memory to the tensor buffer.

[0073] At step 506 , view tensor related operations are performed.

[0074] At step 507 , is it determined whether all view tensors serving as parameters of the kernel function are used? If the determination at step 507 is no, the process returns to step 505 to continue loading data of unused and non-overlapping view tensors in the kernel function parameters into the tensor buffer.

[0075] If the determination in step 507 is yes, then in step 508 , other operations of the kernel function (eg, bias summation independent of the loaded tensor, etc.) are continued to be performed, and the operation of the kernel function is completed.

[0076] In step 509, are all view tensors used for all kernel functions determined? If the determination in step 509 is no, the process returns to step 504 and uses the unused portions of all view tensors as parameters to initiate other kernel functions. If the determination in step 509 is yes, in step 510, other operations (e.g., operations of other kernel functions not related to tensors) are performed. In other words, once a view tensor is loaded into the tensor buffer, all kernel functions that require operations on that view tensor can be executed without repeatedly loading the view tensor.

[0077] In this way, the tensor is split into multiple tensor parts so that they can be loaded into the tensor buffer separately, which can make full use of the tensor buffer. When the tensor is operated on, the tensor parts of the tensor are read separately from the tensor buffer, instead of reading data from the tensor buffer and the high-bandwidth memory at the same time, so that the speed of accessing data is uniform and the operation pipeline can be better arranged, which makes full use of the performance advantages of the tensor buffer and improves the operation efficiency of the tensor.

[0078] Figure 6 FIG. 6 is a block diagram of a tensor processing device 600 according to an embodiment of the present application.

[0079] like Figure 6 As shown, the tensor processing device 600 includes a loading device 610 and a reading device 620 .

[0080] The loading device 610 can be configured to load multiple tensor parts obtained by splitting the tensor to be calculated into the tensor buffer respectively, wherein the size of each tensor part in the multiple tensor parts is less than or equal to the size of the tensor buffer.

[0081] The reading device 620 may be configured to read loaded tensor parts from the tensor buffer to respectively perform one or more operations to implement calculation of the tensor to be calculated.

[0082] In this way, the tensor is split into multiple tensor parts so that they can be loaded into the tensor buffer separately, which can make full use of the tensor buffer. When the tensor is operated on, the tensor parts of the tensor are read separately from the tensor buffer, instead of reading data from the tensor buffer and the high-bandwidth memory at the same time, so that the speed of accessing data is uniform and the operation pipeline can be better arranged, which makes full use of the performance advantages of the tensor buffer and improves the operation efficiency of the tensor.

[0083] In one embodiment, the loading device 610 is configured to load one or more tensor parts from the plurality of tensor parts into a tensor buffer.

[0084] The reading device 620 is configured to read the loaded one or more tensor parts from the tensor buffer and perform one or more operations on the one or more tensor parts.

[0085] The loading device 610 is configured to: after performing one or more operations on the one or more tensor parts, load another one or more tensor parts from the plurality of tensor parts into the tensor buffer.

[0086] The reading device 620 is configured to read the loaded additional tensor part(s) from the tensor buffer and perform one or more operations on the additional tensor part(s).

[0087] The loading device 610 is configured to select a combination of one or more tensor parts from the plurality of tensor parts to load into the tensor buffer according to a correspondence between each tensor part and a corresponding part of the tensor buffer.

[0088] In one embodiment, the correspondence is represented by the starting address and size of each tensor part in the tensor to be calculated and the starting address and size of at least a corresponding part of the tensor buffer.

[0089] In one embodiment, according to the corresponding relationship, corresponding parts of the tensor buffer corresponding to the combination do not overlap on the tensor buffer to maximize the utilization of the tensor buffer.

[0090] The device 600 may further include a splitting device (not shown) configured to split the tensor to be calculated into multiple tensor parts.

[0091] In one embodiment, the tensor to be calculated is split into multiple tensor parts, each of which can be operated on independently.

[0092] In one embodiment, the tensor to be calculated is split into multiple tensor parts of the same size, or the tensor to be calculated is split into multiple tensor parts of any size.

[0093] In one embodiment, the reading device 620 is configured to: read the loaded tensor parts of the multiple tensor parts from the tensor buffer by the kernel function using the multiple tensor parts as parameters of the kernel function to perform one or more operations respectively to realize the calculation of the tensor to be calculated.

[0094] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown.

[0095] The electronic device may include a processor 710 and a storage medium 720 , wherein the storage medium 720 is coupled to the processor 710 and stores computer-executable instructions therein for performing the steps of the various methods of the embodiments of the present application when executed by the processor.

[0096] The processor 710 may include, but is not limited to, one or more processors or microprocessors.

[0097] The storage medium 720 may include, but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0098] In addition, the electronic device may also include (but is not limited to) a data bus 730 , an input / output (I / O) bus 740 , a display 750 , and input / output devices 760 (eg, keyboard, mouse, speaker, etc.).

[0099] The processor 710 may communicate with an external display 750 and an input / output device 760 through the I / O bus 740 via a wired or wireless network (not shown).

[0100] The storage medium 720 may also store at least one computer-executable instruction for performing various functions and / or method steps in the embodiments described in this technology when executed by the processor 710 .

[0101] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a computer program product or software product, wherein one or more computer executable instructions, when executed by a processor, perform the various functions and / or method steps in the embodiments described in the present technology.

[0102] Figure 8 A schematic diagram of a non-transitory computer-readable storage medium according to an embodiment of the present application is shown.

[0103] like Figure 8 As shown, instructions are stored on a non-transitory computer-readable storage medium 820, such as computer instructions 810. When the computer instructions 810 are executed by a processor, the various methods described above can be performed. Non-transitory computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-transitory non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. For example, the non-transitory computer-readable storage medium 820 can be connected to a computing device such as a computer, and then, when the computing device executes the computer instructions 810 stored on the computer-readable storage medium 820, the various methods described above can be performed.

[0104] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are intended to be illustrative examples only and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and may be used interchangeably therewith. The words "such as / for example" used herein refer to the phrase "such as / for example but not limited to," and may be used interchangeably therewith.

[0105] The step flow charts and the above method descriptions in this disclosure are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter," "then," "next," and the like are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to an element in the singular, such as using the articles "a," "an," or "the," is not to be construed as limiting the element to the singular.

[0106] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, according to the concept of this application, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of this application.

[0107] The methods disclosed herein include steps for implementing the described methods. The methods and / or steps may be interchangeable with one another without departing from the scope of the claims. In other words, unless a specific order of steps is specified, the order and / or use of the specific steps may be modified without departing from the scope of the claims.

[0108] The above methods can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the methods can be stored as instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, wherein disks typically reproduce data magnetically, while discs reproduce data optically using lasers.

[0109] Therefore, the present disclosure may also include computer program products that can perform the methods, steps, and operations described herein. For example, such a computer program product can be a computer software package, computer code instructions, or a computer-readable tangible medium having computer instructions tangibly stored (and / or encoded) thereon, the instructions being executable by a processor to perform the operations described herein. The computer program product can include packaging materials.

[0110] In addition, modules and / or other appropriate means for performing the methods and techniques described herein can be downloaded from a server via wireless communication, as appropriate. Alternatively, the various methods described herein can be provided via a storage component so that the various methods are available when coupled to the storage component. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0111] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A tensor processing method, comprising: Splitting a tensor to be calculated into a plurality of tensor parts, wherein a size of each of the plurality of tensor parts is smaller than a size of a tensor buffer, wherein the tensor to be calculated is split into a plurality of tensor parts each capable of performing operations independently, wherein each of the plurality of tensor parts is a virtual tensor defined by a starting address and a size; Respectively loading a plurality of tensor parts obtained by splitting a tensor to be calculated into a tensor buffer, comprising: selecting a combination of a plurality of tensor parts from the plurality of tensor parts according to a correspondence between each tensor part and a corresponding part of the tensor buffer, and loading the combination into the tensor buffer at one time, wherein, according to the correspondence, corresponding parts of the tensor buffer corresponding to the combination do not overlap on the tensor buffer, wherein the correspondence is represented by a starting address and a size of each tensor part in the tensor to be calculated and a starting address and a size of at least a corresponding part of the tensor buffer, wherein the correspondence comprises one tensor part corresponding to a plurality of parts on the tensor buffer, so that the one tensor part is to be loaded into a plurality of locations on the tensor buffer; Reading loaded tensor parts of the plurality of tensor parts from the tensor buffer to respectively perform one or more operations to implement calculation of the tensor to be calculated, Among them, according to the correspondence between each tensor part and the corresponding part of the tensor buffer, it is determined that the corresponding parts of the tensor buffer corresponding to the combination of multiple tensor parts to be loaded into the tensor buffer at one time do not overlap on the tensor buffer, so as to select to load the combination of the multiple tensor parts into the tensor buffer at one time, read the loaded multiple tensor parts from the tensor buffer, and simultaneously perform multiple operations on the multiple tensor parts. After performing multiple operations on the multiple tensor parts, according to the correspondence between each tensor part and the corresponding part of the tensor buffer, it is determined that the corresponding parts of the tensor buffer corresponding to another combination of the multiple tensor parts to be loaded into the tensor buffer at one time do not overlap on the tensor buffer, so as to select to load another combination of the multiple tensor parts into the tensor buffer at one time, read the loaded multiple tensor parts from the tensor buffer, and simultaneously perform multiple operations on the multiple tensor parts, wherein the corresponding parts of the tensor buffers corresponding to the multiple tensor parts do not overlap on the tensor buffer.

2. The method according to claim 1, wherein The corresponding relationship is represented by the starting address and size of each tensor part in the tensor to be calculated and the starting address and size of at least a corresponding part of the tensor buffer.

3. The method according to claim 1, wherein The tensor to be calculated is split into multiple tensor parts of the same size, or the tensor to be calculated is split into multiple tensor parts of any size.

4. The method according to claim 1, wherein The reading of the loaded tensor parts of the plurality of tensor parts from the tensor buffer to respectively perform one or more operations to implement calculation of the tensor to be calculated includes: The kernel function reads the loaded tensor parts of the multiple tensor parts from the tensor buffer by using the multiple tensor parts as parameters of the kernel function to respectively perform one or more operations to realize the calculation of the tensor to be calculated.

5. A tensor processing device comprising: a splitting device configured to split the tensor to be calculated into a plurality of tensor parts, wherein the size of each of the plurality of tensor parts is smaller than the size of the tensor buffer, wherein the tensor to be calculated is split into a plurality of tensor parts each of which can be independently operated, wherein each of the plurality of tensor parts is a virtual tensor defined by a starting address and a size; A loading device, configured to respectively load multiple tensor parts obtained by splitting a tensor to be calculated into a tensor buffer, including: selecting a combination of multiple tensor parts from the multiple tensor parts according to a corresponding relationship between each tensor part and a corresponding part of the tensor buffer to load into the tensor buffer at one time, wherein, according to the corresponding relationship, the corresponding parts of the tensor buffer corresponding to the combination do not overlap on the tensor buffer, wherein the corresponding relationship is represented by a starting address and size of each tensor part in the tensor to be calculated and a starting address and size of at least a corresponding part of the tensor buffer, wherein the corresponding relationship includes one tensor part corresponding to multiple parts on the tensor buffer, so that the one tensor part is to be loaded into multiple locations on the tensor buffer; a reading device configured to read the loaded tensor parts of the plurality of tensor parts from the tensor buffer to respectively perform one or more operations to realize the calculation of the tensor to be calculated, Among them, according to the correspondence between each tensor part and the corresponding part of the tensor buffer, it is determined that the corresponding parts of the tensor buffer corresponding to the combination of multiple tensor parts to be loaded into the tensor buffer at one time do not overlap on the tensor buffer, so as to select to load the combination of the multiple tensor parts into the tensor buffer at one time, read the loaded multiple tensor parts from the tensor buffer, and simultaneously perform multiple operations on the multiple tensor parts. After performing multiple operations on the multiple tensor parts, according to the correspondence between each tensor part and the corresponding part of the tensor buffer, it is determined that the corresponding parts of the tensor buffer corresponding to another combination of the multiple tensor parts to be loaded into the tensor buffer at one time do not overlap on the tensor buffer, so as to select to load another combination of the multiple tensor parts into the tensor buffer at one time, read the loaded multiple tensor parts from the tensor buffer, and simultaneously perform multiple operations on the multiple tensor parts, wherein the corresponding parts of the tensor buffers corresponding to the multiple tensor parts do not overlap on the tensor buffer.

6. An electronic device comprising: Memory, for storing computer instructions; A processor is configured to read the computer instructions in the memory and execute the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having computer instructions stored thereon, in, When the computer instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 4.

8. A computer program product comprising computer instructions, in, When the computer instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 4.

9. An electronic device comprising: One or more stream processor clusters, each of which includes multiple computing units, tensor buffers, Each of the plurality of computing units executes the method according to any one of claims 1 to 4.

Images