A tensor memory transfer method, device, storage medium and program product
By splitting tensor data blocks and calculating memory offsets, the problem of insufficient generalization of memory handling functions in the existing technology is solved, and accurate mapping and performance improvement of diversified tensor data shapes are achieved.
Patent Information
- Application Number
- CN202411037672.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-30
Smart Images

Figure CN118567580B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a tensor memory transfer method, device, storage medium, and program product. Background Art
[0002] An AI model usually refers to a neural network model that is trained to perform inference and prediction, such as an image inference model, a speech inference model, etc. The computation of an AI model can be implemented by operators in a computation graph. The computation graph of an AI model can include operators that perform many types of computations on tensors. The inference and training processes of an AI model are usually completed on an AI chip.
[0003] In the performance tuning scenarios of artificial intelligence models such as Wenshengtu models, there are often a large number of memory transfer operations for tensor data. Moreover, as the demand increases, the shapes of the corresponding tensor data are also more diverse. For example, the shapes of some tensor data are aligned according to the tensor data blocks in the last dimension; the shapes of some tensor data are not aligned according to the tensor data blocks in the last dimension.
[0004] When implementing the above memory transfer operations, the relevant memory transfer operators can only perform memory transfer operations on tensor data of specific shapes, so they are not generalized enough in terms of functionality. Summary of the invention
[0005] The embodiments of the present application provide a tensor memory transfer method, device, storage medium and program product for improving the generalization and versatility of memory transfer functions, thereby improving the performance of memory transfer operators.
[0006] On the one hand, an embodiment of the present application provides a tensor memory transfer method, including:
[0007] Splitting the original tensor into a plurality of tensor data blocks, the plurality of tensor data blocks including a reference data block;
[0008] For the multiple tensor data blocks, the following operations are performed respectively: obtaining a first memory offset of a tensor data block compared to the reference data block in a thread dimension; and obtaining a second memory offset of the tensor data block compared to the reference data block in a tensor data block dimension; determining a target memory address of the tensor data block based on the reference memory address after memory mapping of the reference data block, the first memory offset, and the second memory offset;
[0009] The multiple tensor data blocks are saved according to their respective target memory addresses to obtain a target tensor.
[0010] On the one hand, an embodiment of the present application provides a tensor memory transfer device, including:
[0011] A splitting module, used for splitting the original tensor into a plurality of tensor data blocks, wherein the plurality of tensor data blocks include a reference data block;
[0012] The address mapping module is used to perform the following operations respectively for the multiple tensor data blocks: obtaining a first memory offset of a tensor data block compared to the reference data block in a thread dimension; and obtaining a second memory offset of the tensor data block compared to the reference data block in a tensor data block dimension; determining a target memory address of the tensor data block based on the reference memory address after memory mapping of the reference data block, the first memory offset, and the second memory offset;
[0013] A saving module is used to save the multiple tensor data blocks according to the respective target memory addresses of the multiple tensor data blocks to obtain a target tensor.
[0014] Optionally, the address mapping module is specifically used to:
[0015] Based on the position index of the tensor data block in the original tensor, the first memory offset is selected from candidate memory offsets in multiple thread dimensions; a candidate memory offset is associated with at least one position index.
[0016] Optionally, the candidate memory offsets in the multiple thread dimensions match the shape and layout of the original tensor and the shape and layout of the target tensor.
[0017] Optionally, the position index is a row index;
[0018] The address mapping module is specifically used for:
[0019] Obtain a first mapping address of a first tensor data block located in a first row of a sample input tensor on a plurality of threads in a reference execution unit; and obtain a second mapping address of a first tensor data block of a row corresponding to a row index in the sample input tensor on a plurality of threads in the reference execution unit;
[0020] Based on the first mapping address and the second mapping address, a candidate memory offset in the thread dimension associated with the row index is obtained.
[0021] Optionally, when a tensor data block of the sample input tensor in the last dimension is not aligned with the number of threads in the reference execution unit, the candidate memory offsets in the plurality of thread dimensions are greater than zero;
[0022] When the tensor data block of the sample input tensor in the last dimension is aligned with the number of threads in the reference execution unit, the candidate memory offsets in the plurality of thread dimensions are equal to zero.
[0023] Optionally, the address mapping module is specifically used to:
[0024] Obtaining a third memory offset of the one tensor data block relative to the reference data block in a thread grid dimension;
[0025] Based on the reference memory address after memory mapping of the reference data block, the first memory offset, the second memory offset and the third memory offset, a target memory address of the tensor data block is determined.
[0026] Optionally, the address mapping module is specifically used to:
[0027] The reference memory address, the first memory offset, the second memory offset and the third memory offset are accumulated to obtain a target memory address of the tensor data block.
[0028] On the one hand, an embodiment of the present application provides a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned tensor memory transfer method when executing the computer program.
[0029] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned tensor memory transfer method.
[0030] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned tensor memory transfer method.
[0031] In an embodiment of the present application, a reference data block is selected from multiple tensor data blocks of the original tensor, and a reference memory address after memory mapping of the reference data block is first obtained, and then the target memory address of each tensor data block is determined based on the reference memory address of the reference data block, the first memory offset of each tensor data block in the thread dimension compared to the reference data block, and the second memory offset of each tensor data block in the tensor data block dimension compared to the reference data block. Since the memory offset in the thread dimension is taken into account when mapping the memory address, accurate memory address mapping can be achieved regardless of whether the shape of the tensor data is aligned according to the tensor data block in the last dimension, thereby improving the generalization and versatility of the memory transfer function, and then improving the performance of the memory transfer operator. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0033] Figure 1 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application;
[0034] Figure 2 A schematic diagram of a sample input tensor provided in an embodiment of the present application Figure 1 ;
[0035] Figure 3 A schematic diagram of a sample input tensor provided in an embodiment of the present application Figure 2 ;
[0036] Figure 4 A flowchart of a tensor memory transfer method provided in an embodiment of the present application;
[0037] Figure 5 A schematic diagram of the structure of a tensor memory transfer device provided in an embodiment of the present application;
[0038] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0040] refer to Figure 1, which is a structural diagram of an artificial intelligence chip applicable to an embodiment of the present application, the artificial intelligence chip 100 at least includes: a video memory 101, a plurality of programmable multiprocessors 102, wherein the programmable multiprocessor 102 can be a streaming processing cluster (SPC for short), and each programmable multiprocessor 102 includes: a plurality of execution units 103.
[0041] Each execution unit 103 includes: an on-chip cache 104, a register 105, and a thread group 106. The thread group 106 is also called a thread warp, and the thread warp includes: a plurality of threads.
[0042] The video memory 101 may be a high bandwidth memory (HBM) or other types of memory. The on-chip cache 104 is a temporary memory, which has a smaller capacity than the video memory 101 but a faster data exchange speed than the video memory 101 .
[0043] Compared to the on-chip cache 104 , the register 105 has a smaller capacity than the on-chip cache 104 , but a faster data exchange speed than the on-chip cache 104 .
[0044] In addition to the above-mentioned structure, the artificial intelligence chip 100 in the present application may also include other structures, which are not specifically limited in the present application.
[0045] The artificial intelligence chip 100 may be: a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a domain specific architecture (DSA), etc.
[0046] In practical applications, in the performance tuning scenarios of artificial intelligence models such as the Vincent graph model, the Vincent graph model includes different modules (for example, residual module, attention mechanism module, etc.), and the layout and shape of tensors supported by different modules may be different. For any two modules that support different layouts and shapes (such as the first module and the second module), in order to enable the output tensor of the first module to be used as the input tensor of the second module for subsequent calculations, it is necessary to first perform a memory transfer operation on the output tensor of the first module to adjust the layout and shape of the output tensor of the first module to the layout and shape supported by the second module; that is, perform a memory address mapping on the original memory address of the output tensor of the first module to obtain the target memory address, and then save the output tensor of the first module according to the target memory address, so as to save the output tensor of the first module in memory according to the layout and shape supported by the second module.
[0047] As the demand increases, the shapes of tensor data that are subjected to memory transfer operations are becoming more diverse. For example, some tensor data shapes are aligned with tensor data blocks on the last dimension; some tensor data shapes are not aligned with tensor data blocks on the last dimension, where the last dimension can be a row dimension, a column dimension, etc.
[0048] When implementing the above memory transfer operations, the relevant memory transfer operators can only perform memory transfer operations on tensor data of specific shapes. For example, memory transfer operations can only be performed on tensor data of a shape that is aligned according to tensor data blocks on the last dimension, and errors will occur when memory transfer operations are performed on tensor data of a shape that is not aligned according to tensor data blocks on the last dimension. Therefore, the relevant memory transfer operators are not generalized enough in terms of function, which in turn affects the performance of the memory transfer operators.
[0049] In view of this, the embodiment of the present application proposes the following tensor memory transfer method, firstly introducing the process of generating a memory transfer operator for tensor memory transfer:
[0050] For each combination of input shape, input layout, output shape, and output layout, the memory offset information required for memory transfer is calculated in advance by classification and traversal. The generated memory offset information is saved in a list or other data structure.
[0051] When generating any memory transfer operator, based on the combination of input shape, input layout, output shape and output layout supported by the memory transfer operator, relevant memory offset information is selected, and multiple registers are selected to save each candidate memory offset in the memory offset information, and an association is established between the memory transfer operator and the selected multiple registers. When the memory transfer operator is subsequently executed, the memory offset information can be directly read from the relevant registers, and the memory transfer operation is performed on the original tensor that meets the input shape and input layout according to the memory offset information to obtain the target tensor that meets the output shape and output layout.
[0052] In the present application, for each combination of input shape, input layout, output shape, and output layout, the memory offset information includes one or more of the following candidate memory offsets: candidate memory offsets in thread dimension, candidate memory offsets in thread warp dimension, candidate memory offsets in tensor data block dimension, and candidate memory offsets in thread grid dimension.
[0053] In the present application, each thread warp includes multiple threads, and the thread warp can be mapped to an execution unit in an artificial intelligence chip; each thread grid includes multiple thread blocks, each thread block includes multiple thread warps, and the thread grid can be mapped to a programmable multiprocessor in an artificial intelligence chip.
[0054] The following describes the specific process of calculating each of the above candidate memory offsets:
[0055] For each combination of input shape, input layout, output shape, and output layout, a sample input tensor that matches the input shape and input layout is pre-selected, and the input memory address of the sample input tensor is obtained. According to the input shape and input layout before memory transfer, and the output shape and output layout after memory transfer, the memory address mapping relationship can be determined. Then, according to the memory address mapping relationship, the input memory address of each tensor data in the sample input tensor is memory-mapped to obtain the output memory address. Each tensor data is saved accordingly according to the output memory address of each tensor data to obtain the sample output tensor; that is, the sample output tensor is saved in memory according to the output shape and output layout. Based on the relationship between the output memory addresses of each tensor data in the sample output tensor, the above-mentioned various candidate memory offsets can be obtained.
[0056] For the candidate memory offset in the thread dimension, the calculation process is as follows:
[0057] For tensor data blocks corresponding to different position indexes in the sample input tensor, the candidate memory offsets in the thread dimension may also be different, and a candidate memory offset in a thread dimension is associated with at least one position index. This application calculates the candidate memory offset in the thread dimension for different position indexes, and then associates and saves each position index with the corresponding candidate memory offset in the thread dimension. The position index can be a row index or a column index.
[0058] Taking the position index as the row index, the candidate memory offset on the thread dimension associated with each row index is obtained in the following way:
[0059] The first mapping address of the first tensor data block located in the first row of the sample input tensor on multiple threads in the reference execution unit is obtained; and the second mapping address of the first tensor data block of a row corresponding to a row index in the sample input tensor is obtained on multiple threads in the reference execution unit. Then, based on the first mapping address and the second mapping address, a candidate memory offset on the thread dimension associated with the row index is obtained.
[0060] Specifically, the row indexes corresponding to the tensor data blocks in different rows are different. For example, the row indexes of the tensor data blocks in the first row are all 1, and the row indexes of the tensor data blocks in the second row are all 2. The first tensor data block in the first row of the sample input tensor can be the reference data block of the sample input tensor. The reference execution unit can be the first execution unit among the multiple execution units in the programmable multiprocessor, or it can be other execution units.
[0061] From the output memory address of the sample output tensor, obtain a first mapping address of a reference data block of the sample input tensor on multiple threads in the reference execution unit, and a second mapping address of a first tensor data block of a row corresponding to a row index in the sample input tensor on multiple threads in the reference execution unit.
[0062] Specifically, multiple first tensor data are selected from the reference data block of the sample input tensor, and the selected multiple first tensor data must meet the following conditions: in the process of performing memory address mapping on the input memory address of the sample input tensor to obtain the output memory address of the sample output tensor, the memory address mapping operation of the multiple first tensor data is executed by multiple threads in the reference execution unit.
[0063] Since the output memory address of the sample output tensor includes the output memory address after the memory address mapping of all tensor data in the sample input tensor, the output memory address obtained after the memory address mapping of the multiple first tensor data can be directly obtained from the output memory address of the sample output tensor. The obtained output memory addresses of the multiple first tensor data are the first mapping addresses of the reference data block on the multiple threads in the reference execution unit.
[0064] From the first tensor data block of a row corresponding to a row index in the sample input tensor (for example, the nth row), multiple second tensor data are selected, and the selected multiple second tensor data must meet the following conditions: in the process of performing memory address mapping on the input memory address of the sample input tensor and obtaining the output memory address of the sample output tensor, the memory address mapping operation of the multiple second tensor data is executed by multiple threads in the reference execution unit.
[0065] From the output memory address of the sample output tensor, an output memory address obtained after performing memory address mapping on the plurality of second tensor data is obtained. The output memory addresses of the plurality of second tensor data obtained are the second mapping addresses.
[0066] An output memory address calculated by a thread with thread index x in the reference execution unit is selected from the first mapping address, and an output memory address calculated by a thread with thread index x in the reference execution unit is selected from the second mapping address. The two selected output memory addresses are subtracted to obtain a candidate memory offset on the thread dimension associated with a row index, as shown in the following formula (1):
[0067]
[0068] Among them, thread_offset_n[x] represents the candidate memory offset corresponding to the row index n (i.e., the nth row), thread_addr_n [x] represents the output memory address selected from the second mapping address and calculated by the thread with thread index x in the reference execution unit, thread_addr_1[x] represents the output memory address selected from the first mapping address and calculated by the thread with thread index x in the reference execution unit, n is greater than 1, and x is greater than or equal to 0.
[0069] It should be noted that, for rows other than the first row, the candidate memory offsets in the thread dimension associated with the rows may be completely different or partially the same.
[0070] The reference data block of the sample input tensor may be not only the first tensor data block in the first row of the sample input tensor, but also other tensor data blocks in the first row, or tensor data blocks in other rows. At this time, the first mapping address of the reference data block on multiple threads in the reference execution unit is also obtained, and the second mapping address of any tensor data block corresponding to a different row index from the reference data block on multiple threads in the reference execution unit is obtained. Based on the first mapping address and the second mapping address, the candidate memory offset of the row index corresponding to any of the above tensor data blocks is obtained.
[0071] In some embodiments, when a tensor data block of the sample input tensor in the last dimension is not aligned with the number of threads in the reference execution unit, the candidate memory offsets in the plurality of thread dimensions are greater than zero.
[0072] Specifically, the shape of a tensor includes one or more dimensions, and the dimensions of the shape of the tensor are arranged in a predefined order. For example, the shape of a tensor includes four dimensions, which are: batch dimension, channel dimension, row dimension, column dimension in a predefined order. At this time, the shape of the tensor data is the last dimension: column dimension.
[0073] When performing memory transfer on a sample input tensor, the sample input tensor is first split into multiple tensor data blocks according to a preset data block size, and then the multiple data blocks are traversed in sequence to perform memory transfer operations, wherein the traversal methods include: the order of row dimension first and then column dimension, the order of column dimension first and then row dimension, etc.
[0074] The tensor data block of the sample input tensor on the last dimension is not aligned with the number of threads in the reference execution unit, which means that the number of elements in the last tensor data block of the sample input tensor on the last dimension is not the same as the number of threads in the reference execution unit, that is, the size of the last tensor data block is smaller than the preset data block size.
[0075] Taking the last dimension as the row dimension, see Figure 2, since the number of elements in the last tensor data block (i.e., tensor data block 5) of the first row in the sample input tensor is different from the number of threads in the reference execution unit, when the memory transfer operation is performed, the mapping address of tensor data block 5 on some threads in the reference execution unit is empty. After performing the memory transfer on tensor data block 5, the first tensor data block (i.e., tensor data block 6) of the second row is then memory transferred to obtain the mapping addresses of tensor data block 6 on multiple threads in the reference execution unit. At this time, it is necessary to fill the blank addresses in the mapping addresses corresponding to tensor data block 5 with part of the mapping addresses in the mapping addresses corresponding to tensor data block 6, so as to ensure the continuity of the memory mapping address. In order to achieve this purpose, the mapping address (i.e., the second mapping address 202) of tensor data block 6 on multiple threads in the reference execution unit is no longer the same as the first mapping address 201 of the reference data block on multiple threads in the reference execution unit. Therefore, based on the second mapping address 202 and the first mapping address 201, the candidate memory offset of the second row in the thread dimension is obtained, and the candidate memory offset in the thread dimension is greater than 0.
[0076] After filling the blank addresses in the mapping addresses corresponding to tensor data block 5 with the partial mapping addresses corresponding to tensor data block 6, the missing partial mapping addresses corresponding to tensor data block 6 are filled with the mapping addresses of the next tensor data block that performs memory transfer, and so on, until the last tensor data block.
[0077] It should be noted that for each row of tensor data blocks, the candidate memory offset in the thread dimension is the same, wherein for each tensor data block in the first row, the candidate memory offset in the thread dimension is 0.
[0078] In some embodiments, when a tensor data block in the last dimension of the sample input tensor is aligned with the number of threads in the execution unit, the candidate memory offsets in the plurality of thread dimensions are equal to zero.
[0079] The tensor data blocks of the sample input tensor in the last dimension are aligned with the number of threads in the execution unit. This means that the number of elements in each tensor data block in the last dimension of the sample input tensor is the same as the number of threads in the execution unit.
[0080] Since the number of elements in each tensor data block in the sample input tensor is the same as the number of threads in the reference execution unit, there are no gaps in the mapping addresses of each tensor data block on multiple threads in the reference execution unit, that is, there is no need to fill in the mapping addresses of other tensor data blocks, so the candidate memory offsets in multiple thread dimensions are equal to zero.
[0081] For example, see Figure 3, because the number of elements in the last tensor data block (i.e., tensor data block 4) of the first row in the sample input tensor is the same as the number of threads in the reference execution unit, there is no blank in the mapping address of tensor data block 4 on multiple threads in the reference execution unit. After performing memory transfer on tensor data block 4, the first tensor data block (i.e., tensor data block 5) of the second row is then memory transferred to obtain the mapping address of tensor data block 5 on multiple threads in the reference execution unit.
[0082] Since there is no blank in the mapping address of tensor data block 4 on multiple threads in the reference execution unit, there is no need to supplement the mapping address of tensor data block 4. Therefore, the mapping address of tensor data block 5 on multiple threads in the reference execution unit (i.e., the second mapping address 302) is the same as the first mapping address 301 of the reference data block on multiple threads in the reference execution unit. Based on the second mapping address 302 and the first mapping address 301, the candidate memory offset of the second row in the thread dimension is equal to 0.
[0083] For the candidate memory offset in the warp dimension, the calculation process is as follows:
[0084] Select the third tensor data and the fourth tensor data from the reference data block of the sample input tensor; the third tensor data and the fourth tensor data must meet the following conditions:
[0085] In the process of performing memory address mapping on the input memory address of the sample input tensor to obtain the output memory address of the sample output tensor, the memory address mapping operation of the third tensor data is executed by the thread with thread index m in the first execution unit, and the memory address mapping operation of the fourth tensor data is executed by the thread with thread index m in the second execution unit, the first execution unit and the second execution unit are two adjacent execution units on the same programmable multiprocessor, where m is greater than or equal to 0.
[0086] Since the output memory address of the sample output tensor includes the output memory address after the memory address mapping of all tensor data in the sample input tensor, the output memory addresses of the third tensor data and the fourth tensor data can be directly obtained from the output memory address of the sample output tensor. Finally, based on the output memory addresses of the third tensor data and the fourth tensor data, the unit memory offset in the warp dimension is calculated. Specifically, the output memory address of the fourth tensor data can be subtracted from the output memory address of the third tensor data to obtain the unit memory offset in the warp dimension.
[0087] Based on the index difference between the logical index of each execution unit and the logical index of the reference execution unit, and the unit memory offset in the thread warp dimension, the candidate memory offset of each execution unit in the thread warp dimension is obtained, wherein the logical index of the execution unit refers to: a virtual number of the execution unit at the software level.
[0088] For the candidate memory offsets on the tensor data block dimension, the calculation process is as follows:
[0089] In the process of performing memory address mapping on the input memory address of the sample input tensor and obtaining the output memory address of the sample output tensor, the sample input tensor is split into multiple parts and distributed to multiple programmable multiprocessors for execution, each part including multiple tensor data blocks. Logical indexes are independently constructed for the multiple tensor data blocks processed by each programmable multiprocessor. For example, four tensor data blocks in the sample input tensor are assigned to programmable multiprocessor 1 for processing, and the other four tensor data blocks in the sample input tensor are assigned to programmable multiprocessor 2 for processing. For the four tensor data blocks processed by programmable multiprocessor 1, the logical indexes of the four tensor data blocks are set to 0, 1, 2, and 3 respectively. For the four tensor data blocks processed by programmable multiprocessor 2, the logical indexes of the four tensor data blocks are set to 0, 1, 2, and 3 respectively.
[0090] When calculating the candidate memory offsets on the tensor data block dimension, first select the fifth tensor data and the sixth tensor data from the sample input tensor; the fifth tensor data and the sixth tensor data meet the following conditions:
[0091] The fifth tensor data and the sixth tensor data are located in two different tensor data blocks in the sample input tensor (the logical indices of the corresponding tensor data volumes are different); and, in the process of performing memory address mapping on the input memory address of the sample input tensor to obtain the output memory address of the sample output tensor, the memory address mapping operations of the two tensor data blocks are assigned to a programmable multiprocessor for execution.
[0092] Specifically, the memory address mapping operation of the fifth tensor data is executed by a thread with a thread index of k in the reference execution unit of the programmable multiprocessor, and the memory address mapping operation of the sixth tensor data is executed by a thread with a thread index of k in the reference execution unit of the programmable multiprocessor, where k is greater than or equal to 0.
[0093] Since the output memory address of the sample output tensor includes the output memory address after the memory address mapping of all tensor data in the sample input tensor, the output memory addresses of the fifth tensor data and the sixth tensor data can be directly obtained from the output memory address of the sample output tensor.
[0094] Based on the output memory addresses of the fifth tensor data and the sixth tensor data, the unit memory offset in the tensor data block dimension is calculated. Specifically, the output memory address of the sixth tensor data can be subtracted from the output memory address of the fifth tensor data to obtain the unit memory offset in the tensor data block dimension.
[0095] Based on the difference between the logical index of each tensor data block in the sample input tensor and the index of the reference data block, and the unit memory offset in the tensor data block dimension, a candidate memory offset of each tensor data block in the sample input tensor in the tensor data block dimension is obtained.
[0096] For the candidate memory offset in the thread grid dimension, the calculation process is as follows:
[0097] The seventh tensor data and the eighth tensor data are selected from the sample input tensor; the seventh tensor data and the eighth tensor data meet the following conditions:
[0098] The seventh tensor data and the eighth tensor data are located in two different tensor data blocks in the sample input tensor; in the process of performing memory address mapping on the input memory address of the sample input tensor to obtain the output memory address of the sample output tensor, the memory address mapping operations of the two tensor data blocks are allocated to the adjacent first programmable multiprocessor and second programmable multiprocessor for execution, and the logical indexes of the two tensor data blocks are the same.
[0099] Specifically, the memory address mapping operation of the seventh tensor data is executed by a thread with a thread index of i in the first execution unit in the first programmable multiprocessor; the memory address mapping operation of the eighth tensor data is executed by a thread with a thread index of i in the first execution unit in the second programmable multiprocessor, where i is greater than or equal to 0.
[0100] From the output memory address of the sample output tensor, the output memory addresses of the seventh tensor data and the eighth tensor data are obtained. Based on the output memory addresses of the seventh tensor data and the eighth tensor data, the unit memory offset in the thread grid dimension is calculated. Specifically, the output memory address of the eighth tensor data can be subtracted from the output memory address of the seventh tensor data to obtain the unit memory offset in the thread grid dimension.
[0101] Based on the logical index of each programmable multiprocessor, the index difference between the logical index of the programmable multiprocessor where the reference execution unit is located, and the unit memory offset in the thread grid dimension, the candidate memory offset of each programmable multiprocessor in the thread grid dimension is obtained, wherein the logical index of the programmable multiprocessor refers to: a virtual number of the programmable multiprocessor at the software level.
[0102] In some embodiments, in the process of generating a memory transfer operator, in addition to obtaining relevant memory offset information, the first mapping address of the reference data block in the sample input tensor on multiple threads in the reference execution unit can also be obtained, and the first mapping address can be saved in a register associated with the memory transfer operator. When the memory transfer operator is subsequently executed, the first mapping address and memory offset information can be directly read from the relevant register, and the memory transfer operation is performed on the original tensor that meets the input shape and input layout according to the first mapping address and memory offset information to obtain a target tensor that meets the output shape and output layout.
[0103] After introducing the process of generating memory transfer operators, the following is based on Figure 1 The structure diagram of the artificial intelligence chip shown in the figure introduces the process of a tensor memory transfer method provided by the present application, such as Figure 4 As shown, the process of the method is executed by an artificial intelligence chip and includes the following steps:
[0104] Step 401, split the original tensor into multiple tensor data blocks.
[0105] After the original tensor is input into the memory move operator, the memory move operator can split the original tensor into multiple tensor data blocks according to different splitting strategies. The splitting strategy can be splitting by row, splitting by column, etc.
[0106] The multiple tensor data blocks include a reference data block. In a specific implementation, the reference data block may be the first data block among the multiple tensor data blocks, or may be a tensor data block specified from the multiple tensor data blocks according to a preset rule, and this application does not make any specific limitation on this.
[0107] Step 402, for multiple tensor data blocks, respectively perform the following steps 4021 to 4023:
[0108] Specifically, multiple tensor data blocks are traversed in sequence according to a preset traversal method to perform memory transfer, wherein the preset traversal method may be an order of row dimension first and then column dimension, an order of column dimension first and then row dimension, etc.
[0109] The order of row dimensions first and column dimensions second means: first traverse each tensor data block in the first row, then traverse each tensor data block in the second row, and so on, until all tensor data blocks are traversed.
[0110] The order of column dimension first and row dimension second means: first traverse each tensor data block in the first column, then traverse each tensor data block in the second column, and so on, until all tensor data blocks are traversed.
[0111] For each tensor data block in the original tensor, perform the following memory transfer operations:
[0112] Step 4021, obtain a first memory offset of a tensor data block compared to a reference data block in the thread dimension.
[0113] Specifically, as described above, when a memory transfer operator is generated, relevant memory offset information is selected and saved in a register based on the combination of input shape, input layout, output shape, and output layout supported by the memory transfer operator. The memory offset information includes at least: candidate memory offsets in multiple thread dimensions.
[0114] For a memory transfer operator whose input tensor is an original tensor, whose output tensor is a target tensor, and whose memory transfer operation is a layout switch, the candidate memory offsets on multiple thread dimensions included in the memory transfer operator match the shape and layout of the original tensor and the shape and layout of the target tensor.
[0115] In practical applications, the candidate memory offsets on multiple thread dimensions include: candidate memory offsets on thread dimensions corresponding to different position indexes. A candidate memory offset on a thread dimension is associated with at least one position index.
[0116] When performing memory movement on each tensor data block in the original tensor, a first memory offset is selected from candidate memory offsets in multiple thread dimensions based on a position index of the tensor data block in the original tensor.
[0117] In a specific implementation, the position index is a row index or a column index. Expanding with the row index, in the candidate memory offsets on multiple thread dimensions, each candidate memory offset is associated with at least one row index. For each tensor data block in the original tensor, based on the row index corresponding to the tensor data block, a first memory offset is selected from the candidate memory offsets on multiple thread dimensions.
[0118] It should be noted that when the tensor data block of the original tensor in the last dimension is not aligned with the number of threads in the reference execution unit, the candidate memory offsets in multiple thread dimensions are greater than zero.
[0119] When the tensor data block of the original tensor in the last dimension is aligned with the number of threads in the reference execution unit, the candidate memory offsets in multiple thread dimensions are equal to zero.
[0120] Step 4022, obtain a second memory offset of the tensor data block compared to the reference data block in the tensor data block dimension.
[0121] Specifically, when generating a memory transfer operator, the memory offset information stored in the register associated with the memory transfer operator also includes: a candidate memory offset of each tensor data block in the tensor data block dimension. Therefore, when executing the memory transfer operator, based on the logical index of the tensor data block in the original tensor, the candidate memory offset of the tensor data block in the tensor data block dimension can be directly obtained from the stored memory offset information, and the candidate memory offset is used as the second memory offset.
[0122] Step 4023, determine the target memory address of the tensor data block based on the reference memory address after the memory mapping of the reference data block, the first memory offset and the second memory offset.
[0123] Specifically, as introduced above, when generating a memory transfer operator, in addition to selecting and saving relevant memory offset information based on the combination of input shape, input layout, output shape and output layout supported by the memory transfer operator, the first mapping address of the reference data block in the sample input tensor on multiple threads in the reference execution unit is obtained and saved in the register associated with the memory transfer operator.
[0124] Therefore, during the execution phase of the memory transfer operator, the first mapping address of the reference data block in the sample input tensor can be read directly from the register associated with the memory transfer operator as the first memory address mapped to the reference data block in the original tensor in the reference execution unit; the reference execution unit can be the first execution unit among multiple execution units, or it can be other execution units.
[0125] When generating a memory transfer operator, the memory offset information stored in the register associated with the memory transfer operator also includes: the candidate memory offset of each execution unit in the thread warp dimension. Therefore, in the execution phase of the memory transfer operator, the logical index of the execution unit that performs the memory transfer operation on the tensor data block in the original tensor is first determined. Based on the logical index of the execution unit, the candidate memory offset of the execution unit in the thread warp dimension is directly read from the register associated with the memory transfer operator as the fourth memory offset.
[0126] The first memory address and the fourth memory offset are accumulated to obtain the reference memory address of the reference data block in the original tensor after memory mapping. The reference memory address, the first memory offset, and the second memory offset are accumulated to obtain the target memory address of the tensor data block after memory mapping.
[0127] Step 403, save multiple tensor data blocks according to their respective target memory addresses to obtain a target tensor.
[0128] In an embodiment of the present application, a reference data block is selected from multiple tensor data blocks of the original tensor, and a reference memory address of the reference data block after memory mapping is first obtained, and then the target memory address of the tensor data block is determined based on the reference memory address of the reference data block, the first memory offset of the tensor data block in the thread dimension compared to the reference data block, and the second memory offset of the tensor data block in the tensor data block dimension compared to the reference data block. Since the memory offset in the thread dimension is taken into account when mapping the memory address, accurate memory address mapping can be achieved regardless of whether the shape of the tensor data is aligned according to the tensor data block in the last dimension, thereby improving the generalization and versatility of the memory transfer function, and then improving the performance of the memory transfer operator.
[0129] In some embodiments, a third memory offset of a tensor data block in a thread grid dimension compared to a reference data block is obtained, and a target memory address of the tensor data block is determined based on the reference memory address after memory mapping of the reference data block, the first memory offset, the second memory offset, and the third memory offset.
[0130] Specifically, when generating a memory transfer operator, the memory offset information stored in the register associated with the memory transfer operator also includes: the candidate memory offset of each programmable multiprocessor in the thread grid dimension. Therefore, when executing the memory transfer operator, the logical index of the programmable multiprocessor that performs the memory transfer operation on the tensor data block in the original tensor is first determined; then, based on the logical index, the candidate memory offset in the thread grid dimension is read from the register associated with the memory transfer operator accordingly as the third memory offset.
[0131] The reference memory address after memory mapping of the reference data block, the first memory offset, the second memory offset, and the third memory offset are accumulated to obtain the target memory address of the tensor data block.
[0132] In the embodiment of the present application, several variable values, namely, candidate memory offsets in thread dimension, candidate memory offsets in thread bundle dimension, candidate memory offsets in tensor data block dimension, and candidate memory offsets in thread grid dimension, are abstracted in advance, and then these variable values are saved in the memory transfer operator, that is, most of the memory address calculation process is transferred to the process of generating the memory transfer operator. In this way, when executing the memory transfer operator, the accumulation operation can be directly performed on the basis of these variable values to obtain the target memory address of the tensor data block, thereby greatly reducing the time for instructions to run on the chip thread during the memory mapping process, thereby improving the model subgraph performance.
[0133] In some embodiments, for each programmable multiprocessor, the reference memory address of the reference data block and the third memory offset are fixed values. Therefore, when calculating the reference data block, the reference memory address and the third memory offset process can be initialized, that is, the reference memory address of the reference data block and the third memory offset are accumulated to a variable temp_sum_init, and then this variable is placed in a register.
[0134] When processing other tensor data blocks in the original tensor through the programmable multiprocessor, it is only necessary to calculate the first memory offset and the second memory offset of the tensor data block compared to the reference data block. Then, the first memory offset and the second memory offset are accumulated on the variable temp_sum_init to obtain the target memory address of the tensor data block, which can further reduce the time that instructions run on the chip during the memory mapping process and improve the performance of the model subgraph.
[0135] Based on the same technical concept, the present application embodiment provides a structural schematic diagram of a tensor memory transfer device, such as Figure 5 As shown, the tensor memory transport device 500 includes:
[0136] A splitting module 501 is used to split the original tensor into multiple tensor data blocks, wherein the multiple tensor data blocks include a reference data block;
[0137] The address mapping module 502 is used to perform the following operations respectively for the multiple tensor data blocks: obtaining a first memory offset of a tensor data block compared to the reference data block in the thread dimension; and obtaining a second memory offset of the tensor data block compared to the reference data block in the tensor data block dimension; determining a target memory address of the tensor data block based on the reference memory address after memory mapping of the reference data block, the first memory offset, and the second memory offset;
[0138] The saving module 503 is used to save the multiple tensor data blocks according to the respective target memory addresses of the multiple tensor data blocks to obtain a target tensor.
[0139] Optionally, the address mapping module 502 is specifically configured to:
[0140] Based on the position index of the tensor data block in the original tensor, the first memory offset is selected from candidate memory offsets in multiple thread dimensions; a candidate memory offset is associated with at least one position index.
[0141] Optionally, the candidate memory offsets in the multiple thread dimensions match the shape and layout of the original tensor and the shape and layout of the target tensor.
[0142] Optionally, the position index is a row index;
[0143] The address mapping module 502 is specifically used for:
[0144] Obtain a first mapping address of a first tensor data block located in a first row of a sample input tensor on a plurality of threads in a reference execution unit; and obtain a second mapping address of a first tensor data block of a row corresponding to a row index in the sample input tensor on a plurality of threads in the reference execution unit;
[0145] Based on the first mapping address and the second mapping address, a candidate memory offset in the thread dimension associated with the row index is obtained.
[0146] Optionally, when a tensor data block of the sample input tensor in the last dimension is not aligned with the number of threads in the reference execution unit, the candidate memory offsets in the plurality of thread dimensions are greater than zero;
[0147] When the tensor data block of the sample input tensor in the last dimension is aligned with the number of threads in the reference execution unit, the candidate memory offsets in the plurality of thread dimensions are equal to zero.
[0148] Optionally, the address mapping module 502 is specifically configured to:
[0149] Obtaining a third memory offset of the one tensor data block relative to the reference data block in a thread grid dimension;
[0150] Based on the reference memory address after memory mapping of the reference data block, the first memory offset, the second memory offset and the third memory offset, a target memory address of the tensor data block is determined.
[0151] Optionally, the address mapping module 502 is specifically configured to:
[0152] The reference memory address, the first memory offset, the second memory offset and the third memory offset are accumulated to obtain a target memory address of the tensor data block.
[0153] In an embodiment of the present application, a reference data block is selected from multiple tensor data blocks of the original tensor, and a reference memory address of the reference data block after memory mapping is first obtained, and then the target memory address of the tensor data block is determined based on the reference memory address of the reference data block, the first memory offset of the tensor data block in the thread dimension compared to the reference data block, and the second memory offset of the tensor data block in the tensor data block dimension compared to the reference data block. Since the memory offset in the thread dimension is taken into account when mapping the memory address, accurate memory address mapping can be achieved regardless of whether the shape of the tensor data is aligned according to the tensor data block in the last dimension, thereby improving the generalization and versatility of the memory transfer function, and then improving the performance of the memory transfer operator.
[0154] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 6 As shown, it includes at least one artificial intelligence chip 100 and a memory 601 connected to the at least one artificial intelligence chip 100. The specific connection medium between the artificial intelligence chip 100 and the memory 601 is not limited in the embodiment of the present application. Figure 6 In the example, the artificial intelligence chip 100 and the memory 601 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.
[0155] In an embodiment of the present application, the memory 601 stores instructions that can be executed by at least one artificial intelligence chip 100. At least one artificial intelligence chip 100 can execute the steps of the above-mentioned tensor memory transfer method by executing the instructions stored in the memory 601.
[0156] Among them, the artificial intelligence chip 100 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and tensor memory transfer can be realized by running or executing instructions stored in the memory 601 and calling data stored in the memory 601. Optionally, the artificial intelligence chip 100 may include one or more processing units, and the artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 601 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.
[0157] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, which can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.
[0158] The memory 601 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 601 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 601 is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 601 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0159] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned tensor memory transfer method.
[0160] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned tensor memory transfer method.
[0161] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0162] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0163] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0164] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0165] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0166] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A tensor memory transfer method, characterized in that: include: Splitting the original tensor into a plurality of tensor data blocks, the plurality of tensor data blocks including a reference data block; For the multiple tensor data blocks, the following operations are performed respectively: based on the position index of a tensor data block in the original tensor, from the candidate memory offsets in the multiple thread dimensions, a first memory offset of the tensor data block in the thread dimension compared to the reference data block is obtained; and based on the logical index of the tensor data block, from the candidate memory offsets in the tensor data block dimension of each tensor data block in the sample input tensor, a second memory offset of the tensor data block in the tensor data block dimension compared to the reference data block is obtained; Determine the target memory address of the tensor data block based on the reference memory address after memory mapping of the reference data block, the first memory offset and the second memory offset; the candidate memory offset in the thread dimension and the candidate memory offset in the tensor data block dimension are calculated in advance based on the relationship between the output memory addresses of the respective tensor data in the sample output tensor; The multiple tensor data blocks are saved according to their respective target memory addresses to obtain a target tensor.
2. The method according to claim 1, characterized in that Also includes: A candidate memory offset in a thread dimension is associated with at least one location index.
3. The method according to claim 2, characterized in that The candidate memory offsets in the plurality of thread dimensions match the shape and layout of the original tensor and the shape and layout of the target tensor.
4. The method according to claim 2, characterized in that The position index is a row index; before splitting the original tensor into multiple tensor data blocks, the method further includes: Obtain a first mapping address of a first tensor data block located in a first row of a sample input tensor on a plurality of threads in a reference execution unit; and obtain a second mapping address of a first tensor data block of a row corresponding to the row index in the sample input tensor on a plurality of threads in the reference execution unit; Based on the first mapping address and the second mapping address, a candidate memory offset in the thread dimension associated with the row index is obtained.
5. The method according to claim 4, characterized in that Also includes: When a tensor data block of the sample input tensor in the last dimension is not aligned with the number of threads in the reference execution unit, the candidate memory offsets in the plurality of thread dimensions are greater than zero; When a tensor data block in the last dimension of the sample input tensor is aligned with the number of threads in the reference execution unit, the candidate memory offsets in the plurality of thread dimensions are equal to zero.
6. The method according to any one of claims 1 to 5, characterized in that: The determining, based on the reference memory address after the memory mapping of the reference data block, the first memory offset, and the second memory offset, a target memory address of the tensor data block comprises: Obtaining a third memory offset of the one tensor data block relative to the reference data block in a thread grid dimension; Based on the reference memory address, the first memory offset, the second memory offset and the third memory offset, a target memory address of the one tensor data block is determined.
7. The method according to claim 6, characterized in that The determining, based on the reference memory address, the first memory offset, the second memory offset, and the third memory offset, a target memory address of the tensor data block includes: The reference memory address, the first memory offset, the second memory offset and the third memory offset are accumulated to obtain a target memory address of the tensor data block.
8. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method as described in any one of claims 1 to 7 are implemented.
9. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of the method as claimed in any one of claims 1 to 7.
10. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 7.
