Operator execution method, device, apparatus, storage medium and program product

Abstract

The application provides an operator execution method and device, equipment, a storage medium and a program product, relates to the technical field of artificial intelligence, and the method comprises the following steps: reading input tensor data in a first memory, and splitting the input tensor data according to preset operation granularity information to obtain a plurality of tensor data blocks; for each tensor data block, the following steps are respectively performed: processing the tensor data block by using a basic calculation unit corresponding to the tensor data block in an artificial intelligence chip, and mapping the position information of the basic calculation unit in the artificial intelligence chip to obtain the source memory address of the tensor data block in the first memory; the source memory address of the tensor data block is transformed into the target memory address of output tensor data in a second memory; and the plurality of tensor data blocks are saved to the second memory according to the target memory addresses of the plurality of tensor data blocks to obtain the output tensor data. The complex division operation in the layout transformation is avoided, and the performance of the reordering operator is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an operator execution method, apparatus, device, storage medium, and program product. Background Technology

[0002] Currently, different hardware, computing libraries, or models have different preferences for tensor data layout. Choosing the appropriate tensor layout for different hardware, computing libraries, or models can bring about significant performance improvements.

[0003] In related technologies, the logical steps of realizing tensor data layout transformation through reordering operators are simple, including: the artificial intelligence chip reads the continuous tensor addresses in the input tensor data in sequence according to the tensor layout, performs complex write-out coordinate calculations (including multiple extremely time-consuming division operations), determines the storage address of the output tensor data, and writes it to the global memory according to the calculated storage address to obtain the data tensor data.

[0004] When using an AI chip to implement the layout transformation of tensor data, the conversion of the address coordinates of the input tensor data into the storage address coordinates of the output tensor data involves multiple complex division calculations that consume a lot of time. Furthermore, the writing process of the output tensor data results in low write bandwidth due to the discontinuous storage addresses.

[0005] Therefore, the use of artificial intelligence chips in related technologies to implement tensor data layout transformation suffers from low performance issues. Summary of the Invention

[0006] This application provides an operator execution method, apparatus, device, storage medium, and program product, which avoids complex division operations, reduces the computational overhead of tensor address transformation, and improves the performance of reordering operators during the layout transformation of tensor data using artificial intelligence chips. Through reasonable tensor data block partitioning, it balances the bandwidth utilization of tensor data reading and tensor data writing.

[0007] On one hand, embodiments of this application provide a method for executing an operator, the method comprising:

[0008] The input tensor data is read from the first memory and segmented according to the preset operation granularity information to obtain multiple tensor data blocks; the preset operation granularity information includes the size of multiple dimensions of the tensor data blocks.

[0009] For each tensor data block, the following steps are performed: the tensor data block is processed by the basic computing unit corresponding to the tensor data block in the artificial intelligence chip, and the source memory address of the tensor data block in the first memory is obtained based on the location information of the basic computing unit in the artificial intelligence chip; the source memory address of the tensor data block is transformed into the target memory address of the output tensor data in the second memory.

[0010] Using multiple basic computing units corresponding to the multiple tensor data blocks, the multiple tensor data blocks are saved to the second memory according to their target memory addresses to obtain output tensor data.

[0011] Optionally, the input tensor data is of a first layout type, and the output tensor data is of a second layout type; the preset operation granularity information is determined using the following steps:

[0012] Compare the first layout type and the second layout type to determine the m tensor dimensions to be reordered, where m is greater than 1;

[0013] For each tensor dimension to be reordered, perform the following steps: divide the tensor dimension into multiple sub-tensor dimensions; use the size of the sub-tensor dimensions as the size of one dimension of the tensor data block;

[0014] The size of multiple dimensions of the tensor data block is used as the preset operation granularity information.

[0015] Optionally, before determining the preset operation granularity information, the method further includes:

[0016] If among the m tensor dimensions to be reordered, there exist n tensor dimensions that are consecutive and have the same order in both the first and second layout types, then the n tensor dimensions are merged to obtain a fused tensor dimension; where n is greater than 1.

[0017] The n tensor dimensions are replaced with the fused tensor dimension to obtain m-n+1 tensor dimensions to be reordered.

[0018] Optionally, the location information of the basic computing unit includes: the coordinates of the basic computing unit in multiple directional dimensions in the physical space of the artificial intelligence chip, wherein the multiple directional dimensions of the basic computing unit correspond one-to-one with the multiple tensor dimensions of the tensor data block;

[0019] Based on the location information of the basic computing unit in the artificial intelligence chip, the source memory address of the tensor data block in the first memory is obtained by mapping, including:

[0020] For the coordinates of each of the multiple directional dimensions of the basic computing unit, the following steps are performed respectively: the coordinates of the directional dimension are used as the coordinates of the tensor dimension corresponding to the directional dimension in the tensor data block;

[0021] The coordinates of multiple tensor dimensions in the tensor data block are used as the source memory addresses of the tensor data block in the first memory.

[0022] Optionally, transforming the source memory address of the tensor data block into the target memory address of the output tensor data in the second memory includes:

[0023] Based on the m tensor dimensions to be reordered, determine the m source tensor dimensions in the first layout type and the m target tensor dimensions in the second layout type, and determine the correspondence between the m source tensor dimensions and the m target tensor dimensions.

[0024] For each source tensor dimension, perform the following steps: based on the source memory address of the tensor data block and the correspondence, transform the coordinates of the source tensor dimension of the tensor data block into the coordinates of the target tensor dimension corresponding to the source tensor dimension;

[0025] The coordinates of multiple target tensor dimensions of the tensor data block are used as the target memory address of the tensor data block;

[0026] Based on the target memory addresses of multiple tensor data blocks in the input tensor data, the target memory address of the output tensor data in the second memory is obtained.

[0027] Optionally, before processing the tensor data block using the basic computing unit corresponding to the tensor data block in the artificial intelligence chip, the method further includes:

[0028] The basic computing unit of the artificial intelligence chip caches the tensor data block in an on-chip cache so that the basic computing unit can read the tensor data block from the on-chip cache.

[0029] On one hand, embodiments of this application provide an operator execution apparatus, the apparatus comprising:

[0030] The input tensor segmentation module is configured to read input tensor data from the first memory and segment the input tensor data according to preset operation granularity information to obtain multiple tensor data blocks; the preset operation granularity information includes the size of multiple dimensions of the tensor data blocks;

[0031] The tensor address transformation module is configured to perform the following for each tensor data block: process the tensor data block using the basic computing unit corresponding to the tensor data block in the artificial intelligence chip, and obtain the source memory address of the tensor data block in the first memory based on the location information of the basic computing unit in the artificial intelligence chip; transform the source memory address of the tensor data block into the target memory address of the output tensor data in the second memory;

[0032] The output tensor writing module is configured to use multiple basic computing units corresponding to the multiple tensor data blocks to save the multiple tensor data blocks to the second memory according to the target memory address of the multiple tensor data blocks, so as to obtain the output tensor data.

[0033] Optionally, the input tensor data is of a first layout type, and the output tensor data is of a second layout type; the input tensor segmentation module is configured to determine the preset operation granularity information using the following steps:

[0034] Compare the first layout type and the second layout type to determine the m tensor dimensions to be reordered, where m is greater than 1;

[0035] For each tensor dimension to be reordered, perform the following steps: divide the tensor dimension into multiple sub-tensor dimensions; use the size of the sub-tensor dimensions as the size of one dimension of the tensor data block;

[0036] The size of multiple dimensions of the tensor data block is used as the preset operation granularity information.

[0037] Optionally, before determining the preset operation granularity information, the input tensor segmentation module is further configured to:

[0038] If among the m tensor dimensions to be reordered, there exist n tensor dimensions that are consecutive and have the same order in both the first and second layout types, then the n tensor dimensions are merged to obtain a fused tensor dimension; where n is greater than 1.

[0039] The n tensor dimensions are replaced with the fused tensor dimension to obtain m-n+1 tensor dimensions to be reordered.

[0040] Optionally, the location information of the basic computing unit includes: the coordinates of the basic computing unit in multiple directional dimensions in the physical space of the artificial intelligence chip, wherein the multiple directional dimensions of the basic computing unit correspond one-to-one with the multiple tensor dimensions of the tensor data block;

[0041] Based on the location information of the basic computing unit in the artificial intelligence chip, the source memory address of the tensor data block in the first memory is obtained through mapping. The tensor address transformation module is configured to:

[0042] For the coordinates of each of the multiple directional dimensions of the basic computing unit, the following steps are performed respectively: the coordinates of the directional dimension are used as the coordinates of the tensor dimension corresponding to the directional dimension in the tensor data block;

[0043] The coordinates of multiple tensor dimensions in the tensor data block are used as the source memory addresses of the tensor data block in the first memory.

[0044] Optionally, the tensor address transformation module is configured to transform the source memory address of the tensor data block into the target memory address of the output tensor data in the second memory.

[0045] Based on the m tensor dimensions to be reordered, determine the m source tensor dimensions in the first layout type and the m target tensor dimensions in the second layout type, and determine the correspondence between the m source tensor dimensions and the m target tensor dimensions.

[0046] For each source tensor dimension, perform the following steps: based on the source memory address of the tensor data block and the correspondence, transform the coordinates of the source tensor dimension of the tensor data block into the coordinates of the target tensor dimension corresponding to the source tensor dimension;

[0047] The coordinates of multiple target tensor dimensions of the tensor data block are used as the target memory address of the tensor data block;

[0048] Based on the target memory addresses of multiple tensor data blocks in the input tensor data, the target memory address of the output tensor data in the second memory is obtained.

[0049] Optionally, before processing the tensor data block using the basic computing unit corresponding to the tensor data block in the artificial intelligence chip, the tensor address transformation module is further configured as follows:

[0050] The basic computing unit of the artificial intelligence chip caches the tensor data block in an on-chip cache so that the basic computing unit can read the tensor data block from the on-chip cache.

[0051] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip. When the artificial intelligence chip executes the computer program, it implements the steps of the above-mentioned operator execution method.

[0052] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the execution method of the above-described operator.

[0053] On one hand, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described operator execution method.

[0054] In this embodiment, the input tensor data is divided into tensor data blocks. The source memory address of the input tensor data is mapped through the location information of the basic computing unit, and the source memory address of the input tensor data is transformed into the target memory address of the output tensor data. This avoids complex division operations, reduces the computational overhead of tensor address transformation, and improves the performance of the reordering operator. Through reasonable tensor data block division, the bandwidth utilization of tensor data reading and tensor data writing is balanced. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application;

[0057] Figure 2 A flowchart illustrating an operator execution method provided in an embodiment of this application;

[0058] Figure 3 This is a flowchart illustrating the process of determining preset operation granularity information provided in an embodiment of this application.

[0059] Figure 4 A schematic diagram illustrating the process of obtaining the source memory address of a tensor data block in the first memory based on the location information of the basic computing unit in the artificial intelligence chip, as provided in the embodiments of this application;

[0060] Figure 5 This is a schematic diagram illustrating the process of transforming the source memory address of a tensor data block into the target memory address of the output tensor data in a second memory, as provided in an embodiment of this application.

[0061] Figure 6A schematic diagram illustrating the layout transformation of tensor data provided in the embodiments of this application;

[0062] Figure 7 A schematic diagram of the structure of the operator execution device 700 provided in the embodiments of this application;

[0063] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0065] refer to Figure 1 This is a structural diagram of an artificial intelligence chip applicable to the embodiments of this application. The artificial intelligence chip 100 includes at least: a video memory 101 and multiple programmable multiprocessors 102. The programmable multiprocessors 102 can be a streaming processing cluster (SPC). Each programmable multiprocessor 102 includes multiple execution units 103.

[0066] Each execution unit 103 includes: on-chip cache 104, register 105, and thread group 106. The thread group 106 is also called thread bundle, which includes multiple threads.

[0067] Video memory 101 can be high-bandwidth memory (HBM) or other types of memory. On-chip cache 104 is a temporary memory with a smaller capacity than video memory 101, but a faster data exchange speed.

[0068] Compared to the on-chip cache 104, the register 105 has a smaller capacity but a faster data exchange speed.

[0069] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.

[0070] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-purpose computing on graphics processing units (GPGPU), Domain Specific Architecture (DSA), etc.

[0071] This application is based on Figure 1 The diagram of the AI ​​chip architecture shown illustrates a method for executing an operator, applicable to various scenarios such as image processing, speech processing, and text processing. The physical meaning of the input tensor data can differ across these applications.

[0072] For example, in a text processing scenario, the input tensor data can be text data used in tasks such as text generation and text recognition.

[0073] For example, in a speech processing scenario, the input tensor data can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.

[0074] For example, in image processing scenarios, the input tensor data can be image data used in tasks such as image preprocessing, image segmentation, and object detection.

[0075] It should be noted that in this application, "first" and "second" are used for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated.

[0076] The following is based on Figure 1 The diagram shown illustrates the architecture of an AI chip and details the execution method of one of its operators. (See also...) Figure 2 This method is executed by a computer device that contains an artificial intelligence chip, which can be... Figure 1 The artificial intelligence chip 100 shown herein includes the following steps:

[0077] Step 201: Read the input tensor data in the first memory, and divide the input tensor data into multiple tensor data blocks according to the preset operation granularity information; wherein, the preset operation granularity information includes the size of multiple dimensions of the tensor data blocks.

[0078] In some embodiments, the input tensor data is of a first layout type, and the output tensor data is of a second layout type. See [link to documentation]. Figure 3 The embodiments of this application use the following steps to determine the preset operation granularity information:

[0079] Step 301: Compare the first layout type and the second layout type to determine the m tensor dimensions to be reordered, where m is greater than 1.

[0080] Step 302: For each tensor dimension to be reordered, perform the following steps: divide the tensor dimension into multiple sub-tensor dimensions; use the size of the sub-tensor dimensions as the size of one dimension of the tensor data block.

[0081] Step 303: Use the size of multiple dimensions of the tensor data block as preset operation granularity information.

[0082] Specifically, the input tensor data includes one or more dimensions; there are usually two types of reorder operators in CV scenarios, one is to convert NHWC to NCHW, and the other is to convert NCHW to NHWC; when the input tensor data is a 4-dimensional matrix, it corresponds to 4 dimensions, such as batch (N) dimension, number of channels (C) dimension, H dimension, and W dimension.

[0083] For example, the input tensor data is of the first layout type NCHW, and the output tensor data is of the second layout type NHWC. The three tensor dimensions to be reordered are determined to be C, H, and W. The dimensions of the input tensor data are (8, 128, 128, 128), where the size of the N dimension is 8, representing the batch size of 8; the size of the C dimension is 128, representing the number of channels of 128; the size of the H dimension is 128, representing the height of the input tensor of 128; and the size of the W dimension is 128, representing the width of the input tensor of 128. The tensor data is divided along the C dimension, resulting in four sub-tensor dimensions of 32 each; the tensor data is divided along the H dimension, resulting in four sub-tensor dimensions of 32 each; and the tensor data is divided along the W dimension, resulting in four sub-tensor dimensions of 32 each. The size of each sub-tensor dimension is used as the size of one dimension of the tensor data block. Therefore, the dimensions of the resulting tensor data block are (8, 32, 32, 32).

[0084] In some embodiments, before determining the preset operation granularity information, if among the m tensor dimensions to be reordered, there are n tensor dimensions that are in the first layout type and the second layout type, and whose arrangement order remains unchanged and are consecutive, then the n tensor dimensions are merged to obtain a fused tensor dimension; where n is greater than 1; the fused tensor dimension is used to replace the n tensor dimensions to obtain m-n+1 tensor dimensions to be reordered.

[0085] For example, if the input tensor data is of the first layout type NCHW and the output tensor data is of the second layout type NHWC, and the three tensor dimensions to be reordered are determined to be C, H, and W, and the two tensor dimensions that are consecutive and whose order remains unchanged are H and W, then the H and W dimensions are merged to obtain the fused tensor dimension HW. The fused tensor dimension HW is then used to replace the original H and W dimensions, resulting in two tensor dimensions to be reordered, namely C and HW.

[0086] For example, when partitioning input tensor data into tensor data blocks, if the partitioning is done with a height-weighted (HW) dimension of 8 and a length-weighted (C) dimension of 16, then each tensor data block will have a size of 8 in both the HW and C dimensions. 16, the operation granularity information is 8. 16. Based on the operation granularity information, the input tensor data is divided into segments of size 8 in both the HW and C dimensions. A 16-byte tensor data block.

[0087] The above steps for merging n tensor dimensions achieve address merging during tensor address transformation, turning two dimensions into one, which also reduces coordinate calculation overhead and improves the performance of the reordering operator.

[0088] Step 202: For each tensor data block, perform the following: process the tensor data block using the basic computing unit corresponding to the tensor data block in the artificial intelligence chip, and obtain the source memory address of the tensor data block in the first memory based on the location information of the basic computing unit in the artificial intelligence chip; transform the source memory address of the tensor data block into the target memory address of the output tensor data in the second memory.

[0089] In some embodiments, the location information of the basic computing unit includes: the coordinates of the basic computing unit in multiple directional dimensions in the physical space of the artificial intelligence chip, wherein the multiple directional dimensions of the basic computing unit correspond one-to-one with the multiple tensor dimensions of the tensor data block.

[0090] For example, the physical space of an artificial intelligence chip includes x, y, and z dimensions. This application uses basic computing units (CUs) in the x, y, and z directions to process input tensor data in the N, C, and HW dimensions respectively. The x, y, and z dimensions correspond one-to-one with the N, C, and HW tensor dimensions.

[0091] Specifically, the source memory address of the tensor data block in the first memory is obtained by mapping the location information of the basic computing unit in the artificial intelligence chip, see [link to relevant documentation]. Figure 4 This includes the following steps:

[0092] Step 401: For the coordinates of each direction dimension in the multiple directional dimensions of the basic computing unit, perform the following steps respectively: use the coordinates of the direction dimension as the coordinates of the tensor dimension of the corresponding direction dimension in the tensor data block.

[0093] Step 402: Use the coordinates of multiple tensor dimensions in the tensor data block as the source memory address of the tensor data block in the first memory.

[0094] For example, if the coordinates of the basic computing unit in the x, y, z directions in the physical space of the artificial intelligence chip are (3, 5, 6), then the source memory address of the tensor data block in the first memory is (3, 5, 6).

[0095] In some embodiments, the source memory address of the tensor data block is transformed into the target memory address of the output tensor data in the second memory, see [link to relevant documentation]. Figure 5 This includes the following steps:

[0096] Step 501: Based on the m tensor dimensions to be reordered, determine the m source tensor dimensions in the first layout type and the m target tensor dimensions in the second layout type, and determine the correspondence between the m source tensor dimensions and the m target tensor dimensions.

[0097] Step 502: For each source tensor dimension, perform the following steps: Based on the source memory address and correspondence of the tensor data block, transform the coordinates of the source tensor dimension of the tensor data block into the coordinates of the target tensor dimension corresponding to the source tensor dimension.

[0098] Step 503: Use the coordinates of multiple target tensor dimensions of the tensor data block as the target memory address of the tensor data block.

[0099] Step 504: Based on the target memory addresses of multiple tensor data blocks in the input tensor data, obtain the target memory address of the output tensor data in the second memory.

[0100] For example, the input tensor data is of the first layout type NCHW, and the output tensor data is of the second layout type NHWC. The H dimension and the W dimension are merged to obtain the fused tensor dimension HW. The two tensor dimensions to be reordered include the C dimension and the HW dimension. The source tensor dimension C and the target tensor dimension HW correspond to each other, and the source tensor dimension HW and the target tensor dimension C correspond to each other. The coordinates of the source tensor dimension are transformed to the coordinates of the target tensor dimension corresponding to the source tensor dimension. The source memory address of the tensor data block in the first memory is (3, 5, 6), and the target memory address of the tensor data block in the second memory is (3, 6, 5). And so on, the address of each tensor data block after the input tensor data is divided is transformed to obtain the target memory address of the output tensor data in the second memory.

[0101] This application's embodiments transform the coordinates of the source tensor dimension into the coordinates of the target tensor dimension corresponding to the source tensor dimension. For example, see... Figure 6 (N dimension not shown) Viewing the HW dimension as the C dimension and the C dimension as the HW dimension allows the NHWC2NCHW operator to reuse the NCHW2NHWC operator. A general reorder operator can be used to implement the functional requirements of various tensor layout transformations, reducing code maintenance costs.

[0102] Step 203: Using multiple basic computing units corresponding to multiple tensor data blocks, save multiple tensor data blocks to the second memory according to the target memory address of the multiple tensor data blocks to obtain output tensor data.

[0103] It should be noted that the first memory and the second memory can be the same or different.

[0104] In some embodiments, before the basic computing unit corresponding to the tensor data block in the artificial intelligence chip processes the tensor data block, the embodiments of this application use the basic computing unit of the artificial intelligence chip to cache the tensor data block in the on-chip cache, so that the basic computing unit can read the tensor data block from the on-chip cache. After the basic computing unit caches the tensor data block in the on-chip L2 cache, when other basic computing units read it, they can directly read it from the on-chip L2 cache, which can make full use of the L2 cache and improve the read bandwidth utilization.

[0105] In this embodiment, the input tensor data is divided into tensor data blocks. The source memory address of the input tensor data is mapped through the location information of the basic computing unit, and the source memory address of the input tensor data is transformed into the target memory address of the output tensor data. This avoids complex division operations, reduces the computational overhead of tensor address transformation, and improves the performance of the reordering operator. Through reasonable tensor data block division, the bandwidth utilization of tensor data reading and tensor data writing is balanced.

[0106] Based on the same technical concept, this application provides a schematic diagram of the structure of an operator execution device, such as... Figure 7 As shown, the operator execution device 700 includes:

[0107] The input tensor segmentation module 701 is configured to read input tensor data from the first memory and segment the input tensor data according to preset operation granularity information to obtain multiple tensor data blocks; the preset operation granularity information includes the size of multiple dimensions of the tensor data blocks.

[0108] Tensor address transformation module 702 is configured to, for each tensor data block, perform the following: process the tensor data block using the basic computing unit corresponding to the tensor data block in the artificial intelligence chip, and map the source memory address of the tensor data block in the first memory based on the location information of the basic computing unit in the artificial intelligence chip; transform the source memory address of the tensor data block into the target memory address of the output tensor data in the second memory;

[0109] The output tensor writing module 703 is configured to use multiple basic computing units corresponding to the multiple tensor data blocks to save the multiple tensor data blocks to the second memory according to the target memory address of the multiple tensor data blocks, so as to obtain output tensor data.

[0110] Optionally, the input tensor data is of a first layout type, and the output tensor data is of a second layout type; the input tensor segmentation module is configured to determine the preset operation granularity information using the following steps:

[0111] Compare the first layout type and the second layout type to determine the m tensor dimensions to be reordered, where m is greater than 1;

[0112] For each tensor dimension to be reordered, perform the following steps: divide the tensor dimension into multiple sub-tensor dimensions; use the size of the sub-tensor dimensions as the size of one dimension of the tensor data block;

[0113] The size of multiple dimensions of the tensor data block is used as the preset operation granularity information.

[0114] Optionally, before determining the preset operation granularity information, the input tensor segmentation module is further configured to:

[0115] If among the m tensor dimensions to be reordered, there exist n tensor dimensions that are consecutive and have the same order in both the first and second layout types, then the n tensor dimensions are merged to obtain a fused tensor dimension; where n is greater than 1.

[0116] The n tensor dimensions are replaced with the fused tensor dimension to obtain m-n+1 tensor dimensions to be reordered.

[0117] Optionally, the location information of the basic computing unit includes: the coordinates of the basic computing unit in multiple directional dimensions in the physical space of the artificial intelligence chip, wherein the multiple directional dimensions of the basic computing unit correspond one-to-one with the multiple tensor dimensions of the tensor data block;

[0118] Based on the location information of the basic computing unit in the artificial intelligence chip, the source memory address of the tensor data block in the first memory is obtained through mapping. The tensor address transformation module is configured to:

[0119] For the coordinates of each of the multiple directional dimensions of the basic computing unit, the following steps are performed respectively: the coordinates of the directional dimension are used as the coordinates of the tensor dimension corresponding to the directional dimension in the tensor data block;

[0120] The coordinates of multiple tensor dimensions in the tensor data block are used as the source memory addresses of the tensor data block in the first memory.

[0121] Optionally, the tensor address transformation module is configured to transform the source memory address of the tensor data block into the target memory address of the output tensor data in the second memory.

[0122] Based on the m tensor dimensions to be reordered, determine the m source tensor dimensions in the first layout type and the m target tensor dimensions in the second layout type, and determine the correspondence between the m source tensor dimensions and the m target tensor dimensions.

[0123] For each source tensor dimension, perform the following steps: based on the source memory address of the tensor data block and the correspondence, transform the coordinates of the source tensor dimension of the tensor data block into the coordinates of the target tensor dimension corresponding to the source tensor dimension;

[0124] The coordinates of multiple target tensor dimensions of the tensor data block are used as the target memory address of the tensor data block;

[0125] Based on the target memory addresses of multiple tensor data blocks in the input tensor data, the target memory address of the output tensor data in the second memory is obtained.

[0126] Optionally, before processing the tensor data block using the basic computing unit corresponding to the tensor data block in the artificial intelligence chip, the tensor address transformation module is further configured as follows:

[0127] The basic computing unit of the artificial intelligence chip caches the tensor data block in an on-chip cache so that the basic computing unit can read the tensor data block from the on-chip cache.

[0128] In this embodiment, the input tensor data is divided into tensor data blocks. The source memory address of the input tensor data is mapped through the location information of the basic computing unit, and the source memory address of the input tensor data is transformed into the target memory address of the output tensor data. This avoids complex division operations, reduces the computational overhead of tensor address transformation, and improves the performance of the reordering operator. Through reasonable tensor data block division, the bandwidth utilization of tensor data reading and tensor data writing is balanced.

[0129] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 8 As shown, it includes at least one artificial intelligence chip 100 and a memory 801 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 801 is not limited. Figure 8 Taking the connection between the AI ​​chip 100 and the memory 801 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0130] In this embodiment of the application, the memory 801 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 801, at least one artificial intelligence chip 100 can execute the steps of the above-mentioned operator execution method.

[0131] The artificial intelligence chip 100 serves as the control center of the computer device, connecting to various parts of the device via various interfaces and lines. It performs tensor data reordering by running or executing instructions stored in the memory 801 and retrieving data stored in the memory 801. Optionally, the artificial intelligence chip 100 may include one or more processing units. It may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 801 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0132] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0133] Memory 801, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 801 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 801 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. In the embodiments of this application, memory 801 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0134] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the execution method of the above-mentioned operator.

[0135] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps of the above-described operator execution method.

[0136] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0141] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method of executing an operator, characterized by, The method comprises: reading input tensor data in a first memory, and splitting the input tensor data according to preset operation granularity information to obtain a plurality of tensor data blocks; the preset operation granularity information comprises sizes of a plurality of size dimensions of the tensor data blocks; for each tensor data block, the following are respectively performed: processing the tensor data block by using a basic computing unit corresponding to the tensor data block in an artificial intelligence chip, and mapping a source memory address of the tensor data block in the first memory based on position information of the basic computing unit in the artificial intelligence chip; and transforming the source memory address of the tensor data block into a target memory address of output tensor data in a second memory; using a plurality of basic computing units corresponding to the plurality of tensor data blocks, saving the plurality of tensor data blocks to the second memory according to the target memory addresses of the plurality of tensor data blocks to obtain the output tensor data.

2. The method of claim 1, wherein, The input tensor data is of a first layout type, and the output tensor data is of a second layout type; The preset operation granularity information is determined by the following steps: comparing the first layout type and the second layout type to determine m tensor dimensions to be reordered, m being greater than 1; for each tensor dimension to be reordered, the following steps are respectively performed: dividing the tensor dimension into a plurality of sub-tensor dimensions; and taking the size of the sub-tensor dimension as the size of one size dimension of the tensor data block; taking the sizes of the plurality of size dimensions of the tensor data block as the preset operation granularity information.

3. The method of claim 2, wherein, Before determining the preset operation granularity information, the method further comprises: if there are n tensor dimensions in the m tensor dimensions to be reordered, which are arranged in the same order and are continuous in the first layout type and the second layout type, then merging the n tensor dimensions to obtain a fusion tensor dimension; wherein n is greater than 1; replacing the n tensor dimensions with the fusion tensor dimension to obtain m-n+1 tensor dimensions to be reordered.

4. The method of claim 1, wherein, The position information of the basic computing unit comprises coordinates of a plurality of direction dimensions of the basic computing unit in a physical space of the artificial intelligence chip, and the plurality of direction dimensions of the basic computing unit correspond one-to-one to a plurality of tensor dimensions of the tensor data block; mapping the source memory address of the tensor data block in the first memory based on the position information of the basic computing unit in the artificial intelligence chip comprises: for the coordinates of each direction dimension of the plurality of direction dimensions of the basic computing unit, the following steps are respectively performed: taking the coordinates of the direction dimension as the coordinates of a tensor dimension of the tensor data block corresponding to the direction dimension; taking the coordinates of the plurality of tensor dimensions of the tensor data block as the source memory address of the tensor data block in the first memory.

5. The method of claim 2, wherein, The transformation of the source memory address of the tensor data block into the target memory address of the output tensor data in the second memory comprises: determine m source tensor dimensions in the first layout type and m target tensor dimensions in the second layout type based on the m tensor dimensions to be reordered, and determine a correspondence between the m source tensor dimensions and the m target tensor dimensions; for each source tensor dimension, perform the following steps: based on a source memory address of the tensor data block and the correspondence, transform a coordinate of a source tensor dimension of the tensor data block into a coordinate of a target tensor dimension corresponding to the source tensor dimension; obtain the target memory address of the output tensor data in the second memory according to the target memory addresses of the plurality of tensor data blocks in the input tensor data. Before processing the tensor data block by using the basic computing unit corresponding to the tensor data block in the artificial intelligence chip, the method further comprises:

6. The method according to any one of claims 1 to 5, wherein cache the tensor data block into an on-chip cache by using the basic computing unit of the artificial intelligence chip, so that the basic computing unit reads the tensor data block from the on-chip cache. The device comprises:

7. An execution device of an operator, characterized by, An input tensor splitting module configured to read input tensor data in a first memory and split the input tensor data according to preset operation granularity information to obtain a plurality of tensor data blocks; the preset operation granularity information includes sizes of a plurality of size dimensions of the tensor data blocks; A tensor address transformation module configured to, for each tensor data block, perform the following steps respectively: process the tensor data block by using a basic computing unit corresponding to the tensor data block in an artificial intelligence chip, and map to obtain a source memory address of the tensor data block in the first memory based on position information of the basic computing unit in the artificial intelligence chip; transform the source memory address of the tensor data block into a target memory address of output tensor data in a second memory; An output tensor writing module configured to use a plurality of basic computing units corresponding to the plurality of tensor data blocks to save the plurality of tensor data blocks to the second memory according to the target memory addresses of the plurality of tensor data blocks, to obtain output tensor data. The artificial intelligence chip implements the steps of the method of any one of claims 1-6 when executing the computer program.

8. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored on the memory and executable on the artificial intelligence chip, characterized in that, The computer program product comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions which, when executed by a computer device, cause the computer device to perform the steps of the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer program product comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions which, when executed by a computer device, cause the computer device to perform the steps of the method of any one of claims 1-6.

10. A computer program product, characterised in that, ​

Images