Operator execution method and device, computer equipment and storage medium
By linearly storing and converting data block arrays in memory, the problem of high conversion development costs between multiple layouts is solved, and a general Layout conversion solution is implemented, and the development costs are reduced.
Patent Information
- Application Number
- CN202411710814.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-03-07
AI Technical Summary
In the prior art, conversion between multiple Layouts requires the development of a conversion program separately, resulting in high development costs.
By opening up a continuous space in memory, the data block array of the first Layout is stored linearly, and then converted from the linear storage to the data block array of the second Layout, a general conversion between various Layouts is realized.
There is no need to develop the conversion between various layouts separately, which reduces the development cost of Layout conversion and supports the conversion of memory architecture and data types.
Smart Images
Figure CN119576564A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an operator execution method, apparatus, computer equipment and storage medium. Background Art
[0002] In the field of artificial intelligence technology, when the output of operator 1 is used as the input of operator 2, if the data structure (Layout) corresponding to the output of operator 1 is different from the Layout corresponding to the input of operator 2, it is necessary to rearrange the output of operator 1 from one Layout to another. In the related art, a set of conversion programs is developed for each two Layouts. Since there are multiple Layouts, the conversion between various Layouts needs to be developed separately, so the development cost is high. Summary of the invention
[0003] The embodiments of the present application provide an operator execution method, apparatus, computer device, and storage medium to provide a general Layout conversion solution.
[0004] In a first aspect, an embodiment of the present application provides an operator execution method, including:
[0005] Before inputting the data block array of the first data structure Layout output by the first operator into the second operator, if it is determined that the input of the second operator corresponds to the second Layout, a continuous memory space is allocated according to the number of elements contained in the data block array;
[0006] Using a plurality of first thread blocks, linearly storing the elements in the data block array into the memory space, wherein the number of the plurality of first thread blocks is determined according to the array size of the data block array;
[0007] Using a plurality of second thread blocks, converting the elements in the memory space into a data block array of the second Layout, wherein the number of the plurality of second thread blocks is determined according to an array size corresponding to an input of the second operator;
[0008] The converted data block array is input into the second operator to execute the second operator.
[0009] In some embodiments, there are N1×M1 first thread blocks in total, N1 is determined based on the number of data blocks included in the data block array in the row direction and the number of data blocks corresponding to a single first thread block, and M1 is equal to the number of data blocks included in the data block array in the column direction.
[0010] In some embodiments, using a plurality of first thread blocks, linearly storing elements in the data block array into the memory space includes:
[0011] Control each first thread block to perform the following operations:
[0012] Determining, according to the position number of the first thread block, a data block corresponding to the first thread block in the data block array;
[0013] Reading the elements in the data block into a plurality of registers of the first thread block through a Layout data loading interface;
[0014] Determine addresses of the elements in the multiple registers in the memory space according to the coordinates of the elements in the multiple registers;
[0015] According to the addresses, the elements in the plurality of registers are written into the memory space.
[0016] In some embodiments, the Layout data loading interface has memory architecture parameters and further includes:
[0017] If the data block array corresponds to a first memory architecture and the input of the second operator corresponds to a second memory architecture, before reading the elements in the data block into multiple registers of the first thread block through the Layout data loading interface, the parameter value of the memory architecture parameter is set to the second memory architecture, so as to convert the elements in the data block into the corresponding second memory architecture through the Layout data loading interface.
[0018] In some embodiments, it also includes:
[0019] If the output of the first operator corresponds to a first data type and the input of the second operator corresponds to a second data type, after reading the elements in the data block into multiple registers of the first thread block through the Layout data loading interface, the elements in the multiple registers are converted into the second data type.
[0020] In some embodiments, there are N2×M2 second thread blocks in total, N2 is determined based on the number of reference blocks included in the reference block array in the row direction and the number of reference blocks corresponding to a single second thread block, M2 is equal to the number of reference blocks included in the reference block array in the column direction, the array shape of the reference block array is the same as the array shape corresponding to the input of the second operator, and the reference blocks in the reference block array are arranged according to the second Layout.
[0021] In some embodiments, using a plurality of second thread blocks, converting the elements in the memory space into a data block array of the second Layout includes:
[0022] Control each second thread block to perform the following operations:
[0023] Determine, according to the position number of the second thread block, a reference block corresponding to the second thread block in the reference block array;
[0024] Determine the address of the element in the reference block in the memory space according to the coordinates of the element in the reference block;
[0025] Reading the element at the address into a plurality of registers of the second thread block;
[0026] Writing the elements in the plurality of registers into the reference block through a Layout data saving interface;
[0027] The reference block array after writing is determined as the converted data block array.
[0028] In a second aspect, an embodiment of the present application provides an operator execution device, including:
[0029] an opening module, configured to open a continuous memory space according to the number of elements contained in the data block array before inputting the data block array of the first data structure Layout output by the first operator into the second operator if it is determined that the input of the second operator corresponds to the second Layout;
[0030] A first conversion module, configured to linearly store the elements in the data block array into the memory space by using a plurality of first thread blocks, wherein the number of the plurality of first thread blocks is determined according to the array size of the data block array;
[0031] A second conversion module, configured to convert the elements in the memory space into a data block array of the second Layout by using a plurality of second thread blocks, wherein the number of the plurality of second thread blocks is determined according to an array size corresponding to an input of the second operator;
[0032] An input module is used to input the converted data block array into the second operator to execute the second operator.
[0033] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements any of the above-mentioned operator execution methods when executing the computer program.
[0034] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of a computer device, the computer device can execute any of the above-mentioned operator execution methods.
[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned operator execution methods when executed by a processor.
[0036] In an embodiment of the present application, before the data block array of the first Layout output by the first operator is input into the second operator, if it is determined that the input of the second operator corresponds to the second Layout, a continuous memory space is opened according to the number of elements contained in the data block array, and then, using multiple first thread blocks, the elements in the data block array are linearly stored in the memory space, and then using multiple second thread blocks, the elements in the memory space are converted into a data block array of the second Layout and input into the second operator to execute the second operator. In this way, the first Layout is converted into linearly stored one-dimensional data through a continuous memory space, and then converted from the one-dimensional data into the second Layout, so that the conversion between various different Layouts can be completed, and a general Layout conversion solution is provided. Since there is no need to develop conversions between various Layouts separately, the development cost of Layout conversion can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0038] Figure 1a A schematic diagram of ColMajor provided in an embodiment of the present application;
[0039] Figure 1b A schematic diagram of storing real dense data of a ColMajor Matrix in memory provided in an embodiment of the present application;
[0040] Figure 2a A schematic diagram of an Activation provided in an embodiment of the present application;
[0041] Figure 2b A schematic diagram of storing real dense data of an Activation Matrix in memory provided in an embodiment of the present application;
[0042] Figure 3 A flowchart of an operator execution method provided in an embodiment of the present application;
[0043] Figure 4 A schematic diagram of a Layout conversion process provided in an embodiment of the present application;
[0044] Figure 5A schematic diagram of the structure of an operator execution device provided in an embodiment of the present application;
[0045] Figure 6 A schematic diagram of the hardware structure of a computer device for implementing an operator execution method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to provide a general Layout conversion solution, the embodiments of the present application provide an operator execution method, apparatus, computer device and storage medium.
[0047] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.
[0048] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0049] In order to facilitate the understanding of this application, the technical terms involved in this application are:
[0050] Layout refers to the data arrangement structure, which can be divided into planar data structure (plain Layout) and non-planar data structure. Among them, plain Layout can arrange elements row by row or column by column in memory, without the concept of sub-blocks and blocks, and no alignment requirements. In the large model operation process, non-planar data structures such as column major sequence (ColMajor), row major sequence (RowMajor), activation (Activation), convolution weight (ConvWeight), depthwise convolution weight (DwcWeight), vector (Vector or Vectors), etc. are used. For each non-planar Layout, the dimension of the Layout is fixed, the Sub-Block composition method and Block composition method corresponding to the Layout are also fixed, and there are alignment requirements. In addition, the data block array of the Layout is stored in memory as Block, and the storage method of Block is also fixed.
[0051] A thread block contains a group of threads, such as 512 threads, and each thread's register can be used to store an element. Threads in the same thread block can communicate through shared memory.
[0052] In the field of artificial intelligence technology, when the output of operator 1 is used as the input of operator 2, if the Layout corresponding to the output of operator 1 is different from the Layout corresponding to the input of operator 2, it is necessary to rearrange the output of operator 1 from one Layout to another. In the related art, a set of conversion programs is developed for each two Layouts. Since there are multiple Layouts, the conversion between various Layouts needs to be developed separately, so the development cost is high.
[0053] To this end, an embodiment of the present application provides a general Layout conversion solution, which is based on an intermediate, simple plain Layout. The Layout corresponding to the output of an operator is first converted to the plain Layout, and then converted from the plain Layout to the Layout corresponding to the input of the next operator, thereby realizing the conversion of various output Layout and input Layout combinations.
[0054] First, several non-planar Layouts are introduced with examples.
[0055] 1.Col Major.
[0056] See also Figure 1a , Figure 1aA schematic diagram of ColMajor provided in an embodiment of the present application, ColMajor has three dimensions: N, H, and W, N represents the number of (H, W), and taking the 32-bit floating-point type and the ColMajor matrix (Matrix) as an example, the shaded area in the figure represents an element of a 32-bit floating-point type, and the 32 elements in the W direction constitute a Sub-Block, and the 4 Sub-Blocks in the H direction constitute a Block.
[0057] Figure 1b A schematic diagram of storage of real dense data (Real Dense Data) of a ColMajor Matrix in memory provided in an embodiment of the present application, where the elements in the ColMajor Matrix are stored in memory in units of Blocks. Figure 1b The middle oblique line shadow shows a Block, and the arrows show the order of the Blocks in the memory. For the Blocks in the ColMajor Matrix, the Blocks in the H direction are arranged first, and then the Blocks in the W direction. After arranging a Block on N, the next Block on N is arranged.
[0058] It should be noted that when arranging the ColMajor Matrix, the Block has a 2KB alignment requirement (that is, the memory size occupied by the Block must be a multiple of 2KB). When the actual data size stored in a Block is an integer multiple of 2KB, the memory boundary will be full, and when the actual data size stored in a Block is not an integer multiple of 2KB, the memory boundary will not be full. Figure 1b Unfilled white areas in the are unused memory.
[0059] 2. Activation.
[0060] See also Figure 2a , Figure 2a A schematic diagram of an Activation provided for an embodiment of the present application, Activation has four dimensions: N, C, H, W, N represents the number (C, H, W), taking the 8-bit integer, Activation Matrix as an example, a Sub-Block contains 4*4*8 elements, first arrange 4 elements in the C direction, then arrange 8 elements in the W direction, and finally arrange 4 elements in the H direction. The 4 Sub-Blocks in the C direction form a Block, and the storage format of a Sub-Block in the memory is shown in the figure on the right.
[0061] Figure 2bA schematic diagram of storing real dense data of an Activation Matrix in a memory provided by an embodiment of the present application, wherein the elements in the Activation Matrix are stored in the memory in units of Blocks. Figure 2b The arrow in the upper right corner points to a single Block of 16*4*8. The circuitous arrows show the order of the Blocks in the memory. For the Blocks in the Activation Matrix, the Blocks in the H direction are arranged first, and then the Blocks in the W, C, and N directions.
[0062] It should be noted that in the Activation Matrix, the Block has a 1KB alignment requirement (that is, the memory size occupied by the Block must be a multiple of 1KB). When the actual data size stored in a Block is an integer multiple of 1KB, the memory boundary will be full, and when the actual data size stored in a Block is not an integer multiple of 1KB, the memory boundary will not be full. Figure 2b Unfilled white areas in the Graph represent unused memory.
[0063] In addition, it should be noted that each Vector requires 512B alignment, but Vector also has Sub-Blocks, and several Sub-Blocks also form a Block, which is stored in memory as Blocks. Vectors are composed of several Vectors. Unlike matrices, Sub-Blocks and Blocks in matrices span rows or columns, so in memory, the elements of some columns or rows of the matrix are arranged first, while Vectors will arrange all the elements in one Vector before arranging the elements in the next Vector. Therefore, Vectors and Vectors can also be regarded as non-planar Layouts.
[0064] Next, the operator execution method proposed in this application is explained.
[0065] refer to Figure 3 , Figure 3 A flowchart of an operator execution method provided in an embodiment of the present application, the method includes the following steps.
[0066] In step 301, before the data block array of the first Layout output by the first operator is input into the second operator, if it is determined that the input of the second operator corresponds to the second Layout, a continuous memory space is allocated according to the number of elements contained in the data block array.
[0067] The first operator and the second operator may be any two adjacent operators in the large model. The first Layout and the second Layout are two different non-planar Layouts. Moreover, each data block in the data block array is arranged according to the first Layout.
[0068] In step 302, the elements in the data block array are linearly stored in the memory space using a plurality of first thread blocks, and the number of the plurality of first thread blocks is determined according to the array size of the data block array.
[0069] The array size of the data block array includes the number of data blocks included in the row direction (ie, H direction), the number of data blocks included in the column direction (ie, W direction), and the like.
[0070] In some embodiments, N1×M1 first thread blocks can be started, where N1 can be determined based on the number of data blocks included in the data block array in the row direction and the number of data blocks corresponding to a single first thread block, and the number of data blocks corresponding to a single first thread block can be determined based on the number of threads included in the single first thread block and the number of elements included in the single data block (elements cannot exceed the boundary in the memory), and the register of each thread is used to store one element. Assuming that the number of data blocks corresponding to a single first thread block is S, and the number of data blocks included in the data block array in the row direction is Y, then N1 is equal to the integer value rounded up after S / Y. M1 is equal to the number of data blocks included in the data block array in the column direction.
[0071] Here, linearly storing the elements in the data block array into the memory space means storing the elements sequentially into the memory space according to the storage order of the elements in the data block array. That is, breaking the Sub-Block and Block constraints of the elements in the data block array, and converting the data block array of the first Layout into one-dimensional data of the plain Layout.
[0072] The following is an introduction to the process of linearly storing the elements in the data block array into the memory space.
[0073] During specific implementation, each first thread block may be controlled to perform the following operations:
[0074] Step 1: Determine the data block (possibly one or more) corresponding to itself in the data block array according to its position number in the N1×M1 thread blocks;
[0075] Step 2: Read the elements in the data block into multiple registers of the thread block through the Layout data loading interface according to the coordinates of the data block;
[0076] Step 3: According to the coordinates of the elements in the multiple registers, determine the addresses of the elements in the multiple registers in the memory space;
[0077] Step 4: Write the elements in the multiple registers into the memory space according to the addresses of the elements in the multiple registers in the memory space.
[0078] In some embodiments, if the data block array corresponds to a first memory architecture and the input of the second operator corresponds to a second memory architecture, then before reading the elements in the data block into multiple registers of the thread block through the Layout data loading interface, the parameter value of the memory architecture parameter in the Layout data loading interface can also be set to the second memory architecture, so as to convert the elements in the data block from the corresponding first memory architecture to the corresponding second memory architecture with the help of the Layout data loading interface.
[0079] Among them, the first memory architecture and the second memory architecture are different memory architectures. For example, the first memory architecture is a non-uniform memory architecture (None-Uniform Memory Access, NUMA), and the second memory architecture is a uniform memory architecture (Uniform Memory Access, UMA). For another example, the first memory architecture is UMA, and the second memory architecture is NUMA.
[0080] In some embodiments, if the output of the first operator corresponds to a first data type and the input of the second operator corresponds to a second data type, after the elements in the data block are read into multiple registers of the thread block through the Layout data loading interface, the elements in the multiple registers can also be converted from the first data type to the second data type.
[0081] That is to say, the solution of the embodiment of the present application not only supports the conversion of the memory architecture, but also supports the conversion of the data type during the Layout conversion process, and the conversion function is relatively powerful.
[0082] In addition, it should be noted that the above-mentioned multiple first thread blocks can be released after use to save thread resources.
[0083] In step 303, the elements in the memory space are converted into a data block array of the second Layout using a plurality of second thread blocks, wherein the number of the plurality of second thread blocks is determined according to the array size corresponding to the input of the second operator.
[0084] In practical applications, the input of the second operator is also a data block array. Although the specific values of the elements in this data block array cannot be known in advance, the array size of this data block array can be known in advance, such as the number of data blocks contained in the row direction (i.e., H direction), the number of data blocks contained in the column direction (i.e., W direction), etc., and it can be known in advance that the data blocks in this data block array are arranged according to the second Layout. To this end, a reference block array can be constructed in advance, and the array shape of the reference block array is the same as the array shape of this data block array, and the reference blocks in the reference block array are also arranged according to the second Layout. In other words, the shape of the reference block array is exactly the same as that of this data block array. Then, the array size corresponding to the input of the second operator can be equivalent to the number of reference blocks contained in the reference block array in the row direction (i.e., H direction), the number of reference blocks contained in the column direction (i.e., W direction), etc.
[0085] In some embodiments, N2×M2 second thread blocks can be started, N2 can be determined according to the number of reference blocks included in the reference block array in the row direction and the number of reference blocks corresponding to a single second thread block, and the number of data blocks corresponding to a single second thread block can be determined according to the number of threads included in the single second thread block and the number of elements included in a single data block (the elements must not exceed the memory boundary), and the register of each thread is used to store one element. Assuming that the number of data blocks corresponding to a single second thread block is S′, and the number of data blocks included in the data block array in the row direction is Y′, then N2 is equal to the integer value rounded up after S′ / Y′. M2 is equal to the number of reference blocks included in the reference block array in the column direction.
[0086] The following describes the process of converting the elements in the memory space into the data block array of the second Layout.
[0087] During specific implementation, each second thread block may be controlled to perform the following operations:
[0088] Step 1: Each second thread block determines its corresponding reference block (which may be one or more) in the reference block array according to its own coordinates;
[0089] Step 2: According to the coordinates of the elements in the reference block, determine the addresses of the elements in the reference block in the memory space;
[0090] It should be noted that the elements in the reference block can all be preset values such as 0, 1, etc.
[0091] Step 3: read the elements at the addresses of the elements in the reference block in the memory space into multiple registers of the second thread block;
[0092] Step 4: Write the elements in these multiple registers into the reference block through the Layout data saving interface.
[0093] That is, the elements in the reference block are overwritten with the elements in the memory space.
[0094] Finally, the reference block array that has been written is determined as the converted data block array.
[0095] Similarly, the plurality of second thread blocks are released after being used to save thread resources.
[0096] In step 304, the converted data block array is input into a second operator to execute the second operator.
[0097] In the embodiment of the present application, a continuous memory space is used to convert the output first Layout into one-dimensional data (i.e., data in the form of plain Layout) stored linearly, and then the one-dimensional data is converted into the input second Layout, so that conversion between various different Layouts can be completed, and a universal Layout conversion solution is provided. Since there is no need to develop conversions between various Layouts separately, the development cost of Layout conversion can be reduced.
[0098] The following takes the output of operator 1 as the input of operator 2, the output of operator 1 corresponds to UMA, 32-bit floating point, ColMajor data, and the input of operator 2 corresponds to NUMA, 8-bit integer, Activation data as an example to introduce the solution of the embodiment of the present application.
[0099] Assume that the data block array 1 output by operator 1 is ColMajor(N1H30W80), that is, the output data of operator 1 is arranged according to ColMajor, there is one (H, W) in the N direction, the number of elements in the H direction is 30, and the number of elements in the W direction is 80, and assume that the array size of data block array 1 is 8 blocks in the H direction and 3 blocks in the W direction.
[0100] Assume that the data block array 2 output by operator 2 is Activation (N1C4H20W30), that is, the output data of operator 2 is arranged according to Activation, there is one (C, H, W) in the N direction, the number of elements in the C direction is 4, the number of elements in the H direction is 20, and the number of elements in the W direction is 30, and assume that the array size of data block array 2 is 1 Block in the C direction, 6 Blocks in the H direction, and 4 Blocks in the W direction.
[0101] Since data block array 2 needs to be obtained by converting data block array 1, a reference block array with the same shape as data block array 2 can be constructed. The reference block array is also Activation (N1C4H20W30), and has 1 Block in the C direction, 6 Blocks in the H direction, and 4 Blocks in the W direction.
[0102] See also Figure 4 , Figure 4 A schematic diagram of a Layout conversion process provided in an embodiment of the present application is shown below in conjunction with Figure 4 The conversion process is described in detail.
[0103] The first step is to open up a continuous memory (i.e. plain memory) space according to the number of elements contained in data block array 1.
[0104] Since the data block array 1 contains 30×80=2400 elements, a plain memory space that can store 2400 elements can be opened.
[0105] The second step is to convert the UMA, 32-bit floating point, ColMajor data to NUMA, 8-bit integer plain data.
[0106] 2.1. According to the array size of the data block array 1, start a corresponding number of first thread blocks.
[0107] In ColMajor, each Block contains 4 Sub-Blocks, each Sub-Block contains 32 elements, so each Block contains 4×32=128 elements, and a single first thread block can contain 512 threads, so a single first thread block corresponds to 4 Blocks.
[0108] Data block array 1 has 8 blocks in the H direction and 3 blocks in the W direction. Three first thread blocks are also reserved in the column direction. Then the number of first thread blocks required in the row direction = 8 / 4 = 2, that is, 6 first thread blocks in total of 2 rows and 3 columns are started.
[0109] 2.2. Each first thread block determines its corresponding Block (ie, data block) in data block array 1 according to its own position number.
[0110] Here, there are 6 first thread blocks, and data block array 1 has 3 rows and 8 columns with a total of 24 blocks. Each first thread block corresponds to at most 4 blocks, then:
[0111] The first thread block at position (0, 0) corresponds to Block0 to Block3;
[0112] The first thread block at position (1, 0) corresponds to Block4 to Block7;
[0113] The first thread block at position (0, 1) corresponds to Block8 to Block11;
[0114] The first thread block at position (1, 1) corresponds to Block12 to Block15;
[0115] The first thread block at position (0, 2) corresponds to Block16 to Block19;
[0116] The first thread block at position (1, 2) corresponds to Block 20 to Block 23.
[0117] 2.3. Each thread block loads the elements (512 in total) in the Block into the registers of its own 512 threads according to the coordinates of its corresponding Block through the Layout dedicated data loading application programming interface (Application Programming Interface, API) (i.e., Layout data loading interface), where the register of each thread stores one element.
[0118] The memory architecture mem arch is a parameter of the Layout-specific data loading API. By setting the parameter value to NUMA, the elements in the Block can be converted into corresponding NUMA through the Layout-specific data loading API.
[0119] 2.4. Each thread in each thread block converts the elements in its own register from 32-bit floating point to 8-bit integer.
[0120] 2.5. Each thread in each thread block calculates the address of the element in plain memory based on the coordinates of the element in its own register.
[0121] For example, the address of an element in plain memory = the starting address of plain memory + offset;
[0122] offset = n*H*W+h*W+w;
[0123] Where (n, h, w) is the coordinate of the element in data block array 1.
[0124] 2.6. Each thread in each thread block writes the elements in its register to the plain memory through the store instruction according to the address of the elements in the plain memory. Figure 4 Plain data in , with a total of 2400 elements.
[0125] The third step is to convert NUMA, 8-bit integer plain data to NUMA, 8-bit integer, Activation data.
[0126] 3.1. Initiate a corresponding number of second thread blocks according to the array size corresponding to the input of the second operator.
[0127] In Activation, each Block contains 4 Sub-Blocks, each Sub-Block contains 128 elements, so each Block contains 4×128=512 elements, but 3 / 4 of the elements in the C direction exceed the boundary when stored, so each Block has only 128 valid elements, and a single second thread block can contain 512 threads, so a single second thread block corresponds to 4 Blocks.
[0128] Data block array 2 has 6 blocks in the H direction and 4 blocks in the W direction. 4 second thread blocks are also reserved in the column direction. Then the number of second thread blocks required in the row direction = 6 / 4 = 1.5, which is rounded up to 2, that is, 8 second thread blocks in total of 2 rows and 4 columns are started.
[0129] 3.2. Each second thread block determines its corresponding Block (ie, reference block) in the reference block array according to its own position number.
[0130] Here, there are 8 second thread blocks, and the reference block array has 24 blocks, then each second thread block corresponds to at most 4 blocks, where:
[0131] The second thread block at position (0,0) corresponds to Block0 to Block3;
[0132] The second thread block at position (1, 0) corresponds to Block4 to Block5;
[0133] The second thread block at position (0, 1) corresponds to Block6 to Block9;
[0134] The second thread block at position (1, 1) corresponds to Block10 to Block11;
[0135] The second thread block at position (0, 2) corresponds to Block12 to Block15;
[0136] The second thread block at position (1, 2) corresponds to Block16 to Block17;
[0137] The second thread block at position (0, 3) corresponds to Block18 to Block21;
[0138] The second thread block at position (1, 3) corresponds to Block22 to Block23.
[0139] Among them, the second thread block at the (1, i) position corresponds to 2 blocks, because the elements of 2 blocks are out of the boundary, do not occupy space in the memory, and do not need to be written out.
[0140] 3.3. Each thread in each thread block calculates the address of the element in the plain memory according to the coordinates of the corresponding element in the Block.
[0141] For example, the address of an element in plain memory = the starting address of plain memory + offset′;
[0142] offset′=n′*C*H*W+c′*H*W+h′*W+w′;
[0143] Where (n′, c′, h′, w′) are the coordinates of the element in the reference block array.
[0144] 3.4. Each thread loads the elements in the plain memory into its own register through the load instruction according to the address of the elements in the plain memory.
[0145] 3.5. Each thread block writes the elements in the registers of each thread inside to the Block through the Layout dedicated data storage API (i.e., Layout data storage interface) according to the coordinates of the corresponding Block.
[0146] Step 4: Input the written reference block array into the second operator to execute the second operator.
[0147] The solution of the embodiment of the present application can complete the conversion between various Layout combinations without having to develop a program for a group of Layouts separately. It has good versatility. Moreover, during the Layout conversion process, it also supports the conversion of memory architecture and data type, and the conversion function is also relatively powerful.
[0148] Based on the same technical concept, the embodiment of the present application also provides an operator execution device. The principle of solving the problem by the operator execution device is similar to that of the above-mentioned operator execution method. Therefore, the implementation of the operator execution device can refer to the implementation of the operator execution method, and the repeated parts will not be repeated.
[0149] Figure 5 A schematic diagram of the structure of an operator execution device provided in an embodiment of the present application includes:
[0150] An opening module 501 is used for opening a continuous memory space according to the number of elements contained in the data block array before inputting the data block array of the first data structure Layout output by the first operator into the second operator if it is determined that the input of the second operator corresponds to the second Layout;
[0151] A first conversion module 502, configured to linearly store the elements in the data block array into the memory space by using a plurality of first thread blocks, wherein the number of the plurality of first thread blocks is determined according to the array size of the data block array;
[0152] A second conversion module 503, configured to convert the elements in the memory space into a data block array of the second Layout by using a plurality of second thread blocks, wherein the number of the plurality of second thread blocks is determined according to an array size corresponding to an input of the second operator;
[0153] The input module 504 is used to input the converted data block array into the second operator to execute the second operator.
[0154] In some embodiments, there are N1×M1 first thread blocks in total, N1 is determined based on the number of data blocks included in the data block array in the row direction and the number of data blocks corresponding to a single first thread block, and M1 is equal to the number of data blocks included in the data block array in the column direction.
[0155] In some embodiments, the first conversion module 502 is specifically configured to:
[0156] Control each first thread block to perform the following operations:
[0157] Determining, according to the position number of the first thread block, a data block corresponding to the first thread block in the data block array;
[0158] Reading the elements in the data block into a plurality of registers of the first thread block through a Layout data loading interface;
[0159] Determine addresses of the elements in the multiple registers in the memory space according to the coordinates of the elements in the multiple registers;
[0160] According to the addresses, the elements in the plurality of registers are written into the memory space.
[0161] In some embodiments, the Layout data loading interface has memory architecture parameters, and the first conversion module 502 is further used to:
[0162] If the data block array corresponds to a first memory architecture and the input of the second operator corresponds to a second memory architecture, before reading the elements in the data block into multiple registers of the first thread block through the Layout data loading interface, the parameter value of the memory architecture parameter is set to the second memory architecture, so as to convert the elements in the data block into the corresponding second memory architecture through the Layout data loading interface.
[0163] In some embodiments, the first conversion module 502 is further configured to:
[0164] If the output of the first operator corresponds to a first data type and the input of the second operator corresponds to a second data type, after reading the elements in the data block into multiple registers of the first thread block through the Layout data loading interface, the elements in the multiple registers are converted into the second data type.
[0165] In some embodiments, there are N2×M2 second thread blocks in total, N2 is determined based on the number of reference blocks included in the reference block array in the row direction and the number of reference blocks corresponding to a single second thread block, M2 is equal to the number of reference blocks included in the reference block array in the column direction, the array shape of the reference block array is the same as the array shape corresponding to the input of the second operator, and the reference blocks in the reference block array are arranged according to the second Layout.
[0166] In some embodiments, the second conversion module 503 is specifically configured to:
[0167] Control each second thread block to perform the following operations:
[0168] Determine, according to the position number of the second thread block, a reference block corresponding to the second thread block in the reference block array;
[0169] Determine the address of the element in the reference block in the memory space according to the coordinates of the element in the reference block;
[0170] Reading the element at the address into a plurality of registers of the second thread block;
[0171] Writing the elements in the plurality of registers into the reference block through a Layout data saving interface;
[0172] The reference block array after writing is determined as the converted data block array.
[0173] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated in a processor, or may exist physically separately, or two or more modules may be integrated in one module. The coupling between the modules may be achieved through some interfaces, which are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.
[0174] After introducing the operator execution method and apparatus according to the exemplary embodiment of the present application, next, a computer device according to another exemplary embodiment of the present application is introduced.
[0175] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 6 As shown, it includes at least one artificial intelligence chip 100 and a memory 601 connected to the at least one artificial intelligence chip 100. The specific connection medium between the artificial intelligence chip 100 and the memory 601 is not limited in the embodiment of the present application. Figure 6 In the example, the artificial intelligence chip 100 and the memory 601 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.
[0176] In an embodiment of the present application, the memory 601 stores instructions that can be executed by at least one artificial intelligence chip 100. At least one artificial intelligence chip 100 can perform the steps of the above-mentioned cache line scheduling method by executing the instructions stored in the memory 601.
[0177] Among them, the artificial intelligence chip 100 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and the operator execution is realized by running or executing instructions stored in the memory 601 and calling the data stored in the memory 601. Optionally, the artificial intelligence chip 100 may include one or more processing units, and the artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 601 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.
[0178] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor, or can be executed by a combination of hardware and software modules in the processor.
[0179] Memory 601, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 601 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 601 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 601 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0180] In an exemplary embodiment, a storage medium is also provided, and when a computer program in the storage medium is executed by a processor of a computer device, the computer device can perform the above-mentioned operator execution method. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0181] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed by a computer device, the computer device can implement any exemplary method provided in the present application.
[0182] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.
[0183] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0185] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.
[0186] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0188] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0189] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also includes these modifications and variations.
Claims
1. An operator execution method, characterized in that: include: Before inputting the data block array of the first data structure Layout output by the first operator into the second operator, if it is determined that the input of the second operator corresponds to the second Layout, a continuous memory space is allocated according to the number of elements contained in the data block array; Using a plurality of first thread blocks, linearly storing the elements in the data block array into the memory space, wherein the number of the plurality of first thread blocks is determined according to the array size of the data block array; Using a plurality of second thread blocks, converting the elements in the memory space into a data block array of the second Layout, wherein the number of the plurality of second thread blocks is determined according to an array size corresponding to an input of the second operator; The converted data block array is input into the second operator to execute the second operator.
2. The method according to claim 1, characterized in that There are N1×M1 first thread blocks in total, N1 is determined according to the number of data blocks included in the data block array in the row direction and the number of data blocks corresponding to a single first thread block, and M1 is equal to the number of data blocks included in the data block array in the column direction.
3. The method according to claim 1 or 2, characterized in that Using a plurality of first thread blocks, linearly storing elements in the data block array into the memory space includes: Control each first thread block to perform the following operations: Determining, according to the position number of the first thread block, a data block corresponding to the first thread block in the data block array; Reading the elements in the data block into a plurality of registers of the first thread block through a Layout data loading interface; Determine addresses of the elements in the multiple registers in the memory space according to the coordinates of the elements in the multiple registers; According to the addresses, the elements in the plurality of registers are written into the memory space.
4. The method according to claim 3, characterized in that The Layout data loading interface has memory architecture parameters and also includes: If the data block array corresponds to a first memory architecture and the input of the second operator corresponds to a second memory architecture, before reading the elements in the data block into multiple registers of the first thread block through the Layout data loading interface, the parameter value of the memory architecture parameter is set to the second memory architecture, so as to convert the elements in the data block into the corresponding second memory architecture through the Layout data loading interface.
5. The method according to claim 3, characterized in that Also includes: If the output of the first operator corresponds to a first data type and the input of the second operator corresponds to a second data type, after reading the elements in the data block into multiple registers of the first thread block through the Layout data loading interface, the elements in the multiple registers are converted into the second data type.
6. The method according to claim 1, characterized in that There are N2×M2 second thread blocks in total, N2 is determined according to the number of reference blocks included in the reference block array in the row direction and the number of reference blocks corresponding to a single second thread block, M2 is equal to the number of reference blocks included in the reference block array in the column direction, the array shape of the reference block array is the same as the array shape corresponding to the input of the second operator, and the reference blocks in the reference block array are arranged according to the second Layout.
7. The method according to claim 6, characterized in that Using a plurality of second thread blocks, converting the elements in the memory space into a data block array of the second Layout, comprising: Control each second thread block to perform the following operations: Determine, according to the position number of the second thread block, a reference block corresponding to the second thread block in the reference block array; Determine the address of the element in the reference block in the memory space according to the coordinates of the element in the reference block; Reading the element at the address into a plurality of registers of the second thread block; Writing the elements in the plurality of registers into the reference block through a Layout data saving interface; The reference block array after writing is determined as the converted data block array.
8. An operator execution device, characterized in that: include: an opening module, configured to open a continuous memory space according to the number of elements contained in the data block array before inputting the data block array of the first data structure Layout output by the first operator into the second operator if it is determined that the input of the second operator corresponds to the second Layout; A first conversion module, configured to linearly store the elements in the data block array into the memory space by using a plurality of first thread blocks, wherein the number of the plurality of first thread blocks is determined according to the array size of the data block array; A second conversion module, configured to convert the elements in the memory space into a data block array of the second Layout by using a plurality of second thread blocks, wherein the number of the plurality of second thread blocks is determined according to an array size corresponding to an input of the second operator; An input module is used to input the converted data block array into the second operator to execute the second operator.
9. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method as described in any one of claims 1 to 7 are implemented.
10. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of a computer device, the computer device can execute the method according to any one of claims 1 to 7.
11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by a processor.