Data access mode derivation method of computational graph and related product

Through the memory fetch mode derivation scheme directly based on tensor data, the problem of inefficiency of traditional instruction generation methods on MLU is solved, and the direct conversion of tensor calculation graph to tensor instructions is realized, supporting high-performance computing of MLU.

CN118916081APending Publication Date: 2024-11-08CAMBRICON TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411124813.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-07
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional instruction generation methods are inefficient when targeting machine learning processor MLUs, and cannot directly generate instructions based on tensor data, resulting in cumbersome and complex processing and cannot support high-performance computing.

Method used

Provide a memory access mode derivation scheme based on tensor data. By selecting the entry operator, splitting the data block and deducing the memory access address, the direct conversion of the tensor calculation graph to the tensor instruction is realized, and the MLU-class tensor processing is supported. instruction mapping of the device.

Benefits of technology

It simplifies the instruction generation process, improves computing efficiency, supports high-performance computing of MLU, can effectively generate tensor instructions, and improves processor performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916081A_ABST
    Figure CN118916081A_ABST
Patent Text Reader

Abstract

The invention discloses a data access mode derivation method of a computational graph, a computing device and a related product. The method comprises the following steps: selecting an entry operator from the computational graph; according to the dimension splitting scheme of the data of the entry operator, splitting the data of the entry operator into one or more data blocks; and deriving a memory access mode of data associated with the entry operator data in the computational graph based on the memory access mode of the entry operator data, the memory access mode comprising a memory access address of each data block obtained by splitting the data. According to the scheme disclosed by the invention, a scheme for deducing the memory access mode of the data in the computational graph is provided, and the memory access mode of each data in the computational graph can be automatically deduced for subsequent memory space allocation and / or instruction generation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to the field of intelligent computing, and more particularly to the field of neural networks. More specifically, the present disclosure relates to a method, a computing device, a computer-readable storage medium, and a computer program product for deriving memory access patterns for data in a computational graph. Background Art

[0002] In intelligent computing systems, common operations in neural network model algorithms are encapsulated into operators through programming frameworks for programmers to call directly, such as convolution and pooling. TensorFlow, PyTorch, etc. are currently popular deep learning frameworks. In these programming frameworks, computational graphs are usually used to describe the calculation process of machine learning algorithms, tensors are used to represent all data in the computational graphs, and operators are used to represent various operations.

[0003] Traditional instruction generation is mainly aimed at target hardware such as CPU and GPU. The instruction generation process is to first convert the operators in the computation graph that are represented by tensors into scalars; then generate tensor instructions based on this scalar intermediate representation (IR). Specifically, if there are vector instructions on the target hardware, tensor optimization will be performed again. Tensors that can be quantized will be quantized, and scalar calculations will be used for those that cannot be calculated. When targeting intelligent accelerators such as machine learning processors MLU, if the traditional instruction generation method is still used, the processing process will be cumbersome and inefficient. Because in MLU, if you want to achieve high performance, basically all operators can or even must be calculated using tensor instructions.

[0004] In the process of generating instructions based on the computation graph, it is necessary to derive the memory access address of the data in the computation graph, so as to provide corresponding information for the subsequent allocation of on-chip storage space and instruction generation. Therefore, if the instruction generation process for MLU is to be simplified, it is necessary to provide a corresponding memory access address derivation solution directly based on tensor data. Summary of the invention

[0005] In order to at least partially solve one or more technical problems mentioned in the background technology, the present disclosure provides solutions from multiple aspects.

[0006] In a first aspect, the present disclosure discloses a method for inferring a memory access pattern of data in a computation graph, comprising:

[0007] Selecting an entry operator from the computation graph;

[0008] Splitting the inlet operator data into one or more data blocks according to the dimension splitting scheme of the inlet operator data; and

[0009] Based on the memory access mode of the entry operator data, a memory access mode of data associated with the entry operator data in the computation graph is derived, wherein the memory access mode includes a memory access address of each data block into which the data is split.

[0010] In a second aspect, the present disclosure provides a computing device for performing memory access pattern derivation, comprising a processor configured to execute program instructions; and a memory configured to store the program instructions, so that when the program instructions are loaded and executed by the processor, the processor executes the method described in the first aspect of the present disclosure.

[0011] In a third aspect, the present disclosure provides a computer-readable storage medium having program instructions stored therein, which, when loaded and executed by a processor, causes the processor to execute the method according to the first aspect of the present disclosure.

[0012] In a fourth aspect, the present disclosure provides a computer program product, comprising a computer program or instructions, which implement the method described in the first aspect of the present disclosure when executed by a processor.

[0013] By using the scheme for deducing the memory access mode of data in the computation graph as provided above, the memory access mode of the associated data on the computation graph can be deduced according to the splitting scheme on a certain dimension of the data and the corresponding memory access mode. This derivation scheme for memory access addresses can provide support for the instruction generation process of generating tensor instructions directly based on the computation graph represented by tensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0015] Figure 1 An exemplary splitting of data and the corresponding memory access mode are shown;

[0016] Figure 2 An exemplary representation of a data when it is split multiple times in the same dimension is shown;

[0017] Figure 3 An exemplary flow chart of a method for deriving a memory access mode for data in a computation graph according to an embodiment of the present disclosure is shown;

[0018] Figure 4 An example of memory access mode derivation according to an embodiment of the present disclosure is shown;

[0019] Figure 5a-5dVarious memory access examples are shown when multiplexing exists in the same memory access mode of data;

[0020] Figure 6 An optimized memory access mode derivation example according to an embodiment of the present disclosure is shown;

[0021] Figure 7 An example is shown in which there are multiple access modes for the same data;

[0022] Figure 8 An example of memory access mode merging according to an embodiment of the present disclosure is shown;

[0023] Fig. 9 An example of multiplexing optimization based on the merged final memory access mode according to an embodiment of the present disclosure is shown;

[0024] Fig.10 An example of memory access mode merging and multiplexing optimization according to an embodiment of the present disclosure is shown;

[0025] Fig.11 Another example of memory access mode merging and multiplexing optimization according to an embodiment of the present disclosure is shown;

[0026] Fig.12 A block diagram showing a hardware configuration of a computing device that can implement various aspects of the disclosed embodiments;

[0027] Fig.13 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0028] Fig.14 A schematic diagram of the structure of a board according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0030] It should be understood that the terms "first", "second", "third", and "fourth" etc. that may appear in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0031] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0032] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0033] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0034] As mentioned in the background technology, traditional instruction generation involves converting a tensor computation graph into a scalar IR, and then generating a tensor instruction based on the scalar IR. In this traditional instruction generation, there are usually two ways to derive the address of the data involved in the operator. One is to derive by describing a scalar derivation formula of a certain dimension, and the other is to derive the derivation formula from the calculation formula by describing the calculation formula of a certain operator. However, both of these derivation methods are based on scalars, not tensors.

[0035] In order to adapt to general systems such as CPUs, traditional instruction generation has flexible operator definitions and strong versatility, but this also brings many disadvantages. For example, due to the flexibility of the operator, the analysis process is complicated, the analysis time is long, and the compilation time is also long. For another example, it is impossible to perform shape inference for operators with control flow (such as pad operators). Moreover, it can only be used for eltwise-type front and back operator fusion solutions, and cannot support deep fusion. In addition, there is no multiplexing between processing time slices. This process of scalar IR to tensor instructions is also very unfavorable for instruction mapping of MLU-type tensor processors.

[0036] In view of this, the disclosed embodiment provides a memory access mode derivation scheme directly based on tensor data, thereby realizing the direct conversion process from tensor computation graph to tensor instruction, and effectively supporting the instruction mapping of MLU type tensor processors.

[0037] In this article, the memory access mode refers to the memory access address for accessing each data block into which the data is split after a certain dimension of the data is split or divided according to the dimension splitting scheme.

[0038] In this article, memory access mode derivation or memory access address derivation means that when a certain dimension of the data is split or divided, it is necessary to derive the memory access address of each time / each data block of the associated operator in the calculation graph for the input data and / or output data after the split according to the split data blocks. The associated operator here refers to the operator in the calculation graph that needs to directly or indirectly access the data, for example, it is necessary to directly or indirectly use the data to perform calculations, or it is necessary to directly or indirectly write the calculation results to the data. In addition, in the following description, unless otherwise specified, the data involved are all tensor data.

[0039] Regarding the terms "node" and "operator (OP)" mentioned in this disclosure, it should be noted that the term "operator" is from the computing level of the computer (or from the software level or algorithm level); and the term "node" is a more figurative term (from the graphics level or a more intuitive level). In terms of the content referred to, the terms "operator" and "node" actually refer to the same. That is, in this disclosure, it can be considered that the terms "operator" and "node" have the same meaning and can be used interchangeably, but are described from different aspects.

[0040] Memory address expression

[0041] In order to conveniently represent the memory access mode and deduce the memory access mode, in the disclosed embodiment, a data structure is proposed, which can express the data splitting method and the corresponding memory access address.

[0042] In some embodiments, a data structure Interval is used to represent a memory access pattern of data in one dimension, and the data structure can represent an address memory access pattern of a loop. Specifically, Interval includes a mapping from a loop variable (range) to an address memory access pattern (pattern), and each loop corresponds to the memory access address of a data block into which the data is split.

[0043] The loop variable range, also known as the loop index, is used to mark different loops. The loop variable range can be represented by a minimum value min and a span extent, for example, as follows:

[0044]

[0045] The address access mode pattern is used to represent the address expression corresponding to each range. There are many ways to express the address.

[0046] One way is to use an expression, including a starting value start and a size size, such as:

[0047]

[0048] Another way is to use a data structure, including a minimum value min, a step length stride (the address interval between adjacent data blocks) and a span extent, which is, for example:

[0049]

[0050] Figure 1 An exemplary splitting of data and the corresponding memory access mode are shown. In this example, it is assumed that the length of the data in a certain dimension is 14, and the data is split according to the splitting factor factor=3.

[0051] As shown in the figure, the data can be split into 5 blocks according to 3*4+2, which means 5 loops / memory accesses are required. The stride of each memory access is 3, the size of the first 4 memory accesses is 3, and the size of the last memory access is 2.

[0052] The loop indexes of these five data blocks and the memory access addresses of the corresponding data blocks are:

[0053] 0→[0,3), 1→[3,3), 2→[6,3), 3→[9,3), 4→[12,2)

[0054] From the above mapping, we can see that the starting address and size of the data block access address corresponding to indexes 0 to 3 change regularly, so they can be reduced to an expression:

[0055] start=i*3 (i is the loop variable)

[0056] size=3

[0057] It is expressed in the form of the Interval data structure provided by the embodiment of the present disclosure, that is:

[0058] range→pattern: [0,4)→[i*3,3)[4,1)→[i*3,2)

[0059] The Interval data structure can also be represented in the form of a for loop, which is not expanded here.

[0060] The above describes the memory access mode of data in one dimension. A tensor data is composed of multiple dimensions. When it is necessary to represent the split of each dimension corresponding to a tensor data, that is, the memory access mode in multiple dimensions, the data structure Region can be used to represent it.

[0061] Specifically, Region contains the dimension order of the data (dim_order) and the memory access mode of each dimension, where the memory access mode of each dimension can be represented using the Interval data structure described above. For example, Region can be represented as:

[0062]

[0063] In some embodiments of the present disclosure, for memory access modes in multiple dimensions, the memory access modes in each dimension are relatively independent and can be represented by nested loops (such as for loops). The nesting order of the loops is consistent with the order of the dimensions, and each memory access mode in each dimension corresponds to a for loop.

[0064] In actual instruction generation, a certain dimension of data often needs to be split multiple times. Such multiple splits can be represented by a tree structure, where each additional split adds a child to the current split. Figure 2 An exemplary representation is shown when a data is split multiple times in the same dimension.

[0065] As shown in the figure, a certain dimension of the data at the root node 210 is split into four memory accesses, which are represented in the form of the Interval data structure provided in the embodiment of the present disclosure, namely:

[0066] range→pattern: [0,3)→[i*6,6)[3,1)→[i*6,4)

[0067] Furthermore, the four memory accesses need to be further split. Specifically, the splitting method of the first three memory accesses is the same, and the splitting method of the last memory access is different. Therefore, based on the root node, a child node child1 is added for the first three memory accesses, and a child node child2 is added for the last memory access. The figure shows the memory access mode splitting of each child node and its Interval data structure. Therefore, multiple splits of data on a certain dimension can be organized through a tree structure.

[0068] Memory access mode derivation scheme

[0069] The following describes a memory access mode derivation scheme provided by the disclosed embodiment. When performing memory access mode derivation, the aforementioned expression method may be used to represent the memory access mode of data.

[0070] Figure 3 An exemplary flow chart of a method for inferring memory access patterns for data in a computation graph according to an embodiment of the present disclosure is shown.

[0071] As shown in the figure, in step 310, an entry operator is selected from the computation graph.

[0072] A computational graph is a directed graph consisting of nodes (operators) and edges. Tensor data is passed between nodes in the computational graph. The execution of the computational graph follows the order of the directed graph. Each time a tensor passes through a node, it is calculated as the input of the node's operation, and the result of the calculation flows along the output edge of the node to the subsequent node.

[0073] A computation graph usually includes many operators. In the disclosed embodiment, it is necessary to determine an entry operator for memory access mode derivation, so as to perform memory access mode derivation with the entry operator as the starting point.

[0074] There are many ways to select the entry operator. Preferably, in some embodiments, the entry operator can be selected according to any of the following rules:

[0075] A. Select the end operator in the computation graph as the entry operator. The end operator is an operator in the computation graph that has no successor operator, and is usually the last operator in the computation graph. When the end operator is selected as the entry operator, the memory access mode is derived in the reverse direction, that is, the memory access mode of the input data is derived from the memory access mode of the output data in the opposite direction of the data flow in the computation graph. The advantage of this reverse derivation method is that it ensures that the addresses of all input data required to generate the output data can be derived.

[0076] B. Select the starting operator in the calculation graph as the entry operator. The starting operator refers to the operator in the calculation graph that has no preceding operator, which is usually the first operator in the calculation graph. When the starting operator is selected as the entry operator, the derivation direction of the memory access mode is forward, that is, along the direction of data flow in the calculation graph, the memory access mode of the output data is forward deduced from the memory access mode of the input data. The advantage of this forward derivation method is that its derivation direction is consistent with the data flow direction, and the operation logic is clear, but the possible problem is that when the input data splitting method is not appropriate, the input data required for an output data block may be distributed on multiple input data blocks, so multiple attempts are required.

[0077] C. According to the scale change trend of the input data and output data in the calculation graph, the inflection point operator at the change inflection point is selected as the entry operator. In some neural networks, the scale of data transmitted between each network layer will change, and the trend of scale change will also change, resulting in a change inflection point. For example, U-net is a fully convolutional neural network composed entirely of convolutional layers, and its structure is symmetrical. In the data flow direction, the data scale first changes from large to small, and then changes from small to large, so there will be a change inflection point in the middle. In the calculation graph of this neural network, the inflection point operator at the change inflection point can be selected as the entry operator, and the entry operator is used as the derivation starting point to reversely derive the data access mode of the associated operator before the entry operator, and at the same time, the data access mode of the associated operator after the entry operator is forward derived. Preferably, an operator with a smaller data scale is used as the entry operator. The advantage of this method of selecting the entry operator is that it can ensure that the addresses of all input data required for the data blocks split in the entry operator to generate output data can be derived.

[0078] The above describes the selection method of the entry operator. It can be understood that the computation graph can also be split into several computation subgraphs, and the same or different selection rules are applied in each computation subgraph to select the entry operator. Therefore, the computation graph mentioned in the embodiments of the present disclosure, unless otherwise specified, can be either the entire computation graph of the neural network model or the computation subgraph after the splitting.

[0079] After the entry operator is selected, then, in step 320, the entry operator data is split into one or more data blocks according to the dimension splitting scheme of the entry operator data.

[0080] The data of the entry operator can be the input data of the entry operator or the output data of the entry operator, and can usually be provided according to the derivation direction. For example, when reverse derivation is required based on the entry operator, a dimension splitting scheme of the output data of the entry operator is usually provided; and when forward derivation is required based on the entry operator, a dimension splitting scheme of the input data of the entry operator is usually provided.

[0081] Finally, in step 330, based on the memory access pattern of the entry operator data, the memory access pattern of the data associated with the entry operator data in the computation graph is derived.

[0082] When the entry operator data is split according to the dimension splitting scheme, its corresponding memory access mode can also be determined. The memory access mode includes the memory access address of each data block into which the data is split. In some embodiments, the memory access mode can be described using the Intervel data structure described above. When the data is split in multiple dimensions, it can be described using the Region data structure. For simplicity, the following description will be based on the splitting of data in one dimension and the derivation of the memory access mode as an example, but those skilled in the art can derive the memory access mode derivation scheme of other dimensions according to the scheme described in this article, for example, in a dimensional sequential loop.

[0083] In some embodiments, memory access pattern derivation may include: based on the position of the entry operator, starting from the entry operator, traversing each layer of operators in the computational graph from back to front and / or from front to back, performing reverse and / or forward shape derivation to obtain the memory access addresses of one or more input data blocks and output data blocks that each layer of operators needs to split to meet the dimensional splitting scheme of the entry operator.

[0084] The inventors discovered through studying deep learning algorithms that although the network structure of a neural network is complex, its basic components are relatively stable, mainly including operators such as convolution (conv), pooling (pool), padding (pad), and vector operations. These basic operators take up most of the time for the overall calculation. Further analysis of these basic operators revealed that they can be decomposed into some basic operations. In this article, these basic operations are referred to as atomic operations. Note that since the processing of data in intelligent computing involves division by dimension, the atomic operations here are also determined in units of dimension. In the disclosed embodiments, by analyzing the memory access mode of the basic operators, memory access mode derivation formulas for several kinds of atomic operations can be abstracted. Thus, the memory access mode derivation of various operator data can be performed based on these memory access mode derivation formulas.

[0085] Specifically, in the traversal process of the above-mentioned computational graph, for each layer of operators in the computational graph, reverse or forward shape deduction may include: decomposing the operation of the operator into several atomic operations by dimension; and based on the input-output memory access address relationship of the atomic operation, that is, the memory access mode deduction formula, reversely deriving the memory access address of the input data block required for the operator to calculate an output data block; or forward deriving the memory access address of the output data block after the operator calculates an input data block.

[0086] In some embodiments, the atomic operation may include one or more of the following:

[0087] Forward sliding window operation, such as used in operators such as convolution (conv) and pooling (pool), usually in width (W) and height (H) dimensions;

[0088] Forward padding operations, such as those used in convolution, pooling, or independent padding operators (pad size is a negative number);

[0089] Reverse sliding window operation, such as used in operators such as deconv and unpool, is usually also in the width (W) and height (H) dimensions;

[0090] Reverse padding operation, such as used in reverse convolution, or in a separate padding operator (pad size is a negative number);

[0091] Vector operations, which are used for various vector operations in operators, such as addition, subtraction, multiplication, activation calculation, etc.

[0092] Broadcast operations, which are used to broadcast copy operations, similar to how numpy broadcasts over a dimension, such as in cycle add; and

[0093] Aggregation operation (reduce) is used to aggregate data, such as the accumulation operation of the Ci dimension in the convolution (conv) operator, reduce calculation, etc.

[0094] Based on the above abstracted atomic operations, we can derive the memory access mode derivation formula according to their specific operation mode, that is, the input and output memory access address relationship of the atomic operation. In this way, when traversing the computation graph, we can decompose the operation of the current operator into several atomic operations by dimension, and then deduce them according to the memory access mode derivation formula of the atomic operation to obtain the memory access mode of the current operator's data.

[0095] For example, when the current operator is a convolution operator, the convolution operator can be split into: H and W dimensions are forward sliding window operations, and Ci dimension is an aggregation operation. Then, the H and W dimension data can be derived according to the memory access mode derivation formula of the forward sliding window operation, and the Ci dimension data can be derived according to the memory access mode derivation formula of the aggregation operation.

[0096] Specifically, when reverse deduction is adopted, the memory access mode of the input data of the convolution operator in the H and W dimensions is derived according to the memory access mode of the output data of the convolution operator in the H and W dimensions using the memory access mode derivation formula of the forward sliding window operation; and the memory access mode of the input data of the convolution operator in the Ci dimension is derived according to the memory access mode of the output data of the convolution operator in the C dimension using the memory access mode derivation formula of the aggregation operation.

[0097] Similarly, when forward deduction is adopted, the memory access pattern of the output data of the convolution operator in the H and W dimensions is derived according to the memory access pattern of the input data of the convolution operator in the H and W dimensions using the memory access pattern derivation formula of the forward sliding window operation; and the memory access pattern of the output data of the convolution operator in the C dimension is derived according to the memory access pattern of the input data of the convolution operator in the Ci dimension using the memory access pattern derivation formula of the aggregation operation.

[0098] Figure 4 An example of memory access mode derivation according to an embodiment of the present disclosure is shown. In this example, only two operators connected sequentially in the computation graph are shown: a pad operator and a convolution operator. This example uses a reverse derivation method, selects the convolution operator as the entry operator, and it is known that the output data size (size) of the convolution operator is 14, the step size of the convolution is 1, and the convolution kernel size is 3. One-dimensional (W-dimensional) data is used as an example here. The parameters of the pad operator include: left padding is 1, and right padding is also 1.

[0099] As shown in the figure, assuming that the dimension splitting scheme of the output data 410 of the convolution operator is: split the W dimension according to the splitting factor factor=3, then the output data 410 is split according to 3*4+2, thereby splitting into 5 data blocks 411~415. The memory access mode of these 5 data blocks can be expressed by the data structure Interval as follows:

[0100] range→pattern: [0,4)→[i*3,3)[4,1)→[i*3,2)

[0101] Next, based on the memory access mode of the output data 410 of the convolution operator, the memory access mode of the input data 420 of the convolution operator is inversely deduced. The operation of the convolution operator in the W dimension corresponds to the atomic operation of the forward sliding window. Therefore, according to the memory access mode derivation formula of the forward sliding window operation, the memory access mode of the input data 420 of the convolution operator in the W dimension can be derived.

[0102] According to the parameters of the convolution operator: convolution step size = 1, convolution kernel = 3, and the mapping relationship between the input and output memory access addresses of the forward sliding window operation, the memory access address of the input data block corresponding to each output data block can be derived. As shown in the figure, according to the derivation result, the input data 420 of the convolution operator also needs to be split into 5 data blocks 421-425, of which the size of the first 4 data blocks is 5, the size of the last data block is 4, and there is overlap between the data blocks. The memory access mode of these 5 data blocks 421-425 can be expressed by the data structure Interval as follows:

[0103] range→pattern: [0,4)→[i*3,5)[4,1)→[i*3,4)

[0104] The input data 420 of the convolution operator is also the output data of the preceding padding operator. Therefore, the memory access mode of the input data 430 of the padding operator can be further inferred based on the memory access mode of the output data 420 of the padding operator. The operation of the padding operator in the W dimension corresponds to the atomic operation of forward padding. Therefore, according to the memory access mode derivation formula of the forward padding operation, the memory access mode of the input data 430 of the padding operator in the W dimension can be derived.

[0105] According to the parameters of the padding operator: left padding = 1, right padding = 1, and the mapping relationship between the input and output memory access addresses of the forward padding operation, the memory access address of the input data block corresponding to each output data block can be derived. As shown in the figure, according to the derivation result, the input data 430 of the padding operator also needs to be split into 5 data blocks 431-435, where the size of the first data block is 4, the size of the middle 3 data blocks is 5, the size of the last data block is 3, and there is overlap between the data blocks. The memory access mode of these 5 data blocks 431-435 can be expressed by the data structure Interval as follows:

[0106] range→pattern: [0,1)→[0,4)[1,4)→[i*3-1,5)[4,1)→[i*3-1,3)

[0107] Thus, the memory access patterns of all associated data on the computation graph are derived. Based on these memory access patterns, corresponding memory allocation and / or instruction generation can be performed.

[0108] Reuse Optimization Solution

[0109] from Figure 4 It can also be seen that some data (such as Figure 4 There is overlap between the memory accesses of adjacent data blocks (data 420, data 430), which will cause repeated reading or writing of the same data.

[0110] In some embodiments of the present disclosure, in order to further optimize the solution, repeated memory access ranges between adjacent memory accesses may be reused, thereby reducing the amount of data actually required to be accessed and improving efficiency.

[0111] Specifically, in some embodiments, the overlapping data range between two adjacent memory accesses in the same memory access mode of data in the computation graph can be detected, where one memory access corresponds to a data block; and based on the overlapping data range, the data range that actually needs to be accessed or calculated for each memory access in the memory access mode is determined.

[0112] Figure 5a-5dVarious memory access examples when multiplexing exists in the same memory access mode of data are shown. In this example, it is assumed that the length of data 500 in a certain dimension is 16, and the data is split according to the splitting factor factor=5.

[0113] like Figure 5a As shown, data 500 is split into 5 blocks according to 5*4+4, which means that 5 loops / memory accesses are required. The stride of each memory access is 3, the size of the first 4 memory accesses is 5, and the size of the last memory access is 4. It can be seen that there is partial overlap between the adjacent data blocks of the 5 data blocks of data 500, so data reuse may occur.

[0114] There are two ways to represent the reusable data or the corresponding memory access address.

[0115] One way is to use the data intersection between the current data block and the previous data block to represent it, that is, the data intersection between the current memory access and the previous memory access can be referred to as reuse. In other words, reuse means that the range of the previous memory access can be reused in the current memory access.

[0116] Figure 5b The reuse representation shows the reuse between adjacent memory accesses of data 500. It can be seen that the calculation formula of reuse can be expressed as follows:

[0117] reuse(i)=intersect(access(i),access(i-1))

[0118] Where access(i) represents the i-th data block, and intersect represents the repeated data range between two adjacent data blocks.

[0119] Another way is to use the data intersection between the current data block and the next data block to represent it, that is, the data intersection between the current memory access and the next memory access, which can be referred to as reuse_by_next. In other words, reuse_by_next represents the range of the current memory access that will be used by the next memory access.

[0120] Figure 5c The reuse_by_next representation shows the reuse between adjacent memory accesses of data 500. It can be seen that the calculation formula of reuse_by_next can be expressed as follows:

[0121] reuse_by_next(i)=reuse(i+1)

[0122] Since adjacent memory accesses can be reused, the range that actually needs to be accessed in a certain memory access is the remaining part of the entire memory access after removing the reused part. This part can be referred to as compute. In other words, compute represents the part of data that the operator actually needs to calculate and write.

[0123] Figure 5d The optimized memory access mode of data 500 is shown. It can be seen that after removing the reuse part in each memory access, the remaining part is the compute part, and there is no overlap between the compute parts, thereby avoiding repeated calculations. The calculation formula of compute can be expressed as follows:

[0124] compute(i)=substract(access(i),reuse(i)),i>0

[0125] compute(0)=access(0)

[0126] When the memory is configured with two buffers (ping buffer space and pong buffer space) and adopts ping-pong pipeline mode to support data pipeline processing, the above-mentioned multiplexing between adjacent memory accesses can be carried out by moving the data to be multiplexed from the current buffer space (such as ping space) to the buffer space of the next pipeline (such as pong space), and vice versa. This can avoid repeated calculation of data and multiple accesses to the external memory.

[0127] Figure 6 An optimized memory access mode derivation example according to an embodiment of the present disclosure is shown. In this example, Figure 4 Based on the various conditions of the example, the reuse optimization scheme described above is applied to further detect possible data reuse between memory accesses to reduce the amount of memory accesses, avoid repeated calculations, and improve processing efficiency.

[0128] As shown in the figure, there is no overlap between the five data blocks 411 to 415 of the data 410 , and therefore there is no possibility of data multiplexing.

[0129] The overlap between the five data blocks 421 to 425 of data 420 is similar to that in the previous reference Figure 5a-Figure 5b The situation described in the example is the same, so after reuse optimization, the memory access data can be decomposed into a reuse part and a calculation part, and the corresponding access mode can be obtained. Specifically, according to the calculation method described above, the reuse part and the calculation part compute in the data blocks 421 to 425 can be calculated as shown in Table 1:

[0130] Range(i) Original access pattern Reuse pattern Compute pattern [0,1) [i*3,5) [5,0) [0,5) [1,3) [i*3,5) [i*3,2) [i*3+2,3) [4,1) [i*3,4) [i*3,2) [i*3+2,2)

[0131] Table 1

[0132] For the convenience of correspondence, the loop indexes are uniformly split into [0,1), [1,3) and [4,1).

[0133] Figure 6 There is also overlap between adjacent data blocks of the five data blocks 431 to 435 of the data 430. Similarly, the access data

[0134] Decompose into reuse part and calculation part, and obtain the corresponding access mode. Specifically, according to the calculation method described above, the reuse part and the calculation part compute in the data blocks 431 to 435 can be calculated as shown in Table 2:

[0135] Range(i) Original access pattern Reuse pattern Compute pattern [0,1) [0,4) [5,0) [0,4) [1,3) [i*3-1,5) [i*3-1,2) [i*3+1,3) [4,1) [i*3-1,3) [i*3-1,2) [i*3+1,1)

[0136] Table 2

[0137] As can be seen from the figure, after the reuse optimization, there is no overlap in the data that actually needs to be calculated, that is, the same data will not be calculated repeatedly, and the data that needs to be reused can be directly moved between the ping-pong spaces of the buffer, reducing memory access to external memory (such as DRAM).

[0138] Optionally or additionally, in some embodiments, the reused part may be further optimized based on the operation characteristics of the operator. Figure 6 In the example, the pad operator only performs padding processing on the data in the first memory access and the last memory access, and does not need to process the data in the intermediate memory access. Therefore, in the memory access mode of the input data 430, the reuse part can be removed, thereby saving on-chip data transfer.

[0139] Memory access mode merging scheme

[0140] The inventors also noticed that there may be a branch structure in the computation graph, which may result in the same data (e.g., the output data of an operator at a branch node) being used by multiple operators (e.g., operators on different branches), that is, the same data may have multiple consumers. According to the memory access mode derivation scheme described above, in this case, there will be a memory access mode for each consumer of the data, so there will be multiple memory access modes for the same data.

[0141] In some embodiments of the present disclosure, in order to further optimize memory access and reduce repeated memory access as much as possible, when it is detected in the computation graph that the same data has multiple memory access modes, the multiple memory access modes can be merged into one final memory access mode. Specifically, merging multiple memory access modes into one final memory access mode may include: determining the data range of each memory access in the final memory access mode to cover the data of the corresponding one memory access of each memory access mode in the multiple memory access modes. Thus, for the same data, only one memory access mode is still required to provide the data required by all its consumers.

[0142] Figure 7 shows an example of multiple access modes for the same data. Figure 4 The example is expanded, that is, in addition to the pad operator and convolution operator branch in the computation graph, an element-wise addition (add) operator branch is added. The add operator performs element-wise addition operations on the original input data (that is, the input data of the pad operator) and the output data of the convolution operator. In this example, the add operator is used as the entry operator to reversely deduce the memory access mode of the associated data in the computation graph.

[0143] Since the add operator corresponds to the atomic operation of vector operation, the memory access address of its input and output data is consistent. Therefore, Figure 6 The derivation results of the exemplary address derivation and reuse optimization described are also applicable to Figure 7 This branch has the same operator. Figure 7 As shown, the memory access mode of each data of the pad operator and the conv operator in the calculation graph is identified, and the compute part of the memory access mode after reuse optimization is identified.

[0144] The add operator also needs to access the original input data, so by inferring from this branch, the memory access mode for the original input data 430 can be derived, and its memory access mode is consistent with the memory access mode of the output data of the add operator.

[0145] As can be seen from the figure, data 430 is read by two operators, the add operator and the pad operator, and the data sizes they need are not the same, that is, there are two memory access modes.

[0146] In some embodiments of the present disclosure, when the same data has multiple memory access modes, the multiple memory access modes can be merged into a final memory access mode. When merging, it is necessary to determine how much data each data needs to be read each time to ensure that the data read each time is sufficient to feed all subsequent consumer operators. The final memory access mode after merging is represented by final_read_access, and the merging method can be expressed as follows:

[0147] final_read_access(i)=union(read_access(i,all_consumer_ops))

[0148] Figure 8 An example of memory access mode merging according to an embodiment of the present disclosure is shown. In this example, Figure 7 The two memory access modes of data 430 are merged in the above manner to obtain a final memory access mode.

[0149] As shown in the figure, data 430 includes memory access mode 1 and memory access mode 2, with different data splitting methods and reading ranges. When merging, each memory access after merging covers the data required for the corresponding memory access of memory access mode 1 and memory access mode 2, that is, the union of the data required by the two. The above merged Interval data structure can be represented as shown in Table 3:

[0150] Range(i) Memory access mode 1 Memory access mode 2 Merged final access mode [0,1) [0,4) [i*3,3) [i*3,4) [1,3) [i*3+1,3) [i*3,3) [i*3,4) [4,1) [i*3+1,1) [i*3,2) [i*3,2)

[0151] Table 3

[0152] Since the memory access mode is merged, when the reuse optimization solution is adopted, it is necessary to determine the actual reuse and compute parts again based on the merged final memory access mode.

[0153] Fig. 9 An example of multiplexing optimization based on the merged final memory access mode according to an embodiment of the present disclosure is shown. Figure 7 An example of a computational graph with branches is provided, and the memory access pattern of data 430 is further merged and reused and optimized.

[0154] The left side of the figure shows two memory access modes before the data 430 is merged, and the right side of the figure shows the memory access mode of the data 430 after it is merged. As shown in the figure, after the two memory access modes of data 430 are merged, there is also overlap between the memory accesses of adjacent data blocks, so the overlapping data can be reused. According to the calculation method of the reuse and compute parts described above, the final reuse and compute parts of data 430 can be calculated. In addition, data 410 and 420 each have only one memory access mode, so there is no need to merge them, and their final compute parts are also the same as described above.

[0155] Although on top Figure 8 and Fig. 9In the example, the compute part of the memory access pattern of data 430 is first calculated, and then the memory access patterns are merged based on the compute part, and reuse optimization is performed again. In other embodiments, the original memory access patterns of data 430 that have not been reuse optimized can also be directly merged, and then the merged memory access patterns are reuse optimized, thereby avoiding multiple calculations of the compute part.

[0156] Fig.10 An example of memory access mode merging and reuse optimization according to an embodiment of the present disclosure is shown. The various conditions of this example follow Figure 7 However, in this example, the two original memory access modes of data 430 that have not been optimized for reuse are directly merged, and then the merged memory access mode is optimized for reuse.

[0157] As shown in the figure, each memory access range of memory access mode 1 covers the range of the corresponding memory access in memory access mode 2. Therefore, the merged memory access mode is actually memory access mode 1. Fig. 9 The compute part of the merged reuse optimization is the same as in the previous example, but the reuse part is different.

[0158] From the above example, we can see that merging directly based on the memory access mode before reuse optimization can save the number of reuse optimizations, but the final reuse part will increase. This is because when merging, we need to consider the entire memory access range before reuse optimization, which includes the reused part, thus introducing redundancy. Therefore, the final reuse part contains redundancy, which increases the on-chip handling volume.

[0159] To more clearly show the difference between the two, Fig.11 Another example of memory access mode merging and reuse optimization according to an embodiment of the present disclosure is shown. In this example, compared to Figure 7 In the example of , a relu activation operator is added to the branch containing the pad operator and the convolution operator, and the rest of the parameters are the same. Similarly, in this example, the add operator is used as the entry operator to reversely deduce the memory access mode of the associated data in the computation graph.

[0160] because Fig.11 The computational graph in only adds the relu operator at the beginning of one branch, so the previous combination Figure 7 The derivation described can still be applied to Fig.11, we only need to add the derivation of the memory access mode of the input of the relu operator based on the input of the pad operator (that is, the output of the relu operator). As shown in the figure, the middle part of the figure identifies the memory access mode of the input and output data obtained by reverse deduction of each operator in the calculation graph, and the right part of the figure identifies the memory access mode after reuse optimization, including the compute part and the reuse part.

[0161] It can be seen from the memory access mode after reuse optimization that since the atomic operation corresponding to the relu operator is a vector operation, that is, it is an element-by-element bitwise processing, there is no need to reuse data between adjacent memory accesses, so the reuse part can be removed (see 1190). Similarly, there is no need to reuse data between adjacent memory accesses of the pad operator, and the reuse part can also be removed (see 1180).

[0162] Therefore, when merging is performed based on the memory access mode before reuse optimization, there will be more reuse redundancy, because the memory access range before reuse optimization needs to be considered during merging, which already includes the reused part. In contrast, when merging is performed based on the memory access mode after reuse optimization, since only the compute part that actually needs to be calculated is merged, reuse redundancy has been eliminated, so the part that actually needs to be reused in the final memory access mode is smaller, thereby reducing the amount of on-chip data transfer.

[0163] The present disclosure also provides a computing device, which can be used to execute the above-described method for deducing a memory access pattern.

[0164] Fig.12 1 is a block diagram showing a hardware configuration of a computing device 1200 that can implement various schemes of the disclosed embodiments. As shown in the figure, the computing device 1200 may include a processor 1210 and a memory 1220. Fig.12 In the computing device 1200, only the components related to this embodiment are shown. Therefore, it is obvious to those skilled in the art that the computing device 1200 may also include Fig.12 The components shown in the figure are different from common components, such as: display.

[0165] The computing device 1200 may correspond to a computing device having various processing functions, such as functions for programming and compiling source code. For example, the computing device 1200 may be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc.

[0166] Processor 1210, which is configured to execute program instructions to control all functions of computing device 1200. For example, processor 1210 controls all functions of computing device 1200 by executing a program stored in memory 1220 on computing device 1200. Processor 1210 may be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an artificial intelligence processor chip (IPU), etc. provided in computing device 1200. However, the present disclosure is not limited thereto.

[0167] The memory 1220 is hardware for storing various data processed in the computing device 1200. For example, the memory 1220 may store processed data and data to be processed in the computing device 1200. The memory 1220 may store data that has been processed or to be processed by the processor 1210, such as source code before compilation, assembly instructions after compilation, etc. In addition, the memory 1220 may store program instructions such as applications and drivers to be driven by the computing device 1200. For example: the memory 1220 may store various programs related to the memory access address derivation method to be executed by the processor 1210, etc. The memory 1220 may be a DRAM, but the present disclosure is not limited thereto. The memory 1220 may include at least one of a volatile memory or a non-volatile memory. The non-volatile memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory may include a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous DRAM (SDRAM), a PRAM, an MRAM, an RRAM, a ferroelectric RAM (FeRAM), etc. In an embodiment, the memory 920 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a high-density flash memory (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, caches, or a memory stick.

[0168] In summary, the specific functions implemented by the memory 1220 and the processor 1210 of the computing device 1200 provided in the embodiments of this specification can be explained in comparison with the aforementioned embodiments in this specification, and can achieve the technical effects of the aforementioned embodiments, and will not be repeated here.

[0169] In the disclosed embodiment, a computer-readable storage medium is also provided, in which program instructions are stored, and when the program instructions are loaded and executed by a processor, the processor executes the method for deriving a memory access mode for data in a computation graph described in the disclosed embodiment. In the disclosed embodiment, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implements the method for deriving a memory access mode for data in a computation graph described in the disclosed embodiment.

[0170] Fig.13 1 is a structural diagram showing a combined processing device 1300 according to an embodiment of the present disclosure. As shown in the figure, the combined processing device 1300 includes a computing processing device 1302, an interface device 1304, other processing devices 1306 and a storage device 1308. According to different application scenarios, the computing processing device may include one or more computing devices 1310, which may be configured as Fig.12 The computing device 1200 shown is used to perform the operations described in this document in conjunction with the accompanying drawings.

[0171] In different embodiments, the computing and processing device disclosed herein may be configured to perform user-specified operations. In an exemplary application, the computing and processing device may be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included in the computing and processing device may be implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core, with respect to the computing and processing device disclosed herein, it may be regarded as having a single-core structure or a homogeneous multi-core structure.

[0172] In an exemplary operation, the computing processing device of the present disclosure can interact with other processing devices through an interface device to jointly complete the operation specified by the user. Depending on the implementation, other processing devices of the present disclosure may include one or more types of processors in general and / or special processors such as a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence processor, etc. These processors may include but are not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing processing device of the present disclosure, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, the two can be regarded as forming a heterogeneous multi-core structure.

[0173] In one or more embodiments, the other processing device can serve as an interface between the computing device disclosed herein (which can be embodied as an artificial intelligence such as a computing device related to neural network computing) and external data and control, and perform basic controls including but not limited to data handling, starting and / or stopping the computing device, etc. In other embodiments, the other processing device can also cooperate with the computing device to jointly complete computing tasks.

[0174] In one or more embodiments, the interface device can be used to transmit data and control instructions between the computing and processing device and other processing devices. For example, the computing and processing device can obtain input data from other processing devices via the interface device and write it into a storage device (or memory) on the computing and processing device chip. Further, the computing and processing device can obtain control instructions from other processing devices via the interface device and write them into a control cache on the computing and processing device chip. Alternatively or optionally, the interface device can also read data from the storage device of the computing and processing device and transmit it to other processing devices.

[0175] Additionally or optionally, the combined processing device of the present disclosure may further include a storage device. As shown in the figure, the storage device is connected to the computing processing device and the other processing device, respectively. In one or more embodiments, the storage device may be used to store data of the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage device of the computing processing device or other processing device.

[0176] In some embodiments, the present disclosure also discloses a chip (e.g. Fig.14 In one implementation, the chip is a system on chip (SoC) and integrates one or more Fig.13 The chip can be connected to an external interface device (such as Fig.14 The external interface device 1406 shown in the figure is connected to other related components. The related components may be, for example, a camera, a display, a mouse, a keyboard, a network card or a wifi interface. In some application scenarios, other processing units (such as a video codec) and / or interface modules (such as a DRAM interface) may be integrated on the chip. In some embodiments, the present disclosure also discloses a chip packaging structure, which includes the above-mentioned chip. In some embodiments, the present disclosure also discloses a board card, which includes the above-mentioned chip packaging structure. The following will be combined with Fig.14 The board is described in detail.

[0177] Fig.14 1 is a schematic diagram showing the structure of a board 1400 according to an embodiment of the present disclosure. As shown in the figure, the board includes a storage device 1404 for storing data, which includes one or more storage units 1410. The storage device can be connected and data can be transmitted with the control device 1408 and the chip 1402 described above by means such as a bus. Further, the board also includes an external interface device 1406, which is configured for data relay or switching between the chip (or the chip in the chip packaging structure) and the external device 1412 (such as a server or computer, etc.). For example, the data to be processed can be transmitted to the chip by the external device through the external interface device. For another example, the calculation result of the chip can be transmitted back to the external device via the external interface device. According to different application scenarios, the external interface device can have different interface forms, for example, it can adopt a standard PCIE interface, etc.

[0178] In one or more embodiments, the control device in the disclosed board may be configured to regulate the state of the chip. To this end, in an application scenario, the control device may include a microcontroller unit (MCU) to regulate the working state of the chip.

[0179] According to the above combination Fig.13 and Fig.14 Based on the description, those skilled in the art can understand that the present disclosure also discloses an electronic device or apparatus, which may include one or more of the above-mentioned boards, one or more of the above-mentioned chips and / or one or more of the above-mentioned combined processing devices.

[0180] According to different application scenarios, the electronic equipment or device disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, transportation, household appliances, and / or medical equipment. The transportation includes airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs. The electronic equipment or device disclosed herein may also be applied to the Internet, IoT, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Further, the electronic equipment or device disclosed herein may also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, electronic devices or devices with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0181] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0182] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0183] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0184] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPU, GPU, FPGA, DSP and ASIC, etc. Further, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0185] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for deriving a memory access mode for data in a computation graph, comprising: Selecting an entry operator from the computation graph; Splitting the entry operator data into one or more data blocks according to the dimension splitting scheme of the entry operator data; as well as Based on the memory access mode of the entry operator data, a memory access mode of data associated with the entry operator data in the computation graph is derived, wherein the memory access mode includes a memory access address of each data block into which the data is split.

2. The method according to claim 1, wherein the selecting the entry operator comprises selecting the entry operator according to any of the following rules: Selecting the end operator in the computation graph as the entry operator; Select the starting operator in the computation graph as the entry operator; or According to the scale change trend of the input data and the output data in the calculation graph, the inflection point operator at the inflection point of the change is selected as the entry operator.

3. The method according to claim 1 or 2, wherein: Based on the memory access mode of the entry operator data, deriving the memory access mode of the data associated with the entry operator data in the computation graph includes: Based on the position of the entry operator, starting from the entry operator, each layer of operators in the computational graph is traversed from back to front and / or from front to back, and reverse and / or forward shape deduction is performed to obtain the memory access addresses of one or more input data blocks and output data blocks that each layer of operators needs to split to meet the dimensional splitting scheme of the entry operator.

4. The method according to claim 3, wherein for each layer of operators in the computation graph, performing reverse or forward shape derivation comprises: Decomposing the operation of the operator into a number of atomic operations according to the dimension; as well as According to the input and output memory access address relationship of the atomic operation, the memory access address of the input data block required by the operator to calculate an output data block is reversely derived; or the memory access address of the output data block calculated by the operator for an input data block is forward derived.

5. The method of claim 4, wherein the atomic operation comprises one or more of the following: Forward sliding window operation; Forward filling operation; Reverse sliding window operation; Reverse filling operation; Vector operations; Broadcast operations; Aggregation operations.

6. The method according to any one of claims 1 to 5, further comprising: When it is detected in the computation graph that the same data has multiple memory access modes, the multiple memory access modes are merged into one final memory access mode.

7. The method according to claim 6, wherein merging the plurality of memory access modes into a final memory access mode comprises: The data range of each memory access in the final memory access mode is determined to cover the data of a corresponding memory access in each memory access mode in the multiple memory access modes.

8. The method according to any one of claims 1 to 7, further comprising: Detecting an overlapping data range between two adjacent memory accesses in the same memory access mode of the data in the computation graph, wherein one memory access corresponds to one data block; as well as Based on the overlapping data range, a data range that actually needs to be accessed or calculated for each memory access in the memory access mode is determined.

9. The method according to claim 8, wherein detecting the overlapping data range between two adjacent memory accesses comprises: Determine the data intersection between the current memory access and the previous memory access; and / or Determine the data intersection between the current memory access and the next memory access.

10. The method according to any one of claims 1 to 9, wherein: The memory access pattern of data in one dimension is characterized by a data structure Interval used to represent the address memory access pattern of a loop. The data structure Interval contains a mapping from loop variables to address memory access patterns, and each loop corresponds to the memory access address of a data block into which the data is split.

11. The method according to claim 10, wherein: The memory access mode of the same data in multiple dimensions is represented by a data structure Region, and the data structure Region includes the dimensional order of the data and the memory access mode of each dimension.

12. The method according to any one of claims 10-11, wherein: When there are multiple splits in the same dimension of data, a tree structure is used to represent the multiple splits, wherein each time a split is added, a child node is added to the data structure Interval corresponding to the memory access mode of the current split.

13. The method according to any one of claims 1-12, wherein the derived memory access pattern of the data is used for space allocation and / or instruction generation.

14. A computing device for performing memory access pattern derivation, comprising: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, enables the processor to perform the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing program instructions, which, when loaded and executed by a processor, causes the processor to execute the method according to any one of claims 1 to 13.

16. A computer program product, comprising a computer program or instructions, which implements the method according to any one of claims 1 to 13 when executed by a processor.

Citation Information

Cited By

  • Atomic operation execution method and device, equipment, storage medium and program product

    CN120909801A

  • Method, device, equipment, storage medium and program product for executing atomic operation

    CN120909801B