Calculation graph optimization method, calculation device and related product

Abstract

The invention discloses a calculation graph optimization method, a calculation device, a computer readable storage medium and a computer program product. A computing device that performs an optimization method of a computational graph may be included in a combination processing device that may further include an interface device and other processing devices. The computing device interacts with other processing devices to jointly complete computing operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the computing device and the other processing devices and used for storing data of the computing device and the other processing devices. According to the scheme disclosed by the invention, by optimizing the view-class operator subgraph, the carrying of a memory at an equipment end can be reduced, so that the processing efficiency of a machine is improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of intelligent computing technology, and more particularly to the field of compilation. More specifically, the present disclosure relates to a method for optimizing a computational graph, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In intelligent computing systems, programming frameworks provide programmers with an interface for using hardware and systems, and are a critical core hub in intelligent computing systems. TensorFlow, PyTorch, etc. are currently popular deep learning frameworks. In these programming frameworks, computational graphs are usually used to describe the computational process of machine learning algorithms, tensors are used to represent all data in the computational graphs, and operators are used to represent various operations.

[0003] There is a class of operators, such as slice operators that slice memory data (such as narrow, select, unbind, chunk, slice, and split operators), and transposition operators that swap data dimensions (such as permute, transpose, T operators), which only change the appearance of tensor data but do not change the actual arrangement of tensor data in memory, that is, they do not move real memory data. This type of operator can be called a view operator.

[0004] Due to the characteristics of view operators, tensor data is usually discontinuous in memory. When reading discontinuous data, it will cause problems such as low memory access efficiency and high time consumption of hardware devices. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a computational graph optimization method, a computing device, a computer-readable storage medium, and a computer program product in multiple aspects.

[0006] In a first aspect, the present disclosure provides a method for optimizing a computational graph, comprising: obtaining a view-class operator subgraph of tensor data in the computational graph; determining whether there is an operator sequence in a specific order in the view-class operator subgraph, wherein the specific order includes a transposition-class operator being located before a slice-class operator; and in response to the existence of the operator sequence in the specific order, adjusting the position of the slice-class operator and / or the transposition-class operator in the operator sequence so that the slice-class operator is pre-processed.

[0007] In a second aspect, the present disclosure provides a computing device for optimizing a computational graph, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, wherein when the program instructions are loaded and executed by the processor, the processor executes the computational graph optimization method described in the first aspect of the present disclosure.

[0008] In a third aspect, the present disclosure provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implement the computational graph optimization method as described in the first aspect of the present disclosure.

[0009] In a fourth aspect, the present disclosure provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the computational graph optimization method described in the first aspect of the present disclosure.

[0010] According to the calculation graph optimization method provided above, when there is an operator sequence in which a transposition operator is located before a slice operator in a view operator subgraph, the slice operator can be pre-processed, thereby reducing memory data transfer, improving data access efficiency, and reducing network latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0012] Figure 1 The different shapes of multidimensional arrays and their storage order in memory are shown by way of example;

[0013] Figure 2 An exemplary flow chart of a method for optimizing a computational graph according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A flow chart of a method for determining a dimension mapping relationship according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A diagram showing an example of a principle for determining a dimension mapping relationship according to an embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram of determining a dimension mapping relationship in an operator sequence including a plurality of transposition type operators according to an embodiment of the present disclosure is shown;

[0017] Figure 6a - Figure 6dAn exemplary flow chart of an optimization method including operator fusion according to multiple embodiments of the present disclosure is shown;

[0018] Figure 7 An example of performing operations on multiple target operators operating in the same dimension is shown;

[0019] Figure 8 A block diagram showing a hardware configuration of a computing device 800 that may implement embodiments of the present disclosure;

[0020] Fig. 9 is a structural diagram showing a combined processing device 900 according to an embodiment of the present disclosure;

[0021] Fig.10 is a schematic diagram showing the structure of a board 1000 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0023] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0024] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0025] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0026] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0027] In the programming framework of intelligent computing systems, data is usually modeled as tensors. A tensor can be viewed as an N-dimensional array, and the dimension of the array is the order of the tensor. Therefore, a 0-order tensor corresponds to scalar data; a 1-order tensor corresponds to a one-dimensional array, that is, a vector; a 2-order tensor corresponds to a two-dimensional array, that is, a matrix; and so on, an N-order tensor corresponds to an N-dimensional array. For example, an RGB image can be represented as a 3-order tensor, and a data set consisting of multiple RGB images can be represented as a 4-order tensor.

[0028] Each tensor has some common properties, including data type, shape, etc. The shape of a tensor indicates the length of each order of the tensor. For example, a 0-order tensor corresponds to a scalar data, and its shape is empty; a 1-order tensor corresponds to a one-dimensional vector, and its shape contains one element, and the element value is the length of the vector; a 2-order tensor corresponds to a matrix, and its shape contains two elements, corresponding to the length of the row and column respectively; a 3-order tensor corresponds to a three-dimensional data, and its shape contains three elements, corresponding to the length of each order.

[0029] Although multidimensional arrays have multiple dimensions, because the layout of memory (for example, memory DRAM and cache RAM) is always one-dimensional, there is a corresponding relationship between multidimensional arrays and the storage order on the memory. Multidimensional arrays are usually allocated in continuous storage space, that is, multidimensional arrays can be expanded one-dimensionally and stored in order on the memory.

[0030] Figure 1 Different shapes of multi-dimensional arrays and their storage order in the memory are exemplified, wherein a one-dimensional array of a continuous memory is used to implement the storage of the multi-dimensional array.

[0031] Figure 1 Figure (a) shows the first data, that is, the three-dimensional array X, which has three dimensions, namely dimension 0 (dim0), dimension 1 (dim1) and dimension 2 (dim2). The size of dimension 0 is 2, the size of dimension 1 is 2, and the size of dimension 2 is 3. Therefore, its shape (size) can be expressed as: X3 = (2, 2, 3).

[0032] Figure 1 Figure (c) shows the storage order of the three-dimensional array X in the memory. In the figure, data with the same background are located in the same dimension. When storing, assuming that the storage is in the order of low dimension priority (for example, from left to right in shape representation corresponds to high dimension to low dimension), the first data is expanded in one dimension, and the following can be obtained:

[0033] X=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].

[0034] More specifically, the data in the lowest dimension (the same row) is continuous, while the data in higher dimensions are spaced at different distances. For example, in the storage method shown in Figure (c), accessing the physical structure of adjacent elements on dimension dim2 requires a shift of 1 position (for example, from data 1 to data 2, data 5 to data 6, and so on); accessing the physical structure of adjacent elements on dimension dim1 requires a shift of 3 positions (for example, from data 1 to data 4, data 2 to data 5, ..., data 9 to data 12, and so on); and accessing the physical structure of adjacent elements on dimension dim0 requires a shift of 6 positions (for example, from data 1 to data 7, data 2 to data 8, ..., data 6 to data 12, and so on). This offset is called the stride. The stride of each dimension of the three-dimensional array X can be expressed as S X =(6,3,1).

[0035] In the programming framework of intelligent computing systems, there are view operators that operate on the external representation of tensors, such as transpose, slice, split, etc. Taking transpose as an example, it converts a dimension according to a certain perm N =(p1, p2, ..., p i , ..., p N ), and get the data arrangement after dimension conversion, where p i The value of (i∈1, 2, ..., N) represents the original dimension of the array, p i In perm N The position in represents the target dimension of the conversion. For example, given the dimension conversion rule perm3 = (0, 2, 1), it means that dimension 1 and dimension 2 are to be exchanged, that is, the original dimension 1 is to be converted into the dimension 2 of the new array, and the original dimension 2 is to be converted into the dimension 1 of the new array.

[0036] Figure 1 Figure (b) shows the transformed array Y obtained by performing the transpose operator on the three-dimensional array X shown in Figure (a). In this example, the above exemplary dimension conversion rule perm3=(0, 2, 1) is applied. It can be seen from the figure that compared with array X, dimension 1 and dimension 2 of array Y are swapped. At this time, the dimension information of the three-dimensional array Y can be expressed as: Y3=(2, 3, 2).

[0037] However, since the view operator does not change the storage location of the data in memory, the storage order of the array Y obtained after the transposition operation in memory remains the same. Figure 1As shown in (c) in the figure. At this time, according to the storage order in (c), the step size of each dimension of array Y becomes S Y =(6, 1, 3). It can be seen that if the order of storing each data in the principle of low dimensionality first is called continuity, then the current storage order of array Y is discontinuous. That is, after the transpose operator, the order of the array dimensions is changed, but the storage position on the memory is not changed, so the storage order of the array on the memory becomes discontinuous.

[0038] If you want array Y to be continuous in memory, then according to the principle of low dimension first, its one-dimensional expansion should be Figure 1 As shown in Figure (d):

[0039] Y=[1, 4, 2, 5, 3, 6, 7, 10, 8, 11, 9, 12].

[0040] The shape of a tensor can help programmers form an intuitive feeling about tensors. In programming frameworks such as Pytorch, the View class operator can change the shape (size), stride (the span of the first index between adjacent dimensions of a tensor), storage offset (storage_offset, the offset of the first element of a tensor relative to the storage start position) and other properties of a tensor, but does not change the actual storage location of the tensor. At this time, the tensor uses size, stride and storage_offset to calculate the memory location of the data on the device side.

[0041] Assume that the size of the tensor is (s0, s1, s2, ..., s i ), stride is (y0,y1,y2,…,y i ), storage_offset is b, then the tensor is at the point (x0,x1,x2,…,x i ) is calculated as follows:

[0042]

[0043] Where dptr is the starting position of the memory storage corresponding to the tensor, and dtype is the data type of the tensor.

[0044] Combine from the front Figure 1As can be seen from the description, the tensors in the computational graph generate discontinuous corresponding data after being processed by the view operator. When accessing these tensors, traditional CPUs and GPUs need to access and read discontinuous data according to the above formula, which will lead to problems such as low memory access efficiency and high time consumption of hardware devices. Another way is to call the contiguous() operator to move the data one by one into continuous storage according to the above formula before performing subsequent access and calculation. However, this method is very time-consuming. When the amount of data is large, moving the data one by one will consume a lot of time.

[0045] In view of this, considering that in the operation of computational graphs such as neural networks, the discontinuity of memory data is often caused by view-type operators, the present disclosure proposes a solution for optimizing the view-type operator subgraph in the computational graph. The optimized view-type operator subgraph can then support the subsequent efficient completion of the process of transferring memory data from discontinuous to continuous.

[0046] Regarding the terms "node" and "operator" mentioned in this disclosure, it should be noted that the term "operator" is from the computing level of the computer (or from the software level or algorithm level); and the term "node" is a more figurative term (from the graphical level or a more intuitive level). In terms of the content referred to, the terms "operator" and "node" actually refer to the same thing. That is, in this disclosure, it can be considered that the terms "operator" and "node" have the same meaning and can be used interchangeably, but are described from different aspects.

[0047] As mentioned earlier, when tensor data becomes discontinuous in memory after passing through view operators, traditional CPUs and GPUs need to access and read discontinuous data, which leads to low memory access efficiency and high time consumption of hardware devices. In the neural network computing library, most operators require the input tensors to be continuous in memory, otherwise an error will occur. In this case, it is necessary to call operators such as contiguous(), which move data one by one into continuous storage. This method of moving data one by one is very time-consuming and brings great time overhead to the calculation of the entire computational graph.

[0048] The inventors have found that by constructing a view-class operator subgraph, when encountering operators that require tensor data to be continuous in memory (such as computing operators) in the future, memory data continuity processing can be performed based on these pre-constructed view-class operator subgraphs, and the corresponding device-side IO operator (such as kernel) can be called to perform data handling processing, and the discontinuous memory data can be changed into continuous memory data, thereby improving the memory access efficiency of subsequent computing operators. However, the slice operator itself has the characteristic of reducing the amount of memory data, while the transposition operator does not change the amount of memory data. When there is an operator sequence in the view operator subgraph where the transposition operator is located before the slice operator, in the case of continuous operator calls, more IO data handling will be generated, which will lead to problems such as reduced memory access efficiency of hardware devices and increased network latency.

[0049] Figure 2 An exemplary flow chart of a method for optimizing a computational graph according to an embodiment of the present disclosure is shown. Figure 2 As shown in , the optimization method 200 may include: in step 202, a view class operator subgraph of tensor data in the calculation graph may be obtained. The calculation graph is a directed graph including nodes and edges, and tensors are passed between nodes of the calculation graph. The execution of the calculation graph follows the order of the directed graph. Each time a tensor passes through a node, it will be calculated as the input of the node operation, and the result of the calculation will flow to the subsequent node along the output edge of the node. Therefore, obtaining the view class operator subgraph of tensor data in the calculation graph can be for tensor data, according to the order of the directed graph, traversing the nodes or operators to be processed on the tensor data, and when the traversed operator is a view class operator, extracting the view class operator subgraph constructed by the view class operator. It can be understood that for each tensor data, one or more segments of view class operator subgraphs can be included according to the continuity of the view class operator in the calculation graph. The optimization method of the disclosed embodiment can optimize the one or more segments of the view class operator subgraph.

[0050] Next, in step 204, it can be determined whether there is a sequence of operators in a specific order in the view class operator subgraph, wherein the specific order may include a transposition class operator being located before a slice class operator. In programming frameworks such as Pytorch, there are various view class operators to achieve different functions. The transposition class operator does not change the scale of the tensor data. Its main function is to exchange the data dimension and does not change the position of the memory block data. It may include operators such as permute, transpose, t(), T, etc. The slice class operator will reduce the scale of the tensor data. Its main function is to slice the memory data. It may include operators such as narrow, select, unbind, chunk, slice, and split.

[0051] In some embodiments, the transposition type operator is located before the slice type operator, which may be one transposition type operator before one slice type operator, or multiple transposition type operators before one slice type operator, or one transposition type operator before multiple slice type operators, or multiple transposition type operators before multiple slice type operators. In other embodiments, when multiple transposition type operators are located before a slice type operator, the multiple transposition type operators may be the same operator or different operators whose functions can replace each other; when there are multiple slice type operators after the transposition type operator, the multiple slice type operators may be the same operator or different operators whose functions can replace each other.

[0052] For example, in the view class operator subgraph, there may be an operator sequence of permute+slice, an operator sequence of transpose+transpose+narrow, an operator sequence of transpose+transpose+narrow+select, an operator sequence of permute+narrow+select, an operator sequence of transpose+permute+split+split, etc. These exemplary operator sequences may be operator sequences of a specific order in the embodiments of the present disclosure, but are not limited thereto.

[0053] For the view class operator subgraph, in accordance with the order of the directed graph, the view class nodes or operators to be processed on the tensor data are traversed to determine whether there is an operator sequence of a specific order in the view class operator subgraph. In other embodiments, the operator sequence of a specific order may include a continuous sequence of transposition class operators and slice class operators. The continuous sequence here can be understood as the absence of other types of operators between the transposition class operator and the slice class operator. Exemplarily, when there are multiple consecutive transposition class operators located before the slice class operator, there are no other types of operators between the last transposition class operator of the multiple transposition class operators and the slice class operator, and such an operator sequence is an operator sequence of a specific order. Similarly, when there is a transposition class operator located before multiple consecutive slice class operators, there are no other types of operators between the first slice class operator of the multiple slice class operators and the transposition class operator, and such an operator sequence is an operator sequence of a specific order.

[0054] like Figure 2 As further shown in FIG. 2 , in step 206, in response to the existence of a specific order of operator sequences, the positions of slice operators and / or transposition operators in the operator sequence are adjusted so that the slice operators are processed first. In some embodiments, adjusting the positions of slice operators and / or transposition operators in the operator sequence may include: moving the slice operator before the transposition operator; and / or moving the transposition operator after the slice operator. By adjusting the front-to-back position relationship of the slice operator and the transposition operator, the slice operator can be enabled to process the tensor data before the transposition operator.

[0055] In other embodiments, when adjusting the position of the slice operator and / or the transposition operator in the operator sequence, it is also necessary to modify the parameters of the slice operator and / or the transposition operator to adapt to the position adjustment. In still other embodiments, step 206 may further include: updating the parameters of the slice operator according to the dimension mapping relationship between the dimension information of the input tensor of the transposition operator and the slice dimension parameter of the slice operator, so as to adjust the slice operator to be processed before the transposition operator.

[0056] The input tensor of the transposition operator is the tensor input to the transposition operator for calculation. In some embodiments, the dimensional information of the input tensor may include information such as the number of dimensions of the input tensor. In one example, the dimensional information of the input tensor may be: X = [dim1, dim2, dim3, dim4, dim5, dim6, dim7, dim8], where dim1, dim2, etc. represent dimension 1, dimension 2, etc., respectively, for a total of 8 dimensions, and the specific values ​​of dim1, dim2, etc. can represent the size of each dimension (or the number of data). For example, assuming that the sizes of dim1, dim2, dim3, dim4, dim5, dim6, dim7 and dim8 are 2, 3, 4, 2, 3, 4, 2, 3, respectively, the dimensional information of the input tensor can also be directly expressed as X = [2, 3, 4, 2, 3, 4, 2, 3], where each number represents the size of the corresponding dimension.

[0057] The slice parameters of the slice operator may include the slice dimension parameter, the slice start position, the slice end position, and the number jump step. Here, the slice dimension parameter includes the dimension to be sliced, the slice start position and the slice end position are used to represent the slice range on the slice dimension, and the number jump step may represent the number step length in the slice range. According to these slice parameters, the tensor of the input slice operator can be sliced.

[0058] Since the slice dimension parameter represents the dimension of the slice processing, the data acquisition operations of the slice processing are all performed on this dimension. Therefore, the parameters of the slice operator can be updated according to the dimensional correspondence between the slice dimension parameter and the dimensional information of the input tensor of the transposition operator. This is so that when the slice operator is adjusted to be processed before the transposition operator, the input tensor of the transposition operator can be sliced ​​first, thereby reducing the amount of data when performing subsequent transposition operator calculations, and reducing the amount of data carried by the IO operator.

[0059] In an example, assuming that the dimension information of the input tensor of the transposition operator is X=[dim1, dim2, dim3, dim4, dim5, dim6, dim7, dim8], and the slice dimension parameter of the slice operator corresponds to dim1 in X, then the dimension mapping relationship between the dimension information of the input tensor of the transposition operator and the slice dimension parameter of the slice operator is the mapping relationship from dim1 to the slice dimension parameter. Then, no matter which dimension the slice dimension parameter originally represents, the updated slice dimension parameter of the slice operator will be directly replaced by dim1, so that X can be directly sliced.

[0060] Combination of the above Figure 2An exemplary description of the optimization method of the computational graph according to the embodiment of the present disclosure is given. It can be understood that since the slicing processing performed by the slicing operator can naturally reduce the amount of data, pre-processing the slicing operator will help reduce the amount of memory data handling and calculation during subsequent continuous processing, thereby helping to improve the memory access efficiency of the hardware device. It can also be understood that the above description is exemplary and not restrictive. For example, when adjusting the position of the slicing operator and / or the transposition operator in the operator sequence, it is not necessary to modify the parameters of the slicing operator. Assuming that the slicing dimension of the slicing operator does not perform a transformation operation in the transformation dimension of the transposition operator, the slicing dimension parameters of the slicing operator can be directly used for slicing the input tensor of the transposition operator without modification. In order to facilitate further understanding of the method for determining the dimension mapping relationship, the following will be combined with Figure 3 Provide further explanation.

[0061] Figure 3 A flow chart of a method for determining a dimension mapping relationship according to an embodiment of the present disclosure is shown. Figure 4 The following is an example diagram showing the principle of determining the dimension mapping relationship according to the embodiment of the present disclosure. Figure 3 The method 300 shown may be a combination of the above Figure 2 The step 206 described above is a specific embodiment of the present invention. Figure 2 The description of step 206 can also be applied to the following description of method 300. Figure 3 and Figure 4 Provide explanation.

[0062] like Figure 3 As shown in , method 300 may include: in step 302, a first mapping relationship between the dimension information of the input tensor and the transformation dimension parameter of the transposition operator may be determined. The transformation dimension parameter of the transposition operator may include a dimension transformation rule for indicating the dimension arrangement after the dimension transformation. In one example, see Figure 4, assuming that there is an operator sequence in a specific order in the view operator subgraph, which is an operator sequence of a transposition operator + a slice operator, and the dimension information of the input tensor is X=[dim1, dim2, dim3, dim4, dim5, dim6, dim7, dim8], where dim1, dim2, etc. represent dimension 1, dimension 2, etc., respectively, with a total of 8 dimensions; assuming that the transformation dimension parameters of the transposition operator include Dims=[dim7, dim8, dim1, dim2, dim5, dim4, dim3, dim6], which means that the dimensions of X are rearranged, where dimension 7 and dimension 8 are transformed to dimension 1 and dimension 2 before dimension 2, and dimension 3 and dimension 5 are interchanged. Such a mapping relationship is the first mapping relationship 401 (the mapping relationship of dim2 is shown as an example in the figure).

[0063] return Figure 3 As shown in , in step 304, a second mapping relationship between the slice dimension parameter of the slice type operator and the transformation dimension parameter of the transposition type operator can be determined. In one example, see Figure 4 , assuming that the slice dimension parameter of the slice type operator is dim4', it means that the fourth dimension of the tensor of the input slice type operator is sliced, and the fourth dimension after transformation in the transformation dimension parameter of the transpose type operator in the diagram is dim2, then the slice dimension parameter dim4' of the slice type operator actually corresponds to dim2 in the transformation dimension parameter in the diagram for slicing processing, and this mapping relationship is the second mapping relationship 402.

[0064] like Figure 3 As further shown in FIG. 3 , in step 306, the dimension mapping relationship may be determined according to the first mapping relationship and the second mapping relationship. Figure 4 Taking the example shown as an example, according to the first mapping relationship 401 and the second mapping relationship 402, it can be determined that the dimension mapping relationship between the dimension information X of the input tensor of the transpose operator and the slice dimension parameter dim4' of the slice operator is dim4' mapped to the dimension position of dim2 in X. Therefore, when the slice operator is pre-processed, the slice dimension parameter can be updated to dim2', indicating that the second dimension of the input tensor is sliced, that is, the dim2 of X in the diagram is sliced.

[0065] Combination of the above Figure 3 and Figure 4 The process of determining the dimension mapping relationship according to the embodiment of the present disclosure is described. It can be understood that the above description and illustration are exemplary rather than restrictive. For example, the dimension information of the input tensor may not be limited to eight dimensions, but may be more or less. For example, although Figure 3The step 304 exemplarily shown in the figure is located after step 302, but it is understandable that step 304 may not be limited to being executed after step 302, and may also be executed synchronously with step 302, or step 302 may be executed after step 304. In one example, the second mapping relationship may be determined first, and then the first mapping relationship may be determined. In this way, after determining the dimension corresponding to the slice dimension parameter in the transformation dimension parameter (such as dim2 in the figure), the first mapping relationship between the dimension and the dimension information X of the input tensor may be determined in a targeted manner. For another example, in a specific order of operator sequences, the slice operator may not be limited to having only one transposition operator before it. In some application scenarios, there is a specific order of operator sequences including multiple consecutive transposition operators in the view operator subgraph. For ease of understanding, the following is combined with Figure 5 Provide explanation.

[0066] Figure 5 FIG. 2 shows a schematic diagram of determining a dimension mapping relationship in an operator sequence including multiple transposition operators according to an embodiment of the present disclosure. Figure 5 As shown in , in one example, it is assumed that there is an operator sequence of a specific order in the view operator subgraph, which is an operator sequence of the first transposition operator + the second transposition operator + the slice operator, and the dimension information of the input tensor is X = [dim1, dim2, dim3, dim4, dim5, dim6, dim7, dim8], where dim1, dim2, etc. respectively represent dimension 1, dimension 2, etc., for a total of 8 dimensions. It is assumed that the transformation dimension parameters of the first transposition operator include Dims1 = [dim7, dim8, dim1, dim2, dim5, dim4, dim3, dim6], which indicates a rearrangement of the dimensions of X, where dimension 7 and dimension 8 are transformed to dimension 1 and dimension 2 before dimension 3 and dimension 5 are swapped. Assume that the transformation dimension parameters of the second transposition operator include Dims2 = [dim7, dim8, dim2, dim1, dim5, dim4, dim3, dim6], which means that the dimensions of Dims1 are rearranged, where the positions of dimensions 2 and 1 are changed. Assume that the slice dimension parameter of the slice operator is dim4', which means that the fourth dimension of the tensor input to the slice operator (i.e., the tensor output after the second transposition operator in the figure) is sliced.

[0067] In such Figure 5In the example shown, the second mapping relationship 402 can be determined based on the slice dimension parameter dim4' of the slice operator and the transformation dimension parameter Dims2 of the second transposition operator. In the second mapping relationship 402, dim4' corresponds to dim1 in Dims2. Next, the third mapping relationship 501 between multiple consecutive transposition operators can be determined, such as the mapping relationship between the transformation dimension parameter Dims2 of the second transposition operator and the transformation dimension parameter Dims1 of the first transposition operator shown in the figure. In the third mapping relationship 501, dim1 in Dims2 corresponds to dim1 in Dims1. According to the dimension information X of the input tensor and the transformation dimension parameter Dims1 of the first transposition operator, the first mapping relationship 401 is determined. In the first mapping relationship 401, dim1 in Dims1 corresponds to dim1 in X. Based on the first mapping relationship 401, the third mapping relationship 501 and the second mapping relationship 402, the dimension mapping relationship between the slice dimension parameter and the dimension information of the input tensor can be determined, that is, dim4' in the diagram corresponds to dim1 in X. When the slice-type operator is pre-processed, the slice dimension parameter of the slice-type operator can be updated to dim1'.

[0068] Combination of the above Figure 5 An exemplary description is given of the manner in which the dimensional mapping relationship of an operator sequence including multiple transposition operators according to the disclosed embodiment is determined. It is to be understood that the above description is exemplary and not restrictive. For example, in other embodiments, when there is an operator sequence including a specific order of multiple consecutive transposition operators in a view operator subgraph, the dimensional mapping relationship may not be limited to being determined according to a determined third mapping relationship. Before determining the dimensional mapping relationship, multiple consecutive transposition operators may be fused to form a single transposition operator, and then the dimensional mapping relationship may be determined according to the fused single transposition operator. In yet other embodiments, when there is an operator sequence including a specific order of multiple consecutive transposition operators in a view operator subgraph, the slice operator may be moved forward or the transposition operator may be moved backward in multiple steps.

[0069] Figure 6a-6d An exemplary flow chart of an optimization method including operator fusion according to multiple embodiments of the present disclosure is shown. Figure 6a FIG. 1 shows a schematic diagram of an optimization process of an operator sequence including a specific order of multiple transposition type operators. Figure 6aAs shown in FIG. 6 , in an exemplary computational graph, the input tensor A 611 will sequentially pass through the following nodes according to the flow of the computational graph: permute operator 612, T operator 613, slice operator 614, and Matmul (matrix multiplication) operator 615. Among these operators, permute operator 612, T operator 613, and slice operator 614 are all view type operators, and Matmul (matrix multiplication) operator 615 is a computation type operator.

[0070] like Figure 6a In the exemplary calculation graph shown in the figure, in the optimization method of the disclosed embodiment, a view class operator subgraph can be obtained according to the view class operators in the calculation graph. For example, the permute operator 612, the T operator 613 and the slice operator 614 in the calculation order in the figure constitute a view class operator subgraph. The view class operator subgraph can be optimized subsequently, and the data output by the optimized view class operator subgraph can be used for subsequent calculation of calculation class operators.

[0071] In some embodiments, the transposition type operator can be normalized to a specified transposition type target operator whose functions can replace each other; and the slice type operator can be normalized to a specified slice type target operator whose functions can replace each other. Although the specific functions implemented by the View type operator are varied, they can also be classified. The main function of the Permute operator, transpose operator, t() operator, T operator, etc. is to exchange data dimensions. Such operators can be collectively referred to as transposition type operators, and the functions of these operators can replace each other. The main function of the narrow operator, select operator, unbind operator, chunk operator, slice operator and split operator is to slice memory data. Such operators can be collectively referred to as slice type operators, and the functions of these operators can replace each other.

[0072] For each category of functions, an operator can be selected to represent the functions of the category. The operator can implement the functions of all operators under the corresponding function category. That is, the operator can functionally replace all operators under the corresponding function category. In this article, the replaced operator is called a "target operator". In some embodiments, the transposition class target operator can be, for example, a transpose operator. In other embodiments, the slice class target operator can be, for example, a slice operator. In some embodiments, the transposition class target operator can be, for example, a permute operator. In yet other embodiments, the slice class target operator can be, for example, a split operator. It can be understood that these operators are merely exemplary and not exhaustive, and those skilled in the art can construct similar functional classifications and functionally replaceable target operators based on the principles of the embodiments disclosed herein.

[0073] In some embodiments, the normalization of the transposition type operator to the transposition type target operator, and the normalization of the slice type operator to the slice type target operator, can be achieved by deformation mapping. For example, in one example, the select operator can be obtained by deformation mapping of the slice operator and the squeeze operator. In another example, the transpose operator can be deformed and mapped to the permute operator by adjusting the dimension value. In other embodiments, the normalization of the transposition type operator to the transposition type target operator, and the normalization of the slice type operator to the slice type target operator, can also be achieved by direct replacement.

[0074] like Figure 6a As shown in , by normalizing the permute operator 612 and the T operator 613 in the view operator subgraph, transpose operators 616 and 617 are obtained respectively. In one example, the specified slice target operator is slice, and the slice operator 614 can be directly used as the normalized slice target operator.

[0075] By classifying these operators according to their functions and replacing them with specified target operators, the types of operators in the operator subgraph can be reduced, which also facilitates subsequent optimization operations.

[0076] According to the normalized result, the position of the slice target operator and / or the transpose target operator in the normalized operator sequence can be adjusted so that the slice target operator is processed first. In one example, the adjustment can be made in two steps, moving the slice operator 614 before the transpose operator 617 and then before the transpose operator 616. Each move can be made according to the above description, for example Figure 4 The described rules determine the parameters of the slice operator 614. In another example, the adjustment can be made in two steps, moving the transpose operator 617 to after the slice operator 614, and then moving the transpose operator 616 to after the slice operator 614, and each move can be made according to the above, for example Figure 4 The described rules determine the parameters of the slice operator 614. In yet another example, the parameters of the slice operator 614 may be determined according to, for example, Figure 5 The method shown in , by determining the mapping relationship between two consecutive transpose operators, moves the slice operator 614 directly before the two consecutive transpose operators, or moves the two consecutive transpose operators after the slice operator 614, and updates the parameters of the slice operator 614.

[0077] like Figure 6aAs further shown in FIG. 1 , after the position adjustment is performed, multiple consecutive identical target operators appear, such as transpose operator 616 and transpose operator 617 (shown in dashed boxes in the figure). The multiple consecutive identical target operators can be fused into a single target operator (such as transpose operator 619 in the figure) to further optimize the view class operator subgraph. By fusing multiple consecutive identical target operators, the number of operators can be reduced, thereby reducing the number of operators that need to be called later.

[0078] In one example, when multiple transposition type target operators are fused, since the parameters of the transposition type operators usually include transformation dimension parameters, that is, transformation rules for data dimensions, the transformation dimension parameters of the last transposition type target operator among multiple consecutive transposition type target operators can be equivalent to the transformation of the multiple consecutive transposition type target operators. Figure 6a In the example, transpose operator 616 and transpose operator 617 can be fused into a single target operator, and the transformation dimension parameter of the fused single target operator transpose619 can be the same as that of transpose operator 617.

[0079] Figure 6a The optimization process of including multiple transposition operators in the operator sequence is shown. It can be understood that the operator sequence in a specific order may not be limited to including multiple transposition operators, but may also include multiple slice operators. Figure 6b An exemplary description is given.

[0080] Figure 6b FIG. 2 shows a schematic diagram of an optimization process of an operator sequence including multiple slice operators. Figure 6b As shown in FIG. 6 , in an exemplary computational graph, the input tensor B621 will sequentially pass through the following nodes according to the flow of the computational graph: permute operator 622, select operator 623, slice operator 624, and Matmul (matrix multiplication) operator 625. Among these operators, permute operator 622, select operator 623, and slice operator 624 are all view type operators, and Matmul (matrix multiplication) operator 625 is a computation type operator.

[0081] In an example, taking the transpose type target operator as the transpose operator and the slice type target operator as the slice operator as an example, the permute operator 622 can be normalized to the transpose operator 626, and the select operator 623 can be normalized to the slice operator 627, and the slice operator 624 can continue to be retained in the normalization operation.

[0082] According to the normalized result, the position of the slice target operator and / or the transposition target operator in the normalized operator sequence can be adjusted to make the slice target operator pre-processed. Figure 6b In the example shown, the slice operator 627 and the slice operator 624 may be moved before the transpose operator 626 , or the transpose operator 626 may be moved after the slice operator 627 and the slice operator 624 .

[0083] In one example, the adjustment can be made in two steps: the slice operator 627 is moved before the transpose operator 626, and then the slice operator 624 is moved before the transpose operator 626. Each move can be made according to the above description, for example Figure 4 In another example, the adjustment can be made in two steps, moving the transpose operator 626 to after the slice operator 627, and then moving the transpose operator 626 to after the slice operator 624, and each move can be made according to the above, for example Figure 4 The described rules determine the parameters of the slice operator.

[0084] like Figure 6b As further shown in the figure, after the position adjustment is performed, multiple consecutive identical target operators appear, such as slice operators 627 and 624 in the figure (shown in dotted boxes in the figure). The multiple consecutive identical target operators can be fused into a single target operator (such as slice operator 628 in the figure) to achieve further optimization of the view class operator subgraph.

[0085] Combination of the above Figure 6b The optimization process of the operator sequence including multiple slice operators is described. It can be understood that the above description is exemplary and not restrictive. The normalization operation can be performed before or after the position adjustment. Figure 6c An exemplary description is given.

[0086] Figure 6c Figure 2 shows a schematic diagram of the optimization process for adjusting the operator position before normalization. Figure 6cAs shown in FIG. 6 , in an exemplary computation graph, the input tensor C 631 will sequentially pass through the following nodes according to the flow of the computation graph: permute operator 632, select operator 633, slice operator 634, and Matmul (matrix multiplication) operator 635. Among these operators, permute operator 632, select operator 633, and slice operator 634 are all view type operators, and Matmul (matrix multiplication) operator 635 is a computation type operator.

[0087] By traversing the computational graph, a view operator subgraph including permute operator 632, select operator 633 and slice operator 634 can be obtained. Since permute operator 632 belongs to the transposition operator, select operator 633 and slice operator 634 belong to the slice operator, there exists an operator sequence permute operator 632+select operator 633+slice operator 634 in a specific order in which the transposition operator is located before the slice operator.

[0088] In some embodiments, the select operator 633 and the slice operator 634 may be moved before the permute operator 632, or the permute operator 632 may be moved after the select operator 633 and the slice operator 634, so that the select operator 633 and the slice operator 634 may be pre-processed.

[0089] After the position adjustment is performed, the transposition type operator can be normalized to the specified transposition type target operator whose functions can be substituted for each other; and the slice type operator can be normalized to the specified slice type target operator whose functions can be substituted for each other. Taking the transposition type target operator as the transpose operator and the slice type target operator as the slice operator as an example, the permute operator 632 can be normalized to the transpose operator 637, and the select operator 633 can be normalized to the slice operator 638, and the slice operator 634 can continue to be retained in the normalization operation.

[0090] According to the normalized result, multiple consecutive identical target operators appear, and the multiple consecutive identical target operators can be fused into a single target operator. Figure 6c As further shown in the figure, the slice operators 636 and 634 (shown in dashed boxes in the figure) in the figure can be fused into a single target operator (such as the slice operator 638 in the figure) to achieve further optimization of the view class operator subgraph.

[0091] Combination of the above Figure 6cThe optimization process of normalization and fusion after position adjustment is described as an example. It is understandable that it is not limited to fusing multiple consecutive identical target operators into a single target operator after performing position adjustment. It can also be fused into a single target operator before performing position adjustment, which helps to reduce the complexity of subsequent operator position adjustment operations. Figure 6d An exemplary description is given.

[0092] Figure 6d FIG. 4 shows a schematic diagram of the optimization process of fusing operators before adjusting the position. Figure 6d As shown in FIG. 6 , in an exemplary computational graph, the input tensor D641 will sequentially pass through the following nodes according to the flow of the computational graph: permute operator 642, select operator 643, slice operator 644, and Matmul (matrix multiplication) operator 645. Among these operators, permute operator 642, select operator 643, and slice operator 644 are all view type operators, and Matmul (matrix multiplication) operator 645 is a computation type operator.

[0093] By traversing the computational graph, a view operator subgraph including permute operator 642, select operator 643 and slice operator 644 can be obtained. Since permute operator 642 belongs to the transposition operator, select operator 643 and slice operator 644 belong to the slice operator, there exists a specific order of operator sequence permute operator 642+select operator 643+slice operator 644 in which the transposition operator is located before the slice operator.

[0094] In an example, taking the transpose operator as the target operator and the slice operator as the target operator, the permute operator 642 can be normalized to the transpose operator 646, and the select operator 643 can be normalized to the slice operator 647, and the slice operator 644 can continue to be retained in the normalization operation.

[0095] Then, according to the normalized result, multiple consecutive identical target operators appear, and the multiple consecutive identical target operators can be fused into a single target operator. Figure 6d As further shown in the figure, the slice operators 647 and 644 (shown in dashed boxes in the figure) in the figure can be fused into a single target operator (such as the slice operator 648 in the figure).

[0096] After the continuous identical multiple target operators are fused into a single target operator, the operator sequence after the fusion operation can be further optimized because the transpose operator 646 is located before the slice operator 648. In some embodiments, the slice operator 648 can be moved forward to before the transpose operator 646, or the transpose operator 646 can be moved back to after the slice operator 648, so that the slice operator 648 can be processed in advance.

[0097] The above describes the optimization method flow including normalization and fusion operations according to the embodiment of the present disclosure in combination with multiple accompanying drawings. It can be understood that although the figure only shows the case where the operator sequence includes multiple transposition operators or multiple slice operators, when there is an operator sequence composed of multiple consecutive transposition operators and multiple consecutive slice operators, the method of optimizing the view operator subgraph by combining the above-mentioned normalization, position adjustment, operator fusion and other operations is still applicable, and will not be repeated here.

[0098] In the foregoing, the embodiment of fusing multiple consecutive identical target operators when they are slice-type operators has been mentioned many times. In order to make it easier to understand the implementation method of fusing slice-type target operators, it will be further explained below.

[0099] In some embodiments, when multiple consecutive identical target operators are slice-type operators, fusing the multiple consecutive identical target operators into a single target operator may include: in response to the multiple target operators operating in the same dimension, the equivalent parameters of the fused single target operator can be determined according to the corresponding slice ranges and data access steps of the multiple target operators in the same dimension; in response to the multiple target operators operating in different dimensions, the dimensional operations of the multiple target operators in different dimensions can be merged.

[0100] When multiple target operators operate in the same dimension, the target operator that operates later will further slice the data obtained by the target operator that operates earlier. Therefore, the equivalent parameters of the single target operator after fusion can be determined according to parameters such as the slicing range and data skipping step of each target operator, so that the single target operator can be equivalent to the multiple target operators before fusion, and the equivalent parameters can be equivalent to the parameters of the multiple target operators before fusion.

[0101] In other embodiments, determining the equivalent parameters of the fused single target operator may include: determining the equivalent slice starting position of the fused single target operator based on the first data access jump step of the target operator that performs the operation first among the multiple target operators and the slice starting position of each target operator; determining the equivalent slice ending position of the fused single target operator based on the first data access jump step and the first slice starting position of the target operator that performs the operation first among the multiple target operators, and based on the slice ending position of the target operator that performs the operation later among the multiple target operators; and determining the equivalent data access jump step of the fused single target operator based on the data access jump step of each target operator among the multiple target operators.

[0102] In some embodiments, the determined equivalent parameters may include:

[0103] dim new =dim,

[0104] start new =start1+start2*step1,

[0105] end new =start1+end2*step1,

[0106] step new =step1*step2,

[0107] Among them, dim new Indicates the equivalent slice dimension after fusion, dim indicates the same dimension on which multiple target operators operate, start new Indicates the starting position of the equivalent slice after fusion. start1 indicates the starting position of the first slice of the target operator that performs the operation first among multiple target operators. start2 indicates the starting position of the second slice of the target operator that performs the operation later among multiple target operators. end new Indicates the end position of the equivalent slice after fusion, end1 indicates the end position of the first slice of the target operator that performs the operation first among multiple target operators, end2 indicates the end position of the second slice of the target operator that performs the operation later among multiple target operators, step new Indicates the equivalent data fetching jump step after fusion, step1 indicates the first data fetching jump step of the target operator that performs the operation first among multiple target operators, and step2 indicates the second data fetching jump step of the target operator that performs the operation later among multiple target operators.

[0108] Figure 7 An example of multiple target operators operating on the same dimension is shown. Figure 7As shown in , in this example, taking the fusion of two adjacent slice-type target operators as an example, it is assumed that the slice-type target operator sliceel that performs the operation first and the slice-type target operator slice2 that performs the operation later have slice dimensions that are both dim, and the data included in the slice dimension dim is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], and other parameters of slice1 also include the first slice starting position start1=0, the first slice ending position end1=6, and the first data access step step1=2, and start1 and end1 constitute the slice range of slice1; other parameters of slice2 also include the second slice starting position start2=0, the second slice ending position end2=2, and the second data access step step2=1, and start2 and end2 constitute the slice range of slice2.

[0109] like Figure 7 As shown in , under normal circumstances, according to the slicing range of slice1 and the first access step, the access result of slice1 on dim is [0, 2, 4]; according to the slicing range of slice2 and the second access step, the access result of slice2 on [0, 2, 4] is [0, 2].

[0110] According to the method for determining equivalent parameters of the disclosed embodiment, since slice1 and slice2 operate in the same dimension, the equivalent slice dimension dim in the equivalent parameters after the two are fused is new Same as dim, that is, the slicing operation after merging slice1 and slice2 is still performed on the slice dimension of the target operator. In addition, the starting position of the equivalent slice after merging slice1 and slice2 is start new =start1+start2*step1=0+0*2=0, equivalent slice end position end new =start1+end2*step1=0+2*2=4, equivalent to data jump step new =step1*step2=2*1=2. Based on the determined equivalent parameters, when equivalent slicing is performed on [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], the equivalent number-taking result is [0, 2]. This equivalent number-taking result is the same as the final number-taking result obtained when slice1 and slice2 are operated in sequence. Therefore, a single target operator represented by the equivalent parameters can be used to replace multiple consecutive identical target operators.

[0111] It can be understood that when there are more than two consecutive identical target operators, the equivalent parameters of two consecutive target operators can be calculated in sequence based on the above-mentioned equivalent parameter calculation formula until all consecutive identical target operators are fused into a single target operator, or iterative calculation can be performed based on the above-mentioned equivalent parameter calculation formula to obtain the equivalent parameters of the fused single target operator.

[0112] In other embodiments, when multiple target operators operate in different dimensions, the dimensional operations of multiple consecutive target operators are executed sequentially, each target operator performs dimensional operations on its input tensor data in turn, and the target operators are continuous and identical. These dimensional operations can be merged to achieve the effect of multiple dimensional operations through a single target operator.

[0113] In one embodiment, equivalent parameters after fusion of multiple consecutive target operators operating in different dimensions may include:

[0114] dim new =(dim1, dim2, ... dimN),

[0115] start new =(start1,start2,...startN),

[0116] end new =(end1, end2, ...endN),

[0117] step new =(step1, step2,...stepN),

[0118] Among them, dim new Indicates the equivalent slice dimension after fusion, dim1 indicates the slice dimension of the first target operator to perform the operation, dim2 indicates the slice dimension of the second target operator to perform the operation, N indicates the number of consecutive identical target operators operating in different dimensions, start new Indicates the starting position of the equivalent slice after fusion. start1 indicates the starting position of the first slice in the first target operator that performs the operation. start2 indicates the starting position of the second slice in the second target operator that performs the operation. end new Indicates the end position of the equivalent slice after fusion, end1 indicates the end position of the first slice of the first target operator performing the operation, end2 indicates the end position of the second slice of the second target operator performing the operation, step new Indicates the equivalent data fetching jump step after fusion, step1 indicates the first data fetching jump step of the first target operator performing the operation, and step2 indicates the second data fetching jump step of the second target operator performing the operation.

[0119] In an example, suppose there are two consecutive slice-type target operators slice in the operator sequence. The dimension operations of these two slice operators can be merged into one. Assume that the first slice performs a slice operation on the dim1 dimension, and the second slice performs a slice operation on the dim2 dimension. Since the two slices operate on different dimensions, the parameters of the two slices can be directly merged. When these two slice operators are merged into one slice operator, what it wants to achieve is to perform the operation of the first slice on the dim1 dimension and the operation of the second slice on the dim2 dimension at the same time.

[0120] The above describes the optimization method of the computational graph of the disclosed embodiment in conjunction with the accompanying drawings. By adjusting the position of the slice type operator and / or the transposition type operator in the operator sequence of a specific order, the slice type operator is pre-processed, which can reduce the amount of memory data carried, improve the memory access efficiency of the hardware device, and reduce network time consumption. In some embodiments, by fusing multiple consecutive identical target operators, the number of operator calls can be effectively reduced, and continuous calls of the same operator can be avoided, thereby improving the operator call efficiency, and further improving the memory access efficiency of the hardware device and the end-to-end operation efficiency of the network.

[0121] The present disclosure also provides a computing device, which can be used to optimize a computing graph. Figure 8 800 is a block diagram showing a hardware configuration of a computing device 800 that can implement the disclosed embodiments. As shown in the figure, the computing device 800 may include a processor 810 and a memory 820. Figure 8 In the computing device 800, only the components related to this embodiment are shown. Therefore, it is obvious to those skilled in the art that the computing device 800 may also include Figure 8 The components shown in the figure are different from common components, such as: display.

[0122] The computing device 800 may correspond to a computing device having various processing functions, such as a function for compiling a computational graph. For example, the computing device 800 may be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc.

[0123] The processor 810 is configured to execute program instructions to control all functions of the computing device 800. For example, the processor 810 controls all functions of the computing device 800 by executing a program stored in the memory 820 on the computing device 800. The processor 810 may be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an artificial intelligence processor chip (IPU), etc. provided in the computing device 800. However, the present disclosure is not limited thereto.

[0124] The memory 820 is hardware for storing various data processed in the computing device 800. For example, the memory 820 can store processed data and data to be processed in the computing device 800. The memory 820 can store data that has been processed or to be processed by the processor 810, such as a calculation graph before compilation, a calculation graph after compilation, etc. In addition, the memory 820 can store program instructions such as applications, drivers, etc. to be driven by the computing device 800. For example: the memory 820 can store various programs related to the optimization algorithm of the calculation graph to be executed by the processor 810. The memory 820 can be a DRAM, but the present disclosure is not limited thereto. The memory 820 can include at least one of a volatile memory or a non-volatile memory. The non-volatile memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory may include a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous DRAM (SDRAM), a PRAM, an MRAM, an RRAM, a ferroelectric RAM (FeRAM), etc. In an embodiment, the memory 820 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a high-density flash memory (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, caches, or a memory stick.

[0125] In summary, the specific functions implemented by the memory 820 and the processor 810 of the computing device 800 provided in the embodiments of this specification can be explained in comparison with the aforementioned embodiments in this specification, and can achieve the technical effects of the aforementioned embodiments, and will not be repeated here.

[0126] In the embodiments of the present disclosure, a computer-readable storage medium is also provided, in which program instructions are stored. When the program instructions are loaded and executed by a processor, the processor executes the calculation graph optimization method described in the embodiments of the present disclosure.

[0127] In the embodiments of the present disclosure, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implements the optimization method of the computational graph described in the embodiments of the present disclosure.

[0128] Fig. 9 900 according to an embodiment of the present disclosure. As shown in the figure, the combined processing device 900 includes a computing processing device 902, an interface device 904, other processing devices 906 and a storage device 908. According to different application scenarios, the computing processing device may include one or more computing devices 910, which may be configured as Figure 8 The computing device 800 shown is used to perform the operations described herein in conjunction with the accompanying drawings.

[0129] In different embodiments, the computing and processing device disclosed herein may be configured to perform user-specified operations. In an exemplary application, the computing and processing device may be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included in the computing and processing device may be implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core, with respect to the computing and processing device disclosed herein, it may be regarded as having a single-core structure or a homogeneous multi-core structure.

[0130] In an exemplary operation, the computing processing device of the present disclosure can interact with other processing devices through an interface device to jointly complete the operation specified by the user. Depending on the implementation, other processing devices of the present disclosure may include one or more types of processors in general and / or special processors such as a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence processor, etc. These processors may include but are not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing processing device of the present disclosure, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, the two can be regarded as forming a heterogeneous multi-core structure.

[0131] In one or more embodiments, the other processing device can serve as an interface between the computing device disclosed herein (which can be embodied as an artificial intelligence such as a computing device related to neural network computing) and external data and control, and perform basic controls including but not limited to data handling, starting and / or stopping the computing device, etc. In other embodiments, the other processing device can also cooperate with the computing device to jointly complete computing tasks.

[0132] In one or more embodiments, the interface device can be used to transmit data and control instructions between the computing and processing device and other processing devices. For example, the computing and processing device can obtain input data from other processing devices via the interface device and write it into a storage device (or memory) on the computing and processing device chip. Further, the computing and processing device can obtain control instructions from other processing devices via the interface device and write them into a control cache on the computing and processing device chip. Alternatively or optionally, the interface device can also read data from the storage device of the computing and processing device and transmit it to other processing devices.

[0133] Additionally or optionally, the combined processing device of the present disclosure may further include a storage device. As shown in the figure, the storage device is connected to the computing processing device and the other processing device, respectively. In one or more embodiments, the storage device may be used to store data of the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage device of the computing processing device or other processing device.

[0134] In some embodiments, the present disclosure also discloses a chip (e.g. Fig.10 In one implementation, the chip is a system on chip (SoC) and integrates one or more Fig. 9 The chip can be connected to an external interface device (such as Fig.10 The external interface device 1006 shown in the figure is connected to other related components. The related components may be, for example, a camera, a display, a mouse, a keyboard, a network card or a wifi interface. In some application scenarios, other processing units (such as a video codec) and / or interface modules (such as a DRAM interface) may be integrated on the chip. In some embodiments, the present disclosure further discloses a chip packaging structure, which includes the above-mentioned chip. In some embodiments, the present disclosure further discloses a board card, which includes the above-mentioned chip packaging structure. The following will be combined with Fig.10 The board is described in detail.

[0135] Fig.101 is a schematic diagram showing the structure of a board 1000 according to an embodiment of the present disclosure. As shown in the figure, the board includes a storage device 1004 for storing data, which includes one or more storage units 1010. The storage device can be connected and transmit data with the control device 1008 and the chip 1002 described above by means such as a bus. Further, the board also includes an external interface device 1006, which is configured for data relay or switching between the chip (or the chip in the chip packaging structure) and the external device 1012 (such as a server or computer, etc.). For example, the data to be processed can be transmitted to the chip by the external device through the external interface device. For another example, the calculation result of the chip can be transmitted back to the external device via the external interface device. According to different application scenarios, the external interface device can have different interface forms, for example, it can adopt a standard PCIE interface, etc.

[0136] In one or more embodiments, the control device in the disclosed board may be configured to regulate the state of the chip. To this end, in an application scenario, the control device may include a microcontroller unit (MCU) to regulate the working state of the chip.

[0137] According to the above combination Fig. 9 and Fig.10 Based on the description, those skilled in the art can understand that the present disclosure also discloses an electronic device or apparatus, which may include one or more of the above-mentioned boards, one or more of the above-mentioned chips and / or one or more of the above-mentioned combined processing devices.

[0138] According to different application scenarios, the electronic equipment or device disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, transportation, household appliances, and / or medical equipment. The transportation includes airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs. The electronic equipment or device disclosed herein may also be applied to the Internet, IoT, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Further, the electronic equipment or device disclosed herein may also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, electronic devices or devices with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0139] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0140] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0141] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0142] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPU, GPU, FPGA, DSP and ASIC, etc. Further, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0143] The foregoing content can be better understood in accordance with the following terms:

[0144] Clause 1. A method for optimizing a computational graph, comprising:

[0145] Get the view operator subgraph of the tensor data in the computation graph;

[0146] Determining whether there is a sequence of operators in a specific order in the view operator subgraph, wherein the specific order includes that the transposition operator is located before the slice operator; and

[0147] In response to the existence of the operator sequence in the specific order, the position of the slicing operator and / or the transposition operator in the operator sequence is adjusted so that the slicing operator is processed in advance.

[0148] Clause 2. The optimization method according to clause 1, further comprising:

[0149] Normalizing the transposition type operator to a specified transposition type target operator whose functions can replace each other;

[0150] Normalizing the slice-type operators to designated slice-type target operators whose functions can replace each other; and

[0151] According to the normalized result, the position of the slice-type target operator and / or the transposition-type target operator in the normalized operator sequence is adjusted so that the slice-type target operator is pre-processed.

[0152] Clause 3. An optimization method according to clause 1 or 2, wherein the operator sequence in a specific order comprises a continuous sequence of transposition type operators and slicing type operators.

[0153] Clause 4. The optimization method according to any one of clauses 1 to 3, wherein the position of the slicing operator and / or the transposition operator in the operator sequence is adjusted so that the pre-processing of the slicing operator includes:

[0154] According to the dimensional mapping relationship between the dimensional information of the input tensor of the transposition operator and the slice dimension parameters of the slice operator, the parameters of the slice operator are updated so as to adjust the slice operator to be processed before the transposition operator.

[0155] Clause 5. The optimization method according to clause 4, further comprising:

[0156] Determine a first mapping relationship between the dimension information of the input tensor and the transformation dimension parameter of the transposition operator;

[0157] Determining a second mapping relationship between a slice dimension parameter of the slice operator and a transformation dimension parameter of the transpose operator; and

[0158] The dimension mapping relationship is determined according to the first mapping relationship and the second mapping relationship.

[0159] Clause 6. The optimization method according to clause 2, further comprising:

[0160] Before performing the position adjustment, multiple consecutive identical target operators are fused into a single target operator; and / or

[0161] After performing the adjustment position, multiple consecutive identical target operators are fused into a single target operator.

[0162] Clause 7. The optimization method according to clause 1, further comprising: after performing the position adjustment,

[0163] Normalizing the transposition type operator to a specified transposition type target operator whose functions can replace each other;

[0164] Normalizing the slice-type operators to designated slice-type target operators whose functions can replace each other; and

[0165] According to the normalized results, multiple consecutive identical target operators are fused into a single target operator.

[0166] Clause 8. The optimization method according to clause 6 or 7, wherein when the multiple consecutive identical target operators are slice operators, fusing the multiple consecutive identical target operators into a single target operator comprises:

[0167] In response to the multiple target operators operating in the same dimension, determining equivalent parameters of a fused single target operator according to corresponding slicing ranges and data fetching steps of the multiple target operators in the same dimension;

[0168] In response to the multiple target operators operating in different dimensions, the dimensional operations of the multiple target operators in different dimensions are merged.

[0169] Clause 9. The optimization method according to clause 8, wherein determining the equivalent parameters of the fused single target operator comprises:

[0170] Determine an equivalent slice starting position of a fused single target operator according to a first data fetching jump step of a target operator that first performs operation among the multiple target operators and a slice starting position of each target operator;

[0171] Determine an equivalent slice end position of a fused single target operator according to a first data fetch jump step and a first slice start position of a target operator that performs operations first among the multiple target operators, and according to a slice end position of a target operator that performs operations later among the multiple target operators; and

[0172] According to the data acquisition jump step of each target operator in the multiple target operators, an equivalent data acquisition jump step of a fused single target operator is determined.

[0173] Clause 10. The optimization method according to clause 8 or 9, wherein determining the equivalent parameter comprises:

[0174] dim new =dim,

[0175] start new =start1+start2*step1,

[0176] end new =start1+end2*step1,

[0177] step new =step1*step2,

[0178] Among them, dim new represents the equivalent slice dimension after fusion, dim represents the same dimension on which the multiple target operators operate, start newrepresents the starting position of the equivalent slice after fusion, start1 represents the starting position of the first slice of the target operator that performs the operation first among the multiple target operators, start2 represents the starting position of the second slice of the target operator that performs the operation later among the multiple target operators, and end new represents the end position of the equivalent slice after fusion, end1 represents the end position of the first slice of the target operator that performs the operation first among the multiple target operators, end2 represents the end position of the second slice of the target operator that performs the operation later among the multiple target operators, step new It represents the equivalent data fetching jump step after fusion, step1 represents the first data fetching jump step of the target operator that performs the operation first among the multiple target operators, and step2 represents the second data fetching jump step of the target operator that performs the operation later among the multiple target operators.

[0179] Clause 11. The optimization method according to clause 2, wherein:

[0180] The transposition type target operator is the transpose operator;

[0181] The target operator for slice type is the slice operator.

[0182] Clause 12. A computing device for optimizing a computational graph, comprising:

[0183] a processor configured to execute program instructions; and

[0184] A memory configured to store the program instructions, which, when loaded and executed by the processor, enables the processor to perform the optimization method of the computational graph according to any one of clauses 1-11.

[0185] Clause 13. A computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implements the computational graph optimization method as described in any one of Clauses 1-11.

[0186] Clause 14. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for optimizing a computational graph as described in any one of Clauses 1-11.

[0187] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for optimizing a computational graph, comprising: Get the view operator subgraph of the tensor data in the computation graph; Determining whether there is a sequence of operators in a specific order in the view operator subgraph, wherein the specific order includes that the transposition operator is located before the slice operator; and In response to the existence of the operator sequence in the specific order, the position of the slicing operator and / or the transposition operator in the operator sequence is adjusted so that the slicing operator is processed in advance.

2. The optimization method according to claim 1, further comprising: Normalizing the transposition type operator to a specified transposition type target operator whose functions can replace each other; Normalizing the slice-type operators to designated slice-type target operators whose functions can replace each other; as well as According to the normalized result, the position of the slice-type target operator and / or the transposition-type target operator in the normalized operator sequence is adjusted so that the slice-type target operator is pre-processed.

3. The optimization method according to claim 1 or 2, wherein the operator sequence in a specific order comprises a continuous sequence of transposition operators and slicing operators.

4. The optimization method according to any one of claims 1 to 3, wherein adjusting the position of the slicing operator and / or the transposition operator in the operator sequence so that the pre-processing of the slicing operator comprises: According to the dimensional mapping relationship between the dimensional information of the input tensor of the transposition operator and the slice dimension parameters of the slice operator, the parameters of the slice operator are updated so as to adjust the slice operator to be processed before the transposition operator.

5. The optimization method according to claim 4, further comprising: Determine a first mapping relationship between the dimension information of the input tensor and the transformation dimension parameter of the transposition operator; Determine a second mapping relationship between a slice dimension parameter of the slice operator and a transformation dimension parameter of the transpose operator; as well as The dimension mapping relationship is determined according to the first mapping relationship and the second mapping relationship.

6. The optimization method according to claim 2, further comprising: Before performing the position adjustment, multiple consecutive identical target operators are fused into a single target operator; and / or After performing the adjustment position, multiple consecutive identical target operators are fused into a single target operator.

7. The optimization method according to claim 1, further comprising: After performing the repositioning, Normalizing the transposition type operator to a specified transposition type target operator whose functions can replace each other; Normalizing the slice-type operators to designated slice-type target operators whose functions can replace each other; as well as According to the normalized results, multiple consecutive identical target operators are fused into a single target operator.

8. The optimization method according to claim 6 or 7, wherein when the multiple consecutive identical target operators are slice operators, fusing the multiple consecutive identical target operators into a single target operator comprises: In response to the multiple target operators operating in the same dimension, determining equivalent parameters of a fused single target operator according to corresponding slicing ranges and data fetching steps of the multiple target operators in the same dimension; In response to the multiple target operators operating in different dimensions, the dimensional operations of the multiple target operators in different dimensions are merged.

9. The optimization method according to claim 8, wherein determining the equivalent parameters of the fused single target operator comprises: Determine an equivalent slice starting position of a fused single target operator according to a first data fetching jump step of a target operator that first performs operation among the multiple target operators and a slice starting position of each target operator; Determine an equivalent slice end position of a fused single target operator according to a first data fetch jump step and a first slice start position of a target operator that performs operations first among the multiple target operators, and according to a slice end position of a target operator that performs operations later among the multiple target operators; as well as According to the data acquisition jump step of each target operator in the multiple target operators, an equivalent data acquisition jump step of a fused single target operator is determined.

10. The optimization method according to claim 8 or 9, wherein determining the equivalent parameters comprises: dim new =dim, start new =start1+start2*step1, end new =start1+end2*step1, step new =step1*step2, Among them, dim new represents the equivalent slice dimension after fusion, dim represents the same dimension on which the multiple target operators operate, start new represents the starting position of the equivalent slice after fusion, start1 represents the starting position of the first slice of the target operator that performs the operation first among the multiple target operators, start2 represents the starting position of the second slice of the target operator that performs the operation later among the multiple target operators, and end new represents the end position of the equivalent slice after fusion, end1 represents the end position of the first slice of the target operator that performs the operation first among the multiple target operators, end2 represents the end position of the second slice of the target operator that performs the operation later among the multiple target operators, step new It represents the equivalent data fetching jump step after fusion, step1 represents the first data fetching jump step of the target operator that performs the operation first among the multiple target operators, and step2 represents the second data fetching jump step of the target operator that performs the operation later among the multiple target operators.

11. The optimization method according to claim 2, wherein: The transposition type target operator is the transpose operator; The target operator for slice type is the slice operator.

12. A computing device for optimizing a computing graph, comprising: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, and when the program instructions are loaded and executed by the processor, the processor executes the calculation graph optimization method according to any one of claims 1-11.

13. A computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implement the calculation graph optimization method as described in any one of claims 1-11.

14. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for optimizing the computational graph according to any one of claims 1 to 11.

Images