Computational graph optimization method, computing apparatus, and related product

Abstract

A computational graph optimization method, a computing apparatus (910), a computer-readable storage medium, and a computer program product. The computing apparatus (910), which executes the computational graph optimization method, may be included in a combined processing apparatus (900), and the combined processing apparatus (900) may further comprise an interface apparatus (904) and another processing apparatus (906). The computing apparatus (910) interacts with the other processing apparatus (906) to jointly complete a computing operation specified by a user. The combined processing apparatus (900) may further comprise a storage apparatus (908), and the storage apparatus (908) is respectively connected to the computing apparatus (910) and the other processing apparatus (906), and is used for storing data of the computing apparatus (910) and data of the other processing apparatus (906). By means of optimizing View-type operator sub-graphs, transfers for a memory of a device end can be reduced, thereby improving the processing efficiency of a machine.

Description

Computational graph optimization method, computing device and related products

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed on October 23, 2023, with application number 202311377240.9 and titled “Computational Graph Optimization Method, Computing Device and Related Products”. Technical Field

[0003] The present disclosure generally relates to the field of intelligent computing technology, and more particularly to the field of compilation. More specifically, the present disclosure relates to a computation graph optimization method, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0004] In intelligent computing systems, programming frameworks provide programmers with an interface for accessing hardware and systems, and are a crucial core component of these systems. TensorFlow, PyTorch, and other popular deep learning frameworks are commonly used in these frameworks to describe the computational process of machine learning algorithms. Tensors are used to represent all data in the computational graph, and operators are used to represent various operations.

[0005] There is a class of operators, such as slicing operators (such as Narrow, Select, Unbind, Chunk, Slice, and Split) that slice memory data, and transposition operators (such as Permute, Transpose, and T) that swap data dimensions. These operators only change the appearance or appearance of tensor data, but do not change the actual arrangement of tensor data in memory. In other words, they do not actually move memory data. These operators can be called view operators.

[0006] Due to this characteristic of View operators, tensor data is often discontinuous in memory. Reading operations on discontinuous data can lead to low hardware memory access efficiency and high memory consumption.

[0007] Summary of the Invention

[0008] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a computation graph optimization method, a computing device, a computer-readable storage medium, and a computer program product in multiple aspects.

[0009] In a first aspect, the present disclosure provides a method for optimizing a computational graph, comprising: obtaining a View-class operator subgraph of tensor data in the computational graph; determining whether there is an operator sequence in a specific order in the View-class operator subgraph, wherein the specific order includes a transposition-class operator being located before a slice-class operator; and in response to the existence of the operator sequence in the specific order, adjusting the position of the slice-class operator and / or the transposition-class operator in the operator sequence so that the slice-class operator is pre-processed.

[0010] In a second aspect, the present disclosure provides a computing device for optimizing a computational graph, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, wherein when the program instructions are loaded and executed by the processor, the processor executes the computational graph optimization method described in the first aspect of the present disclosure.

[0011] In a third aspect, the present disclosure provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by one or more processors, the method for optimizing the computational graph as described in the first aspect of the present disclosure is implemented.

[0012] In a fourth aspect, the present disclosure provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the computation graph optimization method described in the first aspect of the present disclosure.

[0013] According to the calculation graph optimization method provided above, when there is an operator sequence in the View operator subgraph where the transposition operator is located before the slice operator, the slice operator can be pre-processed, thereby reducing memory data handling, improving data access efficiency, and reducing network latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an illustrative and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0015] FIG1 exemplarily shows different shapes of multidimensional arrays and their storage order in memory;

[0016] FIG2 shows an exemplary flow chart of a method for optimizing a computation graph according to an embodiment of the present disclosure;

[0017] FIG3 shows a flow chart of a method for determining a dimension mapping relationship according to an embodiment of the present disclosure;

[0018] FIG4 shows an exemplary diagram of the principle of determining a dimension mapping relationship according to an embodiment of the present disclosure;

[0019] FIG5 is a schematic diagram showing a method for determining a dimension mapping relationship in an operator sequence including multiple transposition operators according to an embodiment of the present disclosure;

[0020] 6a-6d show exemplary flow charts of optimization methods including operator fusion according to various embodiments of the present disclosure;

[0021] FIG7 shows an example of performing operations on multiple target operators operating in the same dimension;

[0022] FIG8 is a block diagram illustrating a hardware configuration of a computing device that may implement an embodiment of the present disclosure;

[0023] FIG9 is a structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0024] FIG10 is a schematic diagram showing the structure of a board according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of this disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this disclosure, not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this disclosure.

[0026] It should be understood that the terms “include” and “comprising” used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the disclosure. As used in this disclosure and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should be further understood that the term "and / or" as used in this disclosure and the claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0028] As used in this specification and claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0029] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0030] In the programming framework of intelligent computing systems, data is typically modeled as tensors. A tensor can be thought of as an N-dimensional array, where the number of dimensions of the array is the rank of the tensor. Therefore, a 0-order tensor corresponds to scalar data; a 1-order tensor corresponds to a one-dimensional array, or vector; a 2-order tensor corresponds to a two-dimensional array, or matrix; and so on, with an N-order tensor corresponding to an N-dimensional array. For example, an RGB (red, green, blue) image can be represented as a 3-order tensor, while a dataset consisting of multiple RGB images can be represented as a 4-order tensor.

[0031] Each tensor has some common properties, including data type and shape. The shape of a tensor indicates the length of each order of the tensor. For example, a 0-order tensor corresponds to a scalar data and its shape is empty; a 1-order tensor corresponds to a one-dimensional vector and its shape contains one element, which is the length of the vector; a 2-order tensor corresponds to a matrix and its shape contains two elements, corresponding to the row and column lengths respectively; a 3-order tensor corresponds to a 3-dimensional data and its shape contains three elements, corresponding to the length of each order.

[0032] Although multidimensional arrays have multiple dimensions, because memory (for example, DRAM and RAM) is always laid out in one dimension, there is a correspondence between multidimensional arrays and the order in which they are stored. Multidimensional arrays are typically allocated in contiguous memory, meaning that they can be flattened one-dimensionally and stored sequentially in memory.

[0033] FIG1 exemplarily shows different shapes of multi-dimensional arrays and their storage order in a memory, wherein a one-dimensional array in a continuous memory is used to implement the storage of the multi-dimensional array.

[0034] Figure 1(a) illustrates the first data, namely a three-dimensional array X, which has three dimensions: dimension 0 (dim0), dimension 1 (dim1), and dimension 2 (dim2). Dimension 0 has a size of 2, dimension 1 has a size of 2, and dimension 2 has a size of 3, so its shape (size) can be expressed as: X3 = (2, 2, 3).

[0035] Figure 1(c) illustrates the storage order of a three-dimensional array X in memory. Data with the same background in the figure are located in the same dimension. Assuming storage is done in a low-dimensional order (e.g., from left to right in shape representation, corresponding to high to low dimensions), expanding the first data in one dimension yields: X = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].

[0036] More specifically, the data of the lowest dimension (the same row) is continuous, while the data of higher dimensions are spaced at different distances. For example, in the storage method shown in Figure (c), accessing the physical structure of adjacent elements on dimension dim2 requires an offset of 1 position (for example, from data 1 to data 2, data 5 to data 6, and so on); accessing the physical structure of adjacent elements on dimension dim1 requires an offset of 3 positions (for example, from data 1 to data 4, data 2 to data 5, ..., data 9 to data 12, and so on); and accessing the physical structure of adjacent elements on dimension dim0 requires an offset of 6 positions (for example, from data 1 to data 7, data 2 to data 8, ..., data 6 to data 12, and so on). This offset is called the stride. The stride of each dimension of the three-dimensional array X can be expressed as S X =(6,3,1).

[0037] In the programming framework of intelligent computing systems, there are View operators that operate on the external representation of tensors, such as Transpose, Slice, Split, etc. Taking Transpose as an example, it operates according to a certain dimension conversion rule perm N =(p1, p2, ..., p i ,…,p N ), get the data arrangement after dimension conversion, where p i The value of (i∈1,2,…,N) represents the original dimension of the array, p i In perm N The position in represents the target dimension of the transformation. For example, given the dimension transformation rule perm3 = (0,2,1), it means that dimension 1 and dimension 2 are to be swapped. That is, the original dimension 1 is to be converted to dimension 2 of the new array, and the original dimension 2 is to be converted to dimension 1 of the new array.

[0038] Figure 1 (b) shows the transformed array Y obtained by applying the Transpose operator to the three-dimensional array X shown in Figure (a). In this example, the aforementioned exemplary dimension transformation rule perm3 = (0, 2, 1) is applied. As can be seen from the figure, compared to array X, dimensions 1 and 2 of array Y are swapped. At this point, the dimensional information of the three-dimensional array Y can be expressed as: Y3 = (2, 3, 2).

[0039] However, since the View operator does not change the storage location of the data in memory, the storage order of the array Y obtained after the transposition operation is still as shown in (c) in Figure 1. At this time, according to the storage order in (c), the step size of each dimension of the array Y becomes S Y =(6,1,3). As can be seen, if storing data sequentially with lower dimensions first is called continuity, then the current storage order of array Y is discontinuous. In other words, after the Transpose operator changes the order of the array's dimensions but does not change its storage location in memory, the array's storage order in memory becomes discontinuous.

[0040] If you want the array Y to be continuous in the memory, then according to the principle of low dimension first, its one-dimensional expansion should be as shown in Figure 1 (d): Y = [1, 4, 2, 5, 3, 6, 7, 10, 8, 11, 9, 12].

[0041] The shape of a tensor can help programmers develop an intuitive understanding of it. In programming frameworks like Pytorch, View operators can modify properties such as the shape (size), stride (the distance between the first index of adjacent dimensions), and storage offset (the offset of the first element of a tensor relative to the storage start), without changing the actual storage location of the tensor. In this case, the size, stride, and storage offset are used to calculate the memory location of the data on the device.

[0042] Assume that the size of the tensor is (s0,s1,s2,…,s i ), stride is (y0,y1,y2,…,y i ), storage_offset is b, then the tensor is at the point (x0,x1,x2,…,x i ) is calculated as follows:

[0043] Where dptr is the starting position of the memory storage corresponding to the tensor, and dtype is the data type of the tensor.

[0044] As can be seen from the previous description in conjunction with Figure 1, the tensors in the computational graph generate discontinuous corresponding data after being processed by the View class operator. When accessing these tensors, the traditional CPU (Central Processing Unit) and GPU (Graphics Processing Unit) need to perform discontinuous data access and reading according to the above formula, which will lead to problems such as low memory access efficiency and high time consumption of hardware devices. Another way is to call the Contiguous() operator, first move the data one by one into continuous storage according to the above formula, and then perform subsequent access and calculations. However, this method is very time-consuming. When the amount of data is large, moving the data one by one will consume a lot of time.

[0045] In view of this, considering that in the operation of computational graphs such as neural networks, the discontinuity of memory data is often caused by View-type operators, the present disclosure proposes a solution for optimizing the View-type operator subgraph in the computational graph. The optimized View-type operator subgraph can then support the subsequent efficient completion of the process of transferring memory data from discontinuous to continuous.

[0046] Regarding the terms "node" and "operator" mentioned in this disclosure, it should be noted that the term "operator" is from the perspective of computer computing (or from the perspective of software or algorithm); while the term "node" is a more figurative term (from the perspective of graphics or a more intuitive level). In terms of what they refer to, the terms "operator" and "node" actually refer to the same thing. That is, in this disclosure, the terms "operator" and "node" can be considered to have the same meaning and can be used interchangeably, but are described from different perspectives.

[0047] As mentioned earlier, when tensor data becomes discontinuous in memory after passing through View operators, traditional CPUs and GPUs must access and read the data discontinuously, leading to low memory access efficiency and high memory consumption on the hardware devices. In neural network computing libraries, most operators require input tensors to be contiguous in memory; otherwise, errors will occur. In such cases, operators such as Contiguous() are called, which move the data one by one into contiguous storage. This method of moving data one by one is very time-consuming, adding significant overhead to the computation of the entire computation graph.

[0048] The inventors have discovered that by constructing a View-class operator subgraph, when encountering operators that require tensor data to be continuous in memory (such as computing operators) in the future, memory data continuity processing can be performed based on these pre-constructed View-class operator subgraphs, and the corresponding device-side IO (Input / Output) operator (such as kernel) can be called to perform data transfer processing, and discontinuous memory data can be changed into continuous memory data, thereby improving the memory access efficiency of subsequent computing operators. However, the slicing operator itself has the characteristic of reducing the amount of memory data, while the transposition operator does not change the amount of memory data. When there is an operator sequence in the View-class operator subgraph where the transposition operator is located before the slicing operator, in the case of continuous operator calls, more IO data transfer will be generated, which will lead to problems such as reduced memory access efficiency of hardware devices and increased network latency.

[0049] Figure 2 shows an exemplary flow chart of a computation graph optimization method according to an embodiment of the present disclosure. As shown in Figure 2, optimization method 200 may include: in step 202, obtaining a View operator subgraph for tensor data in the computation graph. A computation graph is a directed graph consisting of nodes and edges, with tensors passing between nodes in the computation graph. The computation graph is executed according to the order of the directed graph. Each time a tensor passes through a node, it is calculated as the input of the node's operation, and the result of the calculation flows along the node's output edge to the subsequent node. Therefore, obtaining a View operator subgraph for tensor data in the computation graph can be performed by traversing the nodes or operators to be processed for the tensor data according to the order of the directed graph. When the traversed operator is a View operator, extracting the View operator subgraph constructed by the View operator. It is understood that for each tensor data, one or more View operator subgraphs may be included based on the continuity of the View operators in the computation graph. The optimization method of the present disclosure embodiment can optimize one or more View operator subgraphs.

[0050] Next, in step 204, it can be determined whether there is a sequence of operators in a specific order in the View class operator subgraph, where the specific order may include a transpose class operator being located before a slice class operator. In programming frameworks such as Pytorch, there are various View class operators to achieve different functions. The transpose class operator does not change the scale of the tensor data. Its main function is to exchange the data dimensions and does not change the position of the memory block data. It may include operators such as Permute, Transpose, t(), T, etc. The slice class operator will reduce the scale of the tensor data. Its main function is to slice the memory data. It may include operators such as Narrow, Select, Unbind, Chunk, Slice, and Split.

[0051] In some embodiments, a transposition operator is located before a slice operator, and it may be one transposition operator before one slice operator, or multiple transposition operators before one slice operator, or one transposition operator before multiple slice operators, or multiple transposition operators before multiple slice operators. In other embodiments, when multiple transposition operators are located before a slice operator, the multiple transposition operators may be the same operator or different operators whose functions can replace each other; when multiple slice operators exist after a transposition operator, the multiple slice operators may be the same operator or different operators whose functions can replace each other.

[0052] For example, in a View class operator subgraph, there may be an operator sequence of Permute+Slice, an operator sequence of Transpose+Transpose+Narrow, an operator sequence of Transpose+Transpose+Narrow+Select, an operator sequence of Permute+Narrow+Select, an operator sequence of Transpose+Permute+Split+Split, etc. These exemplary operator sequences may be operator sequences in a specific order in the embodiments of the present disclosure, but are not limited thereto.

[0053] For the View-class operator subgraph, by traversing the View-class nodes or operators to be used to process the tensor data in the order of the directed graph, it can be determined whether there is a sequence of operators in a specific order in the View-class operator subgraph. In other embodiments, the operator sequence in a specific order may include a continuous sequence of transposition operators and slice operators. The continuous sequence here can be understood as the absence of other types of operators between the transposition operator and the slice operator. Exemplarily, when there are multiple consecutive transposition operators preceding a slice operator, there is no other type of operator between the last transposition operator in the multiple transposition operators and the slice operator, and such an operator sequence is an operator sequence in a specific order. Similarly, when there is a transposition operator preceding multiple consecutive slice operators, there is no other type of operator between the first slice operator in the multiple slice operators and the transposition operator, and such an operator sequence is an operator sequence in a specific order.

[0054] As further shown in FIG. 2 , in step 206 , in response to the presence of a specific sequence of operators, the position of the slice operator and / or the transpose operator in the operator sequence is adjusted so that the slice operator is processed first. In some embodiments, adjusting the position of the slice operator and / or the transpose operator in the operator sequence may include: moving the slice operator before the transpose operator; and / or moving the transpose operator after the slice operator. By adjusting the front-to-back position relationship of the slice operator and the transpose operator, the slice operator can be enabled to process tensor data before the transpose operator.

[0055] In other embodiments, when adjusting the position of a slice operator and / or a transpose operator in an operator sequence, it is also necessary to modify the parameters of the slice operator and / or the transpose operator to accommodate the position adjustment. In still other embodiments, step 206 may further include: updating the parameters of the slice operator based on a dimensional mapping relationship between the dimensional information of the input tensor of the transpose operator and the slice dimension parameters of the slice operator, so as to adjust the slice operator to be processed before the transpose operator.

[0056] The input tensor of the transposition operator is the tensor input to the transposition operator for calculation. In some embodiments, the dimension information of the input tensor may include information such as the number of dimensions of the input tensor. In an example, the dimension information of the input tensor can be: X = [dim1, dim2, dim3, dim4, dim5, dim6, dim7, dim8], where dim1, dim2, etc. represent dimension 1, dimension 2, etc., respectively, for a total of 8 dimensions. The specific values ​​of dim1, dim2, etc. can represent the size of each dimension (or the number of data). For example, assuming that the sizes of dim1, dim2, dim3, dim4, dim5, dim6, dim7 and dim8 are 2, 3, 4, 2, 3, 4, 2, 3 respectively, the dimension information of the input tensor can also be directly expressed as X = [2, 3, 4, 2, 3, 4, 2, 3], where each number represents the size of the corresponding dimension.

[0057] Slicing parameters for slicing operators include the slicing dimension, the starting and ending positions, and the access step. The slicing dimension parameter specifies the dimension to be sliced, the starting and ending positions represent the slicing range within the slicing dimension, and the access step specifies the access step size within the slicing range. Based on these slicing parameters, the tensor input to the slicing operator can be sliced.

[0058] Since the slice dimension parameter represents the dimension of the slice processing, the data acquisition operations of the slice processing are all performed on this dimension. Therefore, the parameters of the slice operator can be updated according to the dimension correspondence between the slice dimension parameter and the dimension information of the input tensor of the transposition operator, so that when the slice operator is adjusted to be processed before the transposition operator, the input tensor of the transposition operator can be sliced ​​first, thereby reducing the amount of data when performing subsequent transposition operator calculations, and reducing the amount of data carried by the IO operator.

[0059] In an example, assuming that the dimension information of the input tensor of the transpose operator is X = [dim1, dim2, dim3, dim4, dim5, dim6, dim7, dim8], and the slice dimension parameter of the slice operator corresponds to dim1 in X, then the dimension mapping relationship between the dimension information of the input tensor of the transpose operator and the slice dimension parameter of the slice operator is the mapping relationship from dim1 to the slice dimension parameter. Then, no matter which dimension the slice dimension parameter originally represents, the updated slice dimension parameter of the slice operator will be directly replaced by dim1, so that X can be directly sliced.

[0060] The above is an exemplary description of the optimization method of the computational graph according to the embodiment of the present disclosure in conjunction with Figure 2. It can be understood that, since the slicing processing performed by the slicing operator can naturally reduce the amount of data, pre-processing the slicing operator will be beneficial to reducing the amount of memory data handling and calculation during subsequent continuous processing, thereby helping to improve the memory access efficiency of the hardware device. It can also be understood that the above description is exemplary and not restrictive. For example, when adjusting the position of the slicing operator and / or the transposition operator in the operator sequence, it is not necessary to modify the parameters of the slicing operator. Assuming that the slicing dimension of the slicing operator does not perform a transformation operation in the transformation dimension of the transposition operator, the slicing dimension parameters of the slicing operator can be directly used for slicing the input tensor of the transposition operator without modification. In order to facilitate further understanding of the determination method of the dimension mapping relationship, further explanation will be given below in conjunction with Figure 3.

[0061] Figure 3 shows a flow chart of a method for determining a dimensional mapping relationship according to an embodiment of the present disclosure. Figure 4 shows an example diagram of the principle for determining a dimensional mapping relationship according to an embodiment of the present disclosure. As will be seen from the following description, method 300 shown in Figure 3 can be a specific embodiment of step 206 described above in conjunction with Figure 2. Therefore, the description of step 206 in conjunction with Figure 2 above also applies to the following description of method 300. The following description will be combined with Figures 3 and 4.

[0062] As shown in FIG3 , method 300 may include determining a first mapping relationship between dimension information of an input tensor and a transformation dimension parameter of a transpose operator in step 302. The transformation dimension parameter of the transpose operator may include a dimension transformation rule for indicating a dimension arrangement after the dimension transformation. In an example, referring to Figure 4, it is assumed that there is an operator sequence in a specific order in the View class operator subgraph, which is an operator sequence of a transposition class operator + a slice class operator, and the dimension information of the input tensor is X = [dim1, dim2, dim3, dim4, dim5, dim6, dim7, dim8], where dim1, dim2, etc. represent dimension 1, dimension 2, etc., respectively, for a total of 8 dimensions; it is assumed that the transformation dimension parameters of the transposition class operator include Dims = [dim7, dim8, dim1, dim2, dim5, dim4, dim3, dim6], which indicates a rearrangement of the dimensions of X, where dimension 7 and dimension 8 are transformed to dimension 1 and dimension 2 before dimension 3 and dimension 5 are swapped. Such a mapping relationship is the first mapping relationship 401 (the mapping relationship of dim2 is shown as an example in the figure).

[0063] Returning to FIG3 , in step 304 , a second mapping relationship between the slice dimension parameter of the slice operator and the transformation dimension parameter of the transpose operator can be determined. In one example, referring to FIG4 , assuming that the slice dimension parameter of the slice operator is dim 4' , which means slicing the fourth dimension of the tensor of the input slicing operator. The fourth dimension after transformation in the transformation dimension parameter of the transpose operator in the figure is dim2, so the slice dimension parameter dim of the slicing operator is dim. 4' The slicing process is actually performed corresponding to dim2 in the transformation dimension parameter in the diagram, and this mapping relationship is the second mapping relationship 402 .

[0064] As further shown in FIG3 , in step 306 , the dimension mapping relationship can be determined based on the first mapping relationship and the second mapping relationship. Still taking the example shown in FIG4 as an example, the dimension information X of the input tensor of the transposition operator and the slice dimension parameter dim of the slice operator can be determined based on the first mapping relationship 401 and the second mapping relationship 402. 4' The dimension mapping relationship between them is dim 4' Mapped to the dimension position of dim2 in X, so when pre-processing the slice operator, the slice dimension parameter can be updated to dim 2' , which means slicing the second dimension of the input tensor, that is, slicing the dim2 of X in the figure.

[0065] The above is a description of the process of determining the dimension mapping relationship according to the embodiment of the present disclosure in conjunction with Figures 3 and 4. It is understood that the above description and illustration are exemplary and non-restrictive. For example, the dimensional information of the input tensor may not be limited to including eight dimensions, but may be more or less. For another example, although step 304 shown in Figure 3 is located after step 302, it is understandable that step 304 may not be limited to being performed after step 302, but may also be performed synchronously with step 302, or step 302 may be performed after step 304. In one example, the second mapping relationship may be determined first, and then the first mapping relationship may be determined. In this way, after determining the dimension corresponding to the slice dimension parameter in the transformation dimension parameter (for example, dim2 in the figure), the first mapping relationship between the dimension and the dimensional information X of the input tensor may be determined in a targeted manner. For another example, in a specific order of operator sequences, the slice class operator may not be limited to having only one transposition class operator before it. In some application scenarios, there is a specific order of operator sequences including multiple consecutive transposition class operators in the View class operator subgraph. For ease of understanding, the following is explained in conjunction with Figure 5.

[0066] Figure 5 shows a schematic diagram of determining a dimension mapping relationship in an operator sequence including multiple transposition operators according to an embodiment of the present disclosure. As shown in Figure 5, in an example, it is assumed that there is an operator sequence of a specific order in the View operator subgraph, which is an operator sequence of the first transposition operator + the second transposition operator + the slice operator, and the dimension information of the input tensor is X = [dim1, dim2, dim3, dim4, dim5, dim6, dim7, dim8], where dim1, dim2, etc. represent dimension 1, dimension 2, etc., for a total of 8 dimensions. It is assumed that the transformation dimension parameters of the first transposition operator include Dims1 = [dim7, dim8, dim1, dim2, dim5, dim4, dim3, dim6], which represents a rearrangement of the dimensions of X, where dimension 7 and dimension 8 are transformed to dimension 1 and dimension 2 before dimension 3 and dimension 5 are swapped. Assume that the transformation dimension parameters of the second transposition operator include Dims2 = [dim7, dim8, dim2, dim1, dim5, dim4, dim3, dim6], which means that the dimensions of Dims1 are rearranged, where dimension 2 and dimension 1 are swapped. Assume that the slice dimension parameters of the slice operator are dim 4' , which means slicing the fourth dimension of the tensor input to the slicing operator (i.e., the tensor output after the second transpose operator in the figure).

[0067] In the example shown in FIG5 , the slice dimension parameter dim of the slice operator can be used to calculate the slice dimension parameter dim. 4' and the transformation dimension parameter Dims2 of the second transposition operator to determine a second mapping relationship 402, in which dim 4' Corresponding to dim1 in Dims2. Then, a third mapping relationship 501 between multiple consecutive transposition operators can be determined, such as the mapping relationship between the transformation dimension parameter Dims2 of the second transposition operator and the transformation dimension parameter Dims1 of the first transposition operator in the figure. In the third mapping relationship 501, dim1 in Dims2 corresponds to dim1 in Dims1. According to the dimension information X of the input tensor and the transformation dimension parameter Dims1 of the first transposition operator, a first mapping relationship 401 is determined. In the first mapping relationship 401, dim1 in Dims1 corresponds to dim1 in X. Based on the first mapping relationship 401, the third mapping relationship 501 and the second mapping relationship 402, the dimension mapping relationship between the slice dimension parameter and the dimension information of the input tensor can be determined, that is, dim1 in the figure. 4' Corresponding to dim1 in X, when the slice operator is pre-processed, the slice dimension parameter of the slice operator can be updated to dim 1' .

[0068] The above description, in conjunction with FIG5 , provides an exemplary description of the method for determining the dimensional mapping relationship of an operator sequence including multiple transposition operators according to an embodiment of the present disclosure. It is understandable that the above description is exemplary and not restrictive. For example, in other embodiments, when there is an operator sequence including a specific order of multiple consecutive transposition operators in a View operator subgraph, it is not limited to determining the dimensional mapping relationship based on a determined third mapping relationship. Before determining the dimensional mapping relationship, multiple consecutive transposition operators may be fused to form a single transposition operator, and then the dimensional mapping relationship may be determined based on the fused single transposition operator. In yet other embodiments, when there is an operator sequence including a specific order of multiple consecutive transposition operators in a View operator subgraph, the slice operator may be moved forward or the transposition operator may be moved backward in multiple steps.

[0069] Figures 6a-6d show exemplary flow charts of optimization methods including operator fusion according to multiple embodiments of the present disclosure. Figure 6a shows a schematic diagram of an optimization process for an operator sequence in a specific order including multiple transposition type operators. As shown in Figure 6a, in an exemplary computational graph, the input tensor A 611 will pass through the following nodes in sequence according to the flow of the computational graph: Permute operator 612, T operator 613, Slice operator 614 and Matmul (matrix multiplication) operator 615. Among these operators, Permute operator 612, T operator 613 and Slice operator 614 all belong to View type operators, and Matmul (matrix multiplication) operator 615 is a computation type operator.

[0070] As shown in the exemplary calculation graph in Figure 6a, in the optimization method of the disclosed embodiment, a View class operator subgraph can be obtained based on the View class operator in the calculation graph. For example, the Permute operator 612, T operator 613 and Slice operator 614 in the calculation order in the diagram constitute a View class operator subgraph. The View class operator subgraph can be optimized subsequently, and the data output by the optimized View class operator subgraph can be used for subsequent calculation of calculation class operators.

[0071] In some embodiments, the transposition type operator can be normalized to a transposition type target operator whose specified functions can replace each other; and the slice type operator can be normalized to a slice type target operator whose specified functions can replace each other. Although the specific functions implemented by the View type operator vary, they can also be classified. The main function of the Permute operator, Transpose operator, t() operator, T operator, etc. is to exchange data dimensions. This type of operator can be collectively referred to as a transposition type operator, and these operators are operators whose functions can replace each other. The main function of the Narrow operator, Select operator, Unbind operator, Chunk operator, Slice operator and Split operator, etc. is to slice memory data. This type of operator can be collectively referred to as a slice type operator, and these operators are operators whose functions can replace each other.

[0072] For each category of functions, an operator can be selected to represent the functions of that category. The operator can implement the functions of all operators under the corresponding function category. That is, the operator can functionally replace all operators under the corresponding function category. In this article, the replaced operator is called a "target operator". In some embodiments, the transposition type target operator can be, for example, a Transpose operator. In other embodiments, the slice type target operator can be, for example, a Slice operator. In some embodiments, the transposition type target operator can be, for example, a Permute operator. In yet other embodiments, the slice type target operator can be, for example, a Split operator. It can be understood that these operators are merely exemplary and not exhaustive, and those skilled in the art can construct similar functional classifications and functionally replaceable target operators based on the principles of the embodiments disclosed herein.

[0073] In some embodiments, normalizing a transposition-type operator to a transposition-type target operator, and normalizing a slice-type operator to a slice-type target operator, can be achieved by means of deformation mapping. For example, in one example, the Select operator can be obtained by deformation mapping of the Slice operator and the Squeeze operator. In another example, the Transpose operator can be deformed and mapped to the Permute operator by adjusting the dimension value. In other embodiments, normalizing a transposition-type operator to a transposition-type target operator, and normalizing a slice-type operator to a slice-type target operator, can also be achieved by direct replacement.

[0074] As shown in Figure 6a, by normalizing the Permute operator 612 and the T operator 613 in the View operator subgraph, Transpose operators 616 and 617 are obtained respectively. In one example, if the specified slice target operator is Slice, then Slice operator 614 can be directly used as the normalized slice target operator.

[0075] By classifying these operators according to their functions and replacing them with specified target operators, the types of operators in the operator subgraph can be reduced, which also facilitates subsequent optimization operations.

[0076] According to the normalized result, the position of the slice-type target operator and / or the transpose-type target operator in the normalized operator sequence can be adjusted so that the slice-type target operator is pre-processed. In one example, the adjustment can be made in two steps, and the Slice operator 614 is moved before the Transpose operator 617, and then moved before the Transpose operator 616. Each time the movement is made, the parameters of the Slice operator 614 can be determined according to the rules described in the foregoing text, such as Figure 4. In another example, the adjustment can be made in two steps, and the Transpose operator 617 is moved after the Slice operator 614, and then the Transpose operator 616 is moved after the Slice operator 614. Each time the movement is made, the parameters of the Slice operator 614 can be determined according to the rules described in the foregoing text, such as Figure 4. In another example, according to the method shown in Figure 5, by determining the mapping relationship between two consecutive Transpose operators, the Slice operator 614 can be moved directly before the two consecutive Transpose operators, or the two consecutive Transpose operators can be moved after the Slice operator 614, and the parameters of the Slice operator 614 can be updated.

[0077] As further shown in Figure 6a, after performing the position adjustment, multiple consecutive identical target operators appear, such as Transpose operators 616 and 617 (shown in dashed boxes). These multiple consecutive identical target operators can be fused into a single target operator (such as Transpose operator 619), further optimizing the View class operator subgraph. By fusing multiple consecutive identical target operators, the number of operators can be reduced, thereby reducing the number of operators that need to be called subsequently.

[0078] In one example, when multiple transposition target operators are fused, since the parameters of the transposition operators typically include transformation dimension parameters, i.e., transformation rules for data dimensions, the transformation dimension parameters of the last of the consecutive multiple transposition target operators can be equivalent to the transformation of the consecutive multiple transposition target operators. For example, in the example of FIG6a, Transpose operator 616 and Transpose operator 617 can be fused into a single target operator, and the transformation dimension parameters of the fused single target operator Transpose619 can be the same as those of Transpose operator 617.

[0079] FIG6a illustrates an optimization process where an operator sequence includes multiple transposition operators. It is understood that an operator sequence in a specific order may not be limited to including multiple transposition operators but may also include multiple slicing operators. An exemplary description will be provided below in conjunction with FIG6b.

[0080] Figure 6b shows a schematic diagram of an optimization process for an operator sequence including multiple slice-type operators. As shown in Figure 6b, in an exemplary computation graph, the input tensor B621 passes through the following nodes in sequence according to the computation graph flow: Permute operator 622, Select operator 623, Slice operator 624, and Matmul (matrix multiplication) operator 625. Of these operators, Permute operator 622, Select operator 623, and Slice operator 624 are all View-type operators, while Matmul (matrix multiplication) operator 625 is a Compute-type operator.

[0081] In an example, taking the transpose type target operator as the Transpose operator and the slice type target operator as the Slice operator as an example, the Permute operator 622 can be normalized to the Transpose operator 626, and the Select operator 623 can be normalized to the Slice operator 627, and the Slice operator 624 can continue to be retained in the normalization operation.

[0082] Based on the normalized result, the position of the slice target operator and / or the transpose target operator in the normalized operator sequence can be adjusted so that the slice target operator is processed first. In the example shown in Figure 6b, the Slice operator 627 and the Slice operator 624 can be moved before the Transpose operator 626, or the Transpose operator 626 can be moved after the Slice operator 627 and the Slice operator 624.

[0083] In one example, the adjustment can be performed in two steps: Slice operator 627 is moved before Transpose operator 626, and then Slice operator 624 is moved before Transpose operator 626. The parameters of the Slice operator can be determined according to the rules described above, for example, in FIG4 , during each move. In another example, the adjustment can be performed in two steps: Transpose operator 626 is moved after Slice operator 627, and then Transpose operator 626 is moved after Slice operator 624. The parameters of the Slice operator can be determined according to the rules described above, for example, in FIG4 , during each move.

[0084] As further shown in Figure 6b, after the position adjustment is performed, multiple consecutive identical target operators appear, such as the Slice operators 627 and 624 in the figure (shown in dotted boxes in the figure). These multiple consecutive identical target operators can be fused into a single target operator (such as the Slice operator 628 in the figure) to achieve further optimization of the View class operator subgraph.

[0085] The above description of the optimization process for an operator sequence including multiple slicing operators is described in conjunction with FIG6b. It is understood that the above description is exemplary and non-restrictive, and the normalization operation can be performed not only before but also after the position adjustment. An exemplary description will be provided below in conjunction with FIG6c.

[0086] Figure 6c shows a schematic diagram of the optimization process for adjusting operator positions before normalization. As shown in Figure 6c, in an exemplary computation graph, the input tensor C 631 passes through the following nodes in sequence according to the computation graph flow: Permute operator 632, Select operator 633, Slice operator 634, and Matmul (matrix multiplication) operator 635. Of these operators, Permute operator 632, Select operator 633, and Slice operator 634 are all View operators, while Matmul (matrix multiplication) operator 635 is a Compute operator.

[0087] By traversing the computational graph, we can obtain a View operator subgraph including the Permute operator 632, the Select operator 633, and the Slice operator 634. Since the Permute operator 632 belongs to the transposition operator, and the Select operator 633 and the Slice operator 634 belong to the slice operator, there exists a specific order of operator sequence Permute operator 632+Select operator 633+Slice operator 634 in which the transposition operator is located before the slice operator.

[0088] In some embodiments, the Select operator 633 and the Slice operator 634 may be moved before the Permute operator 632, or the Permute operator 632 may be moved after the Select operator 633 and the Slice operator 634, so that the Select operator 633 and the Slice operator 634 may be pre-processed.

[0089] After performing the position adjustment, the transposition operator can be normalized to a specified transposition target operator whose functions can be substituted for each other; and the slice operator can be normalized to a specified slice target operator whose functions can be substituted for each other. Taking the transposition target operator as the Transpose operator and the slice target operator as the Slice operator as an example, the Permute operator 632 can be normalized to the Transpose operator 637, and the Select operator 633 can be normalized to the Slice operator 638. The Slice operator 634 can continue to be retained in the normalization operation.

[0090] Based on the normalized results, multiple consecutive identical target operators appear, which can be fused into a single target operator. For example, as further shown in FIG6c , Slice operators 636 and 634 (shown in dashed boxes) can be fused into a single target operator (e.g., Slice operator 638 in the figure) to further optimize the View operator subgraph.

[0091] The above, in conjunction with FIG6c, provides an exemplary description of the optimization process for normalization and fusion after position adjustment. It is understood that the fusion of multiple consecutive identical target operators into a single target operator is not limited to after the position adjustment is performed. The fusion of multiple consecutive identical target operators into a single target operator can also be performed before the position adjustment is performed, which helps reduce the complexity of subsequent operator position adjustment operations. The following exemplary description is provided in conjunction with FIG6d.

[0092] Figure 6d shows a schematic diagram of the optimization process for fusion operators before position adjustment. As shown in Figure 6d, in an exemplary computational graph, the input tensor D641 passes through the following nodes in sequence according to the computational graph flow: Permute operator 642, Select operator 643, Slice operator 644, and Matmul (matrix multiplication) operator 645. Of these operators, Permute operator 642, Select operator 643, and Slice operator 644 are all View operators, while Matmul (matrix multiplication) operator 645 is a Compute operator.

[0093] By traversing the computational graph, we can obtain a View operator subgraph including the Permute operator 642, the Select operator 643, and the Slice operator 644. Since the Permute operator 642 belongs to the transposition operator, and the Select operator 643 and the Slice operator 644 belong to the slice operator, there exists a specific order of operator sequence Permute operator 642+Select operator 643+Slice operator 644 in which the transposition operator precedes the slice operator.

[0094] In an example, taking the transpose type target operator as the Transpose operator and the slice type target operator as the Slice operator as an example, the Permute operator 642 can be normalized to the Transpose operator 646, and the Select operator 643 can be normalized to the Slice operator 647, and the Slice operator 644 can continue to be retained in the normalization operation.

[0095] Then, based on the normalized result, multiple consecutive identical target operators appear, and these multiple consecutive identical target operators can be fused into a single target operator. For example, as further shown in FIG6 d , Slice operators 647 and 644 (shown in dashed boxes in the figure) can be fused into a single target operator (e.g., Slice operator 648 in the figure).

[0096] After multiple consecutive identical target operators are fused into a single target operator, the fused operator sequence can be further optimized because Transpose operator 646 is located before Slice operator 648. In some embodiments, Slice operator 648 can be moved before Transpose operator 646, or Transpose operator 646 can be moved after Slice operator 648, so that Slice operator 648 can be processed first.

[0097] The above describes the optimization method flow including normalization and fusion operations according to the embodiment of the present disclosure in combination with multiple drawings. It can be understood that although the figure only shows the case where the operator sequence includes multiple transposition operators or multiple slice operators, when there is an operator sequence composed of multiple consecutive transposition operators and multiple consecutive slice operators, the method of optimizing the View class operator subgraph by combining the above-mentioned normalization, position adjustment, operator fusion and other operations is still applicable and will not be repeated here.

[0098] In the foregoing, the embodiment of fusing multiple consecutive identical target operators when they are slice-type operators has been mentioned many times. In order to make it easier to understand the implementation method of fusing slice-type target operators, it will be further explained below.

[0099] In some embodiments, when multiple consecutive identical target operators are slicing operators, fusing the multiple consecutive identical target operators into a single target operator may include: in response to the multiple target operators operating in the same dimension, the equivalent parameters of the fused single target operator can be determined based on the corresponding slicing ranges and data jump steps of the multiple target operators in the same dimension; in response to the multiple target operators operating in different dimensions, the dimensional operations of the multiple target operators in different dimensions can be merged.

[0100] When multiple target operators operate in the same dimension, the target operator of the later operation will further slice the data obtained by the target operator of the earlier operation. Therefore, the equivalent parameters of the single target operator after fusion can be determined according to parameters such as the slicing range and data skipping step of each target operator, so that the single target operator can be equivalent to the multiple target operators before fusion, and the equivalent parameters can be equivalent to the parameters of the multiple target operators before fusion.

[0101] In other embodiments, determining the equivalent parameters of the fused single target operator may include: determining the equivalent slice starting position of the fused single target operator based on the first data access jump step of the target operator that performs the operation first among the multiple target operators and the slice starting position of each target operator; determining the equivalent slice ending position of the fused single target operator based on the first data access jump step and the first slice starting position of the target operator that performs the operation first among the multiple target operators, and based on the slice ending position of the target operator that performs the operation later among the multiple target operators; and determining the equivalent data access jump step of the fused single target operator based on the data access jump step of each target operator among the multiple target operators.

[0102] In some embodiments, the determined equivalent parameters may include: dim new =dim, start new =start1+start2*step1, end new =start1+end2*step1, step new =step1*step2,

[0103] Among them, dim new Indicates the equivalent slice dimension after fusion, dim indicates the same dimension of multiple target operators, start new Indicates the starting position of the equivalent slice after fusion. start1 indicates the starting position of the first slice of the target operator that performs the operation first among multiple target operators. start2 indicates the starting position of the second slice of the target operator that performs the operation later among multiple target operators. end new Indicates the end position of the equivalent slice after fusion, end1 indicates the end position of the first slice of the target operator that performs the operation first among multiple target operators, end2 indicates the end position of the second slice of the target operator that performs the operation later among multiple target operators, step new Indicates the equivalent data jump step after fusion. Step1 indicates the first data jump step of the target operator that performs the operation first among multiple target operators. Step2 indicates the second data jump step of the target operator that performs the operation later among multiple target operators.

[0104] Figure 7 shows an example of multiple target operators operating on the same dimension. As shown in Figure 7, in this example, taking the fusion of two adjacent slice-type target operators as an example, it is assumed that the slice dimension of the slice-type target operator Slice1 that performs the operation first and the slice-type target operator Slice2 that performs the operation later are both dim, and the data included in the slice dimension dim is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. Other parameters of Slice1 include the first slice starting position start1 = 0, the first slice ending position end1 = 6, and the first access step step1 = 2. Start1 and end1 constitute the slice range of Slice1. Other parameters of Slice2 include the second slice starting position start2 = 0, the second slice ending position end2 = 2, and the second access step step2 = 1. Start2 and end2 constitute the slice range of Slice2.

[0105] As shown in FIG7 , under normal circumstances, according to the slicing range of Slice1 and the first access step, the access result of Slice1 on dim is [0, 2, 4]; according to the slicing range of Slice2 and the second access step, the access result of Slice2 on [0, 2, 4] is [0, 2].

[0106] According to the method for determining equivalent parameters of the embodiment of the present disclosure, since Slice1 and Slice2 operate in the same dimension, the equivalent slice dimension dim in the equivalent parameters after the two are fused is new Same as dim, that is, the slicing operation after Slice1 and Slice2 are fused is still performed on the slice dimension of the target operator. In addition, the starting position of the equivalent slice after Slice1 and Slice2 are fused is start new =start1+start2*step1=0+0*2=0, equivalent slice end position end new =start1+end2*step1=0+2*2=4, equivalent to jump step start new =step1*step2=2*1=2. Based on these equivalent parameters, when equivalent slicing is performed on [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], the equivalent number acquisition result is [0, 2]. This equivalent number acquisition result is the same as the final number acquisition result obtained when Slice1 and Slice2 are performed sequentially. Therefore, a single target operator represented by the equivalent parameters can be used to replace multiple consecutive identical target operators.

[0107] It can be understood that when there are more than two consecutive identical target operators, the equivalent parameters of two consecutive target operators can be calculated in sequence based on the above-mentioned calculation formula for equivalent parameters until all consecutive identical target operators are fused into a single target operator, or iterative calculation can be performed based on the above-mentioned calculation formula for equivalent parameters to obtain the equivalent parameters of the fused single target operator.

[0108] In other embodiments, when multiple target operators operate in different dimensions, the dimensional operations of multiple consecutive target operators are executed sequentially, each target operator performs dimensional operations on its input tensor data in turn, and the target operators are continuous and identical. These dimensional operations can be merged to achieve the effects of multiple dimensional operations through a single target operator.

[0109] In one embodiment, the equivalent parameters after fusion of multiple consecutive target operators operating in different dimensions may include: dim new =(dim1,dim2,…dim N ), start new =(start1, start2, ... start N ), end new =(end1,end2,…end N ), step new =(step1,step2,…step N ),

[0110] Among them, dim new Indicates the equivalent slice dimension after fusion, dim1 indicates the slice dimension of the first target operator performing the operation, dim2 indicates the slice dimension of the second target operator performing the operation, N indicates the number of consecutive identical target operators operating in different dimensions, start new Indicates the starting position of the equivalent slice after fusion, start1 indicates the starting position of the first slice in the target operator that performs the first operation, start2 indicates the starting position of the second slice in the target operator that performs the second operation, and end new Indicates the end position of the equivalent slice after fusion, end1 indicates the end position of the first slice of the target operator that performs the first operation, end2 indicates the end position of the second slice of the target operator that performs the second operation, step new Indicates the equivalent data jump step after fusion, step1 indicates the first data jump step of the target operator that first performs the operation, and step2 indicates the second data jump step of the target operator that second performs the operation.

[0111] In one example, assume there are two consecutive slice-type target operators in an operator sequence. The dimensional operations of these two Slice operators can be merged into one. Suppose the first Slice performs a slicing operation on the dim1 dimension, and the second Slice performs a slicing operation on the dim2 dimension. Since the two Slices operate on different dimensions, the parameters of the two Slices can be directly merged. When these two Slice operators are merged into one Slice operator, it is implemented to perform the operation of the first Slice on the dim1 dimension and the operation of the second Slice on the dim2 dimension.

[0112] The above describes the computational graph optimization method of the disclosed embodiment in conjunction with the accompanying drawings. By adjusting the position of slicing operators and / or transposition operators in a specific sequence of operators, the slicing operators are pre-processed, which can reduce the amount of memory data moved, improve the memory access efficiency of hardware devices, and reduce network time consumption. In some embodiments, by fusing multiple consecutive identical target operators, the number of operator calls can be effectively reduced, avoiding continuous calls to the same operator, thereby improving the operator call efficiency, and further improving the memory access efficiency of hardware devices and the end-to-end operation efficiency of the network.

[0113] The present disclosure also provides a computing device that can be used to optimize a computational graph. FIG8 shows a block diagram of a hardware configuration of a computing device 800 that can implement an embodiment of the present disclosure. As shown in the figure, the computing device 800 may include a processor 810 and a memory 820. In the computing device 800 of FIG8 , only the components related to this embodiment are shown. Therefore, it is obvious to a person skilled in the art that the computing device 800 may also include common components different from the components shown in FIG8 , such as a display.

[0114] The computing device 800 may correspond to a computing device having various processing functions, such as functions for compiling computation graphs. For example, the computing device 800 may be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc.

[0115] The processor 810 is configured to execute program instructions to control all functions of the computing device 800. For example, the processor 810 controls all functions of the computing device 800 by executing a program stored in the memory 820 on the computing device 800. The processor 810 can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an artificial intelligence processor chip (IPU), etc. provided in the computing device 800. However, the present disclosure is not limited to this.

[0116] The memory 820 is hardware for storing various data processed in the computing device 800. For example, the memory 820 can store processed data and data to be processed in the computing device 800. The memory 820 can store data that has been processed or is to be processed by the processor 810, such as a computational graph before compilation, a computational graph after compilation, and the like. In addition, the memory 820 can store program instructions such as applications and drivers to be driven by the computing device 800. For example, the memory 820 can store various programs related to optimization algorithms for computational graphs to be executed by the processor 810. The memory 820 can be a DRAM, but the present disclosure is not limited thereto. The memory 820 can include at least one of a volatile memory and a non-volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, phase-change random access memory (PRAM), magnetic random access memory (MRAM), resistive random access memory (RRAM), ferroelectric random access memory (FRAM), etc. Volatile memory can include dynamic RAM (DRAM), static RAM (SRAM, Static Random Access Memory), synchronous DRAM (SDRAM, Synchronous Dynamic Random Access Memory), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM, Ferro-electric Random Access Memory), etc.In an embodiment, the memory 820 may include at least one of a hard disk drive (HDD), a solid state drive (SSD), a compact flash card (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, a cache, or a memory stick.

[0117] In summary, the specific functions implemented by the memory 820 and processor 810 of the computing device 800 provided in the embodiments of this specification can be interpreted in comparison with the aforementioned embodiments in this specification, and can achieve the technical effects of the aforementioned embodiments, so they will not be repeated here.

[0118] In an embodiment of the present disclosure, a computer-readable storage medium is also provided, in which program instructions are stored. When the program instructions are loaded and executed by a processor, the processor executes the calculation graph optimization method described in the embodiment of the present disclosure.

[0119] In an embodiment of the present disclosure, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implements the optimization method of the computational graph described in the embodiment of the present disclosure.

[0120] Figure 9 is a block diagram illustrating a combined processing device 900 according to an embodiment of the present disclosure. As shown, the combined processing device 900 includes a computing device 902, an interface device 904, other processing devices 906, and a storage device 908. Depending on the application scenario, the computing device may include one or more computing devices 910, which may be configured as the computing device 800 shown in Figure 8 to perform the operations described herein in conjunction with the accompanying drawings.

[0121] In various embodiments, the computing and processing device of the present disclosure may be configured to perform user-specified operations. In exemplary applications, the computing and processing device may be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included in the computing and processing device may be implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as an artificial intelligence processor core or a partial hardware structure of an artificial intelligence processor core, the computing and processing device of the present disclosure may be considered to have a single-core structure or a homogeneous multi-core structure.

[0122] In exemplary operation, the computing processing device of the present disclosure can interact with other processing devices through interface means, to jointly complete the operation specified by the user. Depending on the difference in implementation, the other processing devices of the present disclosure may include one or more types of processors in general and / or special processors such as central processing unit (Central Processing Unit, CPU), graphics processing unit (Graphics Processing Unit, GPU), artificial intelligence processor. These processors may include but are not limited to digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As previously mentioned, only with respect to the computing processing device of the present disclosure, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, the two can be regarded as forming a heterogeneous multi-core structure.

[0123] In one or more embodiments, the other processing device may serve as an interface between the computing device disclosed herein (which may be embodied as an artificial intelligence computing device such as a neural network computing device) and external data and control, performing basic control including but not limited to data transfer, starting and / or stopping the computing device, and so on. In other embodiments, the other processing device may also collaborate with the computing device to jointly complete computing tasks.

[0124] In one or more embodiments, the interface device can be used to transmit data and control instructions between the computing and processing device and other processing devices. For example, the computing and processing device can obtain input data from other processing devices via the interface device and write it to the storage device (or memory) on the computing and processing device chip. Furthermore, the computing and processing device can obtain control instructions from other processing devices via the interface device and write them to the control cache on the computing and processing device chip. Alternatively or optionally, the interface device can also read data from the storage device of the computing and processing device and transmit it to other processing devices.

[0125] Additionally or optionally, the combined processing device of the present disclosure may further include a storage device. As shown in the figure, the storage device is connected to the computing processing device and the other processing device, respectively. In one or more embodiments, the storage device may be used to store data of the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage device of the computing processing device or other processing device.

[0126] In some embodiments, the present disclosure further discloses a chip (e.g., chip 1002 shown in FIG10 ). In one implementation, the chip is a system-on-chip (SoC) and is integrated with one or more combined processing devices as shown in FIG9 . The chip can be connected to other related components through an external interface device (e.g., external interface device 1006 shown in FIG10 ). The related components may be, for example, a camera, a display, a mouse, a keyboard, a network card, or a Wifi interface. In some application scenarios, other processing units (e.g., a video codec) and / or interface modules (e.g., a DRAM interface) may be integrated on the chip. In some embodiments, the present disclosure further discloses a chip packaging structure, which includes the above-mentioned chip. In some embodiments, the present disclosure further discloses a board card, which includes the above-mentioned chip packaging structure. The board card will be described in detail below in conjunction with FIG10 .

[0127] Figure 10 is a schematic diagram showing the structure of a board 1000 according to an embodiment of the present disclosure. As shown in the figure, the board includes a storage device 1004 for storing data, which includes one or more storage units 1010. The storage device can be connected and transmit data with the control device 1008 and the chip 1002 described above by means of, for example, a bus. Further, the board also includes an external interface device 1006, which is configured to relay or transfer data between the chip (or the chip in the chip packaging structure) and the external device 1012 (such as a server or computer, etc.). For example, the data to be processed can be transferred to the chip by the external device through the external interface device. For another example, the calculation result of the chip can be transmitted back to the external device via the external interface device. According to different application scenarios, the external interface device can have different interface forms, for example, it can adopt a standard PCIE (Peripheral Component Interconnect express, high-speed peripheral component interconnect) interface, etc.

[0128] In one or more embodiments, the control device in the disclosed board can be configured to regulate the state of the chip. To this end, in one application scenario, the control device can include a microcontroller unit (MCU) for regulating the working state of the chip.

[0129] According to the above description in combination with Figures 9 and 10, those skilled in the art can understand that the present disclosure also discloses an electronic device or device, which may include one or more of the above-mentioned boards, one or more of the above-mentioned chips and / or one or more of the above-mentioned combined processing devices.

[0130] According to different application scenarios, the electronic devices or devices disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, Internet of Things terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, vehicles, household appliances, and / or medical equipment. The vehicles include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging (MRI), ultrasound machines and / or electrocardiographs. The electronic devices or devices disclosed herein may also be applied to the Internet, Internet of Things, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, medical care and other fields. Furthermore, the electronic devices or devices disclosed herein may also be used in cloud, edge, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, electronic devices or apparatuses with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smartphones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0131] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will understand that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teachings of the present disclosure, those skilled in the art will understand that some of the steps therein can be performed in other orders or simultaneously. Further, those skilled in the art will understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily necessary for the implementation of one or more schemes of the present disclosure. In addition, depending on the different schemes, the description of some embodiments of the present disclosure also has different emphases. In view of this, those skilled in the art will understand that the parts that are not described in detail in a certain embodiment of the present disclosure may also refer to the relevant descriptions of other embodiments.

[0132] In terms of specific implementation, based on the disclosure and teachings of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this document divides them based on the consideration of logical functions, and there may be other ways of division in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0133] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solution described in the embodiments of this disclosure. In addition, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically separately.

[0134] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0135] The foregoing content can be better understood in accordance with the following terms:

[0136] Item 1. A method for optimizing a computational graph, comprising:

[0137] Get the View operator subgraph of the tensor data in the calculation graph;

[0138] Determining whether there is a sequence of operators in a specific order in the View operator subgraph, wherein the specific order includes a transpose operator being located before a slice operator; and

[0139] In response to the existence of the operator sequence in the specific order, the position of the slicing operator and / or the transposition operator in the operator sequence is adjusted so that the slicing operator is processed in advance.

[0140] Clause 2. The optimization method according to clause 1, further comprising:

[0141] Normalizing the transposition type operator to a specified transposition type target operator whose functions can replace each other;

[0142] Normalizing the slice-type operators to designated slice-type target operators whose functions can replace each other; and

[0143] According to the normalized result, the position of the slice-type target operator and / or the transposition-type target operator in the normalized operator sequence is adjusted so that the slice-type target operator is pre-processed.

[0144] Clause 3. The optimization method according to clause 1 or 2, wherein the sequence of operators in a specific order comprises a continuous sequence of transposition operators and slice operators.

[0145] Clause 4. The optimization method according to any one of clauses 1 to 3, wherein adjusting the position of the slicing operator and / or the transposition operator in the operator sequence so that the pre-processing of the slicing operator includes:

[0146] According to the dimensional mapping relationship between the dimensional information of the input tensor of the transposition operator and the slice dimension parameters of the slice operator, the parameters of the slice operator are updated so as to adjust the slice operator to be processed before the transposition operator.

[0147] Clause 5. The optimization method according to clause 4, further comprising:

[0148] Determining a first mapping relationship between the dimension information of the input tensor and the transformation dimension parameter of the transpose operator; determining a second mapping relationship between the slice dimension parameter of the slice operator and the transformation dimension parameter of the transpose operator; and

[0149] The dimension mapping relationship is determined according to the first mapping relationship and the second mapping relationship.

[0150] Clause 6. The optimization method according to Clause 2, further comprising:

[0151] Before performing the repositioning, fusing consecutive identical target operators into a single target operator; and / or

[0152] After performing the position adjustment, multiple consecutive identical target operators are fused into a single target operator.

[0153] Clause 7. The optimization method according to Clause 1, further comprising: after performing the position adjustment, normalizing the transposition type operator to a specified transposition type target operator whose functions can replace each other;

[0154] Normalizing the slice-type operators to designated slice-type target operators whose functions can replace each other; and

[0155] According to the normalized results, multiple consecutive identical target operators are fused into a single target operator.

[0156] Clause 8. The optimization method according to clause 6 or 7, wherein when the multiple consecutive identical target operators are slicing operators, fusing the multiple consecutive identical target operators into a single target operator comprises:

[0157] In response to the multiple target operators operating in the same dimension, determining equivalent parameters of a fused single target operator according to corresponding slicing ranges and data jump steps of the multiple target operators in the same dimension;

[0158] In response to the multiple target operators operating in different dimensions, the dimensional operations of the multiple target operators in different dimensions are merged.

[0159] Clause 9. The optimization method according to clause 8, wherein determining the equivalent parameters of the fused single target operator comprises:

[0160] Determining an equivalent slice starting position of a fused single target operator according to a first data jump step of a target operator that performs operation first among the multiple target operators and a slice starting position of each target operator;

[0161] Determining an equivalent slice end position of a fused single target operator based on a first data jump step and a first slice start position of a target operator that performs operations first among the multiple target operators, and based on a slice end position of a target operator that performs operations later among the multiple target operators; and

[0162] According to the data acquisition jump step of each target operator in the multiple target operators, an equivalent data acquisition jump step of the fused single target operator is determined.

[0163] Clause 10. The optimization method according to clause 8 or 9, wherein determining the equivalent parameter comprises: dim new =dim, start new =start1+start2*step1, end new =start1+end2*step1, step new =step1*step2,

[0164] Among them, dim new Indicates the equivalent slice dimension after fusion, dim indicates the same dimension on which the multiple target operators operate, start new Indicates the starting position of the equivalent slice after fusion, start1 indicates the starting position of the first slice of the target operator that performs the operation first among the multiple target operators, start2 indicates the starting position of the second slice of the target operator that performs the operation later among the multiple target operators, and end newIndicates the end position of the equivalent slice after fusion, end1 indicates the end position of the first slice of the target operator that performs the operation first among the multiple target operators, end2 indicates the end position of the second slice of the target operator that performs the operation later among the multiple target operators, step new Indicates the equivalent data jump step after fusion, step1 indicates the first data jump step of the target operator that performs the operation first among the multiple target operators, and step2 indicates the second data jump step of the target operator that performs the operation later among the multiple target operators.

[0165] Clause 11. The optimization method according to clause 2, wherein:

[0166] The target operator of the transposition type is the Transpose operator;

[0167] The target operator for slice type is Slice operator.

[0168] Clause 12. A computing device for optimizing a computation graph, comprising:

[0169] a processor configured to execute program instructions; and

[0170] A memory configured to store the program instructions, which, when loaded and executed by the processor, enables the processor to perform the computation graph optimization method according to any one of clauses 1 to 11.

[0171] Clause 13. A computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implement the computational graph optimization method described in any one of Clauses 1-11.

[0172] Clause 14. A computer program product comprising a computer program or instructions, which, when executed by a processor, implements the computation graph optimization method described in any one of Clauses 1-11.

[0173] Although a plurality of embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may conceive of many modifications, changes, and alternatives without departing from the ideas and spirit of the present disclosure. It should be understood that in practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The appended claims are intended to define the scope of protection of the present disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for optimizing a computational graph, comprising: Get the View class operator subgraph of tensor data in the calculation graph; Determine whether there is a sequence of operators in a specific order in the View operator subgraph, wherein the specific order includes that the transposition operator is located before the slice operator; and In response to the existence of the operator sequence in the specific order, the position of the slicing operator and / or the transposition operator in the operator sequence is adjusted so that the slicing operator is processed in advance.

2. The optimization method according to claim 1, further comprising: Normalizing the transposition type operator to a specified transposition type target operator whose functions can replace each other; Normalizing the slice-type operators to designated slice-type target operators whose functions can replace each other; as well as According to the normalized result, the position of the slice-type target operator and / or the transposition-type target operator in the normalized operator sequence is adjusted so that the slice-type target operator is pre-processed.

3. The optimization method according to claim 1 or 2, wherein the operator sequence in a specific order comprises a continuous sequence of transposition operators and slicing operators.

4. The optimization method according to any one of claims 1 to 3, wherein adjusting the position of the slicing operator and / or the transposition operator in the operator sequence so that the pre-processing of the slicing operator comprises: According to the dimensional mapping relationship between the dimensional information of the input tensor of the transposition operator and the slice dimension parameters of the slice operator, the parameters of the slice operator are updated so as to adjust the slice operator to be processed before the transposition operator.

5. The optimization method according to claim 4, further comprising: Determine a first mapping relationship between the dimension information of the input tensor and the transformation dimension parameter of the transposition operator; Determine a second mapping relationship between a slice dimension parameter of the slice operator and a transformation dimension parameter of the transpose operator; as well as The dimension mapping relationship is determined according to the first mapping relationship and the second mapping relationship.

6. The optimization method according to claim 2, further comprising: Before performing the position adjustment, multiple consecutive identical target operators are fused into a single target operator; and / or After performing the adjustment position, multiple consecutive identical target operators are fused into a single target operator.

7. The optimization method according to claim 1, further comprising: After performing the repositioning, Normalizing the transposition type operator to a specified transposition type target operator whose functions can replace each other; Normalizing the slice-type operators to designated slice-type target operators whose functions can replace each other; as well as According to the normalized results, multiple consecutive identical target operators are fused into a single target operator.

8. The optimization method according to claim 6 or 7, wherein when the multiple consecutive identical target operators are slice operators, fusing the multiple consecutive identical target operators into a single target operator comprises: In response to the multiple target operators operating in the same dimension, determining equivalent parameters of a fused single target operator according to corresponding slicing ranges and data fetching steps of the multiple target operators in the same dimension; In response to the multiple target operators operating in different dimensions, the dimensional operations of the multiple target operators in different dimensions are merged.

9. The optimization method according to claim 8, wherein determining the equivalent parameters of the fused single target operator comprises: Determine an equivalent slice starting position of a fused single target operator according to a first data fetching jump step of a target operator that first performs operation among the multiple target operators and a slice starting position of each target operator; Determine an equivalent slice end position of a fused single target operator according to a first data fetch jump step and a first slice start position of a target operator that performs operations first among the multiple target operators, and according to a slice end position of a target operator that performs operations later among the multiple target operators; as well as According to the data acquisition jump step of each target operator in the multiple target operators, an equivalent data acquisition jump step of a fused single target operator is determined.

10. The optimization method according to claim 8 or 9, wherein determining the equivalent parameters comprises: dim new =dim, start new =start1+start2*step1, end new =start1+end2*step1, step new =step1*step2, Among them, dim new represents the equivalent slice dimension after fusion, dim represents the same dimension on which the multiple target operators operate, start new represents the starting position of the equivalent slice after fusion, start1 represents the starting position of the first slice of the target operator that performs the operation first among the multiple target operators, start2 represents the starting position of the second slice of the target operator that performs the operation later among the multiple target operators, and end new represents the end position of the equivalent slice after fusion, end1 represents the end position of the first slice of the target operator that performs the operation first among the multiple target operators, end2 represents the end position of the second slice of the target operator that performs the operation later among the multiple target operators, step new It represents the equivalent data fetching jump step after fusion, step1 represents the first data fetching jump step of the target operator that performs the operation first among the multiple target operators, and step2 represents the second data fetching jump step of the target operator that performs the operation later among the multiple target operators.

11. The optimization method according to claim 2, wherein: The transposition type target operator is the Transpose operator; The target operator for slice type is Slice operator.

12. A computing device for optimizing a computing graph, comprising: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, and when the program instructions are loaded and executed by the processor, the processor executes the calculation graph optimization method according to any one of claims 1-11.

13. A computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by one or more processors, implement the calculation graph optimization method as described in any one of claims 1-11.

14. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for optimizing the computational graph described in any one of claims 1 to 11.

Images