Optimization method of computational graph, data processing method and related product

Abstract

The disclosure discloses an optimization method of a computing graph, a data processing method, a computing device, a computer readable storage medium and a computer program product. The computing device performing the optimization method of the computing graph can be included in a combined processing device, which can also include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can also include a storage device connected to the computing device and the other processing devices, respectively, for storing data of the computing device and the other processing devices. The scheme of the disclosure can optimize data access by constructing a view class operator subgraph, thereby improving the processing efficiency of the machine.

Description

Technical Field

[0001] This disclosure generally relates to the field of intelligent computing, and more particularly to the field of compilers. More specifically, this disclosure relates to a method for optimizing a computational graph, a data processing method, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In intelligent computing systems, programming frameworks provide programmers with an interface to use hardware and the system, serving as a crucial core hub. On one hand, programming frameworks can encapsulate commonly used operations in algorithms into operators for programmers to directly call, such as convolution and pooling. On the other hand, as the interface between software and hardware, programming frameworks can encapsulate hardware architecture, thereby reducing the complexity and difficulty of writing or applying deep learning algorithms and improving the efficiency of algorithm implementation.

[0003] TensorFlow and PyTorch are currently popular deep learning frameworks. These frameworks typically use computation graphs to describe the computation process of machine learning algorithms, tensors to represent all data within the computation graph, and operators to represent various operations. There exists a class of operators, such as transpose, slice, and split, that only change the outward appearance or look of the tensor data without altering its actual arrangement in memory; that is, they do not perform actual memory data movement. These operators can be called view operators.

[0004] Due to this characteristic of view-class operators, tensor data is usually not contiguous in memory. Reading and operating on this discontinuous data can lead to low memory access efficiency and high processing time for hardware devices. Summary of the Invention

[0005] To at least partially address one or more of the technical problems mentioned in the background section, this disclosure provides solutions from several aspects. On one hand, it provides an optimization method for computation graphs by constructing view-class operator subgraphs for subsequent memory data continuity processing. On the other hand, it provides a data processing method that can perform memory data continuity processing based on pre-constructed view-class operator subgraphs, thereby improving data memory access efficiency.

[0006] In a first aspect, this disclosure discloses a method for optimizing a computation graph, comprising: traversing operators associated with tensor data in the computation graph; and when the operator is a view-type operator, extracting the operator to construct a view-type operator subgraph, wherein the view-type operator subgraph is used to perform memory data continuity processing.

[0007] In a second aspect, this disclosure discloses a data processing method, comprising: in response to the tensor data to be processed being non-contiguous in memory, obtaining a view class operator subgraph of the tensor data, wherein the view class operator subgraph is constructed according to the method of the first aspect of this disclosure; and, based on the information of the view class operator subgraph, calling the corresponding kernel to perform data transfer processing to convert the tensor data into tensor data that is contiguous in memory.

[0008] In a third aspect, this disclosure discloses a computing apparatus for optimizing a computational graph or performing data processing, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, which, when loaded and executed by the processor, cause the processor to perform an optimization method for a computational graph according to a first aspect of this disclosure, or a data processing method according to a second aspect of this disclosure.

[0009] In a fourth aspect, this disclosure discloses a computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform an optimization method for a computational graph according to a first aspect of this disclosure, or a data processing method according to a second aspect of this disclosure.

[0010] In a fifth aspect, this disclosure discloses a computer program product, including a computer program or instructions that, when executed by a processor, implement the optimization method of the computational graph of the first aspect of this disclosure or the data processing method of the second aspect of this disclosure.

[0011] Based on the optimization methods and data processing methods of the computation graph provided above, subgraphs can be constructed for view-type operators in the computation graph. This allows for optimization of data memory continuity processing and improvement of data access efficiency. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0013] Figure 1 Examples illustrate different shapes of multidimensional arrays and their storage order in memory;

[0014] Figure 2 An exemplary flowchart of an optimization method for a computational graph according to an embodiment of this disclosure is shown;

[0015] Figure 3An exemplary flowchart of an optimization method for a computational graph according to another embodiment of this disclosure is shown;

[0016] Figures 4a-4c The structures of several exemplary computation graphs and the corresponding constructed view class operator subgraphs are shown.

[0017] Figure 5 An exemplary flowchart of a data processing method according to an embodiment of this disclosure is shown;

[0018] Figure 6 A block diagram of the hardware configuration of a computing device that can implement various embodiments of the present disclosure is shown;

[0019] Figure 7 A structural diagram of a combined processing apparatus according to an embodiment of the present disclosure is shown; and

[0020] Figure 8 A schematic diagram of the structure of a board card according to an embodiment of this disclosure is shown. Detailed Implementation

[0021] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0022] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0023] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0024] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0025] The embodiments disclosed herein are described in detail below with reference to the accompanying drawings.

[0026] In the programming framework of intelligent computing systems, data is typically modeled as tensors. A tensor can be viewed as an N-dimensional array, and the dimension of the array is the order of the tensor. Therefore, a 0th-order tensor corresponds to scalar data; a 1st-order tensor corresponds to a one-dimensional array, i.e., a vector; a 2nd-order tensor corresponds to a two-dimensional array, i.e., a matrix; and so on, an Nth-order tensor corresponds to an N-dimensional array. For example, an RGB image can be represented as a 3rd-order tensor, while a dataset consisting of multiple RGB images can be represented as a 4th-order tensor.

[0027] Each tensor has some common properties, including data type and shape. The shape of a tensor represents the length of each order. For example, a 0th-order tensor corresponds to a scalar data, and its shape is empty; a 1st-order tensor corresponds to a one-dimensional vector, and its shape contains one element, the value of which is the length of the vector; a 2nd-order tensor corresponds to a matrix, and its shape contains two elements, corresponding to the lengths of the rows and columns respectively; a 3rd-order tensor corresponds to three-dimensional data, and its shape contains three elements, each corresponding to the length of its order.

[0028] Although multidimensional arrays have multiple dimensions, because the layout of memory (e.g., DRAM and cache RAM) is always one-dimensional, there is a correspondence between multidimensional arrays and their storage order in memory. Multidimensional arrays are typically allocated in contiguous storage space, meaning they can be unfolded one-dimensionally and stored sequentially in memory.

[0029] Figure 1 Examples illustrate different shapes of multidimensional arrays and their storage order in memory, where a one-dimensional array of a contiguous block of memory is used to implement the storage of the multidimensional array.

[0030] Figure 1 Figure (a) shows the first data, namely the three-dimensional array X, which has three dimensions: dimension 0 (dim0), dimension 1 (dim1), and dimension 2 (dim2). The size of dimension 0 is 2, the size of dimension 1 is 2, and the size of dimension 2 is 3. Therefore, its shape (size) can be represented as: X3 = (2, 2, 3).

[0031] Figure 1Figure (c) illustrates the storage order of the three-dimensional array X in memory. Data with the same background in the figure represent data located in the same dimension. Assuming storage follows a low-dimensional priority order (e.g., from left to right in the shape representation corresponding to high to low dimensions), expanding the first data in one dimension yields:

[0032] X=[1,2,3,4,5,6,7,8,9,10,11,12].

[0033] More specifically, the data in the lowest dimension (within the same row) is continuous, while the data in higher dimensions are spaced at different intervals. For example, in the storage configuration shown in Figure (c), accessing the physical structure of adjacent elements in dimension dim2 requires an offset of 1 position (e.g., from data 1 to data 2, data 5 to data 6, etc.); accessing the physical structure of adjacent elements in dimension dim1 requires an offset of 3 positions (e.g., from data 1 to data 4, data 2 to data 5, ..., data 9 to data 12, etc.); while accessing the physical structure of adjacent elements in dimension dim0 requires an offset of 6 positions (e.g., from data 1 to data 7, data 2 to data 8, ..., data 6 to data 12, etc.). This offset is called the stride. The stride of each dimension of the three-dimensional array X can be represented as S. X = (6,3,1).

[0034] In the programming framework of intelligent computing systems, there exist view-class operators that manipulate the external representation of tensors, such as transpose, slice, and split. Taking transpose as an example, it transforms tensors according to a certain dimensional transformation rule (perm). N = (p1, p2, ..., p i ,…,p N ), to obtain the data arrangement after dimensional transformation, where p i The values ​​of (i∈1,2,…,N) represent the original dimensions of the array, p i In perm N The position in the array represents the target dimension of the transformation. For example, given the dimension transformation rule perm3 = (0, 2, 1), it means that dimension 1 and dimension 2 should be swapped, that is, the original dimension 1 should be transformed into dimension 2 of the new array, and the original dimension 2 should be transformed into dimension 1 of the new array.

[0035] Figure 1Figure (b) shows the transformed array Y obtained after performing the transpose operator on the 3D array X shown in Figure (a). In this example, the exemplary dimension transformation rule perm3 = (0, 2, 1) is applied. As can be seen from the figure, compared to array X, dimensions 1 and 2 of array Y have been swapped. At this time, the dimension information of the 3D array Y can be represented as: Y3 = (2, 3, 2).

[0036] However, since view class operators do not change the storage location of data in memory, the array Y obtained after the transpose operation is still stored in memory in the same order as before. Figure 1 As shown in (c). At this point, according to the storage order in (c), the step size of each dimension of array Y becomes S. Y = (6,1,3). It can be seen that if storing data in order of lower dimension priority is called continuity, then the current storage order of array Y is discontinuous. That is, after the transpose operator, because the dimensional order of the array is changed, but the storage location in memory is not changed, the storage order of the array in memory becomes discontinuous.

[0037] If we want the array Y to be contiguous in memory, then following the principle of prioritizing lower dimensions, its one-dimensional expansion should be: Figure 1 As shown in Figure (d):

[0038] Y=[1,4,2,5,3,6,7,10,8,11,9,12].

[0039] The shape of a tensor helps programmers develop an intuitive understanding of it. In programming frameworks such as PyTorch, View class operators can change the tensor's shape (size), stride (the span between the first indices of adjacent dimensions), and storage offset (the offset of the tensor's first element relative to its initial storage position), but they do not change the tensor's actual storage location. In this case, the tensor uses size, stride, and storage offset to calculate the data's memory location on the device.

[0040] Assume the size of the tensor is (s0, s1, s2, ..., s). i ), where the stride is (y0, y1, y2, ..., y i If storage_offset is b, then the tensor at point (x0, x1, x2, ..., x...) is at the point (x0, x1, x2, ..., x...). i The basic formula for calculating the memory location corresponding to point ) is:

[0041]

[0042] Where dptr is the starting position of the tensor in memory, and dtype is the data type of the tensor.

[0043] Based on the above Figure 1 As the description shows, tensors in the computation graph, after being processed by view-class operators, produce discontinuous corresponding data. When accessing these tensors, traditional CPUs and GPUs need to read and access this discontinuous data according to the formula described above, which leads to low memory access efficiency and high time consumption in hardware devices. Another approach is to call the contiguous() operator, which first moves the data into contiguous storage according to the formula described above before performing subsequent access and operations. However, this method is very time-consuming; when dealing with large amounts of data, moving the data one by one will result in significant time consumption.

[0044] In view of this, considering that in computation graph operations such as neural networks, the generation of discontinuous memory data is often caused by view-class operators, this disclosure proposes a scheme for constructing view-class operator subgraphs for view-class operators in computation graphs. This view-class operator subgraph can then support the efficient subsequent process of moving memory data from discontinuous to continuous.

[0045] Regarding the terms "node" and "operator" mentioned in this disclosure, it should be noted that the term "operator" refers to the computational aspect of computing (or the software or algorithmic aspect); while the term "node" is a more descriptive term (from a graphical or more intuitive perspective). In terms of their meaning, "operator" and "node" are essentially the same. That is, in this disclosure, the terms "operator" and "node" can be considered to have the same meaning and can be used interchangeably, simply describing things from different perspectives.

[0046] Figure 2 An exemplary flowchart of an optimization method for a computation graph according to an embodiment of this disclosure is shown. In this optimization method, subsequent in-memory data continuity processing is supported by constructing a view-class operator subgraph.

[0047] As shown in the figure, in step 210, for the tensor data in the computation graph, the operators associated with the tensor data are traversed.

[0048] A computation graph is a directed graph consisting of nodes and edges, where tensors are passed between nodes. Execution in a computation graph follows the order of the directed graph; each time a tensor passes through a node, it is used as input to the operation performed at that node, and the result flows along the output edge of that node to the next node. Therefore, when constructing an operator subgraph for a view class, you can traverse the nodes or operators that need to be processed on the tensor data, following the order of the directed graph.

[0049] Next, in step 220, when the operator encountered during traversal is a view-type operator, the operator is extracted to construct a view-type operator subgraph.

[0050] In some embodiments, extracting view-class operators to construct a view-class operator subgraph may include: associatively caching the operator information and operator index of the operator; and adding the operator index to the view-class operator subgraph. In these embodiments, by storing the operator information and the view-class operator subgraph separately and establishing the relationship between the two through the operator index, the structure of the view-class operator subgraph can be simplified, facilitating subsequent memory data continuity processing.

[0051] Each operator has attributes to identify relevant information during operation execution. Common attributes include: operator name, operator type, operator input data, operator output data, and operation parameters. In some embodiments, the cached operator information may include at least one of the following: description information of the operator's input data, description information of the output data, and operation parameters. It can be understood that the input and output data of the operator are both tensor data, and the description information of tensor data mainly includes the shape, stride, and storage offset mentioned earlier.

[0052] The operational parameters of an operator are related to the function it performs. For example, the transpose operator's operational parameters may include the two dimensions to be swapped (dim0, dim1). Similarly, the chunk operator's function is to divide a tensor into equal parts along dimension dim; its operational parameters may include the number of chunks to be divided (chunks) and the dimension to be divided (dim).

[0053] The above describes extracting view-class operators that cause discontinuities in memory data to construct view-class operator subgraphs to support subsequent processing of memory data continuity. It can be understood that a view-class operator subgraph can be constructed for each tensor data. Furthermore, it can be understood that each tensor data can include multiple view-class operator subgraphs based on the continuity of view-class operators in the computation graph.

[0054] Figure 3 An exemplary flowchart of an optimization method for a computation graph according to another embodiment of this disclosure is shown. In this embodiment, the construction of the view class operator subgraph can be further optimized to simplify information storage.

[0055] As shown in the figure, when extracting operators to construct view-class operator subgraphs, for each encountered view-class operator, step 310 first checks whether the operator information has been cached in memory. Operator information may include, for example, the description information of the operator's input data, the description information of its output data, and the operation parameters mentioned above.

[0056] If the operator information is not cached, it means that the operator is a new operator relative to the operators in memory. The process proceeds to step 320, where an operator sequence number is generated for the operator and the previously described associated caching of the operator information and operator sequence number is performed. Then, in step 330, the operator sequence number is added to the view class operator subgraph.

[0057] If the operator information is already cached, there is no need to cache the same information again. Instead, the process proceeds directly to step 330, where only the operator index of the cached operator is added to the view class operator subgraph.

[0058] By using the above processing methods, the amount of cached information can be effectively reduced, and the construction of view class operator subgraphs can be simplified.

[0059] Figures 4a-4c Several exemplary computation graph structures and the corresponding constructed view class operator subgraph structures are shown.

[0060] Figure 4a A unidirectional computation graph is shown, in which the input tensor A 410 sequentially passes through the following nodes according to the flow of the computation graph: transpose operator 411, slice operator 412, slice operator 413, and Matmul (matrix multiplication) operator 414. Among these operators, transpose operator 411, slice operator 412, and slice operator 413 are all view-type operators, while Matmul (matrix multiplication) operator 414 is a computation-type operator.

[0061] According to the view class operator subgraph construction scheme of this disclosure embodiment, these view class operators are extracted to form a view class operator subgraph. For example... Figure 4a As shown on the right, for the input tensor A, the view class operator subgraphs sequentially include the transpose operator 411, the slice operator 412, and the slice operator 413.

[0062] In some embodiments, assuming the operator number generated by the transpose operator 411 is 1, the operator number of the slice operator 412 is 2, and the operator number of the slice operator 413 is 3, the constructed view class operator subgraph can be represented as 1->2->3 using these operator numbers. The operator information of the corresponding operator can be extracted from the cached information using the operator numbers.

[0063] In other embodiments, assuming that slice operator 412 and slice operator 413 have the same operator information, they can store only one copy of the operator information and share the same operator index. In this embodiment, when processing slice operator 413, if it is found that the operator information of slice operator 413 is the same as the operator information cached for the previous slice operator 412, then there is no need to perform the caching step. Instead, the operator index 2 of the cached slice operator 412 is directly assigned to slice operator 413 and added to the view class operator subgraph. At this time, the constructed view class operator is represented by the operator index 1->2->2.

[0064] Figure 4b A computational graph of a residual structure is shown, where the input tensor B 420 flows through the following nodes according to the computational graph: view operator 421, Conv (convolution) operator 422, Act (activation) operator 423, and Add (addition) operator 424. The output of view operator 421 is also input to Add operator 424 as its other addend. Among these operators, only view operator 421 belongs to the view class, while the rest are computation class operators.

[0065] According to the view-class operator subgraph construction scheme of this disclosure embodiment, view-class operators included in the computation graph are extracted to form a view-class operator subgraph. For example... Figure 4b As shown on the right, for the input tensor B, the view class operator subgraph only includes the view operator 421.

[0066] Figure 4cA multi-branch computation graph is shown, in which the input tensor C 430 flows through the following nodes according to the direction of the computation graph: split operator 431, transpose1 operator 432, transpose2 operator 433, transpose3 operator 434 located on the three branches respectively, BMM1 operator 435, Softmax operator 436, which operates on the outputs of the first and second branches, and BMM2 operator 437, which operates on the results of the first two branches and the output of the third branch. Among these operators, split operator 431 and the three transpose operators 432-434 are view operators, and the rest are computation operators.

[0067] According to the view-class operator subgraph construction scheme of this disclosure embodiment, view-class operators included in the computation graph are extracted to form a view-class operator subgraph. For example... Figure 4c As shown on the right, for the input tensor C, the view-class operator subgraph can be divided into three branches based on the operation parameters of the split operator 431, such as the number of data blocks it is divided into. Each branch includes the split operator 431 and one of the corresponding transpose operators 432 to 434. It can be seen that when the view-class operator is a multi-branch operator, a view-class operator subgraph including the corresponding number of branches can be constructed based on the multi-branch operator.

[0068] The above examples illustrate the construction scheme of the view class operator subgraph provided in this disclosure embodiment.

[0069] As mentioned earlier, when tensor data becomes discontinuous in memory after passing through view-class operators, traditional CPUs and GPUs need to access and read this discontinuous data using the formula described earlier. This leads to low memory access efficiency and high time consumption in hardware devices. In neural network computing libraries, most operators require the input tensors to be contiguous in memory; otherwise, errors will occur. In this case, operators such as `contiguous()` need to be called. These operators also move the data one by one into contiguous storage according to the formula described above. This method of moving data one by one is very time-consuming, bringing a significant time overhead to the computation of the entire computation graph.

[0070] In this disclosed embodiment, once the view-type operator subgraph is constructed, when encountering operators (such as computation-type operators) that require tensor data to be continuous in memory, memory data continuity processing can be performed based on these pre-constructed view-type operator subgraphs, thereby reducing data transfer time and improving computational efficiency.

[0071] Figure 5 An exemplary flowchart of a data processing method according to some embodiments of this disclosure is shown.

[0072] As shown in the figure, in step 510, in response to the fact that the tensor data to be processed is not contiguous in memory, a view class operator subgraph of the tensor data is obtained. The view class operator subgraph of the tensor data is constructed, for example, according to the method described above.

[0073] In some embodiments, the `is_contiguous` function can be used to determine whether the tensor data is contiguous in memory. If the tensor data is contiguous, no additional processing is required. If the tensor data is discontiguous, the view-class operator subgraph associated with that tensor data can be obtained.

[0074] It is understandable that if there is no view class operator subgraph associated with the tensor data, then the tensor data can only be made continuous by moving the data one by one in the existing way, such as calling the contiguous function.

[0075] Next, in step 520, based on the information of the obtained view class operator subgraph, the corresponding kernel is called to perform data transfer processing to convert the tensor data into tensor data that is continuous in memory.

[0076] Specifically, to avoid the time overhead of moving data one by one, we can analyze the operator types in the view class operator subgraph and call the kernel that matches the operator type to perform data moving processing. These kernels move data in blocks according to the operator type.

[0077] In programming frameworks such as PyTorch, there are various view operators to implement different functionalities. These operators include, but are not limited to: transpose, permute, select, chunk, narrow, slice, expand, view, and so on.

[0078] While these operators implement various specific functions, they can also be categorized. In some embodiments, they can be classified into three categories based on their impact on the data size: size reduction, size expansion, and size invariance. For example, among the operators listed above, transpose, permute, and view do not change the size of the tensor data and belong to the size invariance category; select, chunk, narrow, and slice reduce the size of the tensor data and belong to the size reduction category; while expand increases the size of the tensor data and belong to the size expansion category.

[0079] For each functional category, a suitable kernel can be selected from a high-performance computing library to perform the corresponding data manipulation. This kernel can implement the functions of all operators within the corresponding functional category. For example, for view operators of the constant-size rearrangement type, the transpose kernel in a high-performance computing library (such as CNNL) can be called to implement the data rearrangement function. As another example, for view operators of the scale expansion type, the expandkernel in CNNL can be called to implement the data expansion function.

[0080] Therefore, by traversing each view class operator and calling the kernel according to the order of the view class operator subgraph, the tensor data can be transformed from a discontinuous memory state to a contiguous memory state.

[0081] The foregoing description, in conjunction with the accompanying drawings, outlines a method for constructing a view-class operator subgraph and a method for memory data continuity processing based on the view-class operator subgraph. This disclosure also provides a computing device that can be used to construct a view-class operator subgraph or perform memory data continuity processing.

[0082] Figure 6 A block diagram illustrating the hardware configuration of a computing device 600 that can implement various embodiments of the present disclosure is shown. As shown, the computing device 600 may include a processor 610 and a memory 620. Figure 6 In the computing device 600, only the components relevant to this embodiment are shown. Therefore, it will be apparent to those skilled in the art that the computing device 600 may also include components related to... Figure 6 The following are common components with different constituent elements, such as: display.

[0083] The computing device 600 can correspond to a computing device with various processing functions, such as a computing device for compiling computation graphs. For example, the computing device 600 can be implemented as various types of devices, such as personal computers (PCs), server devices, mobile devices, etc.

[0084] Processor 610 is configured to execute program instructions to control all functions of computing device 600. For example, processor 610 controls all functions of computing device 600 by executing programs stored in memory 620 on computing device 600. Processor 610 may be implemented by a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), artificial intelligence processor chip (IPU), etc., provided in computing device 600. However, this disclosure is not limited thereto.

[0085] Memory 620 is hardware used to store various data processed in computing device 600. For example, memory 620 can store processed data and data to be processed in computing device 600. Memory 620 can store data processed or to be processed by processor 610, such as computational graphs before compilation, computational graphs after compilation, etc. Furthermore, memory 620 can store program instructions such as applications and drivers to be driven by computing device 600. For example, memory 620 can store various programs related to optimization algorithms for computational graphs to be executed by processor 610. Memory 620 can be DRAM, but this disclosure is not limited thereto. Memory 620 can include at least one of volatile memory or non-volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. In an embodiment, memory 620 may include at least one of hard disk drive (HDD), solid-state drive (SSD), high-density flash memory (CF), secure digital card (SD), micro-secure digital card (Micro-SD), mini-secure digital card (Mini-SD), extreme digital card (xD), cache, or memory stick.

[0086] In summary, the specific functions implemented by the memory 620 and processor 610 of the computing device 600 provided in the embodiments of this specification can be explained in comparison with the foregoing embodiments in this specification, and can achieve the technical effects of the foregoing embodiments, so they will not be repeated here.

[0087] In this disclosure embodiment, a computer-readable storage medium is also provided, wherein program instructions are stored, which, when loaded and executed by a processor, cause the processor to perform the computation graph optimization method or data processing method described in this disclosure embodiment.

[0088] In this disclosure embodiment, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement an optimization method or data processing method for a computational graph as described in this disclosure embodiment.

[0089] Figure 7 This is a structural diagram illustrating a combined processing apparatus 700 according to an embodiment of the present disclosure. As shown, the combined processing apparatus 700 includes a computing device 702, an interface device 704, other processing devices 706, and a storage device 708. Depending on different application scenarios, the computing processing apparatus may include one or more computing devices 710, which can be configured to... Figure 6 The computing device 600 shown is used to perform the operations described herein in conjunction with the accompanying drawings.

[0090] In different embodiments, the computing processing apparatus disclosed herein can be configured to perform user-specified operations. In exemplary applications, the computing processing apparatus can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included within the computing processing apparatus can be implemented as an artificial intelligence processor core or a portion of the hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as artificial intelligence processor cores or portions of the hardware structure of artificial intelligence processor cores, the computing processing apparatus disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure.

[0091] In exemplary operation, the computing processing device disclosed herein can interact with other processing devices through an interface device to jointly complete user-specified operations. Depending on the implementation, the other processing devices disclosed herein may include one or more types of processors such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and an artificial intelligence processor, as well as general-purpose and / or special-purpose processors. These processors may include, but are not limited to, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing processing device disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, they can be considered to form a heterogeneous multi-core structure.

[0092] In one or more embodiments, the other processing device may serve as an interface between the computing processing device disclosed herein (which may be specifically embodied in artificial intelligence, such as neural network operations) and external data and control, performing basic controls including but not limited to data transfer, starting and / or stopping the computing device. In another embodiment, the other processing device may also cooperate with the computing processing device to jointly complete computational tasks.

[0093] In one or more embodiments, the interface device can be used to transfer data and control commands between a computing processing device and other processing devices. For example, the computing processing device can obtain input data from other processing devices via the interface device and write it to on-chip storage (or memory) of the computing processing device. Further, the computing processing device can obtain control commands from other processing devices via the interface device and write them to on-chip control cache of the computing processing device. Alternatively or optionally, the interface device can also read data from the storage device of the computing processing device and transmit it to other processing devices.

[0094] Additionally or optionally, the combined processing apparatus disclosed herein may further include a storage device. As shown in the figures, the storage device is connected to both the computing processing device and the other processing device. In one or more embodiments, the storage device may be used to store data from the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage of the computing processing device or other processing device.

[0095] In some embodiments, this disclosure also discloses a chip (e.g. Figure 8 The chip shown is 802. In one implementation, the chip is a system-on-a-chip (SoC) and integrates one or more such... Figure 7 The combined processing unit shown is illustrated. This chip can be connected to external interface devices (such as...). Figure 8 The external interface device 806 shown is connected to other related components. These related components may be, for example, a camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. In some applications, the chip may integrate other processing units (e.g., video codecs) and / or interface modules (e.g., DRAM interfaces). In some embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip. In some embodiments, this disclosure also discloses a board that includes the aforementioned chip package structure. The following will be combined with… Figure 8 This board is described in detail.

[0096] Figure 8 This is a schematic diagram illustrating the structure of a board 800 according to an embodiment of this disclosure. As shown, the board includes a storage device 804 for storing data, which includes one or more storage cells 810. This storage device can be connected and transmit data with a controller 808 and the aforementioned chip 802 via, for example, a bus. Furthermore, the board also includes an external interface device 806, configured for data relay or switching between the chip (or a chip in a chip package structure) and an external device 812 (e.g., a server or computer). For example, data to be processed can be transmitted from the external device to the chip via the external interface device. Alternatively, the calculation results of the chip can be transmitted back to the external device via the external interface device. Depending on the application scenario, the external interface device can have different interface forms, for example, it can adopt a standard PCIe interface.

[0097] In one or more embodiments, the controller in the disclosed board can be configured to regulate the state of the chip. Therefore, in one application scenario, the controller may include a microcontroller (MCU) for regulating the operating state of the chip.

[0098] Based on the above combination Figure 7 and Figure 8 Based on the description, those skilled in the art will understand that this disclosure also discloses an electronic device or apparatus that may include one or more of the aforementioned boards, one or more of the aforementioned chips, and / or one or more of the aforementioned combined processing apparatus.

[0099] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0100] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0101] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0102] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0103] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0104] The foregoing can be better understood in accordance with the following terms:

[0105] Clause 1. An optimization method for a computational graph, comprising:

[0106] For tensor data in the computation graph, traverse the operators associated with the tensor data; and

[0107] When the operator is a view-type operator, the operator is extracted to construct a view-type operator subgraph, wherein the view-type operator subgraph is used to perform memory data continuity processing.

[0108] Clause 2. The method described in Clause 1, wherein extracting the operator to construct a view class operator subgraph includes:

[0109] The operator information and operator number of the operator are cached in a related manner; and

[0110] Add the operator number to the view class operator subgraph.

[0111] Clause 3, the method according to Clause 2, wherein extracting the operator to construct a view-class operator subgraph further includes:

[0112] Check if the operator information of the operator has been cached;

[0113] If the operator information is not cached, then an operator sequence number is generated for the operator and the caching and addition are performed; and

[0114] If the operator information of the operator is cached, then only the cached operator index of the operator is added to the view class operator subgraph.

[0115] Clause 4. The method according to any one of Clauses 2-3, wherein the operator information of the operator includes at least one of the following: description information of the input data of the operator, description information of the output data, and operation parameters.

[0116] Clause 5. The method described in any of Clauses 1-4, wherein extracting the operator to construct the view class operator subgraph further comprises:

[0117] When the operator is a multi-branch operator, a view-class operator subgraph including the corresponding number of branches is constructed based on the multi-branch operator.

[0118] Clause 6. A data processing method, comprising:

[0119] In response to the fact that the tensor data to be processed is non-contiguous in memory, a view class operator subgraph of the tensor data is obtained, wherein the view class operator subgraph is constructed according to any one of the methods described in Items 1-5;

[0120] Based on the information of the view class operator subgraph, the corresponding kernel is called to perform data transfer processing to convert the tensor data into tensor data that is continuous in memory.

[0121] Clause 7. The method described in Clause 6, wherein calling the corresponding kernel for data transfer processing includes:

[0122] The operator types in the view class operator subgraph are analyzed, and the kernel matching the operator type is called to perform data transfer processing, wherein the kernel performs data transfer processing in blocks according to the operator type.

[0123] Clause 8. A computing device for optimizing a computational graph or performing data processing, comprising:

[0124] A processor, configured to execute program instructions; and

[0125] A memory configured to store the program instructions, which, when loaded and executed by the processor, cause the processor to perform an optimization method of a computation graph according to any one of clauses 1-5, or a data processing method according to any one of clauses 6-7.

[0126] Clause 9. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform an optimization method of a computation graph according to any one of Clauses 1-5, or a data processing method according to any one of Clauses 6-7.

[0127] Clause 10. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the optimization method of a computational graph as described in any of Clauses 1-5, or the data processing method as described in any of Clauses 6-7.

[0128] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A data processing method, comprising: In response to the fact that the tensor data to be processed is non-contiguous in memory, a view-class operator subgraph of the tensor data is constructed and obtained; Based on the information of the view class operator subgraph, the corresponding kernel is called to perform data transfer processing to convert the tensor data into tensor data that is continuous in memory; The construction of the view class operator subgraph of the tensor data includes: For tensor data in the computation graph, traverse the operators associated with the tensor data; and When the operator is a view-type operator, the operator is extracted to construct a view-type operator subgraph, wherein the view-type operator subgraph is used to perform memory data continuity processing.

2. The method according to claim 1, wherein calling the corresponding kernel for data transfer processing includes: The operator types in the view class operator subgraph are analyzed, and the kernel matching the operator type is called to perform data transfer processing, wherein the kernel performs data transfer processing in blocks according to the operator type.

3. The method according to claim 1, wherein, Extracting the operators to construct a view class operator subgraph includes: The operator information and operator number of the operator are cached in a related manner; and Add the operator number to the view class operator subgraph.

4. The method according to claim 3, wherein extracting the operator to construct a view-class operator subgraph further comprises: Check if the operator information of the operator has been cached; If the operator information of the operator is not cached, then an operator sequence number is generated for the operator and the caching and addition are performed; as well as If the operator information of the operator is cached, then only the cached operator index of the operator is added to the view class operator subgraph.

5. The method according to any one of claims 3-4, wherein, The operator information of the operator includes at least one of the following: description information of the input data, description information of the output data, and operation parameters of the operator.

6. The method according to any one of claims 3-4, wherein extracting the operator to construct a view-class operator subgraph further comprises: When the operator is a multi-branch operator, a view-class operator subgraph including the corresponding number of branches is constructed based on the multi-branch operator.

7. A computing device for optimizing a computational graph or performing data processing, comprising: A processor, configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, cause the processor to perform the data processing method according to any one of claims 1-6.

8. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the data processing method according to any one of claims 1-6.

9. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the data processing method according to any one of claims 1-6.

Images