Machine learning model compiling optimization method and device and computing equipment cluster

Abstract

A machine learning model compilation optimization method comprises the following steps: identifying nodes in a computational graph of a machine learning model to identify sparse nodes; determining an optimal sparse implementation scheme of the sparse nodes, wherein the optimal sparse implementation scheme comprises memory layout of the sparse nodes, configuration parameters and one or more of sparse formats of weight tensors; if the performance data of the sparse node is superior to target performance data under the optimal sparse implementation scheme, generating a first node based on the optimal sparse implementation scheme, and replacing the sparse node with the first node in the computational graph, the target performance data being the performance data of the sparse node under the original sparse implementation scheme; and performing code generation on the computational graph to obtain an execution code of the computational graph. Therefore, by considering the performance benefits of the nodes in the computational graph under different sparse implementations, the sparse nodes in the computational graph are optimized on the basis of the optimal sparse implementation, and the performance benefits of a machine learning model related to the computational graph are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a machine learning model compilation optimization method, device and computing device cluster. Background Art

[0002] At present, deep learning technology is booming and widely used in many fields, which has a profound impact. Deep learning technology has not only made great breakthroughs in traditional computer vision, natural language processing and other fields, but also achieved great success and commercial application value in many cross-application fields, such as e-commerce, autonomous driving, drug discovery, etc. With the widespread application of deep learning, accelerating model deployment has gradually become an important problem. On the one hand, deep learning models are becoming more and more diverse, and many new deep learning model architectures have been proposed, such as convolutional neural networks (CNN), long short-term memory networks (LSTM), transformer and other architectures. On the other hand, deep learning models are becoming larger and larger, and the number of parameters and computational complexity of newly proposed models has increased dramatically. However, with the significant increase in the size of machine learning models, the model inference delay has increased accordingly, the memory usage has also increased accordingly, and higher requirements have been placed on hardware resources, resulting in poor inference performance of machine learning models. Summary of the invention

[0003] The present application provides a machine learning model compilation optimization method, apparatus, computing device cluster, computer storage medium and computer product, which can adjust the memory layout, configuration parameters, sparse format of weight tensors, etc. of the input and output of sparse nodes in the computational graph of the machine learning model, so as to optimize the sparse tensor calculation in the model and improve the reasoning performance of the machine learning model.

[0004] In a first aspect, the present application provides a machine learning model compilation optimization method, including: identifying nodes in a computational graph of a machine learning model to identify sparse nodes in the computational graph, wherein the proportion of 0 elements in the weight tensor of the sparse node is greater than a preset threshold; determining an optimal sparse implementation scheme for the sparse node, the optimal sparse implementation scheme including: one or more of the memory layout of the input and output of the sparse node, the configuration parameters of the sparse node, and the sparse format of the weight tensor of the sparse node; if the performance data of the sparse node is better than the target performance data under the optimal sparse implementation scheme, then based on the optimal sparse implementation scheme, generating a first node, and replacing the sparse node with the first node in the computational graph, wherein the target performance data is the performance data of the sparse node under the original sparse implementation scheme; and performing code generation on the computational graph to obtain the execution code of the computational graph.

[0005] In this way, by considering the performance benefits of different sparse implementations of nodes in the computation graph and taking the optimal sparse implementation as the basis, the memory layout, configuration parameters, sparse format of weight tensors, etc. of the input and output of sparse nodes in the computation graph are adjusted, thereby optimizing the sparse tensor calculations in the model and improving the reasoning performance of the machine learning model.

[0006] In a possible implementation, determining the optimal sparse implementation scheme of a sparse node includes: executing the execution code of the sparse node under each sparse implementation scheme to obtain at least one performance data, wherein one performance data is related to one sparse implementation scheme; when the best performance data in at least one performance data is better than the target performance data, determining the sparse implementation scheme related to the best performance data as the optimal sparse implementation scheme; when the best performance data in at least one performance data is not better than the target performance data, determining the original sparse implementation scheme as the optimal sparse implementation scheme. In this way, by executing the execution code of the sparse node under each sparse implementation scheme, the performance data of each sparse implementation scheme can be obtained, which can be compared with the performance data of the sparse node when no optimization is performed to screen out the optimal sparse implementation scheme.

[0007] In a possible implementation, after replacing the sparse node with the first node in the computation graph, the method further includes: when the memory layouts in the optimal sparse implementation scheme and the original sparse implementation scheme are different, adding a first layout conversion node before the first node and adding a second layout conversion node after the first node, wherein the first layout conversion node is used to perform memory layout conversion on the input of the first node, and the second layout conversion node is used to perform memory layout conversion on the output of the first node. In this way, after adjusting the memory layout of the sparse node, the equivalence of the computation graph can be ensured by adding the layout conversion node.

[0008] In one possible implementation, the computational graph of the machine learning model includes a second node, which is a newly generated node. In this case, before the code generation of the computational graph, it also includes: when there is at least one node between the first node and the second node that has not undergone memory layout adjustment, if the node in at least one node is a node that is not sensitive to memory layout, the memory layout of the input and output of the node in at least one node is adjusted; based on the adjusted memory layout, a new node is generated, and the node in at least one node is replaced. In this way, by adjusting the memory layout of nodes that are not sensitive to memory layout, an opportunity is provided for further optimization of the computational graph.

[0009] In a possible implementation, before the code generation for the computation graph is performed, the following step is further included: when two adjacent nodes in the computation graph are both newly generated nodes, if the memory layout of the output of the first node of the two adjacent nodes is the same as the memory layout of the input of the second node, then the node between the two adjacent nodes is deleted. In this way, redundant nodes in the computation graph can be eliminated, the amount of calculation in the computation graph can be reduced, and the reasoning performance of the machine learning model can be improved.

[0010] In a possible implementation, the method further includes: rearranging the weight tensor of the sparse node by row and column; compressing the rearranged weight tensor and storing the weight tensor in a preset sparse format, the sparse format including: a weight array, a row rearrangement array, a subscript array, an offset array and a block number array, wherein the weight array is used to record the elements in the rearranged weight tensor, the row rearrangement array is used to record the row information of the rearranged weight tensor, the subscript array is used to record the column to which each element recorded in the weight array belongs, the block number array is used to record the number of dense blocks in the rearranged weight tensor, the elements in the dense blocks are used to load from the global memory to the shared memory for calculation during the calculation process, and the offset array is used to identify the number of elements in each row of the rearranged weight tensor. In this way, by rearranging the weight tensor of the sparse node by row and column, the structuring degree of non-zero elements in the weight tensor of the sparse node can be improved, the problem of low execution efficiency of operators in the node caused by the irregularity of sparse distribution is solved, and the performance of the operator is improved.

[0011] In the second aspect, the present application provides a machine learning model compilation optimization device, including: a node identification module, a scheme determination module, a node processing module and a code generation module. Among them, the node identification module is used to identify the nodes in the calculation graph of the machine learning model to identify the sparse nodes in the calculation graph, wherein the proportion of 0 elements in the weight tensor of the sparse node is greater than a preset threshold; the scheme determination module is used to determine the optimal sparse implementation scheme of the sparse node, and the optimal sparse implementation scheme includes: one or more of the memory layout of the input and output of the sparse node, the configuration parameters of the sparse node and the sparse format of the weight tensor of the sparse node; the node processing module is used to generate a first node based on the optimal sparse implementation scheme when the performance data of the sparse node is better than the target performance data under the optimal sparse implementation scheme, and replace the sparse node with the first node in the calculation graph, wherein the target performance data is the performance data of the sparse node under the original sparse implementation scheme; the code generation module is used to generate code for the calculation graph to obtain the execution code of the calculation graph.

[0012] In one possible implementation, when determining the optimal sparse implementation scheme of a sparse node, the scheme determination module is specifically used to: execute the execution code of the sparse node under each sparse implementation scheme to obtain at least one performance data, wherein one performance data is related to a sparse implementation scheme; when the best performance data in at least one performance data is better than the target performance data, determine the sparse implementation scheme related to the best performance data as the optimal sparse implementation scheme; when the best performance data in at least one performance data is not better than the target performance data, determine the original sparse implementation scheme as the optimal sparse implementation scheme.

[0013] In one possible implementation, after the node processing module replaces the sparse node with the first node in the computation graph, the node processing module is further used to: add a first layout conversion node before the first node and add a second layout conversion node after the first node when the memory layouts in the optimal sparse implementation scheme and the original sparse implementation scheme are different, wherein the first layout conversion node is used to perform a memory layout conversion on the input of the first node, and the second layout conversion node is used to perform a memory layout conversion on the output of the first node.

[0014] In a possible implementation, the computation graph includes a second node, and the second node is a newly generated node. At this time, before the code generation module generates code for the computation graph, the node processing module is further used to: when there is at least one node between the first node and the second node that has not undergone memory layout adjustment, if the node in at least one node is a node that is not sensitive to memory layout, adjust the memory layout of the input and output of the node in at least one node; based on the adjusted memory layout, generate a new node, and replace the node in at least one node.

[0015] In a possible implementation, before the code generation module generates code for the computation graph, the node processing module is further used to: when two adjacent nodes in the computation graph are both newly generated nodes, if the memory layout of the output of the first node of the two adjacent nodes is the same as the memory layout of the input of the second node, then delete the node between the two adjacent nodes.

[0016] In a possible implementation, the node processing module is also used to: rearrange the weight tensor of the sparse node by rows and columns; compress the rearranged weight tensor and store the weight tensor in a preset sparse format, which includes: a weight array, a row rearrangement array, a subscript array, an offset array and a block number array, wherein the weight array is used to record the elements in the rearranged weight tensor, the row rearrangement array is used to record the row information of the rearranged weight tensor, the subscript array is used to record the column to which each element recorded in the weight array belongs, the block number array is used to record the number of dense blocks in the rearranged weight tensor, the elements in the dense blocks are used to load from the global memory to the shared memory for calculation during the calculation process, and the offset array is used to identify the number of elements in each row of the rearranged weight tensor.

[0017] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect, or executes the method described in the second aspect or any possible implementation of the second aspect. The computing device cluster may include one or more computing devices.

[0019] In a fifth aspect, the present application provides a computer program product including instructions, which, when executed by a computing device cluster, enables the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect. The computing device cluster may include one or more computing devices.

[0020] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1It is a schematic diagram of the architecture of a machine learning model compilation and optimization system provided in an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a machine learning model compilation optimization method provided in an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of a process for optimizing nodes in a computational graph provided by an embodiment of the present application;

[0024] Figure 4 This is another schematic diagram of a process for optimizing nodes in a computational graph provided by an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of steps for compressing the weight tensor of a sparse node provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of a process of re-arranging weight tensors of sparse nodes provided by an embodiment of the present application;

[0027] Figure 7 It is a structural diagram of a machine learning model compilation and optimization device provided in an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0030] Fig.10 It is a structural diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0032] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0033] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0034] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0035] First, some technical terms involved in this application are introduced.

[0036] (1) Operator

[0037] An operator is a computing unit that can perform simple or complex calculations such as addition, subtraction, multiplication, and division.

[0038] (2) Computational Graph

[0039] A computation graph is a computation function expressed by a directed graph with operators as nodes. A computation graph is mainly represented by nodes and edges. A node in a computation graph can represent an operator. An edge between two operators in a computation graph can represent the relationship between the two operators. Among them, an edge can have a direction, and the direction of the edge can be the flow direction of data between operators. For example, when the edge between operator a and operator b in a computation graph is from operator a to operator b, the data output by operator a is the data required to be input by operator b.

[0040] (3) Sparse format

[0041] A sparse format is a data representation used to efficiently store and process sparse data structures. In a sparse format, only non-zero elements and their position information are stored to reduce memory usage and improve computational efficiency. This data representation can be used to process sparse matrices, sparse tensors, or other high-dimensional data structures in large-scale datasets to save storage space and speed up related operations such as matrix multiplication, data retrieval, and computation. In other words, a sparse format is a data structure designed to efficiently represent and operate sparse data by storing only non-zero elements to minimize the use of memory and computing resources.

[0042] Next, the technical solution provided by this application is introduced.

[0043] Generally speaking, in order to alleviate the demand for hardware resources for the reasoning of machine learning models and accelerate model reasoning, the sparsity of the model becomes a key direction for exploring scalability and reasoning acceleration. As a result, many efficient model pruning techniques have been proposed. Model pruning technology introduces sparsity into deep learning models by deleting redundant parameters in the model's weight matrix, bringing optimization opportunities to further reduce the model's computational workload and memory access and improve performance. However, the diversity of sparsity patterns brought about by pruning technology and the irregularity of sparsity itself make it difficult to perform efficient calculations on hardware, making it difficult to use sparsity for performance optimization.

[0044] At the same time, in order to automatically deploy models and reduce the burden of developers manually developing operators for performance optimization, academia, industry, and open source communities have proposed deep learning compilers, such as tensorvirtual machine (TVM) and ascend kernel compiler (AKG). Deep learning compilers take deep learning models as input, use multiple intermediate representations to efficiently express the models, and combine computational graph optimization and operator optimization to automatically generate specific high-performance code, greatly accelerating the efficiency of model deployment. In addition, in terms of the reasoning performance of some dense models, deep learning compilers have been able to surpass manual optimization and dedicated operator libraries. However, existing deep learning compilers are mainly optimized for dense tensor calculations, and their support for optimizing sparse tensor calculations is relatively weak. Most of them are still in the initial stage, and the optimization for sparsity at the computational graph optimization level and the operator optimization level is insufficient, making it difficult to automatically convert model sparsity into improved model reasoning performance.

[0045] The technical solution provided in the embodiments of the present application is mainly to perceive the sparse nodes in the computational graph of the machine learning model, and adjust one or more of the memory layout of the input and output of the sparse nodes that can obtain performance benefits, configuration parameters (i.e., parameters used to specify the behavior of operators in the nodes and control their operations), and the sparse format of the weight tensor, thereby achieving performance improvement in sparse computing, improving the reasoning performance of the machine learning model, and reducing global additional overhead.

[0046] For example, Figure 1 The schematic diagram of the architecture of a machine learning model compilation optimization system provided by an embodiment of the present application is shown. Figure 1As shown, the machine learning model compilation and optimization system 100 may include: a computational graph optimization module 110, an operator optimization and generation module 120, and a performance evaluation module 130. The computational graph optimization module 110 and the operator optimization and generation module 120 may be configured in a deep learning compiler; the performance evaluation module 130 may be configured on hardware used in conjunction with the deep learning compiler, such as a graphics processing unit (GPU).

[0047] The calculation graph optimization module 110 is mainly used to analyze and optimize the calculation graph of the machine learning model, and generate a more efficient calculation graph by combining optimization methods such as subgraph replacement and algebraic simplification. The machine learning model can be, but is not limited to, obtained by training based on a neural network (NN). The calculation graph of the machine learning model can be, but is not limited to, obtained by processing the machine learning model through a deep learning compiler. The calculation graph optimization module 110 may include: a sparse node identification module 111, a sparse node conversion module 112, and a sparse layout optimization module 113. Among them, the sparse node identification module 111 is mainly used to identify sparse nodes in the calculation graph based on the weight tensor contained in the node attributes of the node in the calculation graph. When the proportion of 0 elements in the weight tensor of a node is greater than a preset sparsity threshold, the node is determined to be a sparse node; when the proportion of 0 elements in the weight tensor of a node is less than or equal to the preset sparsity threshold, the node is determined to be a dense node. In addition, when the node attributes of a node do not contain a weight tensor, the node can be determined to be a dense node. Illustratively, a sparse node is a node that may gain performance benefits by using sparse computing.

[0048] The sparse node conversion module 112 is mainly used to generate new nodes and replace the original sparse nodes with the newly generated nodes when a certain sparse node can obtain performance benefits based on the optimal sparse implementation scheme fed back by the operator optimization and generation module 120. Exemplarily, the optimal sparse implementation scheme may include: one or more of the memory layout of the input and output of the sparse node, the configuration parameters of the sparse node, and the sparse format of the weight tensor of the sparse node. In addition, when the memory layout of the input and output of the newly generated node is different from that of the original sparse node, in order to ensure the equivalence of the calculation graph, layout conversion nodes can be added before and after the newly generated node. Among them, the layout conversion node added before the newly generated node is mainly used to perform memory layout conversion on the input of the newly generated node. The layout conversion node added after the newly generated node is mainly used to perform memory layout conversion on the output of the newly generated node.

[0049] The sparse layout optimization module 113 is mainly used to adjust the memory layout of the input and output of the node that is not optimized by the sparse node conversion module 112 and is located in the white list, so that the memory layout of the input and output of the node is the same as the memory layout of the input and output of the previous sparse node, thereby reducing the overhead introduced by the sparse node conversion. In addition, after completing the adjustment of the memory layout of the input and output of the node, layout conversion nodes can be added before and after the node to ensure the equivalence of the calculation graph. Exemplarily, the operator in the node in the white list is an operator that is insensitive to the memory layout, that is, after changing the memory layout, the performance of the operator will not be reduced. In addition, the sparse layout optimization module 113 can also eliminate redundant layout conversion nodes in the calculation graph. Among them, when two layout conversion nodes are adjacent, and the output of the first layout conversion node is the input of the second layout conversion node, if the input of the first layout conversion node and the output of the second layout conversion node are the same, then the two layout conversion nodes can be considered to be redundant, and the two layout conversion nodes can be eliminated at this time.

[0050] The operator optimization and generation module 120 is mainly used to optimize the operators in the nodes in the calculation graph and generate the execution code of the operators. The operator optimization and generation module 120 may include: a sparse operator tuning module 121, a sparse code generation module 122 and a dense code generation module 123. The sparse operator tuning module 121 is mainly used to optimize the performance of the sparse nodes according to the weight tensor of the sparse nodes identified by the sparse node identification module 111 and the operator attributes of the operators contained therein, so as to search for the sparse implementation scheme with the best performance (i.e., the optimal sparse implementation scheme), so as to optimize the calculation process of the operator according to the hardware characteristics for a specific hardware platform and generate efficient execution code. Exemplarily, the sparse operator tuning module 121 can obtain at least one sparse implementation scheme by adjusting the memory layout of the input and output of the sparse nodes, the configuration parameters of the sparse nodes, the sparse format of the weight tensor of the sparse nodes, etc. Among them, each sparse implementation scheme may include: one or more of the memory layout of the input and output of the sparse node, the configuration parameters of the sparse node, and the sparse format of the weight tensor of the sparse node. In addition, there may be different configuration parameters or different sparse formats under a memory layout. Then, the sparse operator tuning module 121 can call the sparse code generation module 122 to process the sparse nodes under each sparse implementation scheme to generate the execution code of the sparse node. Among them, there is an execution code under a sparse implementation scheme. Then, the sparse operator tuning module 121 can send the execution code it obtains to the performance evaluation module 130, so as to execute the code on the hardware through the performance evaluation module 130, thereby obtaining corresponding performance data. Among them, a sparse implementation scheme is related to a performance data. In addition, the sparse operator tuning module 121 can also send the execution code of the sparse node under the original sparse implementation scheme (i.e., the sparse implementation scheme when the memory layout, configuration parameters and sparse format are not adjusted) to the performance evaluation module 130, so as to execute the code on the hardware through the performance evaluation module 130, thereby obtaining the corresponding performance data. At the same time, the performance evaluation module 130 can return the performance data obtained to the sparse operator tuning module 121. The sparse operator tuning module 121 can use the performance data under the original sparse implementation scheme as a baseline, and compare it with the performance data under each sparse implementation scheme respectively. When the performance data under a certain sparse implementation scheme is higher than the baseline, the sparse implementation scheme can be selected, that is, the sparse implementation scheme is used as the optimal sparse implementation scheme. In addition, when the performance data under multiple sparse implementation schemes are all higher than the baseline, the sparse implementation scheme associated with the best performance data can be used as the optimal sparse implementation scheme. When the performance data of all sparse implementation schemes are lower than the baseline, the memory layout adjustment of the sparse node can be abandoned. At this time, the sparse implementation scheme corresponding to the baseline (that is, the original sparse implementation scheme) can be understood as the optimal sparse implementation scheme.

[0051] The sparse code generation module 122 is mainly used to process the optimized sparse nodes in the computation graph to obtain the execution code of the sparse nodes. The dense code generation module 123 is mainly used to process the unoptimized nodes in the computation graph in combination with loop optimization and other technologies to obtain the execution code of the dense nodes.

[0052] The performance evaluation module 130 mainly executes the codes corresponding to the sparse nodes under various sparse implementation schemes on the hardware to obtain corresponding performance data, such as computing time.

[0053] The above is an introduction to the machine learning model compilation optimization system 100 provided in the embodiment of the present application. The machine learning model compilation optimization system 100 senses the sparse nodes in the computational graph and optimizes the sparse nodes with performance benefits, thereby improving the performance of sparse computing and reducing the global additional overhead.

[0054] Next, a machine learning model compilation optimization method provided in an embodiment of the present application is introduced.

[0055] For example, Figure 2 The flowchart of a machine learning model compilation optimization method provided by an embodiment of the present application is shown. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the machine learning model compilation optimization method may include the following steps:

[0056] S201. Identify the nodes in the computation graph of the machine learning model to obtain n (n≥1) sparse nodes, wherein the proportion of 0 elements in the weight tensor of the sparse nodes is greater than a preset threshold.

[0057] In this embodiment, the proportion of 0 elements in the weight tensor of each node in the computation graph can be counted. When the proportion of 0 elements in the weight tensor of a node is greater than a preset threshold, the node can be determined to be a sparse node; otherwise, the node is considered to be a dense node. In addition, when a node has no weight tensor, it can be considered that the node is not a sparse node, that is, a dense node. Exemplarily, the aforementioned Figure 1 The sparse node identification module 111 in the node is used for identification.

[0058] S202. Determine at least one sparse implementation scheme for the i-th (initial value of i is 1) sparse node, and execute the execution code of the sparse node under each sparse implementation scheme on hardware to obtain k (k≥1) performance data, where one performance data is related to one sparse implementation scheme.

[0059] In this embodiment, at least one sparse implementation scheme can be obtained by adjusting the memory layout of the input and output of the i-th sparse node, the configuration parameters of the sparse node, the sparse format of the weight tensor of the sparse node, etc. Then, the execution code of the sparse node under each sparse implementation scheme can be executed on the hardware to obtain K (K≥1) performance data. Among them, one performance data is related to a sparse implementation scheme. Exemplarily, the performance data can include calculation time.

[0060] S203. Determine whether the best performance data among the k performance data is better than the target performance data, where the target performance data is obtained by executing the execution code of the i-th sparse node under the original sparse implementation solution on hardware.

[0061] In this embodiment, after obtaining k performance data, the k performance data can be compared with the target performance data. The target performance data is obtained by executing the execution code of the i-th sparse node under the original sparse implementation scheme on the hardware. When the best performance data among the k performance data is better than the target performance data, it indicates that the sparse calculation of the i-th sparse node can obtain performance benefits. At this time, the sparse node can be optimized, that is, S204 is executed. When the best performance data among the k performance data are all lower than the target performance data, it indicates that the sparse calculation of the i-th sparse node cannot obtain performance benefits. At this time, the sparse node may not be optimized, and the next sparse node may be judged, that is, S206 is executed.

[0062] S204. When the best performance data among the k performance data is better than the target performance data, a new node is generated based on a sparse implementation solution related to the best performance data.

[0063] In this embodiment, when the best performance data among the k performance data is better than the target performance data, the sparse implementation scheme related to the best performance data can be processed by a compiler, etc. to generate a new node. The sparse implementation scheme related to the best performance data can be understood as the best sparse implementation scheme.

[0064] S205. Replace the i-th sparse node in the computation graph with the newly generated node.

[0065] In this embodiment, after generating a new node, the i-th sparse node can be replaced with the newly generated node in the calculation graph. In addition, after adjusting the memory layout of the input and output of the i-th sparse node, in order to ensure the equivalence of the calculation graph before and after the replacement, a first layout conversion node can be added before the newly generated node, and a second layout conversion node can be added after the newly generated node. Among them, the first layout conversion node is used to perform a memory layout conversion on the input of the newly generated node, so that the memory layout of the data input to the newly generated node can be the same as the memory layout of the input of the newly generated node. The second layout conversion node is used to perform a memory layout conversion on the output of the newly generated node, so that the output of the newly generated node can be the same as the output of the original sparse node.

[0066] S206, i=i+1, and determine whether i is greater than n.

[0067] In this embodiment, after the optimization of the i-th sparse node is completed, the next sparse node can be optimized. At this time, i can be set to i+1, and it is determined whether the new i is greater than n. If i>n, it indicates that the optimization of all sparse nodes has been completed; when i≤n, it indicates that there are still sparse nodes that can be optimized, and it can return to execute S202.

[0068] S207: Generate code for the computation graph to obtain execution code of the computation graph.

[0069] In this embodiment, after the optimization of the sparse nodes in the computation graph is completed, the computation graph can be processed by a compiler or the like to generate code for the computation graph, thereby obtaining the execution code of the computation graph.

[0070] In this way, by considering the performance benefits of nodes in the computation graph under different sparse implementations, the memory layout, configuration parameters, and sparse format of weight tensors of the input and output of sparse nodes in the computation graph are adjusted based on the optimal sparse implementation, thereby improving the performance benefits of machine learning models related to the computation graph. It should be understood that Figure 2 S201 in the calculation graph can identify all sparse nodes at one time, or identify a part of sparse nodes at one time, or identify one sparse node at one time, which can be determined according to the actual situation and is not limited here. Among them, when identifying one sparse node at a time, after identifying a sparse node, when the sparse node does not need to be optimized, other sparse nodes can be identified in the calculation graph; and when the sparse node can be optimized, other sparse nodes can be identified in the calculation graph after the optimization of the sparse node is completed.

[0071] In some embodiments, after the optimization of sparse nodes in the computation graph is completed, there may be two adjacent newly generated nodes where the memory layout of the output of the previous node is the same as the memory layout of the input of the next node. At this time, if a layout conversion node is added between these two nodes, redundant nodes will exist in the computation graph. In other words, between two adjacent newly generated nodes, if the memory layout of the output of the previous node is the same as the memory layout of the input of the next node, the layout conversion node between the two nodes can be deleted, that is, before S207, the computation graph can be further optimized to eliminate redundant layout conversion nodes. For example, if Figure 3 As shown, in Figure 3 In (A), nodes 1 and 2 are newly generated nodes, and the memory layout of the output of node 1 is the same as the memory layout of the input of node 2. At this time, the memory layout of the data output by node 1 after passing through layout conversion nodes 12 and 21 is the same as the memory layout before conversion by layout conversion node 12. Therefore, layout conversion nodes 12 and 21 are redundant nodes, and these two nodes can be eliminated. Figure 3 After removing the redundant nodes in (A), we can get Figure 3 The computational graph shown in (B).

[0072] In some embodiments, after the optimization of sparse nodes in the computation graph is completed, when there are nodes between two newly generated nodes that have not undergone memory layout adjustment, if these nodes that have not undergone memory layout adjustment are not sensitive to memory layout, the memory layout of the input and output of these nodes can be converted to provide opportunities for further optimization of the computation graph. When adjusting, the goal of adjusting can be to eliminate the intermediate layout conversion nodes. For example, Figure 4 As shown, in Figure 4 In (A), nodes 1 and 4 are newly generated nodes, and nodes 2 and 3 are nodes that have not undergone memory layout adjustment. If nodes 2 and 3 are nodes that are not sensitive to memory layout adjustment, the memory layout of the input and output of nodes 2 and 3 can be adjusted, and the following is obtained: Figure 4 The computational graph shown in (B). Figure 4 In (B), the memory layout of node 1's output is the same as the memory layout of node 2's input, the memory layout of node 2's output is the same as the memory layout of node 3's input, and the memory layout of node 3's output is the same as the memory layout of node 4's input. Next, we can eliminate Figure 4 The redundant nodes in (B) can be obtained as follows Figure 4 The computational graph shown in (C) is Figure 4 (A) and Figure 4 By comparing (C), we can see that Figure 4Two nodes are reduced in (C), which makes Figure 4 The amount of computation in the computation graph shown in (C) will be significantly less than Figure 4 The amount of computation in the computation graph shown in (A) can be reduced, thereby significantly reducing the computation time. Exemplarily, the nodes that are insensitive to the memory layout can be, but are not limited to, pre-set nodes.

[0073] In some embodiments, for sparse nodes, when the non-zero elements in the weight tensor of the sparse nodes are irregularly distributed, it will lead to inefficient memory access during the calculation process, resulting in low calculation efficiency. In view of this, this embodiment can rearrange the rows and columns of the weight tensor of the sparse nodes, design a corresponding sparse format, and store the weight tensor in a new sparse format to improve the structuring degree of the non-zero elements in the weight tensor and improve memory access efficiency. Specifically, Figure 5 As shown, the following steps may be included: S501, using a weight reordering algorithm to reorder the weight tensor of the sparse nodes by rows and columns. The rows may be reordered first, and then the columns may be reordered. For example, taking the weight tensor as a two-dimensional matrix, refer to Figure 6 , Figure 6 (A) shows the matrix of the original weight tensor. When rearranging its rows, we can greedily select a group of the closest rows each time based on the repeated elements in the columns between rows, and complete the grouping and rearrangement of all rows, that is, we can get the following Figure 6 The matrix shown in (B) is shown in Figure 6 In (B), for each row, rearrange the elements on the column to obtain Figure 6 The matrix shown in (C). Figure 6 In (C), the weight tensor is divided into a dense part (the part shown in regions 31 and 32) and a sparse part (the part other than regions 61 and 62). During the column rearrangement process, when a column is in the dense part but there are empty elements in the column, the empty elements can be set to 0, such as the elements in the 6th row and 6th column and the elements in the 3rd row and 7th column. Figure 6 In (C), region 61 can be understood as a dense block, and region 62 can be understood as another dense block. The size of the dense block can be, but is not limited to, pre-set.

[0074] S502. Compress the weight tensor after weight rearrangement, and store the weight tensor in a preset sparse format, which may include a weight array, a row rearrangement array, a subscript array, an offset array, and a block number array, so as to express the weight tensor of the rearranged and compressed sparse nodes through these five arrays. Among them, the weight array is used to record the elements in the weight tensor after weight rearrangement, the row rearrangement array is used to record the row information of the weight tensor after weight rearrangement, the subscript array is used to record the column to which each element recorded in the weight array belongs, the block number array is used to record the number of dense blocks, and the offset array is used to identify the number of elements in each row of the weight tensor after weight rearrangement. For example, continue to participate Figure 6 , in the right Figure 6 After compressing the matrix shown in (C), we can get Figure 6 The weight array, row reordering array, subscript array, offset array, and block number array are shown in (D). Figure 6 In (D), the row reordering array represents the row information after reordering; the block number array represents that there are two dense blocks (i.e. Figure 6 The difference between two adjacent numbers in the offset array represents the number of elements in a row (e.g., 4-2 represents the number of elements in the 3rd row, and 20-18 represents the number of elements in the 5th row); the weight array records each element; the subscript array represents the column to which each element in the weight array belongs (e.g., d element is in the 6th column, n element is in the 2nd column). Figure 6 In (D), these five arrays can be used to express Figure 6 The weight matrix shown in (C). In the calculation process, these five arrays are part of the node input. During the calculation process, the elements in the dense block can be loaded from the global memory to the shared memory, so that memory reuse can be achieved, so that multiple threads can collaborate to complete the calculation and improve the calculation efficiency. Subsequently, the elements can be loaded from the shared memory into the register for calculation; and the elements in the sparse block can be loaded from the global memory into the register for calculation. After the calculation is completed, the calculation results can be written back according to the reverse process of row rearrangement and column rearrangement to obtain the desired results.

[0075] In this way, by rearranging the rows and columns of the weight tensor of the sparse nodes, the structuring degree of the non-zero elements in the weight tensor of the sparse nodes can be improved, which solves the problem of low execution efficiency of operators in the nodes caused by the irregularity of sparse distribution and improves the performance of the operators.

[0076] It is understandable that the order of execution of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the various embodiments described above can be combined according to actual conditions, and the combined solutions are still within the scope of protection of the present application.

[0077] Based on the method in the above embodiment, an embodiment of the present application provides a machine learning model compilation optimization device.

[0078] For example, Figure 7 A schematic diagram of the structure of a machine learning model compilation optimization device provided in an embodiment of the present application is shown. Figure 7 As shown, the machine learning model compilation optimization device 700 includes: a node identification module 701, a scheme determination module 702, a node processing module 703 and a code generation module 704. Among them, the node identification module 701 is used to identify the nodes in the calculation graph of the machine learning model to identify the sparse nodes in the calculation graph, wherein the proportion of 0 elements in the weight tensor of the sparse node is greater than the preset threshold. The scheme determination module 702 is used to determine the optimal sparse implementation scheme of the sparse node, and the optimal sparse implementation scheme includes: one or more of the memory layout of the input and output of the sparse node, the configuration parameters of the sparse node and the sparse format of the weight tensor of the sparse node; the node processing module 703 is used to generate a first node based on the optimal sparse implementation scheme when the performance data of the sparse node under the optimal sparse implementation scheme is better than the target performance data, and replace the sparse node with the first node in the calculation graph, wherein the target performance data is the performance data of the sparse node under the original sparse implementation scheme; the code generation module 704 is used to generate code for the calculation graph to obtain the execution code of the calculation graph.

[0079] In some embodiments, when determining the optimal sparse implementation scheme for a sparse node, the scheme determination module 702 is specifically used to: execute the execution code of the sparse node under each sparse implementation scheme to obtain at least one performance data, wherein one performance data is related to a sparse implementation scheme; when the best performance data in at least one performance data is better than the target performance data, determine the sparse implementation scheme related to the best performance data as the optimal sparse implementation scheme; when the best performance data in at least one performance data is not better than the target performance data, determine the original sparse implementation scheme as the optimal sparse implementation scheme.

[0080] In some embodiments, after the node processing module 703 replaces the sparse node with the first node in the computation graph, it is also used to: add a first layout conversion node before the first node and add a second layout conversion node after the first node when the memory layouts in the optimal sparse implementation scheme and the original sparse implementation scheme are different, wherein the first layout conversion node is used to perform memory layout conversion on the input of the first node, and the second layout conversion node is used to perform memory layout conversion on the output of the first node. In some embodiments, the computation graph includes a second node, and the second node is a newly generated node. At this time, before the code generation module 704 generates code for the computation graph, the node processing module 703 is also used to: when there is at least one node between the first node and the second node that has not undergone memory layout adjustment, if the node in at least one node is a node that is not sensitive to memory layout, then adjust the memory layout of the input and output of the node in at least one node; based on the adjusted memory layout, generate a new node and replace the node in at least one node.

[0081] In some embodiments, before the code generation module 704 generates code for the computation graph, the node processing module 703 is further used to: when two adjacent nodes in the computation graph are both newly generated nodes, if the memory layout of the output of the first node of the two adjacent nodes is the same as the memory layout of the input of the second node, then delete the node between the two adjacent nodes.

[0082] In some embodiments, the node processing module 703 is used to: rearrange the weight tensor of the sparse node by rows and columns; compress the rearranged weight tensor to obtain a sparse format of the weight tensor, which sparse format includes: a weight array, a row rearrangement array, a subscript array, an offset array and a block quantity array, wherein the weight array is used to record the elements in the rearranged weight tensor, the row rearrangement array is used to record the row information of the rearranged weight tensor, the subscript array is used to record the column to which each element recorded in the weight array belongs, the block quantity array is used to record the number of dense blocks in the rearranged weight tensor, the elements in the dense blocks are used to load from the global memory to the shared memory for calculation during the calculation process, and the offset array is used to identify the number of elements in each row of the rearranged weight tensor.

[0083] In some embodiments, Figure 7The node identification module 701, solution determination module 702, node processing module 703 and code generation module 704 shown in the figure can be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the node identification module 701 is introduced below by taking the node identification module 701 as an example. Similarly, the implementation of the layout adjustment module 702, the solution determination module 702, the node processing module 703 and the code generation module 704 can all refer to the implementation of the node identification module 701.

[0084] As an example of a software functional unit, the node identification module 701 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the node identification module 701 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0085] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0086] As an example of a hardware functional unit, the node identification module 701 may include at least one computing device, such as a server, etc. Alternatively, the node identification module 701 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0087] The multiple computing devices included in the node identification module 701 can be distributed in the same region or in different regions. The multiple computing devices included in the node identification module 701 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the node identification module 701 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0088] It should be noted that, in other embodiments, the node identification module 701 can be used to execute any step in the machine learning model compilation optimization method described in the above embodiments, and one or more of the solution determination module 702, the node processing module 703 and the code generation module 704 can be used to execute any step in the machine learning model compilation optimization method described in the above embodiments. The steps that the node identification module 701, the solution determination module 702, the node processing module 703 and the code generation module 704 are responsible for implementing can be specified as needed, and the node identification module 701, the solution determination module 702, the node processing module 703 and the code generation module 704 respectively implement different steps in the machine learning model compilation optimization method described in the above embodiments to achieve Figure 7 The entire functionality of the machine learning model compilation optimization device 700 is shown.

[0089] The present application also provides a computing device 800. Figure 8As shown, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate through the bus 802. The computing device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0090] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 804 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 804 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 806, the processor 804, and the communication interface 808).

[0091] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0092] The memory 806 may include a volatile memory, such as a random access memory (RAM). The processor 804 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0093] The memory 806 stores executable program codes, and the processor 804 executes the executable program codes to respectively implement the aforementioned Figure 7 The functions of the node identification module 701, the solution determination module 702, the node processing module 703 and the code generation module 704 shown in the embodiment are implemented to realize the machine learning model compilation optimization method described in the above embodiment. That is, the memory 806 stores instructions for executing the machine learning model compilation optimization method described in the above embodiment.

[0094] Alternatively, the memory 806 stores executable codes, and the processor 804 executes the executable codes to respectively implement the aforementioned Figure 7 The functions of the machine learning model compilation optimization device 700 shown in the embodiment are implemented to realize the machine learning model compilation optimization method described in the above embodiment. That is, the memory 806 stores instructions for executing the machine learning model compilation optimization method described in the above embodiment.

[0095] The communication interface 803 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or a communication network.

[0096] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0097] like Fig. 9 As shown, the computing device cluster includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing the machine learning model compilation optimization method described in the above embodiment.

[0098] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also respectively store some instructions for executing the machine learning model compilation optimization method described in the above embodiment. In other words, the combination of one or more computing devices 800 can jointly execute the instructions for executing the machine learning model compilation optimization method described in the above embodiment.

[0099] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster may store different instructions, which are respectively used to execute the aforementioned Figure 7 The machine learning model compilation and optimization device 700 shown in FIG. 8 is a partial function. That is, the instructions stored in the memory 806 in the different computing devices 800 can implement the functions of one or more modules in the node identification module 701, the solution determination module 702, the node processing module 703 and the code generation module 704.

[0100] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.10 A possible implementation is shown. Fig.10As shown, two computing devices 800A and 800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for executing the functions of the node identification module 701 and the solution determination module 702. At the same time, the memory 806 in the computing device 800B stores instructions for executing the functions of the node processing module 703 and the code generation module 704.

[0101] It should be understood that Fig.10 The functions of the computing device 800A shown in FIG. 8 may also be completed by multiple computing devices 800. Similarly, the functions of the computing device 800B may also be completed by multiple computing devices 800.

[0102] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 9 and Fig.10 The connection mode of the computing device cluster is different in that the memory 806 in one or more computing devices 800 in the computing device cluster may store the same instructions for executing the method in the above embodiment.

[0103] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also respectively store some instructions for executing the aforementioned machine learning model compilation optimization method. In other words, the combination of one or more computing devices 800 can jointly execute instructions for executing the aforementioned machine learning model compilation optimization method.

[0104] Based on the method in the above embodiment, the embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computing device cluster including at least one computing device, the computing device cluster executes the method described in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0105] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product including instructions. When the computer program product is run on a computing device cluster including at least one computing device, the computing device cluster executes the method in the above embodiment.

[0106] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0107] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0109] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A machine learning model compilation optimization method, characterized in that: include: Identify nodes in a computation graph of a machine learning model to identify sparse nodes in the computation graph, wherein a proportion of zero elements in a weight tensor of the sparse nodes is greater than a preset threshold; Determine an optimal sparse implementation scheme for the sparse node, wherein the optimal sparse implementation scheme includes: one or more of a memory layout of an input and an output of the sparse node, a configuration parameter of the sparse node, and a sparse format of a weight tensor of the sparse node; If the performance data of the sparse node under the optimal sparse implementation scheme is better than the target performance data, a first node is generated based on the optimal sparse implementation scheme, and the sparse node is replaced with the first node in the computation graph, wherein the target performance data is the performance data of the sparse node under the original sparse implementation scheme; Code generation is performed on the computation graph to obtain execution code of the computation graph.

2. The method according to claim 1, characterized in that The determining of the optimal sparse implementation scheme of the sparse nodes comprises: Execute the execution code of the sparse node under each sparse implementation scheme to obtain at least one performance data, wherein one performance data is related to one sparse implementation scheme; In a case where the best performance data among the at least one performance data is better than the target performance data, determining a sparse implementation scheme associated with the best performance data as the best sparse implementation scheme; In a case where the best performance data among the at least one performance data is not better than the target performance data, the original sparse implementation scheme is determined to be the best sparse implementation scheme.

3. The method according to claim 1 or 2, characterized in that: After replacing the sparse node with the first node in the computation graph, the method further includes: When the memory layouts in the optimal sparse implementation scheme and the original sparse implementation scheme are different, a first layout conversion node is added before the first node, and a second layout conversion node is added after the first node, wherein the first layout conversion node is used to perform a memory layout conversion on the input of the first node, and the second layout conversion node is used to perform a memory layout conversion on the output of the first node.

4. The method according to any one of claims 1 to 3, characterized in that: The computation graph includes a second node, where the second node is a newly generated node; Before code generation is performed on the computation graph, the following steps are also included: In the case where there is at least one node between the first node and the second node on which the memory layout is not adjusted, if a node in the at least one node is a node insensitive to the memory layout, adjusting the memory layout of the input and output of the node in the at least one node; Based on the adjusted memory layout, a new node is generated and a node in the at least one node is replaced.

5. The method according to any one of claims 1 to 4, characterized in that: Before code generation is performed on the computation graph, the following steps are also included: In the case where two adjacent nodes in the computation graph are both newly generated nodes, if the memory layout of the output of a first node of the two adjacent nodes is the same as the memory layout of the input of a second node, the node between the two adjacent nodes is deleted.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Rearranging rows and columns of the weight tensor of the sparse node; The rearranged weight tensor is compressed and stored in a preset sparse format, wherein the sparse format includes: a weight array, a row rearrangement array, a subscript array, an offset array, and a block quantity array, wherein the weight array is used to record the elements in the rearranged weight tensor, the row rearrangement array is used to record the row information of the rearranged weight tensor, the subscript array is used to record the column to which each element recorded in the weight array belongs, the block quantity array is used to record the number of dense blocks in the rearranged weight tensor, the elements in the dense blocks are used to load from the global memory to the shared memory for calculation during the calculation process, and the offset array is used to identify the number of elements in each row of the rearranged weight tensor.

7. A machine learning model compilation and optimization device, characterized in that: include: A node identification module, used to identify nodes in a computational graph of a machine learning model to identify sparse nodes in the computational graph, wherein the proportion of zero elements in the weight tensor of the sparse nodes is greater than a preset threshold; A scheme determination module is used to determine an optimal sparse implementation scheme for the sparse node, wherein the optimal sparse implementation scheme includes: one or more of the memory layout of the input and output of the sparse node, the configuration parameters of the sparse node, and the sparse format of the weight tensor of the sparse node; a node processing module, configured to generate a first node based on the optimal sparse implementation scheme, and replace the sparse node with the first node in the computation graph when the performance data of the sparse node under the optimal sparse implementation scheme is better than the target performance data, wherein the target performance data is the performance data of the sparse node under the original sparse implementation scheme; The code generation module is used to generate code for the calculation graph to obtain the execution code of the calculation graph.

8. The device according to claim 7, characterized in that When determining the optimal sparse implementation scheme of the sparse node, the scheme determination module is specifically used to: Execute the execution code of the sparse node under each sparse implementation scheme to obtain at least one performance data, wherein one performance data is related to one sparse implementation scheme; In a case where the best performance data among the at least one performance data is better than the target performance data, determining a sparse implementation scheme associated with the best performance data as the best sparse implementation scheme; In a case where the best performance data among the at least one performance data is not better than the target performance data, the original sparse implementation scheme is determined to be the best sparse implementation scheme.

9. The device according to claim 7 or 8, characterized in that After replacing the sparse node with the first node in the computation graph, the node processing module is further configured to: When the memory layouts in the optimal sparse implementation scheme and the original sparse implementation scheme are different, a first layout conversion node is added before the first node, and a second layout conversion node is added after the first node, wherein the first layout conversion node is used to perform a memory layout conversion on the input of the first node, and the second layout conversion node is used to perform a memory layout conversion on the output of the first node.

10. The device according to any one of claims 7 to 9, characterized in that: The computation graph includes a second node, where the second node is a newly generated node; Before the code generation module generates code for the computation graph, the node processing module is further used to: In the case where there is at least one node between the first node and the second node on which the memory layout is not adjusted, if a node in the at least one node is a node insensitive to the memory layout, adjusting the memory layout of the input and output of the node in the at least one node; Based on the adjusted memory layout, a new node is generated and a node in the at least one node is replaced.

11. The device according to any one of claims 7 to 10, characterized in that: Before the code generation module generates code for the computation graph, the node processing module is further configured to: In the case where two adjacent nodes in the computation graph are both newly generated nodes, if the memory layout of the output of a first node of the two adjacent nodes is the same as the memory layout of the input of a second node, the node between the two adjacent nodes is deleted.

12. The device according to any one of claims 7 to 11, characterized in that: The node processing module is further used for: Rearranging rows and columns of the weight tensor of the sparse node; The rearranged weight tensor is compressed and stored in a preset sparse format, wherein the sparse format includes: a weight array, a row rearrangement array, a subscript array, an offset array, and a block quantity array, wherein the weight array is used to record the elements in the rearranged weight tensor, the row rearrangement array is used to record the row information of the rearranged weight tensor, the subscript array is used to record the column to which each element recorded in the weight array belongs, the block quantity array is used to record the number of dense blocks in the rearranged weight tensor, the elements in the dense blocks are used to load from the global memory to the shared memory for calculation during the calculation process, and the offset array is used to identify the number of elements in each row of the rearranged weight tensor.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-6.

14. A computer-readable storage medium storing a computer program, which, when executed on a computing device cluster comprising at least one computing device, enables the computing device cluster to execute the method according to any one of claims 1 to 6.

15. A computer program product, characterized in that When the computer program product is run on a computing device cluster including at least one computing device, the computing device cluster is enabled to execute the method according to any one of claims 1 to 6.

Images