Operator fusion method, computing device, computing apparatus, and readable storage medium

Abstract

The present disclosure provides an operator fusion method, a computing device, a computing apparatus and a readable storage medium. The operator fusion method is applied to a neural network comprising a plurality of operators, comprising: setting a template fusion mode of at least one linear structure based on a category label of the operators; and performing operator fusion on the plurality of operators in the neural network according to the category label of each of the plurality of operators in the neural network and in the template fusion mode of the at least one linear structure, so as to fuse one or more operators in the neural network that meet the template fusion mode of the at least one linear structure.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to an operator fusion method, computing device, computing equipment, and readable storage medium applied to a neural network comprising multiple operators. Background Technology

[0002] To improve the computational efficiency of neural networks, multiple operators that meet certain conditions or rules are typically fused before the computation process begins, forming a fused operator. The fused operator, or a single operator that cannot be fused, can be called a fusion operator or a fusion layer. The computation process of a neural network is performed layer-by-layer, using fusion layers as units. Therefore, operator fusion is a crucial process in neural network graph optimization.

[0003] Generally, operator fusion involves two steps: first, defining the fusion pattern, i.e., defining the target to be fused; second, determining the fusion algorithm to match the fusion patterns appearing in the neural network and to perform fusion without interference. Related operator fusion schemes employ fixed fusion patterns, and there is a one-to-one correspondence between fusion patterns and fusion algorithms; that is, one fusion pattern corresponds to one fusion algorithm. This means that if the fusion pattern changes, the fusion algorithm also needs to be changed to support it, thus limiting the expansion and customization of the fusion pattern. Summary of the Invention

[0004] Some embodiments of this disclosure provide an operator fusion method, computing device, computing equipment, and readable storage medium for providing efficient and scalable operator fusion schemes for various types of neural network structures.

[0005] According to one aspect of this disclosure, an operator fusion method is provided, applied to a neural network comprising multiple operators. The operator fusion method includes: setting at least one linear structure template fusion pattern based on the category labels of the operators; and fusing the multiple operators in the neural network according to the template fusion pattern of at least one linear structure based on the respective category labels of the multiple operators in the neural network, so as to fuse one or more operators in the neural network that conform to the template fusion pattern of at least one linear structure.

[0006] The operator fusion method according to some embodiments of this disclosure further includes: classifying operators based on their functions and / or the computing architecture characteristics of the hardware platform, and assigning category labels to different categories of operators.

[0007] According to some embodiments of this disclosure, the category labels include non-template categories and template categories, wherein the template categories include one or more of the following: matrix category, normalization category, pooling category, data rearrangement category, data reduction category, regression function category, and loss function category.

[0008] According to some embodiments of this disclosure, operators belonging to the non-template category do not consume register resources or synchronization memory resources during the computation of the neural network.

[0009] According to some embodiments of this disclosure, at least one linear structure template fusion pattern consists of one or more linearly connected template categories, wherein setting at least one linear structure template fusion pattern based on the category label of the operator includes: setting at least one linear structure template fusion pattern based on the category label of the template category and ignoring the category labels of non-template categories.

[0010] The operator fusion method according to some embodiments of the present disclosure further includes: generating a directed acyclic graph based on the network structure of a neural network, wherein the directed acyclic graph includes operators and connections between operators, and the connections between operators represent the data dependencies and data flow between operators, wherein fusing multiple operators in the neural network according to at least one linear structure template fusion pattern includes: traversing the operators in the directed acyclic graph of the neural network according to one or more template categories linearly connected in at least one linear structure template fusion pattern based on the category labels of each of the multiple operators in the neural network, matching one or more operators to the corresponding linear structure template fusion pattern respectively, and fusing multiple operators in the neural network.

[0011] According to some embodiments of this disclosure, setting at least one linear structure template fusion mode based on operator category labels includes: a first template fusion mode and a second template fusion mode setting a linear structure based on operator category labels, wherein, when the first template fusion mode and the second template fusion mode have the same part of template categories and the first template fusion mode includes other template categories in addition to the same part of template categories, in the following case, multiple operators in the neural network are first fused according to the first template fusion mode, and then multiple operators in the neural network are fused according to the second template fusion mode.

[0012] According to some embodiments of this disclosure, setting a template fusion mode for at least one linear structure based on the category label of an operator includes: setting a template fusion mode for at least one linear structure using a configurable file.

[0013] The operator fusion method according to some embodiments of the present disclosure further includes: setting a fusion mode for subgraph structures on an operator-by-operator basis; and generating a directed acyclic graph based on a neural network structure, wherein the directed acyclic graph includes operators and connections between operators, and the connections between operators represent the data dependencies and data flow directions between operators.

[0014] According to some embodiments of this disclosure, before performing operator fusion on multiple operators in a neural network according to a template fusion pattern of at least one linear structure, the method further includes: performing operator fusion on operators in a directed acyclic graph of the neural network according to a fusion pattern of a subgraph structure to obtain a graph representation after subgraph structure operator fusion.

[0015] According to some embodiments of this disclosure, operator fusion of multiple operators in a neural network according to a template fusion pattern of at least one linear structure includes: according to the category labels of the multiple operators in the neural network, traversing the operators in the graph representation after subgraph structure operator fusion according to a template fusion pattern of at least one linear structure, matching one or more operators to the corresponding template fusion pattern of the linear structure, and performing operator fusion of the operators in the graph representation.

[0016] According to some embodiments of this disclosure, operators belonging to the non-template category include: activation functions, linear rectification functions, absolute value functions, and addition functions; operators belonging to the matrix category include: forward convolution operators, backward data convolution operators, backward filter convolution operators, and matrix multiplication operators; operators belonging to the normalization category include: batch normalization operators and layer normalization operators; operators belonging to the pooling category include: max pooling layer operators, average pooling layer operators, and global average pooling layer operators; operators belonging to the data rearrangement category include: concatenation operators and transformation operators; operators belonging to the data reduction category include: maximum value functions, minimum value functions, and average value functions; operators belonging to the regression function category include: regression functions for samples and regression functions for channels; and operators belonging to the loss function category include: mean squared error functions and cross-entropy functions.

[0017] According to another aspect of this disclosure, a computing apparatus is provided for operator fusion of a neural network, wherein the neural network includes a plurality of operators. The computing apparatus includes: a fusion mode configuration unit configured to: set at least one linear structure template fusion mode based on the category labels of the operators; and a fusion unit configured to perform operator fusion on the plurality of operators in the neural network according to the template fusion mode of at least one linear structure, based on the category labels of the respective operators in the neural network, to fuse one or more operators in the neural network that conform to the template fusion mode of at least one linear structure.

[0018] The computing device according to some embodiments of the present disclosure further includes a classification unit configured to: classify operators based on the function of the operators and / or the computing architecture characteristics of the hardware platform, and assign category labels to different categories of operators, wherein the category labels include non-template categories and template categories, and the template categories include one or more of the following: matrix category, normalization category, pooling category, data rearrangement category, data reduction category, regression function category, and loss function category.

[0019] According to some embodiments of this disclosure, operators belonging to the non-template category do not consume register resources or synchronization memory resources during the computation of the neural network.

[0020] According to some embodiments of this disclosure, at least one linear structure template fusion mode consists of one or more linearly connected template categories. Setting at least one linear structure template fusion mode based on the category label of the operator includes: setting at least one linear structure template fusion mode based on the category label of the template category and ignoring the category labels of non-template categories.

[0021] The computing device according to some embodiments of the present disclosure further includes a generation unit configured to generate a directed acyclic graph based on a neural network structure, wherein the directed acyclic graph includes operators and connections between operators, and the connections between operators represent data dependencies and data flow between operators.

[0022] According to some embodiments of this disclosure, the fusion unit performs operator fusion on multiple operators in a neural network according to a template fusion pattern of at least one linear structure, including: traversing the operators in the directed acyclic graph of the neural network according to the category labels of the multiple operators in the neural network and the template fusion pattern of at least one linear structure, matching one or more operators to the corresponding template fusion pattern of the linear structure, and performing operator fusion on the multiple operators in the neural network.

[0023] According to some embodiments of this disclosure, the fusion mode configuration unit sets at least one template fusion mode of a linear structure based on the category labels of operators, including: setting a first template fusion mode and a second template fusion mode of a linear structure based on the category labels of operators. In the case that the first template fusion mode and the second template fusion mode have the same part of the template category and the first template fusion mode includes other template categories in addition to the same part of the template category, the fusion unit first performs operator fusion on multiple operators in the neural network according to the first template fusion mode, and then performs operator fusion on multiple operators in the neural network according to the second template fusion mode.

[0024] According to some embodiments of this disclosure, the fusion mode configuration unit sets at least one linear structure template fusion mode based on the category label of the operator, including: setting at least one linear structure template fusion mode using a configurable file.

[0025] According to some embodiments of this disclosure, the fusion mode configuration unit is further configured to: set a fusion mode for the subgraph structure on a per-operator basis. The computing device according to some embodiments of this disclosure further includes a generation unit configured to: generate a directed acyclic graph based on a neural network structure, wherein the directed acyclic graph includes operators and connections between operators, and the connections between operators represent the data dependencies and data flow directions between operators.

[0026] According to some embodiments of this disclosure, before fusing multiple operators in a neural network according to at least one linear structure template fusion pattern, the fusion unit is further configured to: fuse operators in a directed acyclic graph of the neural network according to a subgraph structure fusion pattern to obtain a graph representation after subgraph structure operator fusion. The fusion unit fusing multiple operators in the neural network according to at least one linear structure template fusion pattern includes: traversing the operators in the graph representation after subgraph structure operator fusion according to at least one linear structure template fusion pattern based on the respective category labels of the multiple operators in the neural network, matching one or more operators to the corresponding linear structure template fusion pattern, and fusing the operators in the graph representation.

[0027] According to some embodiments of the computing apparatus disclosed herein, operators belonging to the non-template category include: activation functions, linear rectification functions, absolute value functions, and addition functions; operators belonging to the matrix category include: forward convolution operators, backward data convolution operators, backward filter convolution operators, and matrix multiplication operators; operators belonging to the normalization category include: batch normalization operators and layer normalization operators; operators belonging to the pooling category include: max pooling layer operators, average pooling layer operators, and global average pooling layer operators; operators belonging to the data rearrangement category include: concatenation operators and transformation operators; operators belonging to the data reduction category include: maximum value functions, minimum value functions, and average value functions; operators belonging to the regression function category include: regression functions for samples and regression functions for channels; and operators belonging to the loss function category include: mean squared error functions and cross-entropy functions.

[0028] According to another aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable code that, when executed by the processor, performs the operator fusion method as described above.

[0029] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, having instructions stored thereon that, when executed by a processor, implement the operator fusion method as described above.

[0030] By utilizing the operator fusion method, computing device, computing equipment, and storage medium provided in some embodiments of this disclosure, it is possible to set a template fusion mode for linear structures based on the category labels of operators, and to provide an operator fusion algorithm that is universally applicable to the template fusion mode for linear structures. This enables the designed template fusion mode to have universality and scalability and to be fused on a category-by-category basis, thereby providing an efficient operator fusion process for neural networks, which is beneficial for graph optimization of neural networks and improves the computational efficiency of neural networks on hardware platforms. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1A A schematic diagram of the neural network computation process is shown;

[0033] Figure 1B A schematic diagram of neural network operator fusion in related technologies is shown;

[0034] Figure 2 A flowchart illustrating an operator fusion method according to some embodiments of the present disclosure is shown;

[0035] Figure 3 A schematic diagram of operator classification according to some embodiments of the present disclosure is shown;

[0036] Figure 4 A schematic diagram of a template fusion mode according to some embodiments of the present disclosure is shown;

[0037] Figure 5A A schematic diagram of a directed acyclic graph of a neural network is shown;

[0038] Figure 5B It shows the Figure 5A The diagram shows the operation fusion of a directed acyclic graph.

[0039] Figure 5C The fused directed acyclic graph is shown.

[0040] Figure 6 A schematic diagram illustrating the fusion priority of template fusion modes according to some embodiments of the present disclosure is shown;

[0041] Figure 7 A schematic block diagram of an operator fusion scheme according to some embodiments of the present disclosure is shown;

[0042] Figures 8A-8B A schematic flowchart of an operator fusion scheme according to some embodiments of the present disclosure is shown;

[0043] Figure 9 A schematic block diagram of a computing device according to some embodiments of the present disclosure is shown;

[0044] Figure 10 A schematic block diagram of a computing device according to some embodiments of the present disclosure is shown;

[0045] Figure 11 A schematic diagram of the architecture of an exemplary computing device according to some embodiments of the present disclosure is shown;

[0046] Figure 12 A schematic diagram of a computer-readable storage medium according to some embodiments of the present disclosure is shown. Detailed Implementation

[0047] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0048] Furthermore, as shown in this disclosure and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "comprising" or "including" and similar terms mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, without excluding other elements or objects. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0049] This disclosure uses flowcharts to illustrate the steps of a method according to embodiments of this disclosure. It should be understood that the preceding or following steps are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes.

[0050] It is understood that the technical terms and nouns used in this article have meanings known to those skilled in the art.

[0051] Artificial Neural Networks (ANNs), or simply neural networks, are mathematical models that mimic the behavioral characteristics of animal neural networks to perform distributed parallel information processing. These networks rely on the complexity of the system, adjusting the connections between a large number of internal nodes to process information. Regardless of the type, neural networks share common characteristics such as massively parallel processing, distributed storage, flexible topology, high redundancy, and nonlinear operations. They possess capabilities in areas such as processing speed, associative ability, adaptability, fault tolerance, and self-organization. These characteristics and capabilities form the technological foundation for neural networks to simulate intelligent activities and have found important applications in various technological fields. For example, neural networks can be used in applications such as data compression, image processing, video coding, and signal processing.

[0052] To improve the computational efficiency of neural networks, multiple adjacent operators that meet certain conditions or rules are typically fused together to form a fused operator. A fused operator, or a single operator that cannot be fused, can be represented as a fusion layer, or simply a layer. The computation process of a neural network is usually performed layer-by-layer, using fusion layers as units. Generally, the output of the previous layer (or the first few layers) serves as the input of the next layer (or the next few layers), thus creating data dependencies between fusion layers. (Illustratively,) Figure 1A A schematic diagram of the neural network computation process is shown, such as... Figure 1A As shown, computational units used in neural networks are generally subdivided into matrix multiplication units, vector operation units, and scalar operation units, each used for different computational tasks. During computation, shared memory is used to synchronize data and ensure the execution order of fusion layers with data dependencies. Shared memory can be based on internal or external memory; no specific restrictions are imposed here.

[0053] Therefore, in the above Figure 1A The efficient and flexible implementation of the operator fusion process preceding the computational process shown has a significant impact on the computational efficiency of the neural network. Furthermore, for different and complex internal structures of neural networks, it is necessary to quickly adjust and define fusion modes and adapt corresponding fusion algorithms to achieve operator fusion within the neural network.

[0054] In related technologies, most operator fusion schemes adopt a fixed pattern or fixed rule fusion pattern (i.e., define the target operator to be fused). Figure 1B A schematic diagram of neural network operator fusion in related technologies is shown, illustrating four fixed-type fusion modes 1-4. Based on Figure 1B The four fusion modes defined in the algorithm require adaptation to four corresponding fusion algorithms 1-4. This is done to find operators in the neural network structure that conform to fusion modes 1-4 and fuse them to obtain a fused network structure diagram. Subsequent computations are then performed based on this fused network structure diagram. As an example, in... Figure 1B In the fusion mode 1 shown, a linear fusion target (including operator 1, operator 2, and operator 3) is defined. During operator fusion based on this fusion mode 1, a fusion algorithm 1 needs to be designed to match the operator 1-operator 2-operator 3 structure within the numerous operators included in the neural network, and then fuse them. In related technologies, subgraph matching is commonly used to implement the fusion algorithm; that is, a subgraph corresponding to the fusion mode is designed, and the operator structure graph of the neural network is traversed to find the corresponding subgraph structure. It is understandable that... Figure 1B The fusion modes shown are all performed on an operator-by-operator basis.

[0055] Typically, for neural networks with complex functional structures, it may be necessary to define a dozen or even dozens of fusion modes, which makes the operator fusion step require a significant amount of computational resources and debugging costs. Figure 1B It can be seen that the defined fusion mode is statically related to the fusion algorithm, and the fusion algorithm and fusion mode are directly related; for example, one fusion mode corresponds to one fusion algorithm. If the fusion mode changes, the fusion algorithm also needs to be changed to support it. Therefore, the expansion and customization of the fusion mode are limited, and it lacks universal applicability. Furthermore, the fusion algorithm also needs to be constantly adjusted accordingly. In addition, an efficient fusion mode also needs to be determined by considering the architectural characteristics of the hardware platform running the neural network. Once the hardware platform is switched, it means that the fusion mode will also change, and the change of the fusion mode will lead to the need to redefine and compile the corresponding fusion algorithm.

[0056] This disclosure is applied to the field of high-efficiency computing for neural network inference or training, and solves the operator fusion problem in the graph optimization process of neural networks. With the goal of fusion mode universality and scalability, it provides an operator fusion strategy with variable fusion mode and fixed fusion algorithm.

[0057] Specifically, some embodiments of this disclosure provide an operator fusion method for fusing operators across various types of neural network structures. It designs a template fusion pattern for linear structures based on operator category labels, along with a corresponding operator fusion algorithm. This ensures the designed template fusion pattern is universal and scalable, and enables fusion on a category-by-category basis, thereby providing an efficient operator fusion process for neural networks. This is beneficial for graph optimization of neural networks and improves the computational efficiency of neural networks on hardware platforms. The implementation process of the algorithm fusion method according to some embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0058] Figure 2 A flowchart illustrating an operator fusion method according to some embodiments of the present disclosure is shown, such as... Figure 2 As shown, the operator fusion method includes steps S101 and S102. Specifically, in step S101, at least one linear structure template fusion mode is set based on the operator's category label. In embodiments according to this disclosure, a template-type fusion mode is provided, and it is defined for the operator's category label, compared to... Figure 1B The operator-based fusion pattern shown is more adaptive; any operator that conforms to the category can be fused, which significantly reduces the number of fusion patterns that need to be fused. Next, in step S102, based on the category labels of the multiple operators in the neural network, operator fusion is performed on the multiple operators in the neural network according to a template fusion pattern with at least one linear structure, to fuse one or more operators in the neural network that conform to the template fusion pattern with at least one linear structure. Next, we will first describe how to generate the template fusion pattern in step S101, and then describe the implementation process of the corresponding fusion algorithm, i.e., step S102.

[0059] The operator fusion method according to some embodiments of this disclosure further includes: classifying operators based on their functions and / or the computing architecture characteristics of the hardware platform, and assigning category labels to different categories of operators.

[0060] To define template fusion patterns, it is necessary to pre-define operator classification rules to categorize the various operators that a neural network may include, dividing the numerous operators into several categories. Generally, operator classification needs to consider their functions and / or the architectural characteristics of the hardware platform. For example, different types of computational functions are usually executed by different computational units, such as matrix operations, vector operations, scalar operations, special function units (SFUs), etc. Multiple such operators can be fused together for pipelined parallelism. Conversely, multiple operators with similar computational functions can be fused together if they do not require additional register or synchronization resources, and executed serially within the computational unit, avoiding the time consumption of data transfer and memory operations.

[0061] As an example, Figure 3 The diagram illustrates operator classification according to some embodiments of the present disclosure. It will be understood that... Figure 3 The text only provides some examples of operator classification; neural network operators can also include... Figure 3 Operators and other categories not shown are not limited herein. Furthermore, in other embodiments according to this disclosure, other classification methods may be defined for operators in neural networks.

[0062] like Figure 3 As shown, according to some embodiments of this disclosure, the above category labels may include non-template categories and template categories. Operators belonging to the non-template category do not consume register resources or synchronization memory resources during the computation of the neural network. For operators belonging to the non-template category, during computation, a sample point is usually still a sample point after computation, and will not occupy additional computational resources. Therefore, in some embodiments of this disclosure, this non-template category operator is set separately so that operators of this category are not included in template matching. Figure 3As shown, operators belonging to the non-template category can be, for example, the Element Wise operator, which may include activation functions (Sigmoid, Swish), Rectified Linear Functions (ReLU), Absolute Value Functions (Abs), Addition Functions (Add), etc., and will not be listed here. These non-template category operators are common in neural network structures. Furthermore, it is understood that operators belonging to the non-template category can also be other operators besides the Element Wise operator, which will not be exemplified here. According to some embodiments of this disclosure, the above-described linear template fusion pattern consists of one or more linearly connected template categories. That is, the template fusion pattern according to embodiments of this disclosure only includes template category operators and does not include the above-described non-template category operators. Therefore, the template fusion pattern designed according to embodiments of this disclosure eliminates the influence of such operators that do not occupy additional computing resources on the fusion pattern, which helps to further reduce the complexity of the template fusion pattern and makes it more universally applicable.

[0063] Next, as Figure 3 As shown, the template categories according to embodiments of this disclosure may include one or more of the following: matrix category, normalize category, pooling category, data reordering category, data reduction category, regression function category (Softmax), and loss function category (Loss), etc., which will not be listed here one by one.

[0064] As examples, operators belonging to the matrix category can include: forward convolution, backward data convolution, backward filter convolution, and matrix multiplication (MatMul). Operators belonging to the normalization category can include: batch normalization and layer normalization. Operators belonging to the pooling category can include: max pooling, average pooling, and global average pooling. Operators belonging to the reorder category can include: concatenation, pruning, reshape, and slice. Operators belonging to the data reduction category (Reduce) can include: maximum value function (Max), minimum value function (Min), average value function (Average), and summation function (Sum). Operators belonging to the regression function category (Softmax) can include: softmax on sample regression function (Softmax on sample), softmax on channel regression function (Softmax on channel), etc. Operators belonging to the loss function category (Loss) can include: mean square error function (MSE), cross-entropy function, etc.

[0065] It is understood that, in the embodiments according to this disclosure, the operators of the neural network are first classified, and then template fusion patterns are defined by category, rather than by the operators themselves. This helps to reduce the number of fusion patterns required by the neural network (as will be shown in the description below). Furthermore, in the template fusion patterns defined by category above, the influence of non-template category operators that do not consume additional computing resources on the fusion patterns is eliminated; that is, such operators are not included in the defined template fusion patterns. This further reduces the complexity of the template fusion patterns and makes them more universally applicable.

[0066] As an example, Figure 4 A schematic diagram of template fusion patterns according to some embodiments of the present disclosure is shown. Figure 4 In the diagram on the left, four sets of example operator graphs are schematically shown. In the operator fusion method provided according to embodiments of this disclosure, these four sets of example operator graphs can all be characterized as follows: Figure 4 The template fusion pattern shown on the right side of the diagram represents a matrix operator connected to a normalization operator. In other words, given the template fusion pattern shown on the right, all four example operator graphs shown on the left side of the neural network conform to this pattern and are thus fused. Furthermore, it can be understood that this template fusion pattern can also generalize other... Figure 4 The other operator connection methods shown on the left, i.e., during operator fusion, all operators that satisfy the template fusion pattern will be fused to form a fused operator, i.e., a fusion layer. This fully demonstrates the advantage of the template fusion pattern provided in this disclosure in terms of universal applicability, i.e., it has the properties of a template, thereby reducing the number of fusion patterns required by the neural network. In contrast, in related technologies, if... Figure 4 The fusion of the four operator connection forms shown on the left requires defining four fusion modes and corresponding fusion algorithms.

[0067] The operator fusion method according to some embodiments of the present disclosure may further include: generating a directed acyclic graph based on the network structure of a neural network, wherein the directed acyclic graph includes operators and connections between operators, and the connections between operators represent the data dependencies and data flow directions between operators.

[0068] A neural network can be viewed as a directed acyclic graph (DAG) composed of many operators (also called computation nodes), where each node in the DAG corresponds to an operator in the neural network. It is understandable that the structure of a neural network can be abstracted as a DAG in any known way, which will not be described in detail here. Specifically, Figure 5A A schematic diagram of a directed acyclic graph (DAG) of a neural network is shown. For example... Figure 5A As shown, a directed acyclic graph (DAG) formed by a neural network comprising 13 operators is illustrated. The operators are connected by lines that characterize the data dependencies and data flow between them. For example, the output data of operator 1 flows to operator 2, the output data of operator 2 flows to operators 3 and 6, and so on. This establishes a data dependency between operator 1 and operator 2, and between operator 2 and operators 6 and 3. It is understood that... Figure 5A The network structure shown is merely illustrative, and the operator fusion method according to embodiments of this disclosure can be applied to various types of neural network structures.

[0069] According to embodiments of this disclosure, a template fusion pattern of a linear structure is defined according to the process described above (e.g., as...). Figure 4Following the pattern shown on the right (Matrix-Nomalize), operator fusion can be performed based on the directed acyclic graph of the neural network.

[0070] According to some embodiments of this disclosure, the template fusion mode of at least one linear structure is composed of one or more linearly connected template categories, wherein setting the template fusion mode of at least one linear structure based on the category label of the operator includes: setting the template fusion mode of at least one linear structure based on the category label of the template category and ignoring the category labels of non-template categories.

[0071] According to an embodiment of this disclosure, step S102, which involves fusing multiple operators in a neural network according to a template fusion pattern of a linear structure, includes: traversing the operators in the directed acyclic graph of the neural network according to the category labels of the multiple operators in the neural network and at least one template fusion pattern of a linear structure, matching one or more operators to the corresponding template fusion pattern of the linear structure, and fusing multiple operators in the neural network.

[0072] Specifically, Figure 5B It shows according to Figure 4 The template fusion mode shown is for Figure 5A The diagram illustrates operator fusion in a directed acyclic graph. In embodiments according to this disclosure, since the defined template fusion pattern is performed on a per-operator-category basis, fusion is also required based on the category labels of the multiple operators in the neural network during the operator fusion process. As an example and for ease of description, assume... Figure 5A The information for operators 1-13 shown is shown in Table 1 below:

[0073] Table 1

[0074]

[0075]

[0076] Referring to the information shown in Table 1 above, it can be seen that, in the case of Figure 5A In the directed acyclic graph shown, three fusion operators can be matched, meaning that according to the template fusion pattern Matrix-Nomalize, three operator fusions can be performed. In other words, in Figure 5B In the example, fusion operators A, B, and C all conform to the template fusion pattern Matrix-Nomalize, and are therefore fused. Figure 5C The fused graph representation is shown. It is understood that the operators listed in Table 1 are merely illustrative, used to describe the process of operator fusion of a directed acyclic graph according to a defined template fusion pattern.

[0077] In the operator fusion method provided according to embodiments of this disclosure, a template fusion pattern for a linear structure and a corresponding operator fusion algorithm can be defined based on the category labels of operators. This allows the designed template fusion pattern to be universal and scalable, and enables fusion on a category-by-category basis, thereby providing an efficient operator fusion process for neural networks. This operator fusion process can be represented as an operator fusion algorithm corresponding to the template fusion pattern.

[0078] According to some embodiments of this disclosure, the template fusion mode for setting a linear structure based on operator category labels includes: a first template fusion mode and a second template fusion mode for setting a linear structure based on operator category labels. In some implementations, not only one template fusion mode needs to be defined for the neural network, but multiple template fusion modes can also be defined. Where the first template fusion mode and the second template fusion mode have the same portion of template categories, and the first template fusion mode includes other template categories besides the same portion, the multiple operators in the neural network are first fused according to the first template fusion mode, and then the multiple operators in the neural network are fused according to the second template fusion mode.

[0079] Figure 6 A schematic diagram illustrating the fusion priority of template fusion modes according to some embodiments of this disclosure is shown, wherein template fusion mode 1, template fusion mode 2, and template fusion mode 3 are schematically shown. For ease of description, the case where the first template fusion mode and the second template fusion mode have the same portion of template categories, and the first template fusion mode includes other template categories besides the same portion of template categories, can also be represented as the first template fusion mode including the second template fusion mode. Reference Figure 6 This illustrates the inclusion relationship between template blending patterns. Specifically, template blending pattern 1 includes template blending pattern 2 and template blending pattern 3. For example, template blending pattern 1 and template blending pattern 2 share a common template category (denoted as "Matrix-Normalize"), and template blending pattern 1 also includes other template categories besides this common category (denoted as "Softmax"). Specifically, this common template category corresponds to template blending pattern 2, and template blending pattern 2 does not include other template categories.

[0080] In addition, continue to refer to Figure 6Template blending pattern 2 can also represent template blending pattern 3. That is, template blending pattern 2 and template blending pattern 3 share a common template category (denoted as "Matrix"), and template blending pattern 1 also includes other template categories (denoted as "Normalize"). Specifically, this common template category corresponds to template blending pattern 3, and template blending pattern 3 does not include other template categories. For... Figure 6 As shown, a fusion priority is set for template fusion patterns with inclusion relationships. Specifically, template fusion pattern 1 is matched and fused before template fusion patterns 2 and 3, and template fusion pattern 2 is fused before template fusion pattern 3. In other words, the more template categories included, the higher the priority. This priority setting allows as many operators as possible to be fused into a single fusion layer. Otherwise, without setting the above priority, when performing fusion matching in a directed acyclic graph based on a neural network, operators matching template fusion pattern 3 might be fused directly, failing to guarantee fusion matching for operators matching template fusion pattern 1.

[0081] According to some embodiments of this disclosure, setting a template fusion mode for a linear structure based on the category label of an operator may include: setting the template fusion mode for the linear structure using a configurable file. As an example, the configurable file may be in JSON or YAML format. That is, in the operator fusion method according to embodiments of this disclosure, the fusion mode can be defined based on a configuration file, for example, defining a template fusion mode as follows: Figure 6 The various template fusion modes shown can be dynamically and configurably implemented to adapt to different neural network structures and hardware computing platforms, increasing the flexibility of the operator fusion process. It is understood that the configurable file formats are not limited to the two listed above. In embodiments according to this disclosure, the template fusion modes are configurable, for example, configured via files. Multiple template fusion modes can form a set, which is input into the computing device via file configuration for use in the neural network operator fusion process.

[0082] In the template fusion mode of the linear structure described above, operators are connected in series, meaning that operator fusion can be performed within a single branch. In some applications, the network structure of the neural network may be more complex. Therefore, in addition to the template fusion mode of the linear structure, it may be necessary to define other types of more complex fusion modes, such as the subgraph structure fusion mode. This subgraph structure fusion mode requires operator fusion across multiple parallel branches to achieve higher computational efficiency.

[0083] The operator fusion method provided according to some embodiments of this disclosure may further include: setting a fusion mode for subgraph structures on a per-operator basis. Compared to the template fusion mode for linear structures set on a per-operator basis, the fusion mode for subgraph structures is defined on a per-operator basis. For example, the mode indicates that several operators with specific connection relationships need to be fused.

[0084] Similarly, the operator fusion process based on subgraph matching is also based on the directed acyclic graph of the neural network. For example, each fusion pattern of subgraph structure can be treated as a subgraph and described using the same Graph Intermediate Representation (Graph IR) as the neural network. Then, the operator fusion process based on the fusion pattern of subgraph structure is the subgraph matching process, that is, matching subgraphs that conform to the Graph IR form from the graph representation of the neural network (usually a DAG graph) and fusing operators that meet the matching conditions.

[0085] According to some embodiments of this disclosure, before fusing multiple operators in a neural network according to a template fusion pattern of a linear structure, the operator fusion method may further include: fusing operators in a directed acyclic graph of the neural network according to a fusion pattern of a subgraph structure to obtain a graph representation after subgraph structure operator fusion. In these embodiments, fusing multiple operators in a neural network according to at least one template fusion pattern of a linear structure includes: traversing the operators in the graph representation after subgraph structure operator fusion according to at least one template fusion pattern of a linear structure based on the respective category labels of the multiple operators in the neural network, matching one or more operators to the corresponding template fusion pattern of the linear structure, and fusing the multiple operators in the neural network. Wherein, operators belonging to non-template categories in the neural network are ignored during the operator fusion process.

[0086] Subgraph matching-based operator fusion and linear template fusion can be two independent fusion processes, yet they complement each other functionally. The former handles computational pattern matching for complex structures, while the latter handles computational pattern matching for linear structures. In the overall operator fusion process of a neural network, subgraph matching-based operator fusion can be executed first, followed by linear template fusion. Furthermore, the latter can be based on the fusion result of the former and can be further fused.

[0087] Figure 7 A schematic block diagram of an operator fusion scheme according to some embodiments of the present disclosure is shown. Figures 8A-8B The following is a schematic flowchart of an operator fusion scheme according to some embodiments of the present disclosure. Figure 7 , Figure 8A and Figure 8B An overall description is given of operator fusion schemes according to some embodiments of the present disclosure.

[0088] like Figure 7 As shown, the operator fusion scheme can be divided into two parts: defining the fusion mode and executing the fusion algorithm based on the defined fusion mode to obtain the neural network graph representation after operator fusion. Specifically, in the stage of defining the fusion mode, a configurable file (such as JSON or YAML format) can be used to set the template fusion mode for linear structures and the fusion mode for subgraph structures. This disclosure does not limit the number of fusion modes or the specific form of the modes.

[0089] During the fusion algorithm execution phase, for the directed acyclic graph (DAG) of the neural network, a subgraph-based operator fusion process is first performed according to the defined subgraph structure fusion pattern, such as through strict subgraph matching. After this step, a graph representation fused by subgraph structure operators is obtained. Next, based on the category labels of the operators, operator fusion is performed on the operators in the graph representation fused by subgraph structure operators according to one or more template categories linearly connected in the template fusion pattern of the linear structure. During the operator fusion process, operators belonging to non-template categories in the neural network are ignored, resulting in a graph representation fused by linear operators.

[0090] As an example, such as Figure 8A As shown, for the fusion pattern of the subgraph structure (including operators 6, 7, 8, and 9), strict subgraph matching is first performed in the DAG to find operators that conform to the pattern, and then operator fusion is performed to obtain the graph representation after subgraph structure operator fusion. Here, ellipse P represents the fused layer. Figure 8B The image shows the fusion layer P. The template category corresponding to this fusion layer P can be defined directly. For example, for operator fusion based on subgraph matching, a unique category label, such as category X, can be defined to correspond to... Figure 3 Operators such as Matrix and Normalize, as shown in the diagram, are distinguished for operator fusion in subsequent linear template matching. Then, as... Figure 8B As shown, for the graph representation after fusion by subgraph structure operators, the operator fusion process based on the template fusion mode of linear structure will continue. For example, operators that conform to the template will be fused, which are represented as fusion layer A, fusion layer B, fusion layer C, fusion layer D and fusion layer E respectively, and finally the graph representation after fusion by linear operators is obtained.

[0091] The operator fusion method using some embodiments of this disclosure can perform operator fusion for various types of neural network structures. It provides a template fusion mode for linear structures based on operator category labels, and a corresponding operator fusion algorithm, so that the designed template fusion mode has universality and scalability and can be fused on a unit basis by operator category. This provides an efficient operator fusion process for neural networks, which is beneficial for graph optimization of neural networks and improves the computational efficiency of neural networks on hardware platforms.

[0092] According to another aspect of this disclosure, a computing device is also provided for operator fusion of a neural network, wherein the neural network includes multiple operators. The computing device according to embodiments of this disclosure can be applied to the field of efficient computation for neural network inference or training, solving the operator fusion problem in the graph optimization process of neural networks. With the goal of versatility and scalability of the fusion mode, it provides an operator fusion strategy with a variable fusion mode and a fixed fusion algorithm. Specifically, the computing device according to some embodiments of this disclosure can perform operator fusion for various types of neural network structures, designing a template fusion mode for linear structures based on operator category labels, and a corresponding operator fusion algorithm. This makes the designed template fusion mode versatility and scalability, and allows fusion on a category-by-category basis, thereby providing an efficient operator fusion process for neural networks, which is beneficial for graph optimization of neural networks and improves the computational efficiency of neural networks on hardware platforms.

[0093] Figure 9 A schematic block diagram of a computing device according to some embodiments of the present disclosure is shown. Figure 9 As shown, the computing device 1000 according to an embodiment of the present disclosure includes: a fusion mode configuration unit 1010 and a fusion unit 1020.

[0094] Specifically, the fusion mode configuration unit 1010 can be configured to set at least one linear structure template fusion mode based on the category label of the operator. It is understood that the term "at least one" in this document can be understood as one or more, that is, one, two or more, without limitation. The number of template fusion modes will not be limited below, but can be similarly interpreted as one or more template fusion modes.

[0095] In embodiments according to this disclosure, a template-based fusion pattern is defined, and it is defined for the category labels of operators, compared to, for example... Figure 1B The operator-based fusion pattern shown is more adaptive, as any operator that fits the category can be fused, which can significantly reduce the number of fusion patterns that need to be designed.

[0096] The fusion unit 1020 can be configured to perform operator fusion on multiple operators in the neural network according to the category labels of each operator in the neural network and in accordance with the template fusion pattern of the linear structure, so as to fuse one or more operators in the neural network that conform to the template fusion pattern of the linear structure.

[0097] The computing device according to some embodiments of this disclosure further includes a classification unit 1030 configured to classify operators based on the functionality of the operators and / or the computing architecture characteristics of the hardware platform, and assign category labels to operators of different categories. It is understood that the term "and / or" herein refers to three scenarios: based on the functionality of the operator, based on the computing architecture characteristics of the hardware platform, or based on both the functionality of the operator and the computing architecture characteristics of the hardware platform. According to some embodiments of this disclosure, the category labels include non-template categories and template categories. The template categories include one or more of the following: matrix categories, normalization categories, pooling categories, data rearrangement categories, data reduction categories, regression function categories, and loss function categories.

[0098] To define template fusion patterns, the various operators required by the neural network must first be classified into several categories. Generally, operator classification needs to consider both their function and the architecture of the hardware platform. For example, different types of computational functions are usually executed by different computational units, such as matrix operations, vector operations, scalar operations, special function units, etc. Multiple such operators can be fused together for pipelined parallelism. Conversely, multiple operators with similar computational functions can be fused together if they do not require additional register or synchronization resources, and executed serially within the computational unit, avoiding the time consumption of data transfer and memory operations.

[0099] According to some embodiments of the computing apparatus disclosed herein, operators belonging to the non-template category include: activation functions, linear rectified functions, absolute value functions, and addition functions. Specifically, operators belonging to the non-template category do not consume register resources or synchronization memory resources during the computation of the neural network. For operators belonging to the non-template category, during the computation process, a sample point is still calculated as a single sample point, and no additional resources are occupied. Therefore, in some embodiments of this disclosure, operators of this category are set separately so that operators of this category are not included in template matching. Figure 3As shown, operators belonging to the non-template category can be, for example, the Element Wise operator. As examples, the Element Wise operator can include activation functions (Sigmoid, Swish), linear rectified functions (ReLU), absolute value functions (Abs), addition functions (Add), etc., which will not be listed here. These non-template category operators are common in neural network structures. For example, the template fusion pattern of the above linear structure consists of one or more template categories connected by straight lines. That is, the template fusion pattern according to the embodiments of this disclosure only includes template category operators and does not include the above non-template category operators. Therefore, the template fusion pattern designed according to the embodiments of this disclosure eliminates the influence of such operators that do not occupy additional computing resources on the fusion pattern, which helps to further reduce the complexity of the template fusion pattern and makes it more universally applicable.

[0100] According to some embodiments of this disclosure, operators belonging to the matrix category include: forward convolution operator, backward data convolution operator, backward filter convolution operator, and matrix multiplication operator; operators belonging to the normalization category include: batch normalization operator and layer normalization operator; operators belonging to the pooling category include: max pooling layer operator, average pooling layer operator, and global average pooling layer operator; operators belonging to the data rearrangement category include: concatenation operator and transformation operator; operators belonging to the data reduction category include: maximum value function, minimum value function, and average value function; operators belonging to the regression function category include: regression function for samples and regression function for channels; and operators belonging to the loss function category include: mean squared error function and cross-entropy function.

[0101] It is understood that, in the embodiments according to this disclosure, the operators of the neural network are first classified, and then template fusion patterns are defined by category, rather than by the operators themselves. This helps to reduce the number of fusion patterns required by the neural network (as will be shown below). Furthermore, in the template fusion patterns defined by category above, the influence of non-template category operators that do not consume additional computing resources on the fusion patterns is eliminated; that is, such operators are not included in the defined template fusion patterns. This helps to further reduce the complexity of the template fusion patterns and makes them more universally applicable.

[0102] According to some embodiments of this disclosure, at least one linear structure template fusion pattern consists of one or more linearly connected template categories. Setting the template fusion pattern of at least one linear structure based on the category labels of operators includes: setting the template fusion pattern of at least one linear structure based on the category labels of template categories and ignoring the category labels of non-template categories. As an example, Figure 4The diagram illustrates a template fusion mode according to some embodiments of the present disclosure. For specific structures, please refer to the above description. Figure 4 The description is as follows.

[0103] According to some embodiments of this disclosure, such as Figure 9 As shown, the computing device also includes a generation unit 1040, configured to generate a directed acyclic graph (DAG) based on a neural network structure. The DAG includes operators and connections between them, with the connections representing data dependencies and data flow between operators. An example of a DAG can be found in [reference needed]. Figure 5A This will not be repeated here.

[0104] According to some embodiments of this disclosure, the fusion unit 1020 performs operator fusion on multiple operators in a neural network according to a template fusion pattern of at least one linear structure, including: according to the category labels of each of the multiple operators in the neural network, traversing the operators in the directed acyclic graph of the neural network according to a template fusion pattern of at least one linear structure, matching one or more operators to the corresponding template fusion pattern of the linear structure, and performing operator fusion on the multiple operators in the neural network.

[0105] According to some embodiments of this disclosure, the fusion mode configuration unit 1010 sets at least one template fusion mode for a linear structure based on the category labels of operators, including: setting a first template fusion mode and a second template fusion mode for a linear structure based on the category labels of operators. In the case that the first template fusion mode and the second template fusion mode have the same part of the template category and the first template fusion mode includes other template categories in addition to the same part of the template category, the fusion unit 1020 first performs operator fusion on multiple operators in the neural network according to the first template fusion mode, and then performs operator fusion on multiple operators in the neural network according to the second template fusion mode.

[0106] In the operator fusion method provided according to embodiments of this disclosure, a template fusion pattern for linear structures and a corresponding operator fusion algorithm are defined based on the category labels of operators. This ensures that the designed template fusion pattern has universality and scalability and can be fused on a category-by-category basis, thereby providing an efficient operator fusion process for neural networks. This operator fusion process can be represented as an operator fusion algorithm corresponding to the template fusion pattern.

[0107] According to some embodiments of this disclosure, the template fusion mode for setting a linear structure based on operator category labels includes: a first template fusion mode and a second template fusion mode for setting a linear structure based on operator category labels. In some implementations, not only one template fusion mode needs to be defined for the neural network, but multiple template fusion modes can also be defined. Where the first template fusion mode and the second template fusion mode have the same portion of template categories, and the first template fusion mode includes other template categories besides the same portion, the multiple operators in the neural network are first fused according to the first template fusion mode, and then the multiple operators in the neural network are fused according to the second template fusion mode.

[0108] According to some embodiments of this disclosure, the fusion mode configuration unit 1010 sets a template fusion mode for a linear structure based on the category label of the operator, including setting a template fusion mode for the linear structure using a configurable file.

[0109] According to some embodiments of this disclosure, the fusion mode configuration unit 1010 is further configured to: set a fusion mode for the subgraph structure on a per-operator basis. The computing device according to some embodiments of this disclosure further includes a generation unit 1040, configured to: generate a directed acyclic graph based on a neural network structure, wherein the directed acyclic graph includes operators and connections between operators, and the connections between operators represent the data dependencies and data flow directions between operators.

[0110] According to some embodiments of this disclosure, before fusing multiple operators in a neural network according to at least one linear structure template fusion pattern, the fusion unit 1020 is further configured to: fuse operators in a directed acyclic graph of the neural network according to a subgraph structure fusion pattern to obtain a graph representation after subgraph structure operator fusion. The fusion unit 1020 fusing multiple operators in the neural network according to a linear structure template fusion pattern includes: traversing the operators in the graph representation after subgraph structure operator fusion according to at least one linear structure template fusion pattern based on the respective category labels of the multiple operators in the neural network, matching one or more operators to the corresponding linear structure template fusion pattern, and fusing the operators in the graph representation.

[0111] Regarding the specific implementation process of operator fusion using the computing device according to the embodiments of this disclosure, please refer to the above. Figures 2-8B The operator fusion method described according to some embodiments of this disclosure will not be repeated here. Computing devices utilizing embodiments of this disclosure can perform similar operator fusion processes and achieve similar technical effects.

[0112] According to another aspect of this disclosure, a computing device is also provided. Figure 10 A schematic block diagram of a computing device according to an embodiment of the present disclosure is shown.

[0113] like Figure 10 As shown, the computing device 2000 may include a processor 2010 and a memory 2020. According to an embodiment of this disclosure, the memory 2020 stores computer-readable code that, when executed by the processor 2010, can perform the operator fusion method as described above.

[0114] Processor 2010 can perform various actions and processes according to the program stored in memory 2020. Specifically, processor 2010 can be an integrated circuit with signal processing capabilities. The general-purpose processor can be a microprocessor or any conventional processor. For example, the processor here can refer to a computing device capable of performing neural network calculations.

[0115] Memory 2020 stores computer-executable instruction code that, when executed by processor 2010, is used to implement an operator fusion method according to embodiments of the present disclosure. Memory 2020 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. It should be noted that the memory described herein can be of any suitable type. As an example, by executing the computer-executable instruction code in memory 2020, a processor such as a CPU can implement an operator fusion method for inter-layer synchronization of a neural network.

[0116] The operator fusion method or computing device according to the embodiments of this disclosure can also be used by means of... Figure 11 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 11 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the operator fusion method provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 11 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 11 One or more components in the computing device shown.

[0117] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is also provided. Figure 12A schematic diagram 4000 of a storage medium according to the present disclosure is shown.

[0118] like Figure 12 As shown, computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the operator fusion method described with reference to the above figures can be performed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the computer storage medium 4020 can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions 4010 stored on the computer storage medium 4020, the operator fusion method provided according to the embodiments of this disclosure as described above can be performed.

[0119] In summary, some embodiments of this disclosure provide an operator fusion method, computing device, computing equipment, and storage medium for providing operator fusion solutions for various types of neural network structures, especially neural networks with complex structures. More specifically, for operator fusion of various types of neural network structures, a template fusion mode based on operator category labels and a corresponding operator fusion algorithm are designed to make the designed template fusion mode universal and scalable and capable of fusion by operator category, thereby providing an efficient operator fusion process for neural networks, which is beneficial for graph optimization of neural networks and improves the computational efficiency of neural networks on hardware platforms.

[0120] Those skilled in the art will understand that the contents disclosed herein can be varied and modified in many ways. For example, the various devices or components described above can be implemented in hardware, or in software, firmware, or a combination of some or all of the three.

[0121] Furthermore, while this disclosure makes various references to certain elements of systems according to embodiments of this disclosure, any number of different elements may be used and operated on clients and / or servers. Elements are merely illustrative, and different aspects of the system and method may use different elements.

[0122] This disclosure uses flowcharts to illustrate the steps of a method according to embodiments of this disclosure. It should be understood that the preceding or following steps are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes.

[0123] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This disclosure is not limited to any particular combination of hardware and software.

[0124] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that terms such as those defined in a common dictionary should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0125] The foregoing description is intended to illustrate the present disclosure and should not be construed as limiting it. While several exemplary embodiments of the present disclosure have been described, those skilled in the art will readily understand that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined by the claims. It should be understood that the foregoing description is intended to illustrate the present disclosure and should not be construed as limiting it to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.

Claims

1. An operator fusion method applied to a neural network comprising multiple operators, the method comprising: Set at least one linear structure template fusion mode based on the operator's category label; as well as Based on the category labels of the plurality of operators in the neural network, operator fusion is performed on the plurality of operators in the neural network according to the template fusion pattern of the at least one linear structure, so as to fuse one or more operators in the neural network that conform to the template fusion pattern of the at least one linear structure. The category labels include template categories and non-template categories. Operators belonging to the non-template categories do not consume register resources or synchronization memory resources during the computation of the neural network. The template fusion mode of the at least one linear structure consists of one or more linearly connected template categories. The step of setting the template fusion mode of the at least one linear structure based on the category label of the operator includes setting the template fusion mode of the at least one linear structure based on the category label of the template category and ignoring the category labels of the non-template categories.

2. The method according to claim 1, further comprising: Operators are classified based on their functionality and / or the computing architecture characteristics of the hardware platform, and category labels are assigned to operators of different categories.

3. The method according to claim 1, characterized in that, The template categories include one or more of the following: matrix category, normalization category, pooling category, data rearrangement category, data reduction category, regression function category, and loss function category.

4. The method according to claim 1, further comprising: A directed acyclic graph (DAG) is generated based on the network structure of the neural network. The DAG includes operators and connections between them, where the connections represent data dependencies and data flow directions between the operators. Wherein, the operator fusion of the plurality of operators in the neural network according to the template fusion mode of the at least one linear structure includes: Based on the category labels of the plurality of operators in the neural network, and according to the template fusion pattern of the at least one linear structure, the operators in the directed acyclic graph of the neural network are traversed, and one or more operators are matched with the corresponding template fusion pattern of the linear structure, thereby performing operator fusion on the plurality of operators in the neural network.

5. The method according to claim 1, characterized in that, The template fusion mode based on operator category labels to set at least one linear structure includes: a first template fusion mode and a second template fusion mode based on operator category labels to set a linear structure, wherein, when the first template fusion mode and the second template fusion mode have the same part of template categories and the first template fusion mode includes other template categories in addition to the template categories with the same part, the multiple operators in the neural network are first fused according to the first template fusion mode, and then the multiple operators in the neural network are fused according to the second template fusion mode.

6. The method according to claim 1, characterized in that, The template fusion mode based on operator category labels with at least one linear structure includes: The template fusion mode of the at least one linear structure is set using a configurable file.

7. The method according to claim 1, further comprising: Set the fusion mode of the subgraph structure on an operator-by-operator basis; as well as A directed acyclic graph is generated based on the network structure of the neural network. The directed acyclic graph includes operators and connections between operators. The connections between operators represent the data dependencies and data flow between operators.

8. The method according to claim 7, characterized in that, Before fusing the plurality of operators in the neural network according to the template fusion pattern of the at least one linear structure, the method further includes: Operators in the directed acyclic graph of the neural network are fused according to the fusion mode of the subgraph structure to obtain the graph representation after subgraph structure operator fusion.

9. The method according to claim 8, characterized in that, The step of fusing the plurality of operators in the neural network according to the template fusion pattern of the at least one linear structure includes: Based on the category labels of the multiple operators in the neural network, and according to the template fusion pattern of the at least one linear structure, the operators in the graph representation after the subgraph structure operators are traversed, and one or more operators are matched with the corresponding template fusion pattern of the linear structure to perform operator fusion in the graph representation.

10. The method according to claim 3, characterized in that, Operators belonging to the non-template category include: activation functions, linear rectification functions, absolute value functions, and addition functions; Operators belonging to the matrix category include: forward convolution operator, backward data convolution operator, backward filter convolution operator, and matrix multiplication operator; Operators belonging to the normalization category include: batch normalization operator and layer normalization operator; Operators belonging to the pooling category include: max pooling layer operator, average pooling layer operator, and global average pooling layer operator; Operators belonging to the aforementioned data rearrangement category include: concatenation operators and transformation operators; Operators belonging to the aforementioned data reduction category include: maximum value function, minimum value function, and average value function; Operators belonging to the aforementioned regression function category include: regression functions for samples, regression functions for channels; and Operators belonging to the aforementioned loss function category include: mean squared error function and cross-entropy function.

11. A computing device for operator fusion of a neural network, wherein, The neural network includes multiple operators, and the computing device includes: The fusion mode configuration unit is configured to: set at least one linear structure template fusion mode based on the operator's category label; and The fusion unit is configured to perform operator fusion on the plurality of operators in the neural network according to the category labels of each of the plurality of operators in the neural network and a template fusion pattern of at least one linear structure, so as to fuse one or more operators in the neural network that conform to the template fusion pattern of at least one linear structure. The category labels include template categories and non-template categories. Operators belonging to the non-template categories do not consume register resources or synchronization memory resources during the computation of the neural network. The template fusion mode of the at least one linear structure consists of one or more linearly connected template categories. The step of setting the template fusion mode of the at least one linear structure based on the category label of the operator includes setting the template fusion mode of the at least one linear structure based on the category label of the template category and ignoring the category labels of the non-template categories.

12. The computing device according to claim 11 further includes a classification unit configured to: classify operators based on their functions and / or the computing architecture characteristics of the hardware platform, and assign category labels to operators of different categories.

13. The computing device according to claim 11, in, The computing device further includes a generation unit configured to: generate a directed acyclic graph based on the network structure of the neural network, wherein the directed acyclic graph includes operators and connections between operators, and the connections between operators represent the data dependencies and data flow directions between operators. The fusion unit performs operator fusion on the plurality of operators in the neural network according to the template fusion mode of the at least one linear structure, including: Based on the category labels of the plurality of operators in the neural network, and according to the template fusion pattern of the at least one linear structure, the operators in the directed acyclic graph of the neural network are traversed, and one or more operators are matched with the corresponding template fusion pattern of the linear structure, thereby performing operator fusion on the plurality of operators in the neural network.

14. The computing device according to claim 13, characterized in that, The fusion mode configuration unit sets at least one template fusion mode for a linear structure based on the operator category labels, including: setting a first template fusion mode and a second template fusion mode for a linear structure based on the operator category labels. Where the first template fusion mode and the second template fusion mode have the same portion of template categories, and the first template fusion mode includes other template categories besides the same portion, the fusion unit first performs operator fusion on the plurality of operators in the neural network according to the first template fusion mode, and then performs operator fusion on the plurality of operators in the neural network according to the second template fusion mode.

15. The computing device according to claim 11, characterized in that, The fusion mode configuration unit is further configured to: set a fusion mode for subgraph structure on an operator-by-operator basis, wherein the computing device further includes a generation unit configured to: generate a directed acyclic graph based on the network structure of the neural network, wherein the directed acyclic graph includes operators and connections between operators, and the connections between operators represent the data dependencies and data flow directions between operators.

16. The computing device according to claim 15, characterized in that, Before fusing the plurality of operators in the neural network according to the template fusion pattern of the at least one linear structure, the fusion unit is further configured to: Operators in the directed acyclic graph of the neural network are fused according to the fusion pattern of the subgraph structure to obtain a graph representation after subgraph structure operator fusion. The fusion unit performs operator fusion on the plurality of operators in the neural network according to the template fusion mode of the at least one linear structure, including: Based on the category labels of the multiple operators in the neural network, and according to the template fusion pattern of the at least one linear structure, the operators in the graph representation after the subgraph structure operators are traversed, and one or more operators are matched with the corresponding template fusion pattern of the linear structure to perform operator fusion in the graph representation.

17. A computing device, comprising: processor; and A memory, wherein the memory stores computer-readable code, which, when executed by the processor, performs the operator fusion method as described in any one of claims 1-10.

18. A non-transitory computer-readable storage medium having instructions stored thereon, characterized in that, When the instructions are executed by the processor, they implement the operator fusion method as described in any one of claims 1-10.

Images