Operator parallel scheduling method, device and medium based on computation graph optimization

Abstract

This application belongs to the field of computer technology, specifically relating to a method, device, and medium for parallel scheduling between operators based on computation graph optimization. The method includes: converting the convolutional neural network to be scheduled into a computation graph; dividing the computation graph into multiple disjoint subgraphs; performing parallel scheduling of operators within each subgraph using a dynamic template scheduling algorithm, and selecting operator replacement or operator parallelism scheduling strategies for operators in each subgraph during the scheduling process to obtain the optimal scheduling method for each subgraph; based on the optimal scheduling methods of each subgraph, obtaining the overall scheduling strategy for the computation graph by concatenating the subgraphs, and executing the operations of operators in the convolutional neural network according to the overall scheduling strategy. This method can significantly reduce the search range, improve the algorithm's convergence time, and fully utilize sub-scheduling while ensuring the global optimal solution. It outputs parallel scheduling for specific hardware devices, improving the scheduling inference speed of the neural network model and the utilization rate of hardware resources.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and particularly relates to an inter-operator parallel scheduling method based on computation graph optimization. BACKGROUND

[0002] Multi-operator parallelization is to place operators of different stages in different threads for scheduling. By adopting a series of optimization algorithms, the parallelization within and between operators can be effectively combined. Existing optimization algorithms mainly use heuristic algorithms to realize the parallel scheduling and execution of multiple neural network operators. For example, a greedy strategy is used, that is, all available deep neural network operators are directly executed on a hardware device to maximize resource utilization; or a new inter-operator dynamic programming scheduling algorithm is used to find a highly optimized schedule at a low search cost, so that the scheduling reasoning delay between operators is further optimized; or SOAP is introduced to parallelize the convolutional neural network from four dimensions of sample, operator, attribute and parameter, which is a more comprehensive deep neural network parallelization strategy search space. A depth-first search is used to explore the space, and the search model best strategy needs 0.8 and 18 hours respectively.

[0003] The above methods all use different heuristic algorithms to find the optimal solution, and the solution result is not necessarily the global optimal solution. Secondly, although the method based on machine learning and dynamic programming can obtain an approximate global optimal solution by expanding the search space, the algorithm does not fully utilize the common sub-schedule during the search process of the entire model architecture, and it is difficult to utilize the historical optimization results; and the entire computation graph is optimized at one time, which will cause the problem of long algorithm convergence time when the model size is large. SUMMARY

[0004] (I) Technical problem to be solved

[0005] In view of the above-mentioned defects and deficiencies of the prior art, the application provides an inter-operator parallel scheduling method based on computation graph optimization, a device and a medium.

[0006] (II) Technical solution

[0007] To achieve the above-mentioned purposes, the application adopts the following technical solutions:

[0008] In a first aspect, the application provides an inter-operator parallel scheduling method based on computation graph optimization, which comprises the following steps:

[0009] Converting a convolutional neural network to be scheduled into a computation graph;

[0010] The computing graph is divided into a plurality of disjoint subgraphs, the number of operators in the subgraphs is the maximum number of parallel operators that can be parallel processed by one thread in parallel processing;

[0011] The operators in the subgraphs are parallel scheduled by a dynamic template scheduling algorithm, and during the scheduling process, a scheduling strategy of operator replacement or operator parallel is selected for the operators in the subgraphs to obtain an optimal scheduling mode of each subgraph; wherein, during the optimization scheduling search, the dynamic template scheduling algorithm saves the optimal sub-scheduling of the current scheduling for subsequent scheduling search.

[0012] Based on the optimal scheduling mode of each subgraph, the overall scheduling strategy of the computing graph is obtained by subgraph splicing, and the operation of the operators in the convolutional neural network is performed according to the overall scheduling strategy.

[0013] Optionally, the operator replacement includes:

[0014] The operators in the subgraphs are replaced based on the operator type; and / or

[0015] According to a preset convolution kernel size constraint rule, the operators in the subgraphs are replaced.

[0016] Optionally, the convolution kernel size constraint rule includes:

[0017] The largest convolution kernel size in the same subgraph is taken as a replacement convolution kernel size, and for each convolution operator in the subgraph, the convolution kernel size of the convolution operator whose convolution kernel size is smaller than the replacement convolution kernel size is expanded to the replacement convolution kernel size.

[0018] Optionally, the operator replacement of the operators in the subgraphs based on the operator type includes:

[0019] The source operator is replaced by a target operator of the same type, and the parameters and weights of the target operator are determined based on the parameters and weights in the source operator;

[0020] When the source operator is a general symbol node, the source operator is replaced by any basic neural network operator.

[0021] Optionally, when the operator is a convolution operator, the method for determining the parameters and weights of the target operator based on the parameters and weights in the source operator includes:

[0022] The convolution kernel size, the convolution step, and the feature map padding width of the target operator are equal to those of the source operator;

[0023] The output channel size of the target operator is the sum of the output channels of the two source operators;

[0024] The weights of the target operator are obtained by connecting the weight information in the two source operators.

[0025] Optionally, the operators in the subgraph are scheduled in parallel by a dynamic template scheduling algorithm, including:

[0026] A subset of the set of operators is initialized with an infinite delay and an empty scheduling strategy, where the subset is a set of subgraph operators;

[0027] The operators in the subset are traversed, and a dynamic programming algorithm is used to schedule the subgraph to obtain the minimum delay and the optimal scheduling manner of the subset; wherein, during the scheduling search process:

[0028] The scheduling delay and scheduling manner of the subset and the operator list of the subset are saved as a template, and in the subsequent search process, the current subset operator list is matched with the template, and if the matching is successful, the scheduling delay and scheduling manner in the template are used as the scheduling delay and scheduling manner of the current subset;

[0029] The delay of operator replacement and the delay of operator parallelism are calculated respectively, and the scheduling manner with smaller current delay is selected as the scheduling manner obtained by the current search.

[0030] Optionally, the computation graph is divided into a plurality of disjoint subgraphs, including:

[0031] A computation graph splitting algorithm based on the largest block is used to recursively divide the computation graph into subgraphs.

[0032] Optionally, before obtaining the scheduling manner of the convolutional neural network to be scheduled according to the optimized computation graph, it further includes: performing local backtracking search around each split point to obtain operator replacement across the split point.

[0033] In a second aspect, the embodiments of the present application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, when the computer program is executed by the processor, the steps of the operator parallel scheduling method based on computation graph optimization according to any one of the first aspect are implemented.

[0034] In a third aspect, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the steps of the operator parallel scheduling method based on computation graph optimization according to any one of the first aspect are implemented.

[0035] (III) Beneficial effects

[0036] The beneficial effects of the present application are: the present application proposes an inter-operator parallel scheduling method based on computational graph optimization, an electronic device and a medium. First, based on the maximum block computational graph splitting algorithm, the original computational graph is recursively divided into smaller subgraphs. Through the division of subgraphs, the search range is greatly reduced.

[0037] Secondly, according to the operator characteristics of the neural network computational graph, the operator replacement and operator parallel strategy scheduling strategy is proposed. In the process of model scheduling, the optimal scheduling mode is selected for the divided subgraph. The number of operators is reduced through the replacement strategy, so that the optimization time of the algorithm search is greatly improved. It is more suitable for complex network models, and overall, the algorithm convergence time is greatly improved.

[0038] Finally, combined with the parallelism of operators within and between operators, the inter-operator parallel scheduling algorithm based on dynamic templates is proposed to search the entire model architecture. Under the condition of ensuring the global optimal solution, the sub-scheduling is fully utilized, and the parallel scheduling is output for the specific hardware device, which improves the scheduling reasoning speed of the neural network model and the hardware resource utilization rate. BRIEF DESCRIPTION OF DRAWINGS

[0039] The present application is described with the help of the following drawings:

[0040] Figure 1 The flowchart of the inter-operator parallel scheduling method based on computational graph optimization in an embodiment of the present application is shown.

[0041] Figure 2 The operator replacement example diagram in another embodiment of the present application is shown.

[0042] Figure 3 The example diagram of the change in model scheduling execution order before and after the operator replacement and IPS scheduling algorithm in another embodiment of the present application is shown.

[0043] Figure 4 The operator scheduling reasoning delay example diagram in another embodiment of the present application is shown.

[0044] Figure 5 The scheduling reasoning delay example diagram of the Inception_v3 model with different Batch Size in another embodiment of the present application is shown.

[0045] Figure 6 The architecture diagram of the electronic device in another embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] To better explain and facilitate understanding of the present invention, it is described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described below are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other; for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0047] The method of the present invention will be specifically described below through various embodiments.

[0048] Example 1

[0049] Figure 1 This is a schematic diagram of the inter-operator parallel scheduling method based on computation graph optimization in one embodiment of this application, as shown below. Figure 1 As shown, the operator parallel scheduling method based on computation graph optimization in this embodiment includes:

[0050] S10. Convert the convolutional neural network to be scheduled into a computation graph;

[0051] S20. Divide the computation graph into multiple non-overlapping subgraphs, wherein the number of operators contained in each subgraph is the maximum number of operators that a thread can run in parallel during the parallel process;

[0052] S30. Operators within the subgraph are scheduled in parallel using a dynamic template scheduling algorithm. During the scheduling process, an operator replacement or operator parallelism scheduling strategy is selected for the operators in the subgraph to obtain the optimal scheduling method for each subgraph. The dynamic template scheduling algorithm saves the optimal sub-scheduling of the current scheduling when optimizing the scheduling search for subsequent scheduling searches.

[0053] S40. Based on the optimal scheduling method of each subgraph, the overall scheduling strategy of the computation graph is obtained by splicing the subgraphs, and the operation of the operators in the convolutional neural network is executed according to the overall scheduling strategy.

[0054] The operator parallel scheduling method based on computation graph optimization in this embodiment can significantly reduce the search range and improve the algorithm convergence time. While ensuring the global optimal solution, it makes full use of sub-scheduling, outputs parallel scheduling for specific hardware devices, and improves the scheduling inference speed of the neural network model and the utilization rate of hardware resources.

[0055] To better understand the present invention, the steps in this embodiment will be described in detail below.

[0056] In this embodiment S20, the computation graph is recursively divided into subgraphs by employing a computation graph splitting algorithm based on the maximum block.

[0057] The maximum block size is defined as the maximum number of parallel operators that a single thread can operate in parallel during the parallel process.

[0058] In this embodiment S30, the operators within the subgraph are scheduled in parallel using a dynamic template scheduling algorithm, including:

[0059] The subset delay of the operator subset is initialized to infinity, and the scheduling policy is empty, where the subset is a set of subgraph operators;

[0060] The operators in the subset are traversed, and a dynamic programming algorithm is used to schedule the subgraph to obtain the minimum delay and optimal scheduling method for the subset; wherein, during the scheduling search process:

[0061] Save the scheduling delay, scheduling method, and operator list of the subset as a template. In the subsequent search process, match the current subset operator list with the template. If the match is successful, use the scheduling delay and scheduling method in the template as the scheduling delay and scheduling method of the current subset.

[0062] Calculate the delay of operator replacement and the delay of operator parallelism respectively, and select the scheduling method with the smaller current delay as the current searched scheduling method.

[0063] In this embodiment S30, operator replacement includes:

[0064] Operator replacement is performed on the operators in the subgraph based on the operator type; and / or

[0065] Operator replacement is performed on the operators in the subgraph according to the preset convolution kernel size constraint rules.

[0066] Specifically, operator replacement is performed on the operators in the subgraph based on the operator type, including:

[0067] The source operator is replaced with a target operator of the same type, wherein the parameters and weights of the target operator are determined based on the parameters and weights in the source operator;

[0068] When the source operator is a general symbol node, the source operator is replaced with a basic neural network operator, such as convolution, pooling, activation function, etc.

[0069] Specifically, the convolution kernel size constraint rules may include:

[0070] The largest convolution kernel size in the same subgraph is used as the replacement convolution kernel size. For each convolution operator in the subgraph, the convolution kernel size of convolution operators whose convolution kernel size is smaller than the replacement convolution kernel size is expanded to the replacement convolution kernel size.

[0071] For example, when replacing operators with convolution sizes of 3x3x256 and 1x1x256, the 1x1x256 source convolution operator is first expanded to a 3x3x256 operator. After replacement according to the replacement rules, the merged new target operator is 3x3x512.

[0072] Example 2

[0073] A CNN network can be abstracted as a computation graph G = (V, E). The CNN computation graph is a directed acyclic graph, where V represents the set of operators and E represents the set of edges between operators, with edges representing data dependencies between operators. Figure 2 This is an example diagram of operator substitution in another embodiment of this application. Figure 2 (a) is an example diagram of the source operator, and (b) is an example diagram of the target operator. Figure 2 In the middle (a), there is a computation graph corresponding to a simple convolutional neural network.

[0074] S1. The model is split using a model splitting algorithm based on the maximum block size.

[0075] Many state-of-the-art neural network models are too large to be optimized directly across the entire graph using a search algorithm. This embodiment uses a maximum-block-based model splitting algorithm to recursively divide the computation graph into smaller disjoint subgraphs. In this embodiment, the maximum block size is defined as the maximum number of operators that a single thread can run concurrently during parallel processing. In this embodiment, the maximum number of operators in each block is 50, i.e., Max_part_size = 50. The split subgraphs can then be scheduled using parallel optimization strategies.

[0076] It should be noted that, since operator substitution cannot be performed on any two sub-blocks, the goal of the split is to minimize the number of operator substitutions spanning two subgraphs, for each operator S i ∈G, cap(S) i Operator substitution is defined as the number of operator substitutions mapped to at least one input edge and one output edge of the operator. Operator substitution is disabled if the graph is split using the `split` operator.

[0077] By using cap(S) i As the weight of each operator, the model splitting problem is mapped to a minimum point cut problem.

[0078] The algorithm takes an initialized computation graph as input and outputs the split computation graph. First, it determines the number of operators in the computation graph, using the maximum number of parallel operators as the partitioning criterion. Second, the partitioning aims to return the operator point cut that minimizes the subgraph. For each operator, it checks if a certain G1 operator node S... iIf P can be reached, the operators of this set are assigned to G1, and G1 is subtracted from the original computation graph to obtain the computational subgraph G2. Finally, the model computation graph is recursively processed, with the set of operators of the subgraphs passed in as input parameters. The final result is the partitioned subgraph blocks of the computation graph.

[0079] S2, Operator replacement.

[0080] Operator replacement strategy: Each replacement consists of a source operator and a target operator. The source operator can be mapped to a specific sub-operator in the model computation graph, and the target operator defines how to create a new sub-operator to replace the mapped operator. The replacement is then compared with the operator parallel latency, and the method with the shortest latency is selected.

[0081] Source operators. Source operators define the structure of the valid operators for replacement. Each node in a source operator is associated with a type and can only map to operators of the same type. Source operators can also include generic operator nodes, which are useful when the type of the operator does not affect the replacement process and when the source operator describes multiple similar replacement scenarios. In addition to type constraints, source operators can also include other convolution size constraints on one or more operators to further restrict the replacement.

[0082] Target Operator. The target operator describes how to construct a new operator to replace the mapped operator. For each newly created operator, the target operator defines how to set parameters and compute weights using the parameters and weights from the source operator.

[0083] When the operator is a convolution operator, the parameters and weights of the target operator are determined based on the parameters and weights of the source operator using methods including:

[0084] The kernel size, stride, and feature map padding width of the target operator are equal to those of the source operator.

[0085] The output channel size of the target operator is the sum of the output channels of the two source operators;

[0086] The weights of the target operator are obtained by concatenating the weight information from the two source operators.

[0087] The outer edges of the target operator should correspond to each outer edge of the source operator, and any outer operator that was initially connected to the mapping operator in the source operator should now be connected to the corresponding operator in the target operator.

[0088] Please continue reading. Figure 2 .like Figure 2 As shown, the first step replaces the Conv2 and Conv3 operators with the Conv4 operator, and the second step replaces the Conv6 and add operators with the Conv7 operator. After the Conv4 operator, in order to maintain the dependencies between operators, the replaced operators will be separated.

[0089] S3. Optimize each subgraph using a dynamic template scheduling algorithm.

[0090] During the optimization of the scheduling search, Template[] is used to save the sub-schedulings in the current scheduling process. Since the scheduling set of a set of subgraphs is changing, the optimal sub-scheduling of the current scheduling will be saved and updated according to the computation time delay during the search optimization process. In subsequent scheduling searches, the saved template will be directly used for replacement, which greatly reduces the optimization search time of the model scheduling.

[0091] S4. Piece the optimized subgraphs together to form a complete computation graph; perform a local backtracking search around each split point to obtain operator replacements across the split points.

[0092] Finally, the overall scheduling strategy of the computation graph is determined, and the operations of operators in the convolutional neural network are executed according to the overall scheduling strategy.

[0093] The following is an example of a scheduling algorithm.

[0094] Algorithm 1 Parallel scheduling based on dynamic template.

[0095] Input: Computation graph G

[0096] Output: The schedule for computation graph G.

[0097]

[0098]

[0099] First, the subset delay of the operator subset is initialized to infinity, and the scheduling policy is empty.

[0100] Secondly, let V represent all operators in the computation graph, and determine the subset delay. If the subset delay is not infinite, return the subset delay. Traverse the operators in the subset and use a dynamic programming algorithm to schedule the subgraph. During scheduling, save and record the scheduling delay of the subset. In subsequent search processes, if the current subset operator list equals the saved operator list, directly assign the value and bind its parallel strategy. During the dynamic programming scheduling search process, select the current optimal strategy based on the computation time value. First, determine whether the current operator can be replaced according to the previously defined strategy. If not, then... Assigning a value of infinity means it is not replaceable. Conversely, the delay for the current operator replacement is returned. Lines 18-21, IPS will... and Returns the optimal delay and strategy for the current set of operators. Line 22 is the Cost Model defined in this paper, which uses dynamic programming to explore the scheduling delay of the remaining operators, aiming to minimize the scheduling subproblem (SS). i-1 The scheduling delay. Lines 23-26 will calculate L based on the defined Cost Model. S′ It automatically updates the current optimal scheduling time and strategy, and finally returns the optimal scheduling delay and strategy.

[0101] The following section uses an Inception module from the Inception_v3 model as an example to further illustrate the implementation process of the parallel scheduling method based on computation graph optimization in this application, and compares the model latency with other scheduling methods.

[0102] After using the maximum block-based model splitting algorithm, the modules to be scheduled are obtained with the Inception module as a subgraph. Figure 3 This is an example diagram illustrating the change in the model scheduling execution order before and after operator replacement and the IPS scheduling algorithm in another embodiment of this application; Figure 3 In the diagram, (a) shows the operator execution sequence before scheduling optimization, and (b) shows the operator execution sequence after IPS optimization. Figure 3 As shown in (a), nodes (a,b,h,i,g) represent convolution operators with a size of 3x3, nodes (c,f) represent convolution operators with a size of 3x1, nodes (d,j) represent convolution operators with a size of 1x3, and node (e) is the model's pooling operator. Because different backend IPS algorithms handle operators differently, this example uses an NVIDIA GeForce RTX 2080ti GPU. The IPS scheduling algorithm uses a Pro scheduling method, such as... Figure 3 As shown in (b), node e is scheduled in Pro1, meaning that node e occupies a separate GPU thread for scheduling during this process. Similarly, in Pro4, the IPS compares the performance of inter-operator parallelism versus operator replacement based on the returned latency, finding that operator replacement yields a relatively better latency. Therefore, the convolution operator of node d is expanded to a 3x3 size, and the original nodes c and d are replaced to obtain the replaced operator [c&d], ultimately minimizing the scheduling latency of the entire computation graph.

[0103] The method employed in this application will be compared with sequential scheduling and greedy scheduling. Sequential scheduling executes each operator sequentially according to the topology of the model's computation graph. Greedy scheduling places all operators that can be executed in parallel in the current state into the same process. The method in this application, however, uses a defined operator replacement and dynamic template scheduling algorithm to execute operators in the computation graph in parallel. All three scheduling methods are executed on the same execution engine for comparison of the final results.

[0104] First, the inference latency of different models and scheduling methods was compared when the batch size was 1. In this example, the average value was obtained after 6 scheduling iterations. The final latency examples for each model are shown below. Figure 4 As shown. Figure 4 This is an example diagram of operator scheduling inference latency in another embodiment of this application. Figure 4 In the diagram, the vertical axis represents latency, and the horizontal axis represents the network models (network, VGG_13, SqueezeNet, VGG_19, and Inception_v3) using sequential scheduling, greedy scheduling, Tensor RT deep learning inference software development kit, and IPS algorithm, respectively. Figure 4 As can be seen, compared with traditional dynamic programming scheduling, this embodiment uses parallel scheduling between operators based on dynamic templates, which greatly reduces the convergence time for finding the optimal schedule.

[0105] Secondly, the scheduling and inference latency of the Inception_v3 model were compared for different batch sizes. The experimental results are a comparison of the average latency obtained from 10 experiments. Figure 5 This is an example diagram illustrating the scheduling and inference latency of the Inception_v3 model with different batch sizes in another embodiment of this application. Figure 5 As shown, the algorithm is significantly slower than the other two scheduling methods under different batch sizes. Greedy scheduling generally exhibits better scheduling latency than sequential scheduling. However, the greedy algorithm shows the highest latency when the batch size is 8. Analysis of the scheduling latency of each Inception block in Inception_v3 reveals that when the batch size is 8, the greedy algorithm experiences resource contention in the early stages of the process, followed by insufficient resource utilization in the later stages, ultimately resulting in a higher latency than sequential scheduling. In summary, IPS is generally slower than the other two scheduling methods.

[0106] The parallelism of the Inception_v3 network model under different hardware backends was tested using the method and sequential scheduling algorithm of this embodiment. The experimental results are shown in Table 1. Table 1 is a statistical table of Inception_v3 block latency under different hardware backends.

[0107] Table 1

[0108]

[0109] As shown in Table 1, for the Inception_v3 model, the model splitting algorithm in this embodiment divides it into 13 sub-blocks. For sequential scheduling, the model's scheduling resources are mainly used for the Inception module, i.e., sub-block 11 (block numbering in this paper starts from 0). For the designed inter-operator scheduling algorithm, the number and method of parallel operation of multiple operators can be adjusted according to the maximum availability of resources during model scheduling, depending on the different execution backends. For example, the maximum latency of K80 is used for sub-block 0, i.e., convolution and pooling operators are executed one by one. The RTX 2080ti and RTX 3090 can perform at least two convolution operators in parallel at once, thus significantly reducing scheduling latency and using their main resources for accelerating inference scheduling in the Inception module. The experimental results above demonstrate that the algorithm proposed in this application can be designed for targeted parallelism on different hardware backends.

[0110] Furthermore, experiments were conducted using the existing deep learning library cuDNN, minimizing the cost based on a defined cost function. Computational latency was used as the optimization objective. Experimental environment setup: Python 3.7, CUDA 11.1, cuDNN version 8.0.5, Ubuntu version 18.04, CPU 6x Xeon E5-2678V3, GPU NVIDIA GeForce RTX2080ti. Table 2 shows the neural network model structure parameters; the model structure is shown in Table 2.

[0111] Table 2

[0112]

[0113] Based on the existing low-level acceleration library cuDNN, intra-operator acceleration is achieved, and a scheduling algorithm that guarantees minimum parallel execution latency is designed, realizing joint optimization of intra-operator and inter-operator parallelism. Experiments show that the designed algorithm achieves a speedup of up to 1.6x compared to existing sequential execution methods, resulting in a performance improvement of 1.18 to 1.86 times compared to Tensor RT systems. The algorithm proposed in this application can be widely used in research on parallel architectures of neural network models.

[0114] For neural network inference, model optimization latency is a crucial factor. The algorithm achieves the same results as other papers within 8 minutes. Compared to heuristic enumeration search, the search scope is significantly reduced because the experiment sets the maximum number of operators executed in each process (g=9) and the maximum number of operators in each group (s=4). For complex models such as the RandWire model, the replacement strategy in this paper reduces the number of operators, thus significantly improving the algorithm's search optimization time. This demonstrates that the algorithm submitted in this application is more suitable for complex network models, and overall, it greatly improves the algorithm's convergence time. Specific results are shown in Table 3. Table 3 shows the optimization convergence time for different networks.

[0115] Table 3

[0116]

[0117] Among them, Algorithm 1 uses an improved machine learning algorithm for search, achieving faster convergence compared to heuristics; Algorithm 2 uses A, S, O, and P to represent the parallel strategy of neural network convolution, evaluating its search process from a more comprehensive perspective; Algorithm 3 uses the original initial dynamic programming algorithm for scheduling.

[0118] Therefore, the method in this embodiment addresses the complexity of computation graphs in modern neural networks by combining intra-operator and inter-operator parallelism. It employs a dynamic template-based inter-operator scheduling algorithm to explore the potential scheduling space, fully utilizing sub-scheduling while ensuring the global optimal solution. This outputs parallelized scheduling for specific hardware devices, improving the scheduling inference speed and hardware resource utilization of the neural network model.

[0119] Example 3

[0120] The second aspect of this application provides an electronic device through Embodiment 3, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the operator parallel scheduling method based on computation graph optimization as described in any of the above embodiments.

[0121] Figure 6 This is a schematic diagram of the architecture of an electronic device in another embodiment of this application.

[0122] Figure 6The illustrated electronic device may include at least one processor 101, at least one memory 102, at least one network interface 104, and other user interfaces 103. The various components of the electronic device are coupled together via a bus system 105. It is understood that the bus system 105 is used to implement communication between these components. In addition to a data bus, the bus system 105 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general labeled all buses as Bus System 105.

[0123] The user interface 103 may include a display, keyboard, or clicking device (e.g., mouse, trackball, or touchpad).

[0124] It is understood that the memory 102 in this embodiment can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 62 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0125] In some implementations, memory 102 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 1021 and application program 1022.

[0126] The operating system 1021 includes various system programs, such as a framework layer, a core library layer, and a driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 1022 includes various applications used to implement various application functions. Programs implementing the methods of this embodiment of the invention can be included in the application program 1022.

[0127] In this embodiment of the invention, the processor 101 executes the method steps provided in the first aspect by calling the program or instructions stored in the memory 102, specifically the program or instructions stored in the application program 1022.

[0128] The methods disclosed in the above embodiments of the present invention can be applied to processor 101, or implemented by processor 101. Processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 101 or by instructions in the form of software. The processor 101 may be a general-purpose processor, digital signal processor, application-specific integrated circuit, off-the-shelf programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in the art. The storage medium is located in memory 102, and processor 101 reads the information in memory 102 and completes the steps of the above method in combination with its hardware.

[0129] In addition, in conjunction with the computation graph-based parallel scheduling method for operators in the above embodiments, this invention can provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements any of the computation graph-based parallel scheduling methods for operators in the above embodiments.

[0130] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. The use of the terms first, second, third, etc., is for convenience only and does not indicate any order. These terms can be understood as part of the component names.

[0131] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0132] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0133] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A parallel scheduling method between operators based on computation graph optimization, characterized in that, The method includes: Convert the convolutional neural network to be scheduled into a computation graph; The computation graph is divided into multiple non-overlapping subgraphs, and the number of operators in each subgraph is the maximum number of operators that a single thread can run in parallel during the parallel process. The operators within the subgraph are scheduled in parallel using a dynamic template scheduling algorithm. During the scheduling process, an operator replacement or operator parallelism scheduling strategy is selected for each operator in the subgraph to obtain the optimal scheduling method for each subgraph. The parallel scheduling of operators within the subgraph using the dynamic template scheduling algorithm includes: The subset delay of the operator subset is initialized to infinity, and the scheduling policy is empty, where the subset is a set of subgraph operators; The operators in the subset are traversed, and a dynamic programming algorithm is used to schedule the subgraph to obtain the minimum delay and optimal scheduling method for the subset; wherein, during the scheduling search process: Save the scheduling delay, scheduling method, and operator list of the subset as a template. In the subsequent search process, match the current subset operator list with the template. If the match is successful, use the scheduling delay and scheduling method in the template as the scheduling delay and scheduling method of the current subset. Calculate the delay of operator replacement and the delay of operator parallelism respectively, and select the scheduling method with the smaller current delay as the current searched scheduling method; The dynamic template scheduling algorithm saves the optimal sub-scheduling of the current schedule for use in subsequent scheduling searches when optimizing the scheduling search. Based on the optimal scheduling method of each subgraph, the overall scheduling strategy of the computation graph is obtained by concatenating the subgraphs, and the operation of the operators in the convolutional neural network is executed according to the overall scheduling strategy.

2. The operator parallel scheduling method based on computation graph optimization according to claim 1, characterized in that, The operator replacement includes: Operator replacement is performed on the operators in the subgraph based on the operator type; and / or Operator replacement is performed on the operators in the subgraph according to the preset convolution kernel size constraint rules.

3. The operator parallel scheduling method based on computation graph optimization according to claim 2, characterized in that, The convolution kernel size constraint rules include: The largest convolution kernel size in the same subgraph is used as the replacement convolution kernel size. For each convolution operator in the subgraph, the convolution kernel size of the convolution operator whose convolution kernel size is smaller than the replacement convolution kernel size is expanded to the replacement convolution kernel size.

4. The operator parallel scheduling method based on computation graph optimization according to claim 2, characterized in that, Operator replacement in the subgraph based on operator type includes: The source operator is replaced with a target operator of the same type, wherein the parameters and weights of the target operator are determined based on the parameters and weights in the source operator; When the source operator is a general symbol node, replace the source operator with any basic neural network operator.

5. The operator parallel scheduling method based on computation graph optimization according to claim 4, characterized in that, When the operator is a convolution operator, the method for determining the parameters and weights of the target operator based on the parameters and weights in the source operator includes: The kernel size, stride, and feature map padding width of the target operator are equal to those of the source operator. The output channel size of the target operator is the sum of the output channels of the two source operators; The weight of the target operator is obtained by concatenating the weight information of the two source operators.

6. The operator parallel scheduling method based on computation graph optimization according to claim 1, characterized in that, The computation graph is divided into multiple disjoint subgraphs, including: A computation graph splitting algorithm based on the maximum block is used to recursively divide the computation graph into subgraphs.

7. The operator parallel scheduling method based on computation graph optimization according to claim 1, characterized in that, Before obtaining the scheduling method for the convolutional neural network to be scheduled based on the optimized computation graph, the process also includes: performing a local backtracking search around each split point to obtain operator replacements across the split points.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the operator parallel scheduling method based on computation graph optimization as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the operator parallel scheduling method based on computation graph optimization as described in any one of claims 1 to 7.

Images