Parallel optimization method and device for neural network model based on operator block

Abstract

The invention discloses a neural network model parallel optimization method and device based on operator block, including an operator block process perceived by hardware devices and a weighted operator scheduling process; the operator block process perceived by hardware devices includes: expanding the neural network model and constructing a calculation graph; analyzing the hardware characteristics of each device and establishing a computing capability model of each hardware device accordingly; traversing the calculation graph and blocking the operators based on the computing capability model of the hardware device; the weighted operator scheduling process includes: updating the calculation graph according to the block scheme of each operator, assigning a weight to each operator node, and the calculation graph becomes a weighted directed acyclic graph; traversing the calculation graph, calculating the height of each operator node, and using the height as the priority of the operator node; finding an operator node set with an in-degree of 0, putting the node with the highest priority into an execution queue, and deleting the node and the edge connected to the node in the calculation graph to obtain a finite topological sequence of the operator.

Description

Technical Field

[0001] The present invention belongs to the field of parallel optimization of neural network reasoning, and in particular, relates to a neural network model parallel optimization method based on operator blocking, and in particular, relates to an operator blocking parallel computing optimization method for hardware computing resource perception in neural network reasoning. Background Art

[0002] Parallel optimization refers to improving the reasoning speed of a program through more efficient parallel computing strategies. In neural networks, parallel optimization usually refers to changing the execution order of operators in the neural network so that multiple operators with parallelism can be executed in parallel. At the same time, when operators perform reasoning, they can often be parallelized. For example, the matrix multiplication (Matmul) operator has a large input data scale, and the operator needs to perform the same calculation (multiplication or addition) on data in different rows or columns, so these same calculations can be processed in parallel.

[0003] Parallel optimization methods for neural network inference can be divided into two types: intra-operator acceleration and inter-operator scheduling:

[0004] The most widely used intra-operator acceleration method is operator blocking, which means analyzing the calculation method of a given operator and splitting the operator into several small operators, called micro-operators, and then assigning these micro-operators to different computing cores to accelerate the execution of the operator through parallel computing. However, this method usually only performs simple blocking without fully considering the differences in computing power and cache size of different devices.

[0005] Inter-operator scheduling analyzes the entire neural network calculation graph and optimizes the execution order of operators while satisfying the data dependencies between operators. This can balance the load and improve system throughput. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a method and device for parallel optimization of a neural network model based on operator blocking. The operators in the neural network are divided into operator blocks according to given hardware device parameters, so that the blocked operators can better conform to the hardware parallel computing characteristics, and the operators are re-scheduled to improve the parallelism between operators, thereby performing more efficient reasoning.

[0007] According to one aspect of the present invention, a method for parallel optimization of a neural network model based on operator partitioning is provided, including a hardware device-aware operator partitioning process and a weighted operator scheduling process; wherein:

[0008] The operator segmentation process perceived by the hardware device includes: expanding the neural network model and constructing a calculation graph; analyzing the hardware characteristics of each device and establishing a computing capability model of each hardware device accordingly; traversing the calculation graph and segmenting the operator based on the computing capability model of the hardware device;

[0009] The weighted operator scheduling process includes: updating the calculation graph according to the block scheme of each operator, assigning a weight to each operator node, and the calculation graph becomes a weighted directed acyclic graph; traversing the calculation graph, calculating the height of each operator node, and using the height as the priority of the operator node; finding a set of operator nodes with an in-degree of 0, putting the node with the highest priority into the execution queue, and deleting the node and the edge connected to the node in the calculation graph to obtain a finite topological sequence of the operator.

[0010] As a further technical solution, the tensor computing capability p of hardware device i i for: Where k represents the number of computing cores, l represents the completion length of the computing core of hardware device i, and t i Represents the sum of data transmission time and calculation time of hardware device i.

[0011] As a further technical solution, if the time for the computing core to read the input data and write back the output data is t i,d Than the time t for calculating logic i,c long, and pipeline arrangement cannot be performed between data reading and writing and tensor calculation, then t i =t i，c +2t i，d ; If pipelining between data reading and writing and tensor computation is supported, then t i =max{t i，c , t i，d}.

[0012] As a further technical solution, for the tensor operation operator with an input tensor length of L, the operator block strategy is:

[0013] The length L of the tensor assigned to hardware device i i for:

[0014] The number of blocks N allocated to the micro-operator queue on hardware device i i for:

[0015] As a further technical solution, in the weighted operator scheduling process, the weight of each operator node is used as the shortest delay of the operator, and a weighted directed acyclic graph is constructed according to the shortest delay of the operator.

[0016] As a further technical solution, in the weighted operator scheduling process, an operator execution queue is constructed according to the operator node height.

[0017] According to one aspect of the present invention, there is provided a neural network model parallel optimization device based on operator block, comprising:

[0018] The first main module is used to perform the operator segmentation process perceived by the hardware device, including: expanding the neural network model and building a calculation graph; analyzing the hardware characteristics of each device and building a computing capability model of each hardware device accordingly; traversing the calculation graph and segmenting the operator based on the computing capability model of the hardware device;

[0019] The second main module is used to execute the weighted operator scheduling process, including: updating the calculation graph according to the block scheme of each operator, assigning weights to each operator node, and the calculation graph becomes a weighted directed acyclic graph; traversing the calculation graph, calculating the height of each operator node, and using the height as the priority of the operator node; finding a set of operator nodes with an in-degree of 0, putting the node with the highest priority into the execution queue, and deleting the node and the edges connected to the node in the calculation graph to obtain a finite topological sequence of operators.

[0020] According to one aspect of the present specification, there is provided an electronic device, comprising: at least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the described method.

[0021] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the method described above.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] When executing neural network model reasoning, the present invention introduces hardware computing power into operator blocks, so that the operators after blocking are more in line with the hardware computing characteristics, and optimizes the scheduling strategy of the operators after blocking, so as to perform more efficient reasoning, effectively improving the problem of insufficient utilization of device resources during existing neural network reasoning, improving the utilization rate of device resources during reasoning, and reducing reasoning delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic flow chart of a method for parallel optimization of a neural network model based on operator blocking provided in an embodiment of the present invention.

[0026] Figure 2 A schematic diagram of the process flow of operator segmentation based on hardware device perception provided in an embodiment of the present invention.

[0027] Figure 3 A schematic diagram of the weighted operator scheduling process flow provided by an embodiment of the present invention.

[0028] Figure 4 A schematic diagram of an operator finite topological sequence provided in an embodiment of the present invention.

[0029] Figure 5 A schematic diagram of the structure of a neural network model parallel optimization device based on operator blocking provided in an embodiment of the present invention.

[0030] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be noted that:

[0032] The parallel optimization method of the neural network model based on operator block provided in the present application can be applied to electronic devices with neural network processing functions such as motherboards and computer devices. The electronic devices can be isomorphic electronic devices or heterogeneous electronic devices. The electronic devices can include a first processor, a second processor and a memory. The first processor and the second processor can be, but are not limited to, one of the following combinations: CPU (central processing unit) and artificial intelligence processor, CPU and DSP (Digital Signal Processing, digital signal processor), CPU and GPU (image processor), GPU and artificial intelligence processor. The memory stores a computer program, and the first processor can implement the operator block perceived by the hardware device when executing the computer program, so as to make the operator after block more consistent with the hardware computing characteristics. The second processor can receive the calculation instructions and input data related to the second processor sent by the first processor after the first processor performs operator block, and optimize the scheduling strategy of the operator after block. It can be understood that the above-mentioned first processor or the second processor can also independently implement the operator block and operator scheduling strategy perceived by the above-mentioned hardware device.

[0033] The target neural network in the embodiment of the present invention can be based on a neural network architecture such as caffe (Convolution Architecture For Feature Extraction), or can be constructed using other programming languages. Various types of neural networks can be constructed by combining various types of network layers through directed connections, including but not limited to convolutional layers, pooling layers, fully connected layers, activation layers and other network layers; each network layer corresponds to at least one operator, for example, the activation layer can correspond to the relu operator, and the convolutional layer can correspond to the convolution operator. Generally, parallel optimization methods for neural network reasoning can be divided into two types: intra-operator acceleration and inter-operator scheduling, and the inter-operator scheduling method does not conflict with the intra-operator acceleration method, which means that the two methods can be used in combination to reduce the neural network reasoning delay.

[0034] The neural network model parallel optimization method based on operator blocking of the embodiment of the present invention can divide the operators in the neural network into operator blocks according to the given hardware device parameters, so that the blocked operators can better conform to the hardware parallel computing characteristics, and re-schedule the operators to improve the parallelism between operators, thereby performing more efficient reasoning.

[0035] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely exemplary and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps may be further decomposed, while some operations / steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0038] The embodiment of the present invention provides a neural network model parallel optimization method based on operator block, such as Figure 1 As shown in the figure, it is divided into the hardware device-aware operator block process and the weighted operator scheduling process. Among them, the hardware device-aware operator block process introduces hardware computing power into the operator block, making the block-based operators more in line with the hardware computing characteristics. The weighted operator scheduling process optimizes the scheduling strategy of the block-based operators, thereby performing more efficient reasoning. Through the interaction of the two, the problem of insufficient utilization of device resources during existing neural network reasoning is effectively improved, the utilization rate of device resources during reasoning is improved, and the reasoning delay is reduced.

[0039] To simplify the description, the application of the parallel optimization method of a neural network model based on operator blocking in an embodiment of the present invention to heterogeneous devices is taken as an example for description.

[0040] like Figure 2 As shown in the figure, the hardware device-aware operator partitioning process includes the following steps:

[0041] Step 1: Expand the neural network model and construct a computational graph, where each node represents an operator and the directed edges between nodes represent the dependencies between nodes.

[0042] Step 2: Analyze the hardware characteristics of each heterogeneous device and model the computing capabilities of each heterogeneous device accordingly. The computing core of heterogeneous device i needs time t to complete the multiplication / addition of tensors of length l. i,c , the data transmission time t i,d , t i represents the sum of data transmission time and computing time, and has k cores, then the tensor computing capability P of the heterogeneous device is i for:

[0043]

[0044] Among them, if the time for the computing core to read input data and write back output data is t i,d , than the time t to calculate the logic i,c long, and pipeline arrangement cannot be performed between data reading and writing and tensor calculation, the calculation delay t i :

[0045] t i =t i，c +2t i，d

[0046] If pipelining between data reading and writing and tensor computation is supported, the computation delay t i :

[0047] t i =max{t i，c , t i，d}

[0048] t i The calculation method to be used will be specified according to the characteristics of the hardware device.

[0049] Step 3: Traverse the computation graph and divide the operators into blocks based on the computing capability model of heterogeneous devices. Input tensor operation operators with tensor length L are assigned to tensor length L of computing device i. i for:

[0050]

[0051] Furthermore, the number of blocks N allocated to the micro-operator queue on computing device i i for:

[0052]

[0053] At the same time, the shortest delay T of completing the tensor operation operator through heterogeneous parallelism can be estimated op for:

[0054] T op =N i ·t i

[0055] Repeat step 3 until all operators in the computation graph have determined the block strategy.

[0056] After the cycle of steps 1, 2, and 3, a distribution scheme for parallel computing on heterogeneous devices is constructed for all operators in the computation graph.

[0057] like Figure 3 As shown, the weighted operator scheduling process includes the following steps:

[0058] Step 4: Update the computational graph according to the load-balanced partitioning scheme for each operator based on heterogeneous resources, and assign a weight to each operator node, whose weight is the shortest delay T of the operator. op , the computational graph becomes a weighted directed acyclic graph.

[0059] Step 5: Traverse the computation graph, calculate the height of each operator node, and use the height as the priority of the operator node. The height of a node refers to the length of the longest path starting from the node in the computation graph.

[0060] Step 6: Find the set of operator nodes with in-degree 0, put the node with the highest priority into the execution queue, and delete the node and the edges connected to the node in the computation graph.

[0061] Repeat step 6 until there are no nodes in the computation graph, and you can get the finite topological sequence of operators. The finite topological sequence of operators is as follows: Figure 4 The finite topological sequence of operators is shown in

[0062] After the above steps, an operator scheduling strategy is generated for the neural network model to be optimized, and an operator partitioning strategy based on heterogeneous resources is generated for each operator.

[0063] It should be understood that the parts not elaborated in detail in the aforementioned method embodiments all belong to the prior art.

[0064] The implementation basis of each embodiment of the present invention is implemented by programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a neural network model parallel optimization device based on operator block, which is used to execute the neural network model parallel optimization method based on operator block in the above method embodiment.

[0065] See also Figure 5 The device includes: a first main module, which is used to execute the operator blocking process perceived by the hardware device, including: expanding the neural network model and constructing a calculation graph; analyzing the hardware characteristics of each device, and establishing the computing power model of each hardware device accordingly; traversing the calculation graph, and blocking the operator based on the computing power model of the hardware device; a second main module, which is used to execute the weighted operator scheduling process, including: updating the calculation graph according to the blocking scheme of each operator, assigning a weight to each operator node, and the calculation graph becomes a weighted directed acyclic graph; traversing the calculation graph, calculating the height of each operator node, and using the height as the priority of the operator node; finding a set of operator nodes with an in-degree of 0, putting the node with the highest priority into the execution queue, and deleting the node and the edge connected to the node in the calculation graph to obtain a finite topological sequence of the operator.

[0066] The neural network model parallel optimization device based on operator block provided by the embodiment of the present invention is aimed at the parallel optimization problem in neural network reasoning. Figure 5 Several modules in the neural network divide the operators into blocks according to the given hardware device parameters, so that the divided operators can better meet the hardware parallel computing characteristics, and re-schedule the operators to improve the parallelism between operators, thereby performing more efficient reasoning.

[0067] It should be noted that the device embodiments provided by the present invention are used to implement the methods in the above method embodiments as well as the methods in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set, and the principle is basically the same as the principle of the above device embodiments provided by the present invention. As long as the technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above device embodiments, obtain the corresponding technical means and the technical solutions composed of these technical means by combining the technical features, and on the premise of ensuring the practicality of the technical solutions, they can improve the equipment in the above device embodiments to obtain the corresponding device class embodiments, which are used to implement the methods in other method class embodiments. For example:

[0068] Based on the content of the above device embodiment, as a preferred embodiment, the neural network model parallel optimization device based on operator block provided in the embodiment of the present invention, the computing power model of the hardware device constructed by the first main module also includes:

[0069] The tensor computing capability p of hardware device i i for: Where k represents the number of computing cores, l represents the completion length of the computing core of hardware device i, and t i It represents the sum of data transmission time and computing time of heterogeneous device i.

[0070] Based on the content of the above device embodiment, as a preferred embodiment, in the neural network model parallel optimization device based on operator blocking provided in the embodiment of the present invention, the first main module is also used to execute the following instructions:

[0071] If the time it takes to compute the core to read input data and write back output data is t i,d Than the time t for calculating logic i,c long, and pipeline arrangement cannot be performed between data reading and writing and tensor calculation, then t i =t i，c +2t i，d ; If pipelining between data reading and writing and tensor computation is supported, then t i =max{t i，c , t i，d}.

[0072] Based on the content of the above device embodiment, as a preferred embodiment, the neural network model parallel optimization device based on operator block provided in the embodiment of the present invention further includes:

[0073] For tensor operators with input tensor length L, the operator blocking strategy is:

[0074] The length L of the tensor assigned to hardware device i i for:

[0075] The number of blocks N allocated to the micro-operator queue on hardware device i i for:

[0076] Based on the content of the above-mentioned device embodiment, as a preferred embodiment, the neural network model parallel optimization device based on operator blocking provided in the embodiment of the present invention, the second main module is also used to execute the following instructions: the weight of each operator node is the shortest delay of the operator, and a weighted directed acyclic graph is constructed according to the shortest delay of the operator.

[0077] Based on the contents of the above-mentioned device embodiment, as a preferred embodiment, the neural network model parallel optimization device based on operator blocking provided in the embodiment of the present invention, the second main module is also used to execute the following instructions: construct an operator execution queue according to the operator node height.

[0078] The method of the embodiment of the present invention is implemented by relying on electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 6 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. At least one processor calls the logic instructions in at least one memory to execute all or part of the steps of the method provided by the aforementioned various method embodiments.

[0079] In addition, when the logic instructions in the above-mentioned at least one memory are implemented in the form of a software functional unit and sold or used as an independent product, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or partly in other words, the part that contributes to the prior art or part of the technical solution is embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (a personal computer, a server, or a network device) to perform all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and various media for storing program codes.

[0080] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0084] In summary, the present invention discloses a neural network model parallel optimization method based on operator partitioning, including a hardware device-perceived operator partitioning process and a weighted operator scheduling process. The hardware device-perceived operator partitioning process includes: firstly, modeling the hardware device computing resources according to the specific hardware device parameters, and then, according to the hardware characteristics of the computing core, partitioning the tensor, and allocating it to multiple computing cores for parallel reasoning. The weighted operator scheduling process includes: firstly, generating a neural network model calculation graph, determining the data dependency of each operator according to the calculation graph, and then determining the execution priority for each operator according to the shortest reasoning delay of the parallelized operator, sorting the operators according to the priority, and generating an optimized operator execution queue. When executing the neural network model reasoning, the present invention introduces the hardware computing power into the operator partitioning, so that the partitioned operator is more in line with the hardware computing characteristics, and optimizes the scheduling strategy of the partitioned operator, so as to perform more efficient reasoning, effectively improving the problem of insufficient utilization of device resources during the existing neural network reasoning, improving the utilization rate of device resources during reasoning, and reducing the reasoning delay.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A parallel optimization method for a neural network model based on operator partitioning, characterized in that: It includes the hardware device-aware operator partitioning process and the weighted operator scheduling process; The operator block process perceived by the hardware device includes: expanding the neural network model and constructing a calculation graph; analyzing the hardware characteristics of each device and establishing a computing capability model of each hardware device accordingly, where the tensor computing capability of hardware device i is for: , where k represents the number of computing cores, Indicates the length of the computational core completed by hardware device i, represents the sum of the data transmission time and computing time of heterogeneous device i; traverse the computational graph and divide the operators into blocks based on the computing capability model of the hardware device. For the tensor operation operator with an input tensor length of L, the operator block strategy is: the tensor length L assigned to the hardware device i i for: , the number of blocks N allocated to the micro-operator queue on hardware device i i for: ; The weighted operator scheduling process includes: updating the calculation graph according to the block scheme of each operator, assigning a weight to each operator node, taking the weight of each operator node as the shortest delay of the operator, and the calculation graph becomes a weighted directed acyclic graph; traversing the calculation graph, calculating the sum of the shortest delays of the operators of the longest path starting from each operator node as the height of each operator node, and taking the height as the priority of the operator node; finding a set of operator nodes with an in-degree of 0, putting the node with the highest priority into the execution queue, and deleting the node and the edge connected to the node in the calculation graph to obtain a finite topological sequence of the operator.

2. The method for parallel optimization of a neural network model based on operator block according to claim 1, characterized in that: If the time it takes to compute the core to read input data and write back output data is t i,d Than the time t for calculating logic i,c If the length of the pipeline is too long and data reading and writing cannot be pipelined with tensor computation, then ; If pipelining between data reading and writing and tensor computation is supported, then .

3. The method for parallel optimization of a neural network model based on operator block according to claim 1, characterized in that: During the weighted operator scheduling process, an operator execution queue is constructed according to the operator node height.

4. A parallel optimization device for a neural network model based on operator block, characterized in that: include: The first main module is used to perform the operator block process perceived by the hardware device, including: unfolding the neural network model and building a calculation graph; analyzing the hardware characteristics of each device and building a computing power model for each hardware device based on this, where the tensor computing power of hardware device i for: , where k represents the number of computing cores, Indicates the length of the computational core completed by hardware device i, represents the sum of the data transmission time and computing time of heterogeneous device i; traverse the computational graph and divide the operators into blocks based on the computing capability model of the hardware device. For the tensor operation operator with an input tensor length of L, the operator block strategy is: the tensor length L assigned to the hardware device i i for: , the number of blocks N allocated to the micro-operator queue on hardware device i i for: ; The second main module is used to execute the weighted operator scheduling process, including: updating the calculation graph according to the block scheme of each operator, assigning weights to each operator node, and taking the weight of each operator node as the shortest delay of the operator, so that the calculation graph becomes a weighted directed acyclic graph; traversing the calculation graph, calculating the sum of the shortest delays of the operators of the longest path starting from each operator node as the height of each operator node, and taking the height as the priority of the operator node; finding a set of operator nodes with an in-degree of 0, putting the node with the highest priority into the execution queue, and deleting the node and the edges connected to the node in the calculation graph to obtain a finite topological sequence of the operator.

5. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method of any one of claims 1 to 3.

Images