Neural network collaborative inference method and device for SoC edge server
By splitting the neural network into operators and dividing the sub-model computation graph into branches in the SoC edge server, the execution order of the operators is determined and allocated to different CPUs, thus solving the problem of wasted computing resources in the SoC edge server and realizing efficient DNN inference.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2023-10-12
- Publication Date
- 2026-07-21
Smart Images
Figure CN117829226B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of neural network technology, and in particular to a neural network collaborative inference method and apparatus for System on Chip (SoC) edge servers. Background Technology
[0002] Deep neural networks (DNNs) are ubiquitous in all aspects of daily life. From large data centers to edge devices such as terminal devices, mobile devices, IoT devices, and wearable devices, DNNs are deployed extensively. The widespread use of DNNs has driven research into how to deploy them on different devices suitable for various applications.
[0003] In DNN research, the deployment of DNN inference tasks is crucial. For computationally intensive DNN model inference scenarios, the DNN model is deployed on a cloud server. The specific implementation process involves: the edge device sending an inference request to the cloud server; the cloud server initiating and executing the DNN model's inference based on the request; and sending the inference result back to the edge device for processing. In cloud servers, because their graphics processing units (GPUs) outperform central processing units (CPUs) in parallel computing, DNN model inference is typically performed on the GPU. For computationally less demanding DNN model inference scenarios, the inference is performed directly on edge devices equipped with hardware such as CPUs, GPUs, or digital signal processors (DSPs), such as mobile devices or terminal devices. It can be seen that implementing DNN model inference on cloud servers and edge devices has become widely adopted.
[0004] The cloud server used for DNN model inference described above is a traditional computer server architecture. However, with the development of computer technology, servers with architectures such as ARM can also be used as edge servers. How to implement DNN inference on servers with ARM architecture requires further research.
[0005] An ARM-based server consists of dozens of ARM-based System-on-a-Chip (SoC) chips used as edge servers in a computer network; this is called an SoC edge server. The SoC chips in an SoC edge server communicate with each other via a Peripheral Component Interconnect (PCI) bus. Each SoC chip has multiple CPUs and at least one GPU. SoC edge servers are frequently used to deploy cloud gaming. However, statistics on processor usage of cloud games running on a single SoC chip show that a cloud game cannot simultaneously utilize CPU resources while using GPUs. Furthermore, cloud games cannot be deployed across SoC chips. Even if different sub-algorithms of a cloud game are pre-implemented on different CPUs, a large number of CPUs remain idle, wasting the computing resources of the SoC edge server.
[0006] Currently, there is no mention of how to deploy and implement DNN inference tasks in SoC edge servers. Therefore, how to effectively utilize the computing resources of each SoC chip in SoC edge servers and achieve DNN inference while minimizing the loss of computing resources is a technical problem that urgently needs to be solved. Summary of the Invention
[0007] In view of this, embodiments of this application provide a neural network collaborative inference method for SoC edge servers, which can make full use of the computing resources of the SoC chip to realize the inference of the visual neural network and reduce the loss of computing resources.
[0008] This application also provides a neural network collaborative inference device for SoC edge servers, which can make full use of the computing resources of the SoC chip to realize neural network inference and reduce the loss of computing resources.
[0009] In one embodiment of this application, a neural network collaborative processing method for a system-on-a-chip (SoC) edge server is provided, comprising:
[0010] Based on preset operator splitting rules and preset sub-model computation graph branching rules, the neural network to be inferred is split into operators and divided into sub-model computation graph branches to obtain multiple sub-models, each of which has multiple operators.
[0011] For each sub-model, the execution order of multiple operators in the sub-model is determined based on the computation graph of the sub-model.
[0012] Based on the execution order of multiple operators in the sub-model, the multiple operators of the sub-model are assigned to different central processing units (CPUs) in different SoC chips of the SoC edge server, so that different CPUs in different SoC chips can calculate different operators in the sub-model and complete the inference of the neural network.
[0013] In the above method, the preset operator splitting rules include:
[0014] The classification type of each operator in the neural network is determined according to the preset operator classification type;
[0015] When splitting the operators in the neural network, the splitting method corresponding to the classification type of the operator is used;
[0016] The operator classification types include: Type I, Type II and Type III. Type I operators are operators that are computed on a single input tensor. Type II operators are operators that are computed on an input tensor plane determined by the height and width dimensions. Type III operators are computed operators that are computed on the input tensor plane where the height and width dimensions are both 1.
[0017] The Type I operator uses a splitting method that splits along the height dimension of the input tensor plane;
[0018] The Type II operator uses a splitting method that splits along the height and width dimensions of the input tensor plane;
[0019] The type III operator does not perform any splitting.
[0020] In the above method, the preset sub-model computation graph branching rules include:
[0021] Type I operators are branched using backtracking multi-operator branching, where a Type I operator represents an operator computed on a single input tensor. Backtracking multi-operator branching involves branching multiple connected Type I operators as a single operator to obtain one operator in the sub-model.
[0022] The Type II operator is branched using a single operator branch, where the Type II operator represents an operator computed on the input tensor plane determined by the height and width dimensions. The single operator branch is a branch for the Type II operator transmitting data, and the Type II operator transmitting data is taken as an operator in the sub-model.
[0023] In the above method, determining the execution order of multiple operators in the sub-model based on the computation graph of the sub-model includes:
[0024] The computational graph of the sub-models is divided based on the merging operators of the multiple sub-models.
[0025] The computation graphs of the multiple sub-models are grouped based on their shared input operators;
[0026] For the computation graph of each sub-model, three types of operation operators are divided.
[0027] The execution order of operators in the sub-model is determined based on the execution order of the operators in the computation graph of the sub-model.
[0028] In the above method, after grouping, the method further includes:
[0029] The sub-models from different groups are assigned to run on different SoC chips of the SoC edge server;
[0030] Different operators in different sub-models within the same group are assigned to different CPUs in the corresponding SoC or to the same CPU for execution, according to the execution order.
[0031] In the above method, the three types of operation operators include:
[0032] Sending operators, merging operation type operators, and other types of operators;
[0033] Determining the execution order of operators in the sub-model based on the execution order of the operators in the computation graph of the sub-model includes:
[0034] First, the send operation operator in the sub-model is executed, then other types of operators in the sub-model are executed, and finally the merge operation type operator in the sub-model is executed.
[0035] In the above method, the method further includes:
[0036] Configure the global planner;
[0037] The global planner performs operator decomposition and sub-model computation graph branching on the neural network to be reasoned, resulting in multiple sub-models;
[0038] For each sub-model, the global planner determines the execution order of operators in the sub-model based on the set computation graph of the sub-model;
[0039] The global planner allocates the multiple operators of the sub-model to different CPUs in different SoC chips of the SoC edge server according to the execution order of the multiple operators in the sub-model.
[0040] In another embodiment of this application, a neural network collaborative processing system for SoC edge servers is provided, comprising: a partitioning unit, an execution order determination unit, and a control operation unit, wherein...
[0041] The partitioning unit is used to perform operator partitioning and sub-model computation graph branching on the neural network to be reasoned based on preset operator partitioning rules and preset sub-model computation graph branching rules, to obtain multiple sub-models, each of which has multiple operators.
[0042] The execution order determination unit is used to determine the execution order of multiple operators in each sub-model based on the set computation graph of the sub-model.
[0043] The control and operation unit is used to allocate the multiple operators of the sub-model to different CPUs in different SoC chips of the SoC edge server according to the execution order of the multiple operators in the sub-model, so that the different CPUs in the different SoC chips can calculate the different operators in the sub-model and complete the inference of the neural network.
[0044] In another embodiment of the present invention, a non-transitory computer-readable storage medium is provided, characterized in that the non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to perform the various steps of the above-described neural network co-processing method for a SoC edge server.
[0045] In another embodiment of this application, an electronic device is provided, including a processor for performing the steps of the above-described neural network co-processing method for SoC edge servers.
[0046] As seen above, this embodiment of the application, based on preset operator splitting rules and preset sub-model computation graph branching rules, splits the neural network to be reasoned into operators and divides the sub-model computation graph branches to obtain multiple sub-models, each of which contains multiple operators. For each sub-model, based on the set computation graph of the sub-model, the execution order of the multiple operators in the sub-model is determined. According to the execution order of the multiple operators in the sub-model, the multiple operators of the sub-model are allocated to different CPUs in different SoC chips of the SoC edge server, so that the different CPUs in the different SoC chips can compute the different operators in the sub-model to complete the inference of the neural network. In this way, by organically splitting the operators in the neural network to be reasoned and allocating them to different processors in different SoC chips of the SoC edge server according to their execution order, the computing resources of the SoC chips of the SoC edge server are fully utilized, and the loss of computing resources is reduced. Attached Figure Description
[0047] Figure 1 A flowchart of a neural network collaborative inference method for SoC edge servers is provided in this application embodiment;
[0048] Figure 2 A schematic diagram of a neural network collaborative inference method framework for SoC edge servers provided in this application embodiment;
[0049] Figure 3 This is a schematic diagram illustrating a neural network collaborative reasoning method for SoC edge servers implemented using a global planner, as provided in an embodiment of this application.
[0050] Figure 4 This is a schematic diagram of a neural network collaborative inference device for SoC edge servers provided in an embodiment of this application.
[0051] Figure 5 This is a schematic diagram of an electronic device provided for another embodiment of this application. Detailed Implementation
[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0053] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0054] The technical solution of this application will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0055] As can be seen from the background technology, there is no mention of how to deploy and implement the inference task of the neural network model on the SoC edge server. If the same deployment method as cloud gaming is directly adopted to deploy the inference task of the neural network model, some CPUs in the SoC edge server will be idle, wasting the computing resources in the SoC edge server.
[0056] To address the aforementioned issues, this application proposes a solution for deploying neural network model inference tasks on SoC edge servers. The specific process is as follows: Based on preset operator splitting rules and preset sub-model computation graph branching rules, the neural network to be inferred is split into operators and its computation graph is divided into branches to obtain multiple sub-models, each containing multiple operators. For each sub-model, based on the set computation graph of the sub-model, the execution order of the multiple operators in the sub-model is determined. According to the execution order of the multiple operators in the sub-model, the multiple operators of the sub-model are allocated to different CPUs in different SoC chips of the SoC edge server, so that different CPUs in different SoC chips can compute different operators in the sub-model to complete the inference of the neural network.
[0057] In this way, by organically splitting the operators in the neural network to be reasoned and distributing them to different processors in different SoC chips in the SoC edge server according to their execution order, the computing resources of the SoC chips in the SoC edge server are fully utilized and the loss of computing resources is reduced.
[0058] In this embodiment of the application, the neural network in the neural network collaborative inference of the SoC edge server is specifically a DNN.
[0059] Figure 1 A flowchart of a neural network collaborative inference method for SoC edge servers provided in this application embodiment includes the following specific steps:
[0060] Step 101: Based on the preset operator splitting rules and the preset sub-model computation graph branching rules, the neural network to be inferred is split into operators and the sub-model computation graph branches are divided to obtain multiple sub-models, each of which has multiple operators.
[0061] Step 102: For each sub-model, based on the set computation graph of the sub-model, determine the execution order of multiple operators in the sub-model;
[0062] Step 103: According to the execution order of the multiple operators in the sub-model, the multiple operators of the sub-model are assigned to different CPUs in different SoC chips of the SoC edge server, so that the different CPUs in the different SoC chips can calculate the different operators in the sub-model and complete the inference of the neural network.
[0063] As can be seen from the above scheme, this application first divides the neural network to be processed into multiple sub-models. The multiple sub-models can be assigned to different SoC chips or the same SoC chip for inference. Then, for each sub-model, the execution order of multiple operators in the entire neural network to be processed is also determined. During specific inference, the multiple operators of the sub-model can be assigned to different CPUs or the same CPU in the corresponding SoC chip according to the execution order, so that the CPUs assigned to the operators can collaboratively infer the sub-model according to the execution order to complete the inference of the entire neural network.
[0064] In this embodiment of the application, two rules are used in combination when dividing the sub-model: operator splitting rule and sub-model computation graph branching rule, which are used for operator splitting and computation graph branching, respectively.
[0065] The preset operator splitting rules include:
[0066] The classification type of each operator in the neural network is determined according to the preset operator classification type;
[0067] When splitting the operators in the neural network, the splitting method corresponding to the classification type of the operator is used;
[0068] The operator classification types include: Type I, Type II and Type III. Type I operators are operators that are computed on a single input tensor. Type II operators are operators that are computed on an input tensor plane determined by the height and width dimensions. Type III operators are computed operators that are computed on the input tensor plane where the height and width dimensions are both 1.
[0069] The Type I operator uses a splitting method that splits along the height dimension of the input tensor plane;
[0070] The Type II operator uses a splitting method that splits along the height and width dimensions of the input tensor plane;
[0071] The type III operator does not perform any splitting.
[0072] Specifically, the operator splitting rule divides the neural network to be inferred into several sub-models of similar size. Inference of the neural network on the SoC edge server involves operator computation and data communication between operators. In operator splitting in the neural network, the operator splitting rule that minimizes operator latency is preferred. Since the input and output of the operator are three-dimensional tensors, the operator can be split along three dimensions: height (H), width (W), and channel (C). This application classifies operators and selects a corresponding splitting method for each category. There are three types of operator classifications: Type I operators: each computation of the operator is performed on a single input tensor unit; Type II operators: each computation of the operator is performed on multiple units of the input tensor plane determined by the H and W dimensions; Type III operators: are fully connected operators, where the size of the input tensor in both the H and W dimensions is 1. Based on the operator classification, the corresponding splitting method is selected: Type I operators are split along the H dimension; Type II operators are split along the H and C dimensions, which can reduce the latency of collaborative inference of SoC edge servers; Type III operators are not split.
[0073] In this embodiment of the application, the preset sub-model computation graph branching rules include:
[0074] Type I operators are branched using backtracking multi-operator branching, where a Type I operator represents an operator computed on a single input tensor. Backtracking multi-operator branching involves branching multiple connected Type I operators as a single operator to obtain one operator in the sub-model.
[0075] The Type II operator is branched using a single operator branch, where the Type II operator represents an operator computed on the input tensor plane determined by the height and width dimensions. The single operator branch is a branch for the Type II operator transmitting data, and the Type II operator transmitting data is taken as an operator in the sub-model.
[0076] Specifically, the serial execution of operator computation processes and data communication processes between operators involved in neural network inference on the SoC edge server can lead to CPU idle waiting. Branching can solve the CPU idle problem; therefore, branching is applied to operators within sub-models. The most basic branching rule is single-operator branching, meaning branching is performed on operators that send communication data. Furthermore, since there is no data communication process between sub-operators of multiple Type I operators connected in the computation graph of each sub-model, the connected subgraphs of these multiple Type I operators can be considered as a single operator, resulting in backtracking multi-operator branching. Therefore, the preset sub-model computation graph branching rule is: Type I operators use backtracking multi-operator branching; Type II operators use single-operator branching.
[0077] In this embodiment of the application, after the neural network to be reasoned is divided into multiple sub-models according to the above two rules, and each sub-model is divided into computation graph branches to obtain multiple operators of the sub-model, it is also necessary to determine the pipeline scheme of the multiple operators in the sub-model, that is, the execution order.
[0078] Specifically, determining the execution order of multiple operators in the sub-model based on the computation graph of the sub-model includes:
[0079] The computational graph of the sub-models is divided based on the merging operators of the multiple sub-models.
[0080] The computation graphs of the multiple sub-models are grouped based on their shared input operators;
[0081] For the computation graph of each sub-model, three types of operation operators are divided.
[0082] The execution order of operators in the sub-model is determined based on the execution order of the operators in the computation graph of the sub-model.
[0083] It can be seen that the pipeline scheme for determining the computation graph branches in the sub-model is to enable the CPUs in the SoC edge server to collaboratively infer different computation graph branches in the sub-model, so as to minimize CPU idle time.
[0084] In the above scheme, after grouping based on the shared input operators, it also includes:
[0085] The sub-models from different groups are assigned to run on different SoCs of the SoC edge server;
[0086] Different operators in different sub-models within the same group are assigned to different CPUs in the corresponding SoC or to the same CPU for execution, according to the execution order.
[0087] In the above scheme, the three types of operation operators include:
[0088] Sending operators, merging operation type operators, and other types of operators;
[0089] Determining the execution order of operators in the sub-model based on the execution order of the operators in the computation graph of the sub-model includes:
[0090] First, the send operation operator in the sub-model is executed, then other types of operators in the sub-model are executed, and finally the merge operation type operator in the sub-model is executed.
[0091] In this embodiment of the application, the method further includes:
[0092] Configure the global planner;
[0093] The global planner performs operator decomposition and sub-model computation graph branching on the neural network to be reasoned, resulting in multiple sub-models;
[0094] For each sub-model, the global planner determines the execution order of operators in the sub-model based on the computation graph of the sub-model.
[0095] The global planner allocates the operators of the sub-model to different CPUs in different SoCs of the SoC edge server according to the execution order of the operators in the sub-model.
[0096] In other words, this application embodiment sets up a global planner to obtain sub-models by combining two preset rules. The global planner uses latency prediction and CPU reallocation techniques. For Type II operators in the neural network to be inferred, a method to reduce collaborative inference latency is selected, which can be achieved by using preset rules. In the global planner, CPU reallocation of execution operators is used to ensure that each operator achieves the optimal computational effect during inference. The core of the global planner is the computation graph and the computation of execution operators, and the computation follows the following rules: (1) Type I operators should choose the method of splitting and backtracking multiple operation branches on the H dimension. (2) Type II operators should choose the method with the minimum inference latency among the four options consisting of splitting on the H dimension and C dimension, and whether to use a single operator branch. (3) Type III operators should not be split.
[0097] The input to the global planner is the number of SoC chips allowed in the SOC edge server and the CPU resource limit. The output is the sub-model of the neural network to be processed and the execution order of each operator.
[0098] In the actual deployment of neural network inference tasks in the SOC edge server, the SOC edge server performs inference according to the divided sub-models and the pipeline execution order of each operator. This process only needs to be performed once during deployment and does not need to be repeated in subsequent deployments.
[0099] The embodiments of this application will be described in detail with examples.
[0100] Figure 2 This diagram illustrates a neural network collaborative inference method framework for SoC edge servers, as provided in an embodiment of this application. The diagram shows a neural network to be inferred. Based on preset operator splitting rules and preset sub-model computation graph branching rules, the neural network is split into operators and sub-model computation graph branches to obtain multiple sub-models. Each sub-model has multiple operators, such as... Figure 2 The subgraph shown in the upper left corner of the image.
[0101] For each sub-model in the neural network, the computational graph of the sub-model is partitioned, specifically including: partitioning the computational graph of the sub-model based on the merging operators possessed by multiple sub-models, such as... Figure 2 As shown in the upper right corner, the computational graph of multiple sub-models is obtained.
[0102] The computational graphs of the multiple sub-models are grouped based on their shared input operators, such as... Figure 2 As shown in the bottom right corner.
[0103] For the computation graph of each sub-model in each group, three types of operational operators are divided, such as... Figure 2 The table on the left is shown in the middle.
[0104] Based on the execution order of the operators in the computation graph of the sub-model, determine the execution order of the operators in the sub-model, such as... Figure 2 As shown in the bottom left corner.
[0105] In this way, the execution order of the multiple sub-models in the neural network and the multiple operators in the sub-models is determined.
[0106] Figure 3 This diagram illustrates a method for collaborative inference of neural networks on a SoC edge server using a global planner, as provided in this embodiment of the application. As shown, the number of SoC chips and CPUs allowed in the SoC edge server, along with the neural network to be inferred, are input into the global planner for processing. The global planner employs time prediction and operator kernel reallocation techniques, setting operator splitting rules and sub-model computation graph branching rules. It splits the neural network to be inferred into operators and branches the sub-model computation graph, resulting in multiple sub-models, each containing multiple operators. Then, for each sub-model, the global planner determines the execution order of the multiple operators within that sub-model based on the set computation graph. Finally, according to the execution order of the multiple operators in the sub-model, the global planner allocates the multiple operators of the sub-model to different CPUs in different SoC chips of the SoC edge server, enabling different CPUs in different SoC chips to compute the different operators in the sub-model and complete the inference of the neural network.
[0107] Figure 4 A schematic diagram of a neural network collaborative inference device for SoC edge servers provided in this application embodiment includes: a partitioning unit, an execution order determination unit, and a control operation unit, wherein...
[0108] The partitioning unit is used to perform operator partitioning and sub-model computation graph branching on the neural network to be reasoned based on preset operator partitioning rules and preset sub-model computation graph branching rules, to obtain multiple sub-models, each of which has multiple operators.
[0109] The execution order determination unit is used to determine the execution order of multiple operators in each sub-model based on the set computation graph of the sub-model.
[0110] The control and operation unit is used to allocate the multiple operators of the sub-model to different CPUs in different SoC chips of the SoC edge server according to the execution order of the multiple operators in the sub-model, so that the different CPUs in the different SoC chips can calculate the different operators in the sub-model and complete the inference of the neural network.
[0111] In another embodiment of this application, a non-transitory computer-readable storage medium is provided, which stores instructions that, when executed by a processor, cause the processor to execute a neural network co-processing method for a SoC edge server as described in the foregoing embodiments.
[0112] Figure 5 This is a schematic diagram of an electronic device provided for another embodiment of this application. (See diagram below.) Figure 5 As shown, another embodiment of this application also provides an electronic device that may include a processor 501, wherein the processor 501 is used to execute the steps of the above-described neural network co-processing method for a SoC edge server. From Figure 5 It can also be seen that the electronic device provided in the above embodiments further includes a non-transitory computer-readable storage medium 502, on which a computer program is stored. The computer program is executed by the processor 501 to perform the steps of the above-described neural network co-processing method for SoC edge servers.
[0113] Specifically, the non-transitory computer-readable storage medium 602 can be a general-purpose storage medium, such as a mobile disk, hard disk, FLASH, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or portable compact disk read-only memory (CD-ROM), etc. When the computer program on the non-transitory computer-readable storage medium 602 is run by the processor 501, it can trigger the processor 501 to execute the various steps of the above-described neural network co-processing method for SoC edge servers.
[0114] In practical applications, the non-transitory computer-readable storage medium 502 may be included in the device / apparatus / system described in the above embodiments, or it may exist independently without being assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, can perform the various steps of the aforementioned neural network co-processing method for SoC edge servers.
[0115] Another embodiment of this application provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described neural network co-processing method for a SoC edge server.
[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments disclosed in this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings. For example, two blocks shown connectedly may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0117] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, and all such combinations and / or combinations fall within the scope of this application.
[0118] This document uses specific embodiments to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application, and are not intended to limit this application. For those skilled in the art, changes can be made to the specific implementation methods and application scope based on the ideas, spirit and principles of this application. Any modifications, equivalent substitutions, improvements, etc., made should be included within the scope of protection of this application.
Claims
1. A neural network collaborative processing method for edge servers of a System-on-Chip (SoC), characterized in that, include: Based on preset operator splitting rules and preset sub-model computation graph branching rules, the neural network to be reasoned is split into operators and sub-model computation graph branches to obtain multiple sub-models, each of which has multiple operators. For each sub-model, the execution order of multiple operators in the sub-model is determined based on the computation graph of the sub-model. According to the execution order of multiple operators in the sub-model, the multiple operators of the sub-model are assigned to different central processing units (CPUs) in different SoC chips of the SoC edge server, so that different CPUs in different SoC chips can calculate different operators in the sub-model and complete the inference of the neural network. in, The preset operator splitting rules include: The classification type of each operator in the neural network is determined according to the preset operator classification type; When splitting the operators in the neural network, the splitting method corresponding to the classification type of the operator is used; The operator classification types include: Type I, Type II and Type III. Type I operators are operators that are computed on a single input tensor. Type II operators are operators that are computed on an input tensor plane determined by the height and width dimensions. Type III operators are computed operators that are computed on the input tensor plane where the height and width dimensions are both 1. The Type I operator uses a splitting method that splits along the height dimension of the input tensor plane; The Type II operator uses a splitting method that splits along the height and width dimensions of the input tensor plane; The type III operator does not perform any partitioning. The preset sub-model computation graph branching rules include: Type I operators are branched using backtracking multi-operator branching, where a Type I operator represents an operator computed on a single input tensor. Backtracking multi-operator branching involves branching multiple connected Type I operators as a single operator to obtain one operator in the sub-model. The Type II operator is branched using a single operator branch, where the Type II operator represents an operator computed on the input tensor plane determined by the height and width dimensions. The single operator branch is a branch for the Type II operator transmitting data, and the Type II operator transmitting data is taken as an operator in the sub-model. The determination of the execution order of multiple operators in the sub-model based on the computation graph of the sub-model includes: The computational graph of the sub-models is divided based on the merging operators of the multiple sub-models. The computation graphs of the multiple sub-models are grouped based on their shared input operators; For the computation graph of each sub-model, three types of operation operators are divided. The execution order of operators in the sub-model is determined based on the execution order of the operators in the computation graph of the sub-model.
2. The method as described in claim 1, characterized in that, After grouping, the process also includes: The sub-models from different groups are assigned to run on different SoC chips of the SoC edge server; Different operators in different sub-models within the same group are assigned to different CPUs in the corresponding SoC or to the same CPU for execution, according to the execution order.
3. The method as described in claim 1, characterized in that, The three types of operation operators include: Sending operators, merging operation type operators, and other types of operators; Determining the execution order of operators in the sub-model based on the execution order of the operators in the computation graph of the sub-model includes: First, the send operation operator in the sub-model is executed, then other types of operators in the sub-model are executed, and finally the merge operation type operator in the sub-model is executed.
4. The method as described in claim 1, characterized in that, The method further includes: Configure the global planner; The global planner performs operator decomposition and sub-model computation graph branching on the neural network to be reasoned, resulting in multiple sub-models; For each sub-model, the global planner determines the execution order of operators in the sub-model based on the set computation graph of the sub-model; The global planner allocates the multiple operators of the sub-model to different CPUs in different SoC chips of the SoC edge server according to the execution order of the multiple operators in the sub-model.
5. A neural network collaborative processing system for SoC edge servers, characterized in that, include: The unit is divided into a division unit, an execution sequence determination unit, and a control operation unit, among which, The partitioning unit is used to perform operator partitioning and sub-model computation graph branching on the neural network to be reasoned based on preset operator partitioning rules and preset sub-model computation graph branching rules, to obtain multiple sub-models, each of which has multiple operators. The execution order determination unit is used to determine the execution order of multiple operators in each sub-model based on the set computation graph of the sub-model. The control and operation unit is used to allocate the multiple operators of the sub-model to different CPUs in different SoC chips of the SoC edge server according to the execution order of the multiple operators in the sub-model, so that the different CPUs in the different SoC chips can calculate the different operators in the sub-model and complete the inference of the neural network. in, The preset operator splitting rules include: The classification type of each operator in the neural network is determined according to the preset operator classification type; When splitting the operators in the neural network, the splitting method corresponding to the classification type of the operator is used; The operator classification types include: Type I, Type II and Type III. Type I operators are operators that are computed on a single input tensor. Type II operators are operators that are computed on an input tensor plane determined by the height and width dimensions. Type III operators are computed operators that are computed on the input tensor plane where the height and width dimensions are both 1. The Type I operator uses a splitting method that splits along the height dimension of the input tensor plane; The Type II operator uses a splitting method that splits along the height and width dimensions of the input tensor plane; The type III operator does not perform any partitioning. The preset sub-model computation graph branching rules include: Type I operators are branched using backtracking multi-operator branching, where a Type I operator represents an operator computed on a single input tensor. Backtracking multi-operator branching involves branching multiple connected Type I operators as a single operator to obtain one operator in the sub-model. The Type II operator is branched using a single operator branch, where the Type II operator represents an operator computed on the input tensor plane determined by the height and width dimensions. The single operator branch is a branch for the Type II operator transmitting data, and the Type II operator transmitting data is taken as an operator in the sub-model. The determination of the execution order of multiple operators in the sub-model based on the computation graph of the sub-model includes: The computational graph of the sub-models is divided based on the merging operators of the multiple sub-models. The computation graphs of the multiple sub-models are grouped based on their shared input operators; For the computation graph of each sub-model, three types of operation operators are divided. The execution order of operators in the sub-model is determined based on the execution order of the operators in the computation graph of the sub-model.
6. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to perform various steps in a neural network co-processing method for a SoC edge server as described in any one of claims 1 to 4.
7. An electronic device, characterized in that, Includes a processor for performing various steps in a neural network co-processing method for a SoC edge server as described in any one of claims 1 to 4.