Neural network model processing method, device and electronic device

Abstract

The embodiments of the present application disclose a neural network model processing method, device, and electronic device. The method includes: obtaining a neural network model to be configured; mapping the neural network model into a graph structure based on the dependency relationship of operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model; traversing the graph structure to obtain the execution order of the nodes; and configuring the execution order of the operators represented by the nodes based on the execution order of the nodes. This method converts the neural network model into a graph structure, and then determines the execution order of the operators represented by each node by traversing the nodes in the graph structure, thereby realizing the rapid determination of the execution order of all operators and improving the computational efficiency of the entire model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a neural network model processing method, device, and electronic device. Background Art

[0002] Neural network models are typically trained on devices such as computers. To facilitate the execution of trained neural network models on electronic devices such as mobile phones and tablets, these models can be optimized. However, this optimization process can disrupt the execution order of the neural network model's operators. Summary of the Invention

[0003] In view of the above problems, the present application proposes a neural network model processing method, device and electronic device to improve the above problems.

[0004] In a first aspect, the present application provides a neural network model processing method, which is applied to an electronic device, and the method includes: obtaining a neural network model to be configured; mapping the neural network model into a graph structure based on the dependency relationship of operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model; traversing the graph structure to obtain the execution order of the nodes; and configuring the execution order of the operators represented by the nodes based on the execution order of the nodes.

[0005] In a second aspect, the present application provides a neural network model processing device that runs on an electronic device, and the method includes: a model acquisition unit for acquiring the neural network model to be configured; a model processing unit for mapping the neural network model into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model; a traversal unit for traversing the graph structure to obtain the execution order of the nodes; and a sequence determination unit for configuring the execution order of the operators represented by the nodes based on the execution order of the nodes.

[0006] In a fourth aspect, the present application provides an electronic device comprising a multi-core processor, a boot controller, and a memory, wherein the memory is used to store data to be loaded; one or more programs are stored in the boot controller and configured to be executed by the boot controller to implement the above method.

[0007] In a fifth aspect, the present application provides a computer-readable storage medium, in which program code is stored, wherein the above method is executed when the program code is started and the controller is run.

[0008] The present application provides a neural network model processing method, device, and electronic device. After obtaining the neural network model to be configured, the neural network model is mapped into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model. Then, the graph structure is traversed to obtain the execution order of the nodes, and the execution order of the operators represented by the nodes is configured based on the execution order of the nodes. Thus, by converting the neural network model into a graph structure and then determining the execution order of the operators represented by each node by traversing the nodes in the graph structure, it is possible to quickly determine the execution order of all operators and improve the computational efficiency of the entire model. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flowchart of a neural network model processing method proposed in an embodiment of the present application is shown;

[0011] Figure 2 A schematic diagram corresponding to a neural network model in a neural network model processing method proposed in one embodiment of the present application is shown;

[0012] Figure 3 A schematic diagram showing the mapping relationship between processing resource consumption and time in a neural network model processing method proposed in one embodiment of the present application is shown;

[0013] Figure 4 A flowchart of a neural network model processing method proposed in another embodiment of the present application is shown;

[0014] Figure 5 A flowchart of a neural network model processing method proposed in another embodiment of the present application is shown;

[0015] Figure 6 A structural block diagram of a neural network model processing device proposed in an embodiment of the present application is shown;

[0016] Figure 7 A structural block diagram of a neural network model processing device proposed in another embodiment of the present application is shown;

[0017] Figure 8 A structural block diagram of a neural network model processing device proposed in another embodiment of the present application is shown;

[0018] Figure 9 A structural block diagram of an electronic device for executing a neural network model processing method according to an embodiment of the present application is shown;

[0019] Figure 10 It is a storage unit for storing or carrying program codes for implementing the neural network model processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] Neural networks (NNs) are complex network systems formed by extensively interconnecting a large number of simple processing units (called neurons). NNs possess large-scale parallelism, distributed storage and processing, self-organization, self-adaptation, and self-learning capabilities. Neural network models typically include a large number of operators. Operators can be considered as part of the algorithmic process within a neural network model, mapping functions to functions or vice versa.

[0022] However, the inventors found in their research that when a neural network has many operators or is highly complex, the overall operator execution order cannot be determined quickly.

[0023] Furthermore, the operator order in neural networks is often based on the operator order pre-trained by the server-side neural network framework. For example, in neural network models trained using frameworks like TensorFlow, when adding operators to a trained neural network model, they can only be added between specific operators in the existing network graph. This requires regenerating the neural network graph during operator optimization, significantly slowing down model conversion. This is because each operator in the neural network graph has dependencies. This means that for an operator to execute correctly, the operators it depends on must execute completely. Therefore, operators have an execution order, and models built using PC-based frameworks like TensorFlow generally have a pre-defined operator order. For example, if operator B depends on operator A, then operator A will be ordered before B in the file (which stores the neural network model). If operator C needs to be added between A and B, the typical approach is to regenerate the network, copying operators in order up to operator A, then adding operator C, then copying operator B, and then copying subsequent operators. In other words, C must be added between A and B to ensure correct operator execution. This makes adding operators a very complex operation, which will take a long time to convert the neural network model, and thus cause the model conversion speed to slow down a lot.

[0024] Furthermore, the deployment of neural network models on mobile devices typically involves solidifying the trained model into a file on a PC or other computer. The mobile neural network framework then parses the file, reads it into memory, and sequentially executes the operators in the neural network model. However, due to limited resources on mobile devices, only small models can be run. Large models require a series of optimizations, such as operator fusion, network pruning, model quantization, and network segmentation, to optimize the model for mobile operation. These optimizations often change the original operator execution order, necessitating an effective method to reorder the operators to ensure proper execution of the neural network model.

[0025] Therefore, the inventors proposed a neural network model processing method, device and electronic device in this application that can improve the above-mentioned problems.

[0026] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0027] See also Figure 1 , an embodiment of the present application provides a neural network model processing method, which is applied to an electronic device, and the method includes:

[0028] Step S110: Obtain the neural network model to be configured.

[0029] As one approach, the neural network model to be configured can be a neural network model directly obtained from the network end. Furthermore, the neural network model to be configured can also be a neural network model that has been optimized from the neural network model obtained from the network end. Optimizing the neural network model can be understood as performing operations such as operator fusion, network pruning, model quantization, and network segmentation on the neural network model.

[0030] Step S120: Based on the dependency relationship of operators in the neural network model, the neural network model is mapped into a graph structure, wherein a node in the graph structure represents an operator in the neural network model.

[0031] A graph is a mathematical object that represents relationships between objects. If each edge in a graph is given a direction, the resulting graph is called a directed graph. In a directed graph, edges associated with a node are either outgoing or incoming. Conversely, a graph with undirected edges is called an undirected graph.

[0032] As a method, the graph structure described in this embodiment is a directed acyclic graph (DAG). Generally speaking, a directed graph without a cycle is called a DAG. DAGs are actually data structures, just like arrays, permutations, and blockchains. However, unlike blockchains, DAGs replace the longest chain consensus with the heaviest chain consensus. In traditional blockchains, newly released blocks are added to the original longest chain, and the chain that all nodes consider the longest is used as the standard, and the chain spreads indefinitely. In a DAG, however, each newly added unit is not just added to a block in the long chain, but to all previous blocks.

[0033] It's understandable that a neural network model is composed of many operators. Some operators have dependencies on each other. These dependencies can be understood as data dependencies. For example, a neural network model contains operators A, B, C, D, E, F, G, and H. The neural network model defines that the input data for operator C is the output data for operator B, and the input data for operator B is the output data for operator A. Operator A only outputs data. Therefore, it follows that the computation of operator D requires the output data of operator B, so operator D is dependent on operator B. Similarly, the computation of operator B requires the output data of operator A, so operator B is dependent on operator A.

[0034] Furthermore, the output data of operator C is the output data of operator F, the input data of operator E is the output data of operator A, and the input data of operator F comes from operator C as well as operator B. The input data of operator G comes from operators B, C, and E, and the input data of operator H comes from operators D and F. Based on the above explanation of dependency relationships, operator F depends on both operators B and C, operator E depends on operator A, operator G depends on operators B, C, and E, and operator H depends on both operators D and F.

[0035] Furthermore, based on the above dependencies, we can conclude that Figure 2 The phased acyclic graph shown in Figure 1 contains nodes 0, 1, 2, 3, 4, 5, 6, and 7. Node 0 represents operator A, node 1 represents operator B, node 2 represents operator C, node 3 represents operator D, node 4 represents operator E, node 5 represents operator F, node 6 represents operator G, and node 7 represents operator H. The node to which the arrow points depends on the starting node of the arrow.

[0036] Then in this way, first, the electronic device traverses the graph structure to obtain the execution order of the nodes, including: performing a depth-first traversal on the directed acyclic graph to obtain the execution order of the nodes.

[0037] Step S130: traverse the graph structure to obtain the execution order of the nodes.

[0038] Among them, Figure 2 As shown in FIG, although the execution order between each two adjacent nodes is relatively clear, if the nodes that do not have an adjacent relationship are taken into account, the overall execution order may be wrong. Figure 2 In the example of nodes 1, 2, and 5, it's clear that node 1 executes before node 5, and that node 2 executes before node 5. However, if node 5 needs to be executed after node 1, node 5 will run before node 2. However, the operator corresponding to node 5 may depend on the output data of the operator corresponding to node 2 during execution, which can cause data errors. As a method, electronic devices can traverse each node in the graph structure to determine the overall execution order of each node.

[0039] As one approach, the directed acyclic graph is subjected to a depth-first traversal to obtain the execution order of the nodes.

[0040] It's important to note that graph node traversal methods include depth-first search and breadth-first search. Depth-first search, abbreviated as DFS, is a graph algorithm that searches every possible branch path until it can't go any deeper, traversing each node only once.

[0041] In the specific traversal process, because there are two vertices that depend on node 0, node 1, and node 3, one of the vertices is selected here (it doesn't matter which one is selected). Here, node 1 is selected as the node to continue searching. After finding node 1, you need to find the vertex that depends on node 1. From the figure, you can see that nodes 3, 5, and 6 all depend on node 1. Here, node 3 is selected as the next node to search. Then continue to look for the next node that depends on node 3. Since only node 7 depends on node 3, node 7 is selected. Since no node depends on node 7, node 7 is selected as the first target node. Since no node that depends on node 7 can be found, the next step is to return to the previous node 3. Since the only node that depends on node 3 is node 7, and node 7 has been traversed, it is impossible to find the node that depends on node 3 that has not been traversed, so node 3 is also selected as the target node. The target nodes at this time include node 7 and node 3 in sequence.

[0042] Next, we select node 1, the node immediately preceding node 3, and search for nodes that depend on node 1 but haven't been traversed yet. Nodes 5 and 6 remain, so we select node 5. After finding node 5, we continue searching for nodes that depend on node 5. The only node that depends on node 5 is node 7, which has already been traversed. Therefore, we mark node 5 as the target node. The target nodes now include node 7, node 3, and node 5, in that order.

[0043] Returning to node 5's previous node, 1, we continue searching for nodes that depend on node 1. Only node 6 remains. Since no nodes depend on node 6, we mark node 6 as the target node. The target nodes now include, in order, nodes 7, 3, 5, and 6. Returning to node 6's previous node, node 6, we mark node 1 as the target node. Since no nodes depend on it and it has not been traversed, the target nodes now include, in order, nodes 7, 3, 5, 6, and 1. Returning to node 1's previous node, node 0, we find node 4 that depends on it. Since no untraversed vertices depend on node 4, we mark node 4 as the target node. The target nodes now include, in order, nodes 7, 3, 5, 6, 1, and 4. Returning to node 4's previous node, node 0, we mark node 0 as the target node since no nodes depend on it. The target nodes now include, in order, nodes 7, 3, 5, 6, 1, 4, and 0. Since we have returned to the input node, node 0, we need to start traversing from another input node, node 2. Since all nodes that depend on node 2 have been traversed, node 2 is directly marked as the target node.

[0044] Then based on the marking order of the target nodes mentioned above, the execution order of each node can be obtained, that is, the execution order is node 2, node 0, node 4, node 1, node 6, node 5, node 3 and node 7. Among them, in order to facilitate the electronic device to quickly identify the execution order of the marked target nodes, the storage space of the stack structure can be used to store the marked target nodes. Then in this way, during the traversal process, the electronic device performs a depth-first traversal on the directed acyclic graph, and stores the target nodes obtained during the traversal process into the storage space of the stack structure in sequence. The target node is a node without subordinate nodes or a node whose subordinate nodes have all been traversed; the order in which the nodes in the storage space are popped out of the stack is used as the execution order of the nodes. Based on Figure 2 In the example shown in , the order of target nodes stored in the storage space of the stack structure is node 7, node 3, node 5, node 6, node 1, node 4, node 0, and node 2. The corresponding order of popping the stack is node 2, node 0, node 4, node 1, node 6, node 5, node 3, and node 7.

[0045] It is understood that the electronic device can identify a node that does not depend on other nodes as an input vertex. Then, if the electronic device identifies multiple input vertices, the electronic device can perform a depth-first traversal of the directed acyclic graph based on the specified starting input vertex to obtain the execution order of the nodes.

[0046] It is understandable that in the case of depth-first traversal, the input vertices traversed first will be executed later. For example, in the above example, the traversal starts from input vertex 0 (node ​​0) first, and then starts from input vertex 2 (node ​​2). Then the traversal of node 0 will be executed later. Similarly, for nodes that are dependent on multiple nodes, the nodes that are traversed first will also be executed later. For example, Figure 2 Nodes 3, 5, and 6 in the example all depend on node 1. If the traversal starts from node 3, then node 3 will be executed after node 5 or 6. The operators represented by different nodes may consume different processing resources when executed. Furthermore, the total processing resources of electronic devices are limited. If a large amount of processing resources is consumed during the execution of an operator corresponding to a certain node, and the electronic device always has other programs that also consume a large amount of processing resources, it will cause the electronic device to freeze or even crash.

[0047] As a way to improve the above problem, the electronic device can perform a traversal of all branches during the node traversal process, thereby obtaining multiple execution orders of all nodes. For example, in the case of node 0, node 1, node 2, node 3, node 4, node 5, node 6, and node 7, one execution order can be node 2, node 0, node 4, node 1, node 6, node 5, node 3, and node 7. Another execution order can be node 2, node 0, node 4, node 1, node 6, node 3, node 5, and node 7. Another execution order can be node 2, node 0, node 4, node 1, node 6, node 3, node 5, and node 7. In addition, another execution order can be node 2, node 0, node 4, node 1, node 3, node 5, node 6, and node 7. In the case where the processing resources consumed during the execution of the operators corresponding to different nodes are different, after determining the execution order of all nodes, the electronic device can obtain the relative amount of processing resources required for each stage of the execution of the neural network model. One stage represents the execution process of an operator.

[0048] Correspondingly, the electronic device can also estimate the consumption of processing resources by other programs during the entire execution process of the neural network model, and then match the consumption of processing resources by other programs with the resource consumption corresponding to the aforementioned multiple execution orders, and use the one among the multiple execution orders that has the least conflict with the consumption of processing resources by other programs as the final execution order.

[0049] As a method, the electronic device can associate the processing resource consumption status of various execution orders with the processing time and thus establish a two-dimensional mapping relationship between the processing resource consumption and time. Figure 3As shown, the horizontal axis represents time, the vertical axis represents resource consumption, and each column represents the execution process of a node. For example, t1 to t2 represents the execution process of the operator corresponding to a node. It is understandable that the execution time consumed by the operator represented by each node may be different, and the figure is only an example. In addition, for ease of calculation, the resource consumption corresponding to each node in the figure is not the average resource consumption during the operation of the operator corresponding to the node.

[0050] Then based on Figure 3 By using the method shown, the mapping relationships between resource consumption levels and time corresponding to various execution orders can be obtained. The mapping relationships between resource consumption programs and time for other programs on the electronic device within the corresponding time period can then be obtained. The mapping relationships between resource consumption programs and time for other programs can then be segmented and matched with the mapping relationships between resource consumption levels and time corresponding to the various execution orders to obtain an execution order that minimizes resource consumption conflicts.

[0051] It can be understood that each segment in the segment matching represents the execution process of the operator represented by a node. For example, Figure 3 In the example, t1 to t2 is a segment, and t2 to t3 is also a segment. During the matching process, the electronic device can calculate the processing resource consumption of other programs during the time period from t1 to t2. It then matches this processing resource consumption with the processing resources required for the execution of the operators corresponding to the nodes during the time period from t1 to t2. The matching is then completed for each segment in turn, resulting in an overall matching result.

[0052] In this embodiment, as a way, matching can be completed by scoring. Optionally, the electronic device can establish a mapping relationship between the difference in resource consumption and the score. When matching the resource consumption of each stage, a score can be obtained. For example, the smaller the difference, the higher the score. Then the electronic device can obtain the score of each stage, thereby obtaining the total score of each execution order, and the execution order with the lowest score is used as the selected execution order. Specifically, Figure 3For example, the execution phase corresponding to node a is from t1 to t2, and its corresponding processing resource consumption is a1. The processing resource consumption of other applications on the electronic device during this period is a2. The score associated with the absolute value of the difference between a1 and a2 is then used as the score for this phase. Similarly, scores can be obtained for time periods from t2 to t3, t3 to t4, t4 to t5, t5 to t6, t6 to t7, and t7 to t8, resulting in the overall score.

[0053] It should be noted that the resource consumption of other applications in the electronic device during each time period can also be an average consumption level. Furthermore, it can be understood that the aforementioned selection process of multiple execution orders is performed before the actual execution of the neural network model. Then, the time consumed by the operation of the operator corresponding to the node in each stage and the degree of processing resource occupancy are pre-calculated estimated values. Similarly, the degree of processing resource occupancy of other applications in the electronic device during each stage is also an estimated value.

[0054] As a way, the time consumed by the operation of the operator corresponding to the node in each stage and the degree of processing resource occupancy can be pre-calculated during the training of the neural network model. Then in this way, when the electronic device receives the file storing the neural network model from the network, it will also receive a file that identifies the operation time and resource occupancy of each node in the neural network model. Furthermore, the degree of processing resource occupancy of other applications of the electronic device in each stage can be estimated based on the historical operation records of the application of the electronic device. It is understandable that the electronic device can record what application is run in what time period of each day, and the corresponding degree of processing resource consumption during the operation process can also be recorded. Then the electronic device can obtain certain rules by counting the degree of processing resource consumption over a period of time, and then estimate the degree of processing resource consumption in each subsequent time period.

[0055] It should be noted that each of the aforementioned stages represents a time period within a day, such as 12:10 to 12:12. Another example is 12:01:30 to 12:02, or even shorter or longer time periods. The processing resources mentioned herein can be understood as the processor occupancy rate.

[0056] Step S140: configuring the execution order of the operators represented by the nodes based on the execution order of the nodes.

[0057] The present application provides a neural network model processing method. After obtaining the neural network model to be configured, the neural network model is mapped into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model. Then, the graph structure is traversed to obtain the execution order of the nodes, and the execution order of the operators represented by the nodes is configured based on the execution order of the nodes. Thus, by converting the neural network model into a graph structure and then determining the execution order of the operators represented by each node by traversing the nodes in the graph structure, it is possible to quickly determine the execution order of all operators and improve the computational efficiency of the entire model.

[0058] See also Figure 4 , an embodiment of the present application provides a neural network model processing method, which is applied to an electronic device, and the method includes:

[0059] Step S210: Receive the trained neural network model.

[0060] Step S220: Optimizing the trained neural network model to obtain a neural network model to be configured; wherein the neural network model to be configured is a neural network model adapted to the data computing capability of the electronic device.

[0061] The step of optimizing the trained neural network model includes at least one of the following steps: performing operator fusion on the trained neural network model; performing network pruning on the trained neural network model; performing model quantization on the trained neural network model; and performing network cutting on the trained neural network model.

[0062] Among them, operator fusion can be understood as merging some operators to reduce calculations or memory copies (such as the merging of Conv2D and BatchNorm). Network pruning can be understood as removing some unnecessary operators to simplify the network (such as removing some redundant operators that are only used during training). Furthermore, the internal calculations of general neural network models all use floating-point calculations, which consume relatively large computing resources (space and CPU / GPU time). If other simple numerical types can be used for calculations within the neural network model without affecting the accuracy of the neural network model, the calculation speed will be greatly improved and the computing resources consumed will be greatly reduced. This is especially important for mobile devices. Quantization is to compress the original neural network model by reducing the number of bits required to represent each weight.

[0063] Step S230: Mapping the neural network model into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model;

[0064] Step S240: traverse the graph structure to obtain the execution order of the nodes.

[0065] Step S250: configuring the execution order of the operators represented by the nodes based on the execution order of the nodes.

[0066] The present application provides a neural network model processing method. After obtaining the neural network model to be configured, the neural network model is mapped into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model. Then, the graph structure is traversed to obtain the execution order of the nodes, and the execution order of the operators represented by the nodes is configured based on the execution order of the nodes. Thus, by converting the neural network model into a graph structure and then determining the execution order of the operators represented by each node by traversing the nodes in the graph structure, it is possible to quickly determine the execution order of all operators and improve the computational efficiency of the entire model. In addition, the method provided in this embodiment can be used to calculate the optimized neural network model, thereby quickly obtaining the accurate execution order of operators after the neural network model is optimized.

[0067] See also Figure 5 , an embodiment of the present application provides a neural network model processing method, which is applied to an electronic device, and the method includes:

[0068] Step S310: Receive the trained neural network model.

[0069] Step S320: After specifying the execution order of operators in the trained neural network model, a target operator is added to obtain the neural network model to be configured, and the target operator is used to convert the storage format of the data obtained by executing the specified operator into the target format.

[0070] It's important to note that storage format conversion allows electronic devices to perform data calculations faster. For example, in electronic devices like smartphones and tablets, GPU-based operators generally have two memory layouts: Buffer and Image. The Buffer format doesn't require model processing, but its computational speed is not as fast as the Image format, so the Image format is generally used. However, it does require converting the weights in the network model from plain memory to Image format, necessitating the addition of an operator to the network graph for memory conversion.

[0071] Step S330: Mapping the neural network model into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model;

[0072] Step S340: traverse the graph structure to obtain the execution order of the nodes.

[0073] Step S350: configuring the execution order of the operators represented by the nodes based on the execution order of the nodes.

[0074] The present application provides a neural network model processing method. After obtaining the neural network model to be configured, the neural network model is mapped into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model. Then, the graph structure is traversed to obtain the execution order of the nodes, and the execution order of the operators represented by the nodes is configured based on the execution order of the nodes. Thus, by converting the neural network model into a graph structure and then determining the execution order of the operators represented by each node by traversing the nodes in the graph structure, the execution order of all operators can be quickly determined, thereby improving the computational efficiency of the entire model. In addition, the method provided in this embodiment can be to calculate the neural network model after inserting a new operator, thereby realizing that the accurate operator execution order can be quickly obtained after the new operator is inserted into the neural network model.

[0075] See also Figure 6 The embodiment of the present application provides a neural network model processing device 400, which runs on an electronic device. The device 400 includes:

[0076] The model acquisition unit 410 is used to acquire the neural network model to be configured.

[0077] As a way, Figure 7 As shown, the model acquisition unit 410 includes:

[0078] The model receiving subunit 411 is used to receive the trained neural network model;

[0079] The model optimization subunit 412 is used to optimize the neural network model obtained through training to obtain a neural network model to be configured; wherein the neural network model to be configured is a neural network model adapted to the data computing capability of the electronic device.

[0080] As a way, Figure 8 As shown, the model acquisition unit 410 includes:

[0081] The model receiving subunit 413 is used to receive the trained neural network model.

[0082] The operator adding subunit 414 adds a target operator after specifying the execution order of the operators in the trained neural network model to obtain the neural network model to be configured. The target operator is used to convert the storage format of the data obtained by executing the specified operator into the target format.

[0083] Among them, optionally, the step of optimizing the trained neural network model includes at least one of the following steps: performing operator fusion on the trained neural network model; performing network pruning on the trained neural network model; performing model quantization on the trained neural network model; and performing network cutting on the trained neural network model.

[0084] The model processing unit 420 is used to map the neural network model into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model.

[0085] The traversal unit 430 is used to traverse the graph structure to obtain the execution order of the nodes.

[0086] The order determination unit 440 is configured to configure the execution order of the operators represented by the nodes based on the execution order of the nodes.

[0087] As one approach, the graph structure is a directed acyclic graph. In this approach, the model processing unit 420 is specifically configured to perform a depth-first traversal on the directed acyclic graph to obtain an execution order for the nodes. The traversal unit 430 is specifically configured to perform a depth-first traversal on the directed acyclic graph, sequentially storing target nodes obtained during the traversal into a storage space of a stack structure, wherein the target node is a node without subordinate nodes or a node whose subordinate nodes have all been traversed; and the order in which the nodes in the storage space are popped from the stack is used as the execution order for the nodes.

[0088] As a method, the traversal unit 430 is specifically configured to perform a depth-first traversal on the directed acyclic graph based on a specified starting input vertex to obtain an execution order of the nodes.

[0089] It should be noted that the device embodiment in this application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0090] The following will be combined Figure 9 An electronic device provided by this application is described.

[0091] See also Figure 9 Based on the above-mentioned neural network model processing method and apparatus, the embodiments of the present application also provide another electronic device 200 that can execute the above-mentioned neural network model processing method. The electronic device 200 includes one or more (only one is shown in the figure) processors 102, a memory 104, and a network module 106 that are coupled to each other. The memory 104 stores a program that can execute the content of the above-mentioned embodiments, and the processor 102 can execute the program stored in the memory 104.

[0092] The processor 102 may include one or more cores for processing data. The processor 102 utilizes various interfaces and circuits to connect various components within the electronic device 200. It executes instructions, programs, code sets, or instruction sets stored in the memory 104, and accesses data stored in the memory 104 to perform various functions and process data within the electronic device 200. Optionally, the processor 102 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 102 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 102 and may be implemented separately via a communication chip.

[0093] The memory 104 may include a random access memory (RAM) or a read-only memory (ROM). The memory 104 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal 100 during use (such as a phone book, audio and video data, chat history data), etc.

[0094] The network module 106 is used to receive and transmit electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices, such as communicating with an audio playback device. The network module 106 may include various existing circuit components for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, etc. The network module 106 can communicate with various networks such as the Internet, an enterprise intranet, a wireless network, or communicate with other devices via a wireless network. The above-mentioned wireless network may include a cellular telephone network, a wireless local area network, or a metropolitan area network. For example, the network module 106 can exchange information with a base station.

[0095] Please refer to Figure 10 , which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1100 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0096] Computer-readable storage medium 1100 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Alternatively, computer-readable storage medium 1100 includes a non-transitory computer-readable storage medium. Computer-readable storage medium 1100 has storage space for program code 810 for executing any of the method steps described above. These program codes can be read from or written to one or more computer program products. Program code 1110 can be compressed, for example, in a suitable form.

[0097] The present application provides a neural network model processing method, device, and electronic device. After obtaining the neural network model to be configured, the neural network model is mapped into a graph structure based on the dependency relationship of the operators in the neural network model, wherein a node in the graph structure represents an operator in the neural network model. Then, the graph structure is traversed to obtain the execution order of the nodes, and the execution order of the operators represented by the nodes is configured based on the execution order of the nodes. Thus, by converting the neural network model into a graph structure and then determining the execution order of the operators represented by each node by traversing the nodes in the graph structure, it is possible to quickly determine the execution order of all operators and improve the computational efficiency of the entire model.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A neural network model processing method, characterized in that: Applied to electronic equipment, the method includes: Get the neural network model to be configured; Based on the dependency relationship of operators in the neural network model, mapping the neural network model into a graph structure, wherein the graph structure is a directed acyclic graph, a node in the graph structure represents an operator in the neural network model, and operators represented by different nodes consume different processing resources when executed; Performing multiple depth-first traversals on the directed acyclic graph to obtain multiple execution orders of the nodes, wherein the node traversal order determined in each depth-first traversal is different; Obtaining a final execution order from the execution orders of the multiple nodes, the final execution order being the execution order that minimizes conflicts with other programs in consuming processing resources; the other programs being programs that are estimated to be run during the operation of the neural network model, wherein obtaining the final execution order comprises: obtaining a first mapping relationship between resource consumption levels and time corresponding to the multiple execution orders, and then obtaining a second mapping relationship between resource consumption programs and time of other programs of the electronic device within the corresponding time period, and performing segmented matching on the first mapping relationship and the second mapping relationship to obtain the final execution order, wherein each segment in the segmented matching represents the execution process of the operator represented by a node; The execution order of the operators represented by the nodes is configured based on the final execution order.

2. The method according to claim 1, characterized in that The step of performing a depth-first traversal on the directed acyclic graph to obtain the execution order of the nodes includes: Performing a depth-first traversal on the directed acyclic graph, and sequentially storing target nodes obtained during the traversal into a storage space of a stack structure, wherein the target node is a node without subordinate nodes or a node whose subordinate nodes have all been traversed; The order in which the nodes in the storage space are popped out of the stack is used as the execution order of the nodes.

3. The method according to claim 1, characterized in that The directed acyclic graph has multiple input vertices, and the step of performing a depth-first traversal on the directed acyclic graph to obtain the execution order of the nodes includes: Based on the specified starting input vertex, a depth-first traversal is performed on the directed acyclic graph to obtain the execution order of the nodes.

4. The method according to claim 1, wherein The step of obtaining the neural network model to be configured includes: Receive the trained neural network model; The trained neural network model is optimized to obtain a neural network model to be configured; wherein the neural network model to be configured is a neural network model adapted to the data computing capability of the electronic device.

5. The method according to claim 4, characterized in that The step of optimizing the trained neural network model includes at least one of the following steps: Performing operator fusion on the neural network model obtained by the training; Performing network pruning on the neural network model obtained by the training; Performing model quantization on the neural network model obtained by the training; and Perform network cutting on the neural network model obtained by training.

6. The method according to claim 4, characterized in that The step of optimizing the trained neural network model to obtain the neural network model to be configured includes: After specifying the execution order of operators in the trained neural network model, a target operator is added to obtain the neural network model to be configured. The target operator is used to convert the storage format of data obtained by executing the specified operator into the target format.

7. A neural network model processing device, characterized in that: Running on an electronic device, the device includes: A model acquisition unit, used to acquire a neural network model to be configured; a model processing unit, configured to map the neural network model into a graph structure based on dependencies among operators in the neural network model, wherein the graph structure is a directed acyclic graph, wherein a node in the graph structure represents an operator in the neural network model, and operators represented by different nodes consume different processing resources when executed; a traversal unit, configured to perform multiple depth-first traversals on the directed acyclic graph to obtain multiple execution orders of the nodes, wherein the node traversal order determined in each depth-first traversal is different; A sequence determination unit is used to obtain a final execution order from the execution orders of the multiple nodes, wherein the final execution order is the execution order that has the least conflict with the consumption of processing resources by other programs; the other programs are programs that are estimated to be run during the operation of the neural network model, wherein obtaining the final execution order includes: obtaining a first mapping relationship between the resource consumption levels and time corresponding to the multiple execution orders, and then obtaining a second mapping relationship between the resource consumption programs and time of other programs of the electronic device within the corresponding time period, and performing segmented matching on the first mapping relationship and the second mapping relationship to obtain the final execution order, wherein each segment in the segmented matching represents the execution process of the operator represented by a node; and configuring the execution order of the operator represented by the node based on the final execution order.

8. An electronic device, characterized in that: including a processor and a memory; One or more programs are stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 6 is executed.

Images