Neural network segmentation method, prediction method and related device

Abstract

A neural network segmentation method, prediction method and related device, the neural network segmentation method comprises: obtaining a neural network graph, the neural network graph is used to represent a neural network; segmenting the neural network graph to obtain a depth subgraph; each of the multiple nodes included in the depth subgraph performs data exchange by reading and writing an on-chip buffer, the depth subgraph is used to successively process at least two groups of data obtained by splitting first input data to obtain first output data, the first input data is the input data of the depth subgraph. In the method, the neural network segmentation device segments the neural network graph to obtain one or more depth subgraphs, so as to generate one or more depth subnetworks according to the one or more depth subgraphs, and using these depth subnetworks to perform neural network processing tasks can greatly reduce the number of times the external memory is accessed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a neural network segmentation method, a prediction method and related devices. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc. Machine learning is currently the most important way to achieve artificial intelligence.

[0003] In the field of machine learning and cognitive science, a neural network (NN) or a quasi-neural network is a mathematical model or computational model that imitates the structure and function of a biological neural network (the central nervous system of an animal, especially the brain) and is used to estimate or approximate a function. A neural network is calculated by connecting a large number of artificial neurons. At present, there are two main ways to calculate a neural network: 1) Graphics Processing Unit (GPU); 2) Application Specific Integrated Circuits (ASIC). However, regardless of GPU or ASIC, the conventional technical solution for using a neural network to perform processing tasks is to use a single node (i.e., neuron) of the neural network as the basic unit for layer-by-layer calculation. Since the calculation result data of a single node of a neural network is often large and cannot be stored in the on-chip storage, the calculation result needs to be directed to the off-chip storage. Therefore, it is necessary to access the off-chip storage to save the output result of a single node of the neural network. Since the number of neural network nodes is often large, this will result in frequent access to the off-chip storage during the calculation process of the neural network. Frequent access to the off-chip storage will cause the computing performance to be limited to a certain extent by the off-chip storage bandwidth and high system power consumption. Summary of the invention

[0004] The embodiments of the present application provide a neural network segmentation method, a prediction method and related devices, which segment a neural network graph to obtain a deep subgraph to facilitate the generation of a deep subnetwork. The deep subnetwork does not need to access an external memory during the execution of processing operations, which can effectively reduce the number of times the external memory is accessed or even avoid accessing the external memory.

[0005] In a first aspect, an embodiment of the present application provides a neural network segmentation method, which may include: obtaining a neural network graph, the neural network graph being used to represent a neural network, the neural network graph including multiple nodes, each node representing a computing unit in the neural network; segmenting the neural network graph to obtain a deep subgraph; the deep subgraph being used to represent a deep subnetwork, the deep subnetwork including multiple nodes that exchange data by reading and writing an on-chip buffer, the deep subnetwork being used to successively process at least two groups of data obtained by splitting first input data to obtain first output data, the first input data being input data of the deep subnetwork, and the first input data including one or more computer-processable signals.

[0006] The execution subject of the method is a neural network segmentation device, which can be a server, a mobile phone or other terminals and other computer devices. In practical applications, the neural network segmentation device can segment the neural network graph to obtain one or more deep subgraphs, so as to generate one or more deep subnetworks based on the one or more deep subgraphs. The neural network graph represents a neural network, and the essence of segmenting the neural network graph is to segment the neural network. These deep subnetworks can be understood as subnetworks obtained by segmenting the neural network, that is, each deep subnetwork includes a part of the nodes in the neural network. The segmentation of the neural network mentioned here is only a logical segmentation, that is, only adjusting the process of processing its input data and the data reading and writing process of each node in the neural network, and not dividing the neural network into several parts. The nodes included in the segmented neural network (that is, these deep subnetworks) are also the nodes in the neural network, and the processing operations implemented by each node will not change. The neural network is used to perform the target task, and each deep subnetwork is used to perform a subtask included in the target task. It can be understood that these deep subnetworks can achieve the target task. For example, inputting reference input data into a neural network for prediction processing can obtain a reference prediction result; inputting the reference input data into one or more deep sub-networks obtained by splitting the neural network for processing can also obtain the reference prediction result. In other words, the processing operations implemented by the deep sub-network obtained by splitting the neural network are the same as the processing operations implemented by the neural network. Since the deep sub-network does not need to access external memory in the process of processing its input data, using the deep sub-network obtained by splitting the neural network to perform the target task can reduce the number of accesses to the external memory or even avoid accessing the external memory. External memory refers to memory other than the on-chip buffer. Since the number of accesses to the external memory is reduced, using the split neural network to perform processing tasks can also reduce power consumption.

[0007] In an embodiment of the present application, a neural network segmentation device segments a neural network graph to obtain one or more depth sub-graphs, so as to generate one or more depth sub-networks based on the one or more depth sub-graphs. Using these depth sub-networks to perform neural network processing tasks can greatly reduce the number of times the external memory is accessed and reduce power consumption.

[0008] In an optional manner, the method further includes: dividing the neural network graph to obtain a direct subgraph; the direct subgraph is used to characterize the direct subnetwork, and each of the multiple nodes included in the direct subnetwork interacts with each other by reading and writing the on-chip buffer; the direct subnetwork is used to process the second input data as a whole to obtain second output data, and the second input data is the input data of the direct subnetwork. The neural network dividing device divides the neural network graph to obtain a deep subgraph and a direct subgraph. The deep subnetwork generated by the deep subgraph can split its input data into at least two groups of data for sequential processing, and the direct subnetwork generated by the direct subgraph can process its input data as a whole. Since the direct subnetwork processes its input data as a whole, each node in the direct subnetwork only needs to perform one processing operation, which takes a short time.

[0009] In this implementation, the neural network segmentation device segments the neural network graph to obtain a direct subgraph, so as to generate a direct subnetwork based on the direct subgraph. The direct subnetwork can process its input data as a whole, which can effectively save processing time.

[0010] In an optional manner, the storage space required for the first input data is larger than the storage space available in the on-chip buffer, the storage space required for each of the at least two groups of data is no larger than the storage space available in the on-chip buffer, and the storage space required for the second input data is no larger than the storage space available in the on-chip buffer.

[0011] In this implementation, on the one hand, when the storage space required for the input data is larger than the available storage space of the on-chip buffer, the input data is split into at least two groups of data and processed successively, so that the storage space required for each group of data is no larger than the available storage space of the on-chip buffer, thereby avoiding access to the external memory; on the other hand, when the storage space required for the input data is no larger than the available storage space of the on-chip buffer, the input data is processed as a whole, which saves processing time while not accessing the external memory.

[0012] In an optional manner, each of the multiple nodes included in the deep sub-network performs at least two processing operations in the process of processing the first input data, and at least one node in the direct sub-network performs one processing operation in the process of processing the second input data.

[0013] In this implementation, by dividing the first input data into at least two groups and processing them sequentially, accessing the external memory can be avoided.

[0014] In an optional manner, the segmentation of the neural network graph to obtain a depth subgraph includes: obtaining a first reference subgraph, the first reference subgraph including a first node and a second node, the first node being a node to be allocated in the current neural network graph, and the second node being the next node of the first node in the neural network graph; adding a third node to the first reference subgraph to obtain a second reference subgraph, the third node being the next node of the second node in the neural network graph, and the second reference subgraph being used to process third input data; in the process of the second reference subgraph processing the third input data, allocating the address of the on-chip buffer to the second reference subgraph; in the case where the address of the on-chip buffer is successfully allocated to the second reference subgraph and the third node belongs to the termination node, using the second reference subgraph as the depth subgraph. The reference subgraph is a subgraph including at least one node that is not determined to be a depth subgraph or a direct subgraph, and whose starting node is a subgraph of a node currently to be allocated in the neural network graph. The reference subgraph is a subgraph including a node currently to be allocated in the neural network graph or including a node currently to be allocated in the neural network graph and the next node of the node to be allocated. In the process of the neural network graph segmentation method, the neural network graph segmentation device can add one or more unassigned nodes in the neural network graph on the basis of a reference subgraph to obtain a direct subgraph or a deep subgraph; after obtaining a direct subgraph or a deep subgraph, add one or more unassigned nodes in the neural network graph on the basis of another reference subgraph (a subgraph including a node to be currently assigned in the neural network graph or a subgraph including a node to be currently assigned in the neural network graph and a next node of the node to be assigned) to obtain a new direct subgraph or a deep subgraph. It can be understood that each reference subgraph includes a node to be currently assigned in the neural network graph, and a direct subgraph or a deep subgraph can be obtained based on each reference subgraph.

[0015] In this implementation, the neural network segmentation device can quickly and accurately generate a deep subgraph based on the allocation result of the on-chip buffer address and whether the node is a termination node.

[0016] In an optional manner, after allocating the address of the on-chip buffer to the second reference sub-image, the method further comprises: if the address of the on-chip buffer is not successfully allocated to the second reference sub-image, using the first reference sub-image as the direct sub-image.

[0017] The situation where the address of the on-chip buffer is not successfully allocated to the second reference sub-image includes: in the process of successively processing at least two groups of data obtained by splitting the third input data, the address of the on-chip buffer is not successfully allocated to the second reference sub-image. The failure to successfully allocate the address of the on-chip buffer to the second reference sub-image indicates that the second reference sub-image cannot be used as a depth sub-image. In this implementation, in the case where the address of the on-chip buffer is not successfully allocated to the second reference sub-image, the first reference sub-image is used as a direct sub-image; thus, the generation of an erroneous depth sub-image can be avoided.

[0018] In an optional manner, allocating the address of the on-chip buffer to the second reference sub-graph includes: determining whether the address of the on-chip buffer is successfully allocated to the second reference sub-graph during the process of the second reference sub-graph processing the third input data as a whole; if the address of the on-chip buffer is not successfully allocated to the second reference sub-graph, allocating the address of the on-chip buffer to the second reference sub-graph during the process of the second reference sub-graph successively processing at least two groups of data obtained by splitting the third input data.

[0019] In this implementation, if the address of the on-chip buffer is successfully allocated to the second reference sub-graph during the process of the second reference sub-graph processing the third input data as a whole, subsequent operations do not need to be performed, and operations can be reduced.

[0020] In an optional manner, after splitting the neural network graph to obtain deep subgraphs, the method further includes: when the number of nodes included in the depth subgraph is not less than a first threshold, splitting the depth subgraph to obtain a first second-order subgraph and a second second-order subgraph; the first second-order subgraph is used to characterize the first second-order subnetwork, the second second-order subgraph is used to characterize the second second-order subnetwork, the first second-order subnetwork and the second second-order subnetwork are both included in the deep subnetwork, and the first second-order subnetwork and the second second-order subnetwork contain different nodes.

[0021] The first threshold value may be 5, 6, 7, 8, etc., which is not limited in the embodiments of the present application. The first second-order subgraph and the second second-order subgraph are subgraphs obtained by segmenting the depth subgraph. The first second-order subnetwork and the second second-order subnetwork can be understood as two subnetworks obtained by segmenting the depth subnetwork. Exemplarily, one of the first second-order subgraph and the second second-order subgraph is a direct subgraph, and the other is a depth subgraph. That is, a new depth subgraph and a direct subgraph can be obtained by segmenting a depth subgraph including a number of nodes not less than the first threshold. Exemplarily, both the first second-order subgraph and the second second-order subgraph are depth subgraphs. That is, two new depth subgraphs can be obtained by segmenting a depth subgraph including a number of nodes not less than the first threshold. For example, a depth subgraph includes nodes 1, 2, 3, 4, 5, and 6. The depth subgraph is segmented to obtain a depth subgraph including nodes 1, 2, and 3, and a depth subgraph including nodes 4, 5, and 6. Exemplarily, both the first second-order subgraph and the second second-order subgraph are direct subgraphs. That is, two new direct subgraphs can be obtained by splitting a deep subgraph including a number of nodes not less than a first threshold. In some embodiments, the neural network segmentation device first splits the neural network graph to obtain at least one deep subgraph, and then splits the deep subgraph including a number of nodes not less than a first threshold in the at least one deep subgraph, so that the number of nodes included in each deep subgraph is less than the first threshold. It can be understood that the more nodes a deep subnetwork includes, the more repeated calculations the deep subnetwork has. By reducing the number of nodes included in the deep subnetwork, the amount of repeated calculations can be effectively reduced.

[0022] In this implementation, by splitting the depth subgraphs including nodes that are not less than a first threshold, the number of depth subgraphs including nodes that exceed the first threshold can be reduced, so as to reduce the amount of repeated calculations of the depth subnetwork generated based on the depth subgraphs.

[0023] In an optional manner, each of the multiple nodes included in the first second-order sub-network exchanges data by reading and writing the on-chip buffer, and each of the multiple nodes included in the second second-order sub-network exchanges data by reading and writing the on-chip buffer.

[0024] In this implementation, the number of times the external memory is accessed can be reduced, thus saving power consumption.

[0025] In an optional manner, the input data of the first second-order subnetwork is the first input data, the output data of the second second-order subnetwork is the first output data, the first second-order subnetwork is used to store the first intermediate data obtained by processing the first input data into an intermediate buffer, and the second second-order subnetwork is used to process the first intermediate data obtained from the intermediate buffer, and the intermediate buffer is not the on-chip buffer.

[0026] Optionally, the intermediate buffer may be an external buffer having a read / write speed lower than that of the on-chip buffer and a read / write speed higher than that of the external memory. In other words, the performance of the intermediate buffer is between that of the on-chip buffer and the external memory, the read / write performance of the intermediate buffer is better than that of the external memory but lower than that of the on-chip buffer, and the storage space of the intermediate buffer is smaller than that of the external memory but larger than that of the on-chip buffer.

[0027] In this implementation, the output data of the first second-order sub-network is temporarily stored in an intermediate buffer, so as to reduce the number of deep sub-graphs including nodes exceeding the first threshold.

[0028] In an optional manner, the intermediate buffer is an off-chip buffer having a lower reading and writing speed than the on-chip buffer.

[0029] In an optional manner, after the depth subgraph is segmented to obtain a first second-order subgraph and a second second-order subgraph, the method further includes: when the second second-order subnetwork is used to process the first intermediate data as a whole, merging the second second-order subgraph and the first direct subgraph to obtain a second direct subgraph, the first direct subgraph is used to characterize the first direct subnetwork, the input data of the first direct subnetwork is the first output data output by the second second-order subnetwork, the first direct subnetwork is used to process the first output data as a whole to obtain third output data, the second direct subgraph is used to characterize the second direct subnetwork, and the second direct subnetwork is used to process the first intermediate data as a whole to obtain the third output data.

[0030] In this implementation, two adjacent direct subgraphs are merged into one direct subgraph, so that the two adjacent direct subgraphs are taken as a whole to generate corresponding instructions.

[0031] In an optional implementation, the splitting of the depth subgraph to obtain a first second-order subgraph and a second second-order subgraph includes: determining at least one reference node among the multiple nodes included in the depth subgraph whose storage space required for output data is smaller than the storage space available in the intermediate buffer; splitting the depth subgraph using the output of an intermediate node among the at least one reference node as a split point to obtain the first second-order subgraph and the second second-order subgraph, the intermediate node being any reference node among the at least one reference node, and the output data of the intermediate node being the output data of the first second-order subgraph and the input data of the second second-order subgraph.

[0032] In this implementation, a depth sub-graph can be quickly split into two second-order sub-graphs, and it can be ensured that the output data of the sub-network represented by the first second-order sub-graph can be stored in the intermediate buffer.

[0033] In an optional implementation, before the output of the intermediate node in the at least one reference node is used as a segmentation point to split the depth subgraph to obtain the first second-order subgraph and the second second-order subgraph, the method further includes: obtaining the depth difference of the two second-order subgraphs obtained by splitting the depth subgraph with the output of the at least one reference node as a segmentation point, and obtaining at least one depth difference, wherein the at least one reference node corresponds to the at least one depth difference one by one; determining to use the output of the intermediate node whose corresponding depth difference in the at least one reference node is less than a depth difference threshold as a segmentation point to split the depth subgraph. Optionally, determining to use the output of the intermediate node whose corresponding depth difference in the at least one reference node is the smallest as a segmentation point to split the depth subgraph. The depth difference threshold may be 1, 2, 3, etc. The depth of a subgraph may be the number of nodes included in the subgraph. In this implementation, the depth subgraph can be quickly split into two second-order subgraphs whose depth difference is less than the depth difference threshold.

[0034] In an optional manner, after the neural network graph is segmented to obtain a depth sub-graph, the method further includes: generating a target instruction corresponding to the depth sub-graph, the target instruction is used to execute a target sub-task, the neural network is used to execute a target task, and the target sub-task is a part of the target task.

[0035] In this implementation, a target instruction corresponding to the depth sub-map is generated, so that the target sub-task can be achieved by executing the target instruction.

[0036] In a second aspect, an embodiment of the present application provides a prediction method based on a neural network, which may include: obtaining original input data, wherein the original input data includes one or more computer-processable signals; inputting the original input data into the neural network for prediction processing to obtain a prediction result; the prediction processing includes: successively inputting at least two groups of data obtained by splitting the first input data into a deep sub-network for processing, wherein the deep sub-network is included in the neural network and includes some nodes in the neural network, each node represents a computing unit in the neural network, and each of the multiple nodes included in the deep sub-network exchanges data by reading and writing on-chip buffers; the first input data is obtained in the process of inputting the original input data into the neural network for prediction processing; and outputting the prediction result.

[0037] In an embodiment of the present application, in the process of inputting the original input data into the neural network for predictive processing, a deep sub-network is used to perform the processing tasks of the neural network. Since the deep sub-network does not need to access the external memory in the process of processing its input data, the number of times the external memory is accessed can be greatly reduced, and power consumption can be reduced.

[0038] In an optional manner, the prediction processing also includes: the direct sub-network processes the second input data as a whole, the direct sub-network is contained in the neural network and includes some nodes in the neural network, and the second input data is obtained in the process of inputting the original input data into the neural network for prediction processing.

[0039] In this implementation, the direct sub-network processes its input data as a whole, which is fast.

[0040] In an optional manner, the storage space required for the second input data is no greater than the storage space available in the on-chip buffer.

[0041] In this implementation, when the storage space required for the input data is not greater than the available storage space of the on-chip buffer, the input data is processed as a whole, which saves processing time while not accessing the external memory.

[0042] In an optional manner, at least one node in the direct sub-network performs a processing operation during processing of the second input data.

[0043] In this implementation, processing time can be saved.

[0044] In an optional manner, the storage space required for the first input data is larger than the storage space available in the on-chip buffer, and the storage space required for each of the at least two groups of data is no larger than the storage space available in the on-chip buffer.

[0045] In this implementation, when the storage space required for the input data is larger than the available storage space of the on-chip buffer, the input data is split into at least two groups of data and processed sequentially so that the storage space required for each group of data is no larger than the available storage space of the on-chip buffer, thereby avoiding access to the external memory.

[0046] In an optional manner, each of the multiple nodes included in the deep sub-network performs at least two processing operations during processing the first input data.

[0047] In this implementation, each node in the deep sub-network can avoid accessing the external memory by performing at least two processing operations in succession.

[0048] In an optional manner, in the process of inputting the original input data into the neural network for prediction processing and obtaining the prediction result, there is no need to access the off-chip memory.

[0049] In an optional manner, the plurality of computer-processable signals include at least one of a voice signal, a text signal or an image signal.

[0050] In a third aspect, an embodiment of the present application provides a neural network graph segmentation device, the device comprising: a memory and a processor; the memory is used to store code; the processor reads the code stored in the memory to perform the following operations: obtaining a neural network graph, the neural network graph is used to represent a neural network, the neural network graph includes multiple nodes, each node represents a computing unit in the neural network; the neural network graph is segmented to obtain a deep subgraph; the deep subgraph is used to represent a deep subnetwork, each of the multiple nodes included in the deep subnetwork interacts with data by reading and writing an on-chip buffer, the deep subnetwork is used to successively process at least two groups of data obtained by splitting first input data to obtain first output data, the first input data is the input data of the deep subnetwork, and the first input data includes one or more computer-processable signals.

[0051] In an optional manner, the processor is further used to split the neural network graph to obtain a direct subgraph; the direct subgraph is used to represent the direct subnetwork, and each of the multiple nodes included in the direct subnetwork interacts with each other by reading and writing the on-chip buffer; the direct subnetwork is used to process the second input data as a whole to obtain second output data, and the second input data is the input data of the direct subnetwork.

[0052] In an optional manner, the storage space required for the first input data is larger than the storage space available in the on-chip buffer, the storage space required for each of the at least two groups of data is no larger than the storage space available in the on-chip buffer, and the storage space required for the second input data is no larger than the storage space available in the on-chip buffer.

[0053] In an optional manner, each of the multiple nodes included in the deep sub-network performs at least two processing operations in the process of processing the first input data, and at least one node in the direct sub-network performs one processing operation in the process of processing the second input data.

[0054] In an optional manner, the processor is specifically used to obtain a first reference subgraph, the first reference subgraph including a first node and a second node, the first node being the node to be allocated in the current neural network graph, and the second node being the next node of the first node in the neural network graph; adding a third node to the first reference subgraph to obtain a second reference subgraph, the third node being the next node of the second node in the neural network graph, and the second reference subgraph being used to process third input data; determining whether the on-chip address manager successfully allocates the address of the on-chip buffer to the second reference subgraph during the process of the second reference subgraph processing the third input data; and using the second reference subgraph as the depth subgraph when the address of the on-chip buffer is successfully allocated to the second reference subgraph and the third node is a termination node.

[0055] In an optional manner, the processor is further configured to use the first reference sub-image as the direct sub-image if the address of the on-chip buffer is not successfully allocated to the second reference sub-image.

[0056] In an optional manner, the on-chip address manager is specifically used to allocate the address of the on-chip buffer to the second reference sub-graph when the second reference sub-graph processes the third input data as a whole; the processor is specifically used to allocate the address of the on-chip buffer to the second reference sub-graph when the second reference sub-graph successively processes at least two groups of data obtained by splitting the third input data if the on-chip address manager fails to successfully allocate the address of the on-chip buffer to the second reference sub-graph.

[0057] In an optional manner, the processor is further used to generate a target instruction corresponding to the depth sub-map, the target instruction is used to execute a target sub-task, the neural network is used to execute the target task, and the target sub-task is a part of the target task.

[0058] In a fourth aspect, an embodiment of the present application provides a data processing device, which includes: a memory and a processor; the memory is used to store code and original input data; the processor reads the code stored in the memory to perform the following operations: obtain the original input data, the original input data includes one or more computer-processable signals; input the original input data into a neural network for prediction processing to obtain a prediction result; the prediction processing includes: inputting at least two groups of data obtained by splitting the first input data into a deep sub-network for processing in sequence, the deep sub-network is included in the neural network and includes some nodes in the neural network, each node represents a computing unit in the neural network, and each of the multiple nodes included in the deep sub-network exchanges data by reading and writing on-chip buffers; the first input data is obtained in the process of inputting the original input data into the neural network for prediction processing; and outputting the prediction result.

[0059] In an optional manner, the prediction processing also includes: the direct sub-network processes the second input data as a whole, the direct sub-network is contained in the neural network and includes some nodes in the neural network, and the second input data is obtained in the process of inputting the original input data into the neural network for prediction processing.

[0060] In an optional manner, the storage space required for the second input data is no greater than the storage space available in the on-chip buffer.

[0061] In an optional manner, at least one node in the direct sub-network performs a processing operation during processing of the second input data.

[0062] In an optional manner, the storage space required for the first input data is larger than the storage space available in the on-chip buffer, and the storage space required for each of the at least two groups of data is no larger than the storage space available in the on-chip buffer.

[0063] In an optional manner, each of the multiple nodes included in the deep sub-network performs at least two processing operations during processing the first input data.

[0064] In an optional manner, the processor does not need to access an off-chip memory when inputting the original input data into the neural network for prediction processing to obtain the prediction result.

[0065] In an optional manner, the plurality of computer-processable signals include at least one of a voice signal, a text signal or an image signal.

[0066] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the methods of the first aspect to the second aspect and the optional implementation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is the classic computing form of neural network (NN) in AI chips;

[0068] Figure 2 The computing form of the neural network in the AI ​​chip provided in the embodiment of the present application;

[0069] Figure 3A A framework of a neural network graph segmentation processing system provided in an embodiment of the present application;

[0070] Figure 3B A framework of another neural network graph segmentation processing system provided in an embodiment of the present application;

[0071] Figure 4 A schematic diagram of the structure of a neural network graph segmentation device provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of a method for segmenting a neural network graph provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of a process of segmenting a neural network graph provided in an embodiment of the present application;

[0074] Figure 7A flow chart of a data segmentation judgment method provided in an embodiment of the present application;

[0075] Figure 8 A flow chart of a method for constructing a subgraph using the Direct mode provided in an embodiment of the present application;

[0076] Fig. 9 A flowchart of a method for constructing a subgraph using a Depth mode provided in an embodiment of the present application;

[0077] Fig.10 A flow chart of a data segmentation method provided in an embodiment of the present application;

[0078] Fig.11A It is a schematic diagram of three groups of data obtained by segmenting the input data;

[0079] Fig. 11B A schematic diagram of output data;

[0080] Fig.12 A flow chart of an on-chip storage allocation method provided in an embodiment of the present application;

[0081] Fig.13 A method flow for releasing storage space of an on-chip buffer provided in an embodiment of the present application;

[0082] Fig.14 An on-chip address allocation method process provided in an embodiment of the present application;

[0083] Fig.15 A flow chart of another on-chip address allocation method provided in an embodiment of the present application;

[0084] Fig.16A A computing form based on on-chip single-level storage in an AI chip provided in an embodiment of the present application;

[0085] Fig. 16B A computing form based on multi-level storage in an AI chip provided in an embodiment of the present application;

[0086] Fig.17 A schematic diagram of an optimal discrete block provided in an embodiment of the present application;

[0087] Fig.18 A schematic diagram of the structure of another neural network graph segmentation device provided in an embodiment of the present application;

[0088] Fig.19 This is a comparison diagram of the neural network before and after optimization;

[0089] Fig. 20 A schematic diagram of a Concat node optimizing on-chip storage provided in an embodiment of the present application;

[0090] Fig.21 A schematic diagram of the structure of another neural network graph segmentation device provided in an embodiment of the present application;

[0091] Fig. 22 A schematic diagram of the changes in the neural network structure before and after the optimization of the DetectionOutput node provided in the embodiment of the present application;

[0092] Fig.23 A schematic diagram of the structure of another neural network graph segmentation device provided in an embodiment of the present application;

[0093] Fig.24 A flow chart of a prediction method provided in an embodiment of the present application;

[0094] Fig.25 A schematic diagram of the structure of a neural network graph segmentation device provided in an embodiment of the present application;

[0095] Fig.26 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] In order to enable those skilled in the art to better understand the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all of the embodiments.

[0097] The terms "first", "second", and "third" in the specification embodiments and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0098] At present, the conventional technical solution for using neural networks to perform processing operations (such as image processing, speech processing, etc.) is to use a single node of the neural network (i.e., a neuron or a neural network computing unit) as the basic unit for layer-by-layer calculation, and finally guide the calculation results to external storage. Since the data volume of the calculation results of a single node of a neural network is often large, it cannot be stored in the internal storage unit, i.e., the on-chip buffer. Therefore, it is necessary to access external storage to save the output results of a single node of the neural network. Figure 1It is a form of computing of neural network (NN) in AI chip. In this application, DDR refers to off-chip memory, on-chip buffer refers to on-chip memory, and AI chip can be central processing unit, graphics processing unit, neural network processing unit (NPU), etc. In other words, DDR can be double data rate synchronous dynamic random access memory (Double Data Rage RAM) or other types of memory. Figure 1 The circles in the figure represent neural network computing units (Operations, OPs for short). For example, "1", "2", "3" and "4" indicate four OPs, which can also be called node 1, node 2, node 3 and node 4 (an OP can also be called a node). Figure 1 , the calculation process of the AI ​​chip is as follows: (1) Read the data to be processed (Input) from the DDR; (2) The AI ​​chip calculates each OP, and the result of each OP needs to be read and written back and forth between the AI ​​chip and the DDR; (3) After the entire NN calculation is completed, the final result (Output) is saved to the DDR. It can be seen that in this calculation form, OP is used as the basic unit for calculation, and the result of each OP needs to be read and written back and forth between the AI ​​chip and the DDR. This requires frequent access to external storage.

[0099] The main principle of this application is to convert the expression form of NN from OP as the basic unit to a depth subgraph (Depth SubGraph) and a direct subgraph (Direct SubGraph) as the basic unit. The depth subgraph represents the depth subnetwork, and the direct subgraph represents the direct subnetwork. Each depth subgraph and each direct subgraph are accompanied by on-chip address allocation information (completed by the on-chip address management module), etc., wherein the depth subgraph is also accompanied by data segmentation information (completed by the data segmentation module). Each depth subgraph and each direct subgraph include multiple OPs, and each subgraph includes multiple OPs that interact with each other by reading and writing on-chip buffers. In other words, each depth subgraph is equivalent to an OP in the classical computing form, and each direct subgraph is also equivalent to an OP in the classical computing form. The AI ​​chip can complete instruction mapping according to the new expression form (the form of depth subgraph and direct subgraph), that is, the NN whole network calculation can be completed in the on-chip memory (buffer). Each depth subgraph can be mapped to at least two groups of computing instruction pipelines (i.e., PipeLine), and each direct subgraph can be mapped to a group of computing instruction pipelines, each group of computing instruction pipelines is used to perform a series of operations.

[0100] Figure 2 The computing form of the neural network in the AI ​​chip provided in the embodiment of the present application. Figure 2In FIG. 1 , the depth subgraph 201 includes nodes 1 and 2, and the direct subgraph 202 includes nodes 3 and 4. Figure 2 It can be seen that the depth subgraph 201 processes input data 0 to obtain output data 20 and stores it in the on-chip buffer (corresponding to the depth subgraph calculation instruction pipeline 0), processes input data 1 to obtain output data 21 and stores it in the on-chip buffer (corresponding to the depth subgraph calculation instruction pipeline 1), and processes input data 2 to obtain output data 22 and stores it in the on-chip buffer (corresponding to the depth subgraph calculation instruction pipeline 2); the direct subgraph 202 processes the data input by the depth subgraph (i.e., the data obtained by output data 21, output data 22, and output data 23), and stores the processed output data in the external memory (corresponding to the direct subgraph calculation instruction pipeline 0). Optionally, node 1 and node 2 first process input data 0 to obtain output data 20 and store it in the on-chip buffer, then process input data 1 to obtain output data 21 and store it in the on-chip buffer, and finally process input data 2 to obtain output data 22 and store it in the on-chip buffer. In other words, the depth subgraph processes at least two groups of data obtained by splitting its input data in sequence. It can be understood that the depth subgraph corresponds to at least two groups of computing instruction pipelines, and each group of computing instruction pipelines is used to process a part of the input data of the depth subgraph. The output data obtained by the depth subgraph after multiple calculations can be spliced ​​as the input data of the direct subgraph or other depth subgraphs. Each group of computing instruction pipelines corresponds to a series of data movement and computing instructions. The solution of the present application is to divide the neural network graph representing the NN into the form of deep subgraphs and direct subgraphs as basic units, and use the computing instruction pipelines (i.e. PipeLine) corresponding to the depth subgraph and the direct subgraph as basic units for calculation. See Figure 2 The calculation process performed by the AI ​​chip in this application is as follows: (1) Read the data to be processed (Input) from DDR; (2) The AI ​​chip uses the PipcLinc corresponding to the depth sub-map or direct sub-map as the basic unit for calculation, where the calculation results of multiple groups of PipeLine of the depth sub-map will be gathered to the on-chip buffer, and the PipeLine of the direct sub-map will be directly read and written to the on-chip buffer, and the calculation can be completed OP by OP; (3) After completing the entire NN calculation, the result is saved to DDR. Figure 2 It can be seen that both the deep subgraph and the direct subgraph include multiple OPs, and the multiple OPs included in each subgraph exchange data by reading and writing the on-chip buffer. For the deep subgraph 201, node 1 obtains data from the DDR, and nodes other than node 1 do not need to access the DDR to perform processing operations, and only need to read and write the on-chip buffer. For the direct subgraph 202, node 4 stores the processing results (i.e., output data) to the DDR, and nodes other than node 4 do not need to access the DDR to perform processing operations, and only need to read and write the on-chip buffer.

[0101] The difference between a depth subgraph and a direct subgraph is that a depth subgraph corresponds to at least two groups of computing instruction pipelines, each of which is used to perform processing operations based on a portion of the input data of the depth subgraph; a direct subgraph corresponds to only one group of computing instruction pipelines, and each computing instruction pipeline corresponding to a direct subgraph is used to perform processing operations based on all the input data of the direct subgraph. It can be understood that the multiple groups of PipeLines corresponding to the depth subgraph are used to successively process at least two groups of data obtained by splitting the input data to obtain output data, that is, each group of PipeLines processes a portion of the input data. For a depth subgraph, the amount of output data obtained by the corresponding computing instruction pipeline performing the processing operation is fixed. The more the input data of the depth subgraph is divided into, the less the storage overhead of each group of computing instruction pipelines performing the processing operation. In addition, during the processing operations performed by the multiple groups of PipeLines corresponding to the depth subgraph, some addresses in the on-chip buffer can be reused. That is to say, in the process of processing operations performed by multiple groups of PipeLine corresponding to the depth subgraph, the storage space occupied by invalid data (data not needed in subsequent processing operations) will be released, and only valid data (data needed in subsequent processing operations) will be retained, so that each group of PipeLine performs processing operations with very little storage overhead on the on-chip buffer. It can be understood that by processing at least two groups of data obtained by splitting the input data separately to obtain output data, the storage space occupied by the on-chip buffer can be greatly reduced. The execution operations that originally (unsplit data) need to access the external memory to complete can be realized by reading and writing the on-chip buffer, thus reducing the access to the external memory. For a certain node, when the storage space occupied by the node for processing its input data exceeds the currently available storage space of the on-chip buffer, the input data of the node is split, and the multiple groups of data obtained by splitting the input data are processed separately, so that the node can complete the processing operation without accessing the external memory. In addition, the processing operations performed by each PipeLine in the multiple groups of PipeLines corresponding to the depth subgraph are similar. The more parts the input data of the depth subgraph is divided into, the more PipeLines corresponding to the depth subgraph. Since each PipeLine is executed one after another (i.e., serially), the more PipeLines corresponding to the depth subgraph, the longer it takes for these PipeLines to complete the processing operations. In order to balance storage overhead and computational efficiency, the input data of the depth subgraph needs to be reasonably divided. Each direct subgraph corresponds to a group of PipeLines, and the nodes corresponding to the direct subgraph only need to perform one processing operation; the depth subgraph corresponds to multiple groups of PipeLines, and the nodes corresponding to the depth subgraph need to perform multiple processing operations.

[0102] In summary, the computational efficiency of the nodes corresponding to the deep subgraph and the overhead of the on-chip storage space for executing processing operations are lower than those of the nodes corresponding to the direct subgraph. Therefore, it is necessary to reasonably divide the neural network graph to obtain the deep subgraph and the direct subgraph, and generate the PipeLine based on the deep subgraph and the direct subgraph, so as to improve the computational efficiency while reducing the access to the external memory. In addition, with OP as the basic computing unit, the PipeLine corresponding to the neural network needs to frequently access the external storage during the execution of the processing operation, which has high power consumption and cannot meet the needs of mobile terminal scenarios such as the detection and recognition of faces by the AI ​​chip in the off-screen scenario. With the deep subgraph and / or direct subgraph as the basic computing unit, the PipeLine corresponding to the neural network does not need to frequently access the external storage during the execution of the processing operation, has low power consumption, and can well meet the needs of scenarios such as the off-screen scenario.

[0103] The following describes a method for splitting a neural network graph to obtain a deep subgraph and a direct subgraph provided in an embodiment of the present application.

[0104] Figure 3A A framework of a neural network graph segmentation processing system provided in an embodiment of the present application. Figure 3A The framework in shows the process of splitting the neural network graph to obtain deep subgraphs and direct subgraphs, and generating the computational instruction pipeline corresponding to the deep subgraphs and direct subgraphs. Figure 3B Another framework of a neural network graph segmentation processing system provided in an embodiment of the present application is provided. Figure 3B exist Figure 3A On this basis, a process is added to further split the generated depth sub-graph into a second-order sub-graph. Figure 4 A schematic diagram of the structure of a neural network graph segmentation device provided in an embodiment of the present application, wherein the neural network graph segmentation device adopts Figure 3A The neural network graph segmentation processing system in the present invention is used to segment the neural network graph to obtain deep subgraphs and direct subgraphs, and to generate computational instruction pipelines corresponding to the deep subgraphs and direct subgraphs. The neural network graph segmentation device can be a server, a computer, a mobile phone, or other computer equipment. Figure 4As shown, the neural network graph segmentation device mainly includes: a control module 401, a graph segmentation module 402, a data segmentation module 403 and an on-chip address management module 404. The four modules are interdependent and jointly complete the function of segmenting the neural network graph. The control module 401 is used to process the NN with OP as the basic unit into a NN with subgraph (SubGraph) as the basic unit by controlling the graph segmentation module 402, the data segmentation module 403 and the on-chip address management module 404, and is provided with on-chip address mapping and segmentation information. The graph segmentation module 402 is used to segment the neural network graph (i.e., NN Graph) into two types of subgraphs, i.e., deep subgraphs and direct subgraphs. The operations performed by the graph segmentation module 402 correspond to Figure 3A The data segmentation module 403 is used to segment the input data of the depth subgraph (the data segmentation is completed in the process of creating the depth subgraph) and map it to different computing instruction pipelines. From the perspective of network computing performance, the control module 401 can give priority to forming a direct subgraph. The data segmentation module 403 can give priority to using a non-segmented data method, using the on-chip address management module to allocate address space for the Tensor in the subgraph (i.e., the neural network input and output data). If the non-segmentation method fails, the data segmentation method is entered, and then based on the subgraph after data segmentation, the address space is allocated for the Tensor after segmentation in the subgraph. The operations performed by the data segmentation module 403 correspond to Figure 3A In the embodiment of the present application, Tensor is an expression and definition of input data and output data in a neural network. The graph segmentation module 402 and the data segmentation module 403 can be the same module or two independent modules. The on-chip address management module 404 is mainly used to allocate on-chip storage space for a neural network graph or a certain Tensor, and is also used to determine whether graph segmentation and / or data segmentation is feasible. The address allocation for a certain neural network graph can be address allocation for the Tensor inside the subgraph. The operations performed by the on-chip address management module 404 correspond to Figure 3A In-chip address management and in-chip address allocation. Figure 4 As shown, the neural network graph segmentation device may also include an instruction mapping module 405, which is used to map the deep subgraph and the direct subgraph to corresponding PipeLine. Specifically, each deep subgraph is mapped to at least two groups of PipeLine, and each deep subgraph mapped at least two groups of PipeLine are used to implement the processing operations performed by the nodes included in the deep subgraph; each direct subgraph is mapped to a group of PipeLine, and each direct subgraph mapped PipeLine is used to implement the processing operations performed by the nodes included in the direct subgraph. In the present application, the two types of subgraphs are unified in the form of Pipeline and mapped to AI chip instructions to ensure that their calculation forms are unified.

[0105] Combine the following Figure 3A and Figure 4 Let's briefly describe the process of splitting a neural network graph into direct subgraphs and deep subgraphs. The specific processing flow is as follows: the control module 401 controls the on-chip address management module 404 to determine whether to split the neural network graph; if the neural network graph is to be split, the graph splitting module 401 is controlled to split the neural network graph to obtain a depth sub-graph and a direct sub-graph; if the neural network is not to be split, the on-chip address management module 404 is controlled to allocate on-chip addresses to the Tensors inside the neural network graph; the control module 401 controls the on-chip address management module 404 to determine whether to split the input data of the sub-graph; if the input data of the sub-graph is to be split, the data splitting module 403 and the on-chip address management module 404 are controlled to split the input data of the sub-graph (depth sub-graph); if the input data of the sub-graph is not to be split, the on-chip address management module 404 is controlled to allocate addresses to the Tensors inside the sub-graph (direct sub-graph) to obtain address allocation information; the control module 401 maps the depth sub-graph and the direct sub-graph into a computing instruction pipeline.

[0106] In some embodiments, the neural network graph segmentation device adopts Figure 3A After the neural network graph segmentation processing system in the present invention performs graph segmentation (i.e., first-level graph segmentation) on the neural network graph to obtain direct subgraphs (i.e., first-order subgraphs) and depth subgraphs (i.e., first-order subgraphs), it can further perform graph segmentation (i.e., second-level graph segmentation) on the depth subgraphs obtained by segmentation whose depth is not less than a first threshold to obtain second-order subgraphs, so that the depth of at least one depth subgraph is less than the first threshold. By segmenting the depth subgraph whose depth exceeds the first threshold, the number of processing operations performed by at least one node in the deep subnetwork represented by the depth subgraph can be reduced, thereby improving the computing efficiency. For example, a certain depth subgraph includes 6 nodes. When the depth subnetwork represented by the depth subgraph performs a processing operation, its input data is split into 3 parts and processed successively. Each node in the depth subnetwork needs to perform 3 processing operations successively, that is, 1 part of the data is processed each time. The depth subgraph is divided into a second-order depth subgraph including 3 nodes (i.e., a depth subgraph) and a second-order direct subgraph including 3 nodes (i.e., a direct subgraph). When the second-order depth subgraph represents a second-order depth subnetwork (i.e., a depth subnetwork) performs a processing operation, its input data is split into 3 parts and processed successively. When the second-order direct subnetwork represented by the second-order direct subgraph performs a processing operation, its input data is processed as a whole. It can be seen that in this example, the processing operations implemented by the second-order depth subnetwork and the second-order direct subnetwork are the same as those implemented by the depth subnetwork, but only 3 nodes need to perform 3 processing operations successively, which improves the computational efficiency.

[0107] Combine the following Figure 3B and Figure 4 To describe the process of first splitting the neural network graph to obtain direct subgraphs and depth subgraphs, and then splitting the depth subgraphs whose depth is not less than the first threshold. The specific processing flow is as follows: the control module 401 controls the on-chip address management module 404 to determine whether to split the neural network graph; if the neural network graph is split, the graph splitting module 401 is controlled to split the neural network graph to obtain depth subgraphs and direct subgraphs; if the neural network is not split, the on-chip address management module 404 is controlled to allocate on-chip addresses to the Tensors inside the neural network graph; the control module 401 controls the on-chip address management module 404 to determine whether to split the input data of the subgraph; if the input data of the subgraph is split, the data splitting module 403 and the on-chip address management module 404 are controlled to split the input data of the subgraph (depth subgraph); if the input data of the subgraph is not split, the on-chip address management module 404 is controlled to allocate on-chip addresses to the Tensors inside the subgraph (direct subgraph). or performs address allocation to obtain address allocation information; after completing the first-level graph segmentation of the neural network graph, the control module 401 determines whether the depth of the depth subgraph is not less than the first threshold; if the depth of the depth subgraph is less than the first threshold, the control module 401 controls the on-chip address management module 404 to perform address allocation for the Tensor inside the subgraph (depth subgraph) to obtain address allocation information; if the depth of the depth subgraph is not less than the first threshold, the control module 401 screens out at least one reference node whose storage space required for output data is less than the storage space available in the intermediate buffer among the multiple nodes included in the depth subgraph; the control module 401 calculates the depth difference corresponding to each reference node; the control module 401 determines the segmentation scheme; the control module 401 maps the depth subgraph (including the first-order depth subgraph and the second-order depth subgraph) and the direct subgraph into a computational instruction pipeline. The first-order depth subgraph refers to the depth subgraph obtained by segmenting the neural network graph, and the second-order depth subgraph refers to the depth subgraph obtained by segmenting the depth subgraph whose depth is not less than the first threshold. Optionally, the control module 401 calculates the depth difference between the two second-order subgraphs obtained by dividing the depth subgraph with each reference node as a segmentation point, and obtains the depth difference corresponding to each reference node, wherein each reference node corresponds to a depth difference. Optionally, the control module 401 determines that the segmentation scheme can be to use the intermediate node with the smallest corresponding depth difference among the reference nodes as a segmentation point to divide the depth subgraph, and obtain two second-order subgraphs (corresponding to the segmentation scheme). The implementation method of segmenting the depth subgraph whose depth is not less than the first threshold will be described in detail later.

[0108] Furthermore, the control module 401 also controls the on-chip address management module 404 to allocate addresses for the Tensors inside the depth sub-map. Figure 3AIt can be seen that the on-chip address management module 404 is required to participate in the graph segmentation, data segmentation and on-chip address allocation process. Figure 3A The subgraphs of different sizes in the data segmentation (corresponding to the reference subgraphs in the following text) are transmitted to the on-chip address management module 404, and the on-chip address management module 404 determines the specific method of segmenting the neural network graph. Similarly, in the data segmentation process, the data segmentation module 403 will transmit the data segmentation information ( Figure 3A The on-chip address management module 404 determines the specific method of dividing the input data of the depth sub-graph. During the on-chip address allocation process, the on-chip address management module 404 performs on-chip address allocation according to the on-chip address allocation mapping table, and generates a computational instruction pipeline for the direct sub-graph and the depth sub-graph. The subsequent embodiments will describe the methods of graph segmentation, data segmentation, and on-chip address allocation in detail, which will not be described in detail here.

[0109] The following first introduces the process of the control module 401 controlling the graph segmentation module 402, the data segmentation module 403 and the on-chip address management module 404 to segment the neural network graph to obtain deep subgraphs and direct subgraphs, as well as the address allocation process. Figure 5 A schematic diagram of a method for segmenting a neural network graph provided in an embodiment of the present application. Figure 5 As shown, the method may include:

[0110] 501. A neural network graph segmentation device obtains a neural network graph.

[0111] The neural network graph (NNGraph) is an expression of a neural network. The neural network graph may include information about the processing operations performed by the neural network, as well as information such as the size of the storage space occupied by the input and output data of each OP in the neural network during the processing operation. The neural network graph segmentation device may obtain the neural network graph by receiving the neural network graph input by the user; or the neural network graph segmentation device may obtain the program code corresponding to the neural network and determine the neural network graph based on the program code; or it may obtain reference information representing the neural network graph or the neural network and determine the neural network graph based on the reference information; or it may obtain the neural network graph in other ways, which is not limited in this application. Figure 6 A schematic diagram of a process of segmenting a neural network graph provided in an embodiment of the present application. Figure 6 As shown, the neural network diagram as an example includes 7 nodes, namely Figure 6Nodes 1 to 7 in the neural network diagram can reflect the order in which each node executes the processing operations; dividing the neural network diagram can obtain a deep subgraph (including nodes 1 to 4) and a direct subgraph (including nodes 5 to 7). Figure 6 601 to 603 are three groups of data obtained by dividing the first input data, 604 is the output data obtained by processing 601, 605 is the output data obtained by processing 602, 606 is the output data obtained by processing 603, 607 is the second input data, 608 is the second output data, and the straight line stripes are the overlapping parts of two adjacent groups of input data, namely, the overlapping parts of 601 and 602 and the overlapping parts of 602 and 603. Figure 6 It can be seen that the first input data of the depth sub-map is divided into three parts. There may be overlap between two adjacent parts of the data. Each part is processed as the input data of the depth sub-map to obtain a part of output data. The three parts of output data obtained in total are the first output data of the depth sub-map, and the first output data is the second input data of the direct sub-map.

[0112] 502. The neural network graph segmentation device determines whether the traversal of all nodes in the neural network graph is completed.

[0113] The process of the neural network graph segmentation device segmenting the neural network graph may be to segment each node into a direct subgraph or a depth subgraph in the order in which each node included in the neural network graph performs processing operations, until the traversal of all nodes is completed. Determining whether the traversal of all nodes in the neural network graph is completed may be to determine whether the last node in the neural network graph is segmented into a direct subgraph or a depth subgraph. If so, execute 513; if not, execute 503.

[0114] 503. The neural network graph segmentation device uses the current node as the starting node of the reference subgraph.

[0115] The current node refers to the node to be split in the neural network graph. That is, the nodes in the neural network graph that perform processing operations before the current node are all split into direct subgraphs or deep subgraphs, and other nodes need to perform processing operations after the current node.

[0116] 504. The neural network graph segmentation device determines whether the current node is the last node in the neural network graph.

[0117] If yes, go to step 506; if no, go to step 505.

[0118] 505. The neural network graph segmentation device adds the next node of the current node to the reference subgraph.

[0119] The next node of the current node may be a node in the neural network graph that first performs a processing operation after the current node performs a processing operation.

[0120] 506. The neural network graph segmentation device determines whether the data segmentation is executed successfully.

[0121] If yes, execute 507; if no, execute 508. The implementation of step 506 will be described in detail in the subsequent embodiments, which will not be described in detail here.

[0122] 507. The neural network graph segmentation device adopts a direct mode to construct a subgraph.

[0123] The neural network graph segmentation device constructs a subgraph in a direct mode according to the reference subgraph obtained in step 504 or step 505. For example, the reference subgraph includes Figure 6 For nodes 5 and 6 in the graph, the neural network graph segmentation device adopts a direct mode to construct a subgraph based on the benchmark subgraph to obtain a direct subgraph including nodes 5 to 7.

[0124] 508. The neural network graph segmentation device adopts the depth mode to construct subgraphs.

[0125] The neural network graph segmentation device constructs a subgraph using a depth mode according to the reference subgraph obtained in step 504 or step 505. For example, the reference subgraph includes Figure 6 The neural network graph segmentation device uses the deep mode to construct a subgraph based on the reference subgraph to obtain a deep subgraph including nodes 1 to 4 (see Figure 6 ). Figure 5 The process in mainly describes the control process of segmenting the neural network graph, that is, the control process executed by the control module. The subsequent embodiments will describe in detail the method of constructing a subgraph using the direct mode and the method of constructing a subgraph using the deep mode, which will not be described in detail here.

[0126] 509. The neural network graph segmentation device determines whether the subgraph is created successfully.

[0127] If yes, execute 510; if no, execute 512. The neural network graph segmentation device determines whether the creation of the subgraph is successful by determining whether a direct subgraph or a depth subgraph is successfully constructed. Specifically, if the neural network graph segmentation device executes step 508 to obtain a depth subgraph, it is determined that the subgraph is successfully created; if the neural network graph segmentation device executes step 508 to output information indicating that the construction of the subgraph fails, it is determined that the construction of the subgraph fails. Specifically, if the neural network graph segmentation device executes step 507 to obtain a depth subgraph or a direct subgraph, it is determined that the subgraph is successfully created; if the neural network graph segmentation device executes step 507 to output information indicating that the construction of the subgraph fails, it is determined that the construction of the subgraph fails.

[0128] 510. The neural network graph segmentation device outputs subgraphs and address allocation information.

[0129] The address allocation information is obtained in the process of constructing the subgraph. The address allocation information refers to the on-chip address that is allocated to each node in the subgraph during the execution of the processing operation.

[0130] 511. The neural network graph segmentation device updates the next node of the subgraph to be the starting node of the next reference subgraph.

[0131] For example, the sub-graph outputted in step 510 is Figure 6 In the depth subgraph in, updating the next node of the subgraph to be the starting node of the next benchmark subgraph means taking node 5 as the starting node of the next benchmark subgraph.

[0132] 512. The neural network graph segmentation device returns segmentation failure.

[0133] The neural network graph segmentation device may return segmentation failure by outputting information indicating that the neural network graph segmentation has failed.

[0134] 513. Output all sub-graphs and address allocation information.

[0135] From the perspective of minimizing the overhead of the slice memory buffer, the neural network graph can be divided into multiple deep subgraphs. That is, the subgraphs obtained by dividing the neural network graph do not include direct subgraphs.

[0136] In an embodiment of the present application, the neural network graph segmentation device uses two methods of constructing sub-graphs to segment the neural network graph, which can effectively reduce the number of times the external memory is accessed and ensure that the computing performance is not greatly affected.

[0137] The above embodiments are not described in detail. Figure 5 The specific implementation method of step 506 is as follows: Figure 7 This section describes how to determine whether data segmentation is successful. Figure 7 A flow chart of a data segmentation judgment method provided in an embodiment of the present application. Figure 7 As shown, the method may include:

[0138] 701. A neural network graph segmentation device obtains a reference subgraph.

[0139] The neural network graph segmentation device may obtain the reference subgraph by receiving the reference subgraph from the control module by the on-chip address management module (ie, the reference subgraph obtained in step 504 or step 505). Figure 7 The method flow in can be understood as a method flow in which the control module calls the data segmentation module and the on-chip address management module to perform data segmentation on the reference sub-graph.

[0140] 702. The neural network graph segmentation device determines whether to allocate the address of the on-chip buffer for the input and output data within the reference subgraph.

[0141] If yes, execute 703; if no, execute 707. The input and output data within the reference subgraph refers to the input and output data of each node in the reference subgraph during the execution of the processing operation except the input data and output data of the reference subgraph. Figure 6 Take the depth subgraph in as an example of a reference subgraph, the input data of the reference subgraph is the first input data, the output data is the first output data, and the input and output data inside the reference subgraph are the output data of node 1, the input data and output data of node 3, the input data and output data of node 2, and the input data of node 4. The neural network graph segmentation device determines whether to allocate the address of the on-chip buffer for the input and output data inside the reference subgraph by: determining whether the storage space required for the input and output data other than the input and output data of the reference subgraph during the execution of the processing operation of the reference subgraph is greater than the available storage space of the on-chip buffer. Optionally, the neural network graph segmentation device simulates the process of allocating the address of the on-chip buffer for the input and output data inside the reference subgraph; if the allocation is successful, it is determined that the address of the on-chip buffer is allocated for the input and output data inside the reference subgraph; if the allocation is unsuccessful, it is determined that the address of the on-chip buffer is not allocated for the input and output data inside the reference subgraph.

[0142] 703. The neural network graph segmentation device allocates the address of the on-chip buffer for the input and output data within the reference subgraph.

[0143] The neural network graph segmentation device allocates the address of the on-chip buffer for the input and output data inside the reference subgraph by the control module calling the on-chip address management module to allocate the address of the on-chip buffer for the input and output data inside the reference subgraph. The implementation method of step 703 will be described in detail later.

[0144] 704. The neural network graph segmentation device determines whether to allocate an address for the output data of the reference subgraph.

[0145] If yes, execute 705; if no, execute 707. The neural network graph segmentation device determines whether to allocate an address for the output data of the reference subgraph by: determining whether the currently available storage space of the on-chip buffer is greater than the storage space required for the output data of the reference subgraph; if yes, allocate an address for the output data of the reference subgraph; if no, do not allocate an address for the output data of the reference subgraph.

[0146] 705. The neural network graph segmentation device allocates addresses for output data of the reference subgraph.

[0147] The neural network graph segmentation device may allocate an address to the output data of the reference subgraph by the control module calling the on-chip address management module to allocate the address of the on-chip buffer to the output data of the reference subgraph.

[0148] 706. The neural network graph segmentation device reclaims all allocated addresses of the reference subgraph.

[0149] The neural network graph segmentation device reclaims all allocated addresses of the reference subgraph, which can be a storage space occupied by the on-chip address management module releasing currently invalid data (ie, data not needed in subsequent processing operations).

[0150] 707. The neural network graph segmentation device resets the on-chip address management module.

[0151] The neural network graph segmentation device may reset the on-chip address management module by releasing the address of the on-chip buffer allocated by executing step 703 and / or step 705.

[0152] 708. The neural network graph segmentation device determines whether to allocate an address for the output data of the reference subgraph.

[0153] If yes, execute 709; if no, execute 713. Step 708 and step 704 may be implemented in the same manner.

[0154] 709. The neural network graph segmentation device allocates addresses for output data of the reference subgraph.

[0155] Step 709 may be implemented in the same manner as step 705 .

[0156] 710. The neural network graph segmentation device segments the input data of the reference subgraph according to the segmentation rule.

[0157] The neural network graph segmentation device segments the input data of the reference subgraph according to the segmentation rule, which can be that the control module calls the on-chip address management module to segment the input data of the reference subgraph according to the segmentation rule. The implementation method of step 710 will be described in detail later.

[0158] 711. The neural network graph segmentation device allocates an address of an on-chip buffer for input and output data within the reference subgraph when the input data of the reference subgraph is segmented into at least two parts.

[0159] The neural network graph segmentation device allocates the address of the on-chip buffer for the input and output data inside the reference subgraph when the input data of the reference subgraph is segmented into at least two parts. The control module may call the on-chip address management module to allocate the address of the on-chip buffer for the input and output data inside the reference subgraph when the input data of the reference subgraph is segmented into at least two parts. Assuming that the input data of the reference subgraph is divided into a first part of data, a second part of data and a third part of data, the on-chip address management module allocates the address of the on-chip buffer for the input and output data inside the reference subgraph and the output data of the reference subgraph when the first part of data is used as the input data of the reference subgraph, wherein the on-chip address management module can release the address occupied by invalid data (such as data obtained in the process of processing the first part of data) during the address allocation process; the on-chip address management module allocates the address of the on-chip buffer for the input and output data inside the reference subgraph and the output data of the reference subgraph when the second part of data is used as the input data of the reference subgraph, wherein the on-chip address management module can release the address occupied by invalid data (such as data obtained in the process of processing the second part of data) during the address allocation process; the on-chip address management module allocates the address of the on-chip buffer for the input and output data inside the reference subgraph and the output data of the reference subgraph when the third part of data is used as the input data of the reference subgraph. Step 711 is the process of allocating the address of the on-chip buffer, but this process may cause allocation failure due to insufficient storage space currently available in the on-chip buffer. That is to say, step 711 is not necessarily executed successfully, but is a process of attempting to allocate. Because the neural network graph is divided and packaged before step 711 is executed, it is not possible to accurately determine whether the address of the on-chip buffer can be successfully allocated for the input and output data inside the reference subgraph when the input data of the reference subgraph is divided into at least two parts.

[0160] 712. The neural network graph segmentation device determines whether the address of the on-chip buffer is successfully allocated for the input and output data within the reference subgraph.

[0161] Step 712 is implemented in the same manner as step 703. If yes, execute 714; if no, execute 713.

[0162] 713. The neural network graph segmentation device returns unsupported.

[0163] The neural network graph segmentation device may return a message indicating that data segmentation is not supported. If the neural network graph segmentation device returns a message indicating that data segmentation is not supported, the result of the judgment in step 506 is that data segmentation fails.

[0164] 714. The neural network graph segmentation device outputs the address allocation result of the reference subgraph.

[0165] The address allocation result of the reference subgraph output by the neural network graph segmentation device may be the address allocation information obtained in step 702 and the address allocation information obtained in step 704; or the address allocation information obtained in step 708 and the address allocation information obtained in step 711. If the neural network graph segmentation device outputs the address allocation result of the reference subgraph, the result of the judgment in step 506 is that the data segmentation is successful.

[0166] from Figure 7 It can be seen that the neural network graph segmentation device first allocates an address to the reference subgraph without segmenting the input data of the reference subgraph, and then allocates an address to the reference subgraph when the input data of the reference subgraph is segmented. Compared with allocating an address to the reference subgraph when the input data of the reference subgraph is segmented, fewer operations are required to be performed to allocate an address to the reference subgraph when the input data of the reference subgraph is not segmented. In this way, time can be saved, that is, the execution efficiency of step 506 is higher. Because when the input data of the reference subgraph is segmented, the time required to allocate addresses to the input and output data inside the reference subgraph far exceeds the time required to allocate addresses to the output data of the reference subgraph. When allocating addresses to the reference subgraph when the input data of the reference subgraph is segmented, first allocate addresses to the output data of the reference subgraph, and then allocate addresses to the input and output data inside the reference subgraph; when the output data of the reference subgraph is not successfully allocated, the address allocation process is terminated in time, which can effectively save time.

[0167] The above embodiments are not described in detail. Figure 5 The implementation of step 507 is as follows: Figure 8 To describe how to construct a subgraph using Direct mode. Figure 8 A flow chart of a method for constructing a subgraph using the Direct mode provided in an embodiment of the present application. Figure 8 As shown, the method may include:

[0168] 801. A neural network graph segmentation device obtains a reference subgraph.

[0169] The neural network graph segmentation device obtains the reference subgraph by the graph segmentation module receiving the reference subgraph from the control module (ie, the reference subgraph obtained in step 504 or step 505). Figure 8 The method flow in can be understood as a process in which the control module calls the graph segmentation module and the on-chip address management module to construct a deep subgraph or a direct subgraph according to the reference subgraph.

[0170] 802. The neural network graph segmentation device adds the node to be segmented in the neural network graph to the reference subgraph.

[0171] The neural network graph segmentation device adds the node to be segmented in the neural network graph to the reference subgraph, which can be a graph segmentation module adding the node to be segmented in the neural network graph to the reference subgraph. Adding the node to be segmented in the neural network graph to the reference subgraph can be understood as expanding the reference subgraph by one node. For example, the reference subgraph includes the following nodes: Figure 6 Node 1 and node 2 in the neural network graph are Figure 6 Node 3 in , adds node 3 to the base subgraph, and the base subgraph includes nodes changed to nodes 1 to 3.

[0172] 803. The neural network graph segmentation device performs data segmentation judgment.

[0173] The neural network graph segmentation device performs data segmentation judgment, which can be performed by the control module calling the data segmentation module and the on-chip address management module. Figure 7 The method flow in .

[0174] 804. The neural network graph segmentation device determines whether the data segmentation is successfully executed.

[0175] Step 804 and Figure 5 The same as step 506 in . If the neural network graph segmentation device performs data segmentation judgment and returns not supported, it is determined that the data segmentation execution fails; if the neural network graph segmentation device outputs the address allocation result of the reference subgraph, it is determined that the data segmentation execution is successful. If so, execute step 805; if not, execute step 806.

[0176] 805. The neural network graph segmentation device determines whether the input data of the reference subgraph is segmented.

[0177] If so, execute step 806; if not, execute step 802. In the case where the neural network graph segmentation device successfully performs data segmentation, the address allocation result of the reference subgraph is output. Optionally, the address allocation result may include information indicating whether the input data of the reference subgraph is segmented, and the neural network graph segmentation device determines whether the input data of the reference subgraph is segmented based on the information. It can be understood that the address allocation result output when the input data of the reference subgraph is segmented is different from the address allocation result output when the input data of the reference subgraph is not segmented. Optionally, the neural network graph segmentation device determines whether the input data of the reference subgraph is segmented based on the address allocation result.

[0178] 806. The neural network graph segmentation device generates a direct subgraph or a deep subgraph.

[0179] If the data segmentation is not successful, the benchmark subgraph is used as a direct subgraph; if the data segmentation is successful and the input data of the benchmark subgraph is segmented, the benchmark subgraph is used as a deep subgraph.

[0180] In the embodiment of the present application, the sub-graph is constructed in Direct mode, so that a direct sub-graph or a depth sub-graph can be constructed quickly.

[0181] The above embodiments are not described in detail. Figure 5 The implementation of step 508 is as follows: Fig. 9 To describe how to construct a subgraph using the Depth mode. Fig. 9 A flowchart of a method for constructing a subgraph using the Depth mode provided in an embodiment of the present application. Fig. 9 As shown, the method may include:

[0182] 901. A neural network graph segmentation device obtains a reference subgraph.

[0183] The neural network graph segmentation device obtains the reference subgraph by the graph segmentation module receiving the reference subgraph from the control module (ie, the reference subgraph obtained in step 504 or step 505). Fig. 9 The method flow in can be understood as a process in which the control module calls the graph segmentation module, the data segmentation module and the on-chip address management module to construct a deep sub-graph according to the reference sub-graph.

[0184] 902. The neural network graph segmentation device adds the node to be segmented in the neural network graph to the reference subgraph.

[0185] The neural network graph segmentation device adds the current node to be segmented in the neural network graph to the reference subgraph, which can be a graph segmentation module adding the current node to be segmented in the neural network graph to the reference subgraph. Adding the current node to be segmented in the neural network graph to the reference subgraph can be understood as expanding the reference subgraph by one node.

[0186] 903. The neural network graph segmentation device performs data segmentation judgment.

[0187] The neural network graph segmentation device performs data segmentation judgment, which can be performed by the control module calling the data segmentation module and the on-chip address management module. Figure 7 The method flow in .

[0188] 904. The neural network graph segmentation device determines whether the data segmentation is successfully executed.

[0189] If yes, execute step 905; if no, execute step 907. Steps 901 to 904 correspond to steps 801 to 804 in sequence.

[0190] 905. The neural network graph segmentation device determines whether the current node is a special node.

[0191] Special nodes may include Pooling nodes, etc. The current node refers to a node most recently added in step 902 (i.e., a node expanded from the baseline subgraph in step 902). If the current node is a feature node, execute step 906; if the current node does not belong to a feature node, execute step 902. Determining whether the current node is a special node is only an optional method provided by an embodiment of the present application to determine whether to output the current baseline subgraph as a depth subgraph. In actual applications, the neural network graph segmentation device can also output the current baseline subgraph as a depth subgraph when the current basic subgraph satisfies other components.

[0192] 906. The neural network graph segmentation device uses the reference subgraph as the depth subgraph.

[0193] 907. The neural network graph segmentation device determines that the deep mode fails to construct a subgraph.

[0194] In the embodiment of the present application, the Depth mode is adopted to construct the sub-graph, so that the depth sub-graph can be constructed quickly, and the failure of constructing the sub-graph can be confirmed in time when the sub-graph is not successfully constructed.

[0195] The above embodiments are not described in detail. Figure 7 Implementation of step 710 in . The data segmentation rule is to ensure that the height of each group of output data is as large as possible under the premise that the on-chip buffer can complete the space allocation, so as to reduce the number of PipeLines mapped by the depth submap, thereby improving the calculation efficiency. In some embodiments, the output data is a three-dimensional matrix or a four-dimensional matrix, and the height of the output data refers to the height of the three-dimensional matrix or the four-dimensional matrix. Therefore, it can be attempted by looping on the height of the original output data in the form of continuously decreasing the target value (for example, 1). If the address space allocation requirements are met, the segmentation is stopped, and the height of the obtained output data is the required height. The output data of the depth submap can be a four-dimensional data block of [N, C, H, W], H corresponds to its height, and the height of the output data is determined by continuously cutting "thin". The output data of the depth submap can also be a three-dimensional data block of [C, H, W], H corresponds to its height size, and the corresponding segmentation is obtained by continuously cutting "thin". Among them, N, C, H and W are all integers greater than 0. N can represent the number of data blocks corresponding to the output data, C represents the number of channels, and W represents the width of the input data. Combine the following Fig.10 To describe how to segment the input data of the reference sub-graph (depth sub-graph) according to the segmentation rule, that is, Figure 7 Implementation method of step 710 in .

[0196] Fig.10 A flow chart of a data segmentation method provided in an embodiment of the present application. Fig.10 As shown, the method may include:

[0197] 1001. A neural network graph segmentation device determines whether a height of output data of a depth sub-graph is a height threshold.

[0198] The height threshold may be 1. Assuming that the output data is a three-dimensional matrix of [5, 10, 256], the height of the output data is 10. If yes, end this process; if no, execute step 1002.

[0199] 1002. The neural network graph segmentation device subtracts the target value from the height of the output data of the depth sub-graph to obtain a reference height.

[0200] The target value can be 1, 2, 3, etc., which is not limited in this application. The reference height refers to the height of the output data minus the target value. The target value represents the height value of the output data each time the height is reduced. For example, the output data is a three-dimensional matrix of [5, 10, 256], the target value is 1, and the height of the output data 10 minus the target value 1 (corresponding to the reference height) is 9.

[0201] 1003. The neural network graph segmentation device segments the input data of the depth sub-graph according to the reference height.

[0202] The neural network graph segmentation device segments the input data of the depth subgraph according to the reference height. The data segmentation module can divide the height of the complete output data of the depth subgraph (i.e., the original height) by the reference height, and round up to obtain the number of segmented PipeLines; the input data of the depth subgraph is segmented according to the number of PipeLines; wherein each group of PipeLines corresponds to a group of input data. The height of the complete output data of the depth subgraph refers to the height of the output data of the depth subgraph before the height of the output data of the depth subgraph is adjusted. For example, the original height of the output data of the depth subgraph is 7, and the height of the adjusted input data is 4. 7 / 4 is calculated and rounded up to obtain 2 (the number of PipeLines), and the input data of the depth subgraph is segmented into two parts.

[0203] 1004. The neural network graph segmentation device allocates an address of an on-chip buffer to the depth sub-graph.

[0204] The address of the on-chip buffer allocated by the neural network graph segmentation device to the depth subgraph may be the address of the on-chip buffer allocated by the on-chip address manager to the input and output data of the multiple groups of PipeLine corresponding to the depth subgraph during the processing operation. It can be understood that the on-chip address manager allocates the address of the on-chip buffer to the input and output data of the multiple groups of PipeLine corresponding to the depth subgraph in the process of executing the processing operation according to the information of the storage space required to be occupied by the input and output data of the multiple groups of PipeLine corresponding to the depth subgraph in the process of executing the processing operation, and does not require the multiple groups of PipeLine corresponding to the depth subgraph to perform the processing operation.

[0205] 1005. The neural network graph segmentation device recovers the on-chip address of invalid data (the address of the on-chip buffer).

[0206] Invalid data refers to data that is not needed in subsequent processing operations. For example, the input data of the depth submap is divided into two parts, namely the first part and the second part. When the first part is processed as input data to obtain the first output, the storage space occupied by other data used in the process of processing the first part is released, and only the first output is retained; when the second part is processed as input data to obtain the second output, the storage space occupied by other data used in the process of processing the second part is released, and only the second output is retained. In other words, the on-chip address management module can adopt a memory reuse mechanism to continuously recycle the storage space of the currently available on-chip buffer.

[0207] 1006. The neural network graph segmentation device determines whether the address is allocated successfully.

[0208] If yes, end this process; if no, execute step 1001.

[0209] The input data of the depth subgraph has three forms after being segmented: 1) Graph Head form; 2) Graph Body form; 3) Graph Tail form. Among them, if the OP corresponding to the depth subgraph is a convolution operator, etc., it generally contains padding information. The Graph Head contains Up Padding, the Graph Body does not contain Up Padding and Down Padding, and the Graph Tail contains Down Padding. After the input data of the depth subgraph is segmented, the corresponding address offset needs to be calculated for the corresponding PipeLine (after the input data is segmented, each segmentation block (each group of data) needs to access different storage spaces for independent processing). After the input data of the depth subgraph is segmented, at least two groups of data may or may not have overlapping parts. The overlap between two groups of data means that the two adjacent groups of data that are segmented contain the same data. Fig.11A This is a schematic diagram of three groups of data obtained by segmenting the input data. Fig.11A It can be seen that the input data is divided into three parts, namely the header data, the main data and the tail data, among which there is overlap between the header data and the main data ( Fig.11A There is also overlap between the main data and the tail data ( Fig.11AIn practical applications, when the input data of the depth subgraph may be divided into four or more parts, a header data, a tail data and a plurality of main body data may be obtained. The storage space occupied by each main body data is the same. The on-chip address management module in the neural network graph segmentation device calculates the address offset of each piece of data obtained by segmenting the input data as follows:

[0210] GraphHeadaddr=inputTensor addr;

[0211] GraphBodyaddr=GraphHead addr+GraphHead data size-overlapSize;

[0212] GraphTailaddr=GraphBody addr+(Graph Body data size-overlapSize)*(loop-2).

[0213] Among them, input Tensor addr represents the base address of the input data, GraphHead addr represents the base address of the header data, GraphHead data size represents the storage address occupied by the header data, overlapSize represents the storage address occupied by the data overlapping the main data and the header data, GraphBodyaddr represents the base address of the main data, GraphTailaddr represents the base address of the tail data, Graph Body data size represents the storage address occupied by one copy of the main data, and loop represents the number of copies into which the input data is segmented. Fig. 11B A schematic diagram of output data. Fig. 11B The output data in the depth sub-map is the PipeLine corresponding to the depth sub-map. Fig.11A The three sets of data in are processed as input data to obtain three sets of output data. Among them, Fig.11A The header data, main data and external data obtained by splitting the input data correspond to Fig. 11B The output data includes header data, main body data and external data. The on-chip address management module in the neural network graph segmentation device calculates the address offset of the output data in the same way as the address offset of the input data.

[0214] In the embodiment of the present application, after being processed by the data segmentation module, the maximum height of the output data of the depth sub-map that can be supported by the current on-chip buffer is obtained, which can reduce the number of PipeLines mapped by the depth sub-map to improve the calculation efficiency.

[0215] The above embodiments are not described in detail. Figure 7 The implementation of step 703 and step 711 in Fig.12 To describe how to allocate the address of the on-chip buffer for the input and output data within the reference sub-graph. Fig.12 A flow chart of an on-chip storage allocation method provided in an embodiment of the present application. Fig.12 As shown, the method may include:

[0216] 1201. The neural network graph segmentation device obtains the current node.

[0217] The current node refers to the node in the subgraph that is currently assigned an on-chip address to be assigned. The neural network graph segmentation device can allocate storage space of the on-chip buffer for the output data (ie, output edge) of each node in turn according to the order in which the nodes in the subgraph execute processing operations.

[0218] 1202. The neural network graph segmentation device determines whether the current node is the last node in the subgraph.

[0219] If not, execute step 1203; if so, execute step 1210.

[0220] 1203. The neural network graph segmentation device reclaims the currently releasable storage space of the on-chip buffer.

[0221] The currently releasable storage space of the on-chip buffer refers to the storage space of the on-chip buffer occupied by currently invalid data. By recycling the currently releasable storage space of the on-chip buffer, the reuse rate of the on-chip address can be greatly improved.

[0222] 1204. The neural network graph segmentation device obtains the input edge of the current node.

[0223] In this application, an edge refers to the input data and output data of a node; wherein an input edge refers to the input data of a node, and an output edge refers to the output data of a node.

[0224] 1205. The neural network graph segmentation device determines whether the output edge of the current node is an output edge of the subgraph.

[0225] The outgoing edge of a subgraph refers to the outgoing edge of the last node in the subgraph. Figure 6 For example, the output data of node 4 is the output edge of the subgraph, and the output data of nodes 1 to 3 are all the output edges of the nodes, not the output edges of the subgraph. If not, execute step 1206; if so, execute step 1201.

[0226] 1206. The neural network graph segmentation device allocates the address of the on-chip buffer to the output edge of the current node.

[0227] 1207. The neural network graph segmentation device determines whether the address of the on-chip buffer is successfully allocated to the output edge of the current node.

[0228] If so, execute step 1208; if not, execute step 1210.

[0229] 1208. The neural network graph segmentation device records the processed nodes.

[0230] The neural network graph segmentation device records the processed nodes and can record the current node as a processed node.

[0231] 1209. The neural network graph segmentation device records the processed edges.

[0232] 1210. The neural network graph segmentation device releases all allocated edges.

[0233] The neural network graph segmentation device releases all allocated edges, which may be to release the storage space occupied by all input data and output data corresponding to the subgraph in the on-chip buffer. In addition to performing on-chip address allocation for the output data of the subgraph, the on-chip buffer also provides on-chip address allocation for the input and output data of each node in the subgraph. If there is a situation where the on-chip buffer cannot store a subgraph, all allocated edges are released and failure is directly returned, indicating that the neural network graph segmentation cannot be performed.

[0234] In an embodiment of the present application, the neural network graph segmentation device adopts a memory reuse mechanism to allocate on-chip addresses to the output edges of each node in the subgraph, which can greatly reduce the overhead of the on-chip addresses and is simple to implement.

[0235] The following describes how the on-chip buffer reclaims the currently releasable storage space of the on-chip buffer. Fig.13 A method flow for releasing storage space of an on-chip buffer provided in an embodiment of the present application. Fig.13 As shown, the method may include:

[0236] 1301. The neural network graph segmentation device obtains the current edge of the subgraph.

[0237] The current edge refers to the input data and output data of the current node. The neural network graph segmentation device can obtain the edge of each node in turn according to the order in which the nodes in the subgraph execute the processing operations. Figure 6 Taking the depth subgraph in as an example, the neural network graph segmentation device executes depth subgraph calculation instruction pipeline 0 and then executes depth subgraph calculation instruction pipeline 1. After executing depth subgraph calculation instruction pipeline 0, the neural network graph segmentation device can sequentially recycle the storage space of the on-chip buffer occupied by the edges other than the output edges of the depth subgraph (input data 0 and data 10).

[0238] 1302. The neural network graph segmentation device determines whether the current edge is an output edge of the subgraph.

[0239] If yes, execute step 1301; if no, execute step 1303.

[0240] 1303. The neural network graph segmentation device releases the storage space of the on-chip buffer occupied by the current edge.

[0241] Optionally, the neural network graph segmentation device can use a memory reuse mechanism to release currently available storage space in the on-chip buffer during the execution of PipeLine.

[0242] In the embodiment of the present application, the neural network graph segmentation device can effectively solve the problem of insufficient storage space in the on-chip buffer by releasing the currently available storage space in the on-chip buffer.

[0243] Some operations in the above embodiments involve on-chip address allocation, such as step 702, step 704, and step 708. The following describes the on-chip address allocation process. The on-chip address allocation process can be divided into two stages. The first stage is to find the best discrete block (i.e., the best storage space) that can currently store the data to be stored from the on-chip buffer, and the second stage is to determine the address space in the best discrete block that is more suitable for storing the data to be stored. Each discrete block corresponds to a continuous storage space in the on-chip buffer. Fig.14 The present invention provides an on-chip address allocation method process. Fig.14 As shown, the method may include:

[0244] 1401. The neural network graph segmentation device determines the best discrete block in the on-chip buffer that can currently store the data to be stored.

[0245] The data to be stored may be output data or input data of a node in the subgraph. The neural network graph segmentation device determines that the best discrete block in the on-chip buffer that can currently store the data to be stored may be traversed by the on-chip address management module in the order of the size of the discrete blocks currently available in the on-chip buffer, so as to take the smallest discrete block that can store the data to be stored as the best discrete block. The size of the discrete block refers to the size of the storage space of the discrete block.

[0246] 1402. The neural network graph segmentation device determines the target address space in the optimal discrete block.

[0247] The target address space is used to store the data to be stored. In practical applications, the storage space of the best discrete block is often larger than the storage space required to store the data to be stored. In order to reduce the overhead of on-chip storage space, the neural network graph segmentation device can allocate the target address space in the best discrete block to the data to be stored. The Ping-Pong Allocation strategy provided in the present application can further determine the portion of space in the best discrete block that is more suitable for storing the data to be stored, so as to reduce the overhead of on-chip storage space.

[0248] In the embodiment of the present application, the optimal discrete block is first determined, and then the target address space in the optimal discrete block is further determined, which can greatly reduce the overhead of the on-chip storage space.

[0249] Fig.14 The implementation of step 1402 is not described in detail. Fig.15 To describe how to determine the target address space in optimal discrete blocks. Fig.15 A flow chart of another on-chip address allocation method provided in an embodiment of the present application. Fig.15 The method flow in is the method flow of the ping-pong allocation strategy provided by this application. Fig.15 As shown, the method may include

[0250] 1501. The neural network graph segmentation device obtains the best discrete blocks.

[0251] The neural network graph segmentation device may obtain the best discrete block by the on-chip address management module. The best discrete block is a discrete block for storing the data of the current node. In other words, the best discrete block is a discrete block determined for storing the data of the current node.

[0252] 1502. The neural network graph segmentation device sorts the discrete blocks in the on-chip buffer in ascending order according to the base address.

[0253] Each discrete block in the on-chip buffer includes available discrete blocks and occupied discrete blocks.

[0254] 1503. The neural network graph segmentation device determines whether the total discrete address space of the on-chip buffer is complete.

[0255] The neural network graph segmentation device determines whether the total discrete address space of the on-chip buffer is complete by determining whether the storage space of the on-chip buffer is occupied; if the storage space of the on-chip buffer is not occupied, the total discrete address space of the on-chip buffer is complete, otherwise, the total discrete address space of the on-chip buffer is incomplete. If yes, execute step 1511; if no, execute step 1504.

[0256] 1504. The neural network graph segmentation device determines whether the best discrete blocks are located at both ends of the storage space of the on-chip buffer.

[0257] The addresses of the storage space of the on-chip buffer are arranged from low to high in the order of the base address to the end address. One end of the storage space of the on-chip buffer may be a continuous storage space including the base address of the on-chip buffer, and the other end may be a continuous storage space including the end address of the on-chip buffer. For example, one end of the storage space of the internal buffer may be the storage space from the base address to the first address, and the other end of the storage space of the internal buffer may be the storage space from the second address to the end address; wherein the size of the storage space from the base address to the first address is one tenth, one eighth, etc. of the size of the entire storage space of the on-chip buffer, and the size of the storage space from the second address to the end address is one tenth, one eighth, etc. of the size of the entire storage space of the on-chip buffer. If yes, execute step 1505; if no, execute step 1508.

[0258] 1505. The neural network graph segmentation device determines whether the data to be stored will occupy the address of the on-chip buffer for a long time.

[0259] The neural network graph segmentation device determines whether the data to be stored will occupy the address of the on-chip buffer for a long time by determining whether the data to be stored is only needed in the M groups of PipeLines executed most recently; if so, it is determined that the data to be stored will not occupy the address of the on-chip buffer for a long time; if not, it is determined that the data to be stored will occupy the address of the on-chip buffer for a long time. M is an integer greater than 0 such as 1, 2, 3, 4. If yes, execute step 1506; if not, execute step 1507.

[0260] 1506. The neural network graph segmentation device allocates storage addresses at both ends of the optimal discrete block for the data to be stored.

[0261] One end of the best discrete block may be a continuous storage space including the base address of the best discrete block, and the other end may be a continuous storage space including the end address of the best discrete block. Fig.17 A schematic diagram of an optimal discrete block provided in an embodiment of the present application. Fig.17As shown, the storage addresses of the best discrete blocks are sorted from low to high from the base address to the end address, 1601 is one end of the best discrete block including the base address, 1603 is the other end of the best discrete block including the end address, and 1602 is the storage address of the best discrete block away from both ends. For example, the size of the best discrete block is 100KB, and the data to be stored needs to occupy 40KB, then the neural network graph segmentation device allocates the first 40KB or the last 40KB of the 100KB. In other words, when the data to be stored needs to occupy a part of the storage space in the best discrete block for a long time, the address close to the two ends of the best discrete block is selected, so that a part of the continuous space can be left to store other data, so that the address space can be more fully utilized.

[0262] 1507. The neural network graph segmentation device allocates optimal discrete blocks of storage addresses away from both ends for the data to be stored.

[0263] The storage addresses far from the two ends of the best discrete block refer to the storage addresses corresponding to the best discrete block except the storage addresses at the two ends of the best discrete block. The storage addresses far from the two ends of the best discrete block are allocated to the data to be stored that does not need to occupy the storage space for a long time, so as to recycle this part of the addresses in time.

[0264] 1508. The neural network graph segmentation device determines whether the forward address block stores input data of the current node.

[0265] The forward address block (corresponding to the third storage space) refers to a discrete block in the storage space of the on-chip buffer that is adjacent to the best discrete block (corresponding to the second storage space) and is located before the best discrete block. The data to be stored may be the output data of the current node. If yes, execute step 1510; if no, execute step 1509.

[0266] 1509. The neural network graph segmentation device allocates the low address (corresponding to the third address) of the best discrete block to the data to be stored.

[0267] The low address of the best discrete block can be half of the address including the base address, and the high address of the best discrete block can be the other half of the address including the end address. For example, if the size of the best discrete block is 100KB and the data to be stored needs to occupy 40KB, the neural network graph segmentation device allocates the first 40KB of the 100KB.

[0268] 1510. The neural network graph segmentation device allocates a high address (corresponding to the fourth address) of the best discrete block to the data to be stored.

[0269] When the forward address block stores input data of the current node, allocating a high address of the best discrete block for the data to be stored can form a larger continuous storage space with the storage space corresponding to the low address in the best discrete block when the forward address block is released.

[0270] 1511. The neural network graph segmentation device allocates the low address of the best discrete block to the data to be stored.

[0271] 1512. The neural network graph segmentation device obtains an assigned address.

[0272] In an embodiment of the present application, when the storage space of the best discrete block is larger than the storage space required to store the data to be stored, determining the storage space in the best discrete block that is more suitable for storing the data to be stored can further reduce the number of discrete blocks in the on-chip buffer.

[0273] The above embodiment describes the neural network graph segmentation device using Figure 3A The neural network graph segmentation processing system in the embodiment performs graph segmentation (i.e., first-level graph segmentation) on the neural network graph to obtain direct subgraphs and deep subgraphs. The following describes the implementation method of performing graph segmentation (i.e., second-level graph segmentation) on the deep subgraph whose depth is not less than the first threshold.

[0274] In an optional implementation, after the neural network graph is divided into the neural network graph to obtain a deep subgraph (i.e., a first-order subgraph), the neural network graph segmentation device may also perform the following operations: when the number of nodes included in the deep subgraph is not less than a first threshold, the deep subgraph is segmented to obtain a first second-order subgraph and a second second-order subgraph; the first second-order subgraph is used to characterize the first second-order subnetwork, the second second-order subgraph is used to characterize the second second-order subnetwork, the first second-order subnetwork and the second second-order subnetwork are both included in the deep subnetwork, and the nodes included in the first second-order subnetwork and the second second-order subnetwork are different. Optionally, the input data of the first second-order subnetwork is the first input data, the output data of the second second-order subnetwork is the first output data, the first second-order subnetwork is used to store the first intermediate data obtained by processing the first input data to an intermediate buffer, the second second-order subnetwork is used to process the first intermediate data obtained from the intermediate buffer, and the intermediate buffer is not the on-chip buffer, that is, the intermediate buffer is located outside the AI ​​chip. Exemplarily, the intermediate buffer is an off-chip buffer having a lower read and write speed than the on-chip buffer, and the read and write speed of the intermediate buffer is faster than an external memory, such as DDR.

[0275] Exemplarily, the neural network graph segmentation device segments the depth subgraph to obtain the first second-order subgraph and the second second-order subgraph in the following manner: determine at least one reference node among the multiple nodes included in the depth subgraph whose storage space required for output data is less than the storage space available in the intermediate buffer; segment the depth subgraph using the output of the intermediate node in the at least one reference node as the segmentation point to obtain the first second-order subgraph and the second second-order subgraph, the intermediate node being any reference node in the at least one reference node, the output data of the intermediate node being the output data of the first second-order subgraph, and being the input data of the second second-order subgraph. It should be understood that the neural network graph segmentation device can also segment at least one depth subgraph whose depth is not less than the first threshold in other ways to obtain at least two second-order subgraphs, which is not limited in the embodiments of the present application. The depth subgraph can be any depth subgraph obtained by segmenting the neural network graph. It should be understood that the neural network graph segmentation device can segment each depth subgraph whose depth is not less than the first threshold in a similar manner. Optionally, the neural network graph segmentation device attempts to allocate subgraph addresses to the second-order subgraph obtained by segmenting the depth subgraph using the available storage space in the on-chip buffer (see Figure 7 ), if the data does not need to be segmented, the second-order subgraph is a direct subgraph, otherwise, the second-order subgraph is a depth subgraph. In some embodiments, after the neural network graph segmentation device segments the depth subgraph to obtain at least two second-order subgraphs, the control module 401 can be used to control the on-chip address management module 404 to assign addresses to the obtained second-order subgraphs to obtain address allocation information. The implementation method is similar to the method of assigning addresses to the depth subgraph and the direct subgraph, and will not be repeated here.

[0276] Optionally, before the neural network graph segmentation device segments the depth subgraph with the output of the intermediate node in the at least one reference node as the segmentation point to obtain the first second-order subgraph and the second second-order subgraph, it can perform the following operations: obtain the depth difference between the two second-order subgraphs obtained by segmenting the depth subgraph with the at least one reference node as the segmentation point, and obtain at least one depth difference, wherein the at least one reference node corresponds to the at least one depth difference one by one; determine to use the output of the intermediate node in the at least one reference node whose corresponding depth difference is less than the depth difference threshold as the segmentation point to segment the depth subgraph. Optionally, determine to use the intermediate node in the at least one reference node whose corresponding depth difference is the smallest as the segmentation point to segment the depth subgraph. The depth difference threshold can be 1, 2, 3, etc. For example, the nodes in the depth subgraph are node 1, node 2, node 3, node 4, and node 5 in order of execution. The depth subgraph is divided by the output of node 1 as the dividing point to obtain a second-order subgraph including node 1 (with a depth of 1) and a second-order subgraph including node 2, node 3, node 4, and node 5 (with a depth of 4). The depth difference between the two second-order subgraphs is 3. For another example, the nodes in the depth subgraph are node 1, node 2, node 3, node 4, and node 5 in order of execution. The depth subgraph is divided by the output of node 3 as the dividing point to obtain a second-order subgraph including node 1, node 2, node 3 (with a depth of 3) and a second-order subgraph including node 4 and node 5 (with a depth of 2. The depth difference between the two second-order subgraphs is 1. In this implementation, the depth subgraph can be quickly divided into two second-order subgraphs whose depth difference is less than the depth difference threshold.

[0277] An example of segmenting a depth subgraph whose segmentation depth is not less than a first threshold is introduced below. After completing the segmentation of the first-level graph of the neural network graph, the neural network graph segmentation device can retain the input data address and output data address of each depth subgraph and direct subgraph.

[0278] Assume that the neural network graph segmentation device segments the neural network graph to obtain a deep subgraph including nodes 1 to 5 and a direct subgraph including node 6. The neural network graph segmentation device can first determine which node in the depth subgraph has its output data stored in the intermediate buffer. The specific process is as follows: determine whether the internal output in the depth subgraph, such as the output data of nodes 1, 2, 3, and 4, can be placed in the intermediate buffer, and then filter out each node that can place the output data in the intermediate buffer to obtain an output list. According to the output list, calculate the depth difference between the two second-order subgraphs obtained by segmenting the depth subgraph. For example, the depth subgraph is segmented with the output of OP1 as the segmentation point to obtain two second-order subgraphs, one of which includes node 1 and has a depth of 1, and the other includes nodes 2 to 5 and has a depth of 4, then the depth difference is 3. By analogy, the depth difference corresponding to the output of OP1 as the segmentation point is 3, the depth difference corresponding to the output of OP2 as the segmentation point is 1, the depth difference corresponding to the output of OP3 as the segmentation point is 1, and the depth difference corresponding to the output of OP4 as the segmentation point is 3. Exemplarily, the output of the node with the smallest depth difference and relatively large output data size is determined as the segmentation point. For example, in this example, the depth difference between the outputs of OP2 and OP3 is 1. If the output size of OP3 is larger than the output size of OP2, the output of OP3 is used as the segmentation point of the depth subgraph, and the output of OP3 is set to the intermediate buffer. In this way, two second-order subgraphs (OP1-OP3) and (OP4-OP5) can be formed. Each second-order subgraph obtained by segmenting a depth subgraph can be a direct subgraph or a depth subgraph. Optionally, after the neural network graph segmentation device segments the depth subgraph to obtain at least one direct subgraph, it can merge all the direct subgraphs connected before and after to form a new direct subgraph. For example Fig. 16B The second-order subgraph OP4-OP5 and the first-order subgraph OP6 are merged to form a new direct subgraph OP4-OP6 (data is read from the intermediate buffer). Optionally, the neural network graph segmentation device can compile all the final subgraphs to form an AI chip executable file (including chip instructions, read and write addresses, etc.) according to the result of the subgraph merging.

[0279] In some embodiments, if the depth of N deep subgraphs obtained by the neural network graph segmentation device by segmenting the neural network graph is not less than the first threshold, then any 1, 2, 3, ..., N-1 of the N deep subgraphs are segmented according to the above method, and W groups of neural network graph segmentation results can be obtained, plus the situation where the secondary graph segmentation is not performed, a total of (W+1) situations. In some embodiments, the neural network graph segmentation device can generate AI chip executable files corresponding to the (W+1) groups of neural network graph segmentation results respectively, and use test data to test the execution time of each AI chip executable file to process the test data respectively. The neural network graph segmentation device can compare the execution time of (W+1) AI chip executable files (calculating the same input data scale), select the smallest situation as the final result and deploy it on the terminal device (such as a mobile phone, smart monitoring equipment, etc.) or even on the cloud. Overall, one of the neural network graph segmentation results, due to the smaller depth of the depth subgraph, the amount of repeated calculation data (such as the overlap caused by the convolution operator) is reduced, so that the performance of the entire network is further improved.

[0280] The following is a description of the difference between the calculation form of the nodes included in the depth subgraph (ie, the first-order depth subgraph) and the calculation form of the nodes included in the second-order subgraph, with reference to the accompanying drawings.

[0281] Fig.16A A computing form based on single-level storage in an AI chip provided in an embodiment of the present application. Fig.16A It can be seen that the deep subnetwork includes nodes 1 to 5, and the input data of the deep subnetwork is divided into 4 parts, that is, the deep subnetwork corresponds to 4 groups of PipeLine, the direct subnetwork includes node 6, and the output data of the deep subnetwork (that is, the output data of node 5) is stored in the on-chip buffer, and the direct subnetwork reads data from the on-chip buffer. Among them, node 1 reads data from DDR or reads data from the on-chip buffer, and node 5 writes the output data to the on-chip buffer or DDR. If node 1 is the first node in the neural network graph to perform processing operations, node 1 reads data from DDR, otherwise, it reads data from the on-chip buffer. If node 6 is the last node in the neural network graph to perform processing operations, node 6 writes the output data to DDR, otherwise, node 6 writes the output data to the on-chip buffer. Fig.16AThe processing flow executed by each node in is as follows: 1) Read 1 / 4 Tensor (i.e., 1 / 4 input data), and pass through the calculation of node 1, node 2, node 3, node 4, and node 5 in turn, and store the calculation result of node 5 (i.e., 1 / 4 calculation result) to the on-chip buffer, where data is read from DDR or the on-chip buffer, and address reading and writing are completed in the on-chip buffer; 2) Read the subsequent 1 / 4 Tensor, and pass through the calculation of node 1, node 2, node 3, node 4, and node 5 in turn, and store the calculation result of node 5 (i.e., 1 / 4 calculation result) to the on-chip buffer; 3) Read The subsequent 1 / 4Tensor is calculated by nodes 1, 2, 3, 4, and 5 in sequence, and the calculation result of node 5 (i.e., 1 / 4 of the calculation result) is stored in the on-chip buffer; 4) The subsequent 1 / 4Tensor is read and calculated by nodes 1, 2, 3, 4, and 5 in sequence, and the calculation result of node 5 (i.e., 1 / 4 of the calculation result) is stored in the on-chip buffer; 5) The complete Tensor (i.e., the complete calculation result output by node 5) is read from the on-chip buffer, and after calculation by node 6, the calculation result is written to DDR or the on-chip buffer.

[0282] Fig. 16B A computing form based on multi-level storage in an AI chip provided in an embodiment of the present application. Fig. 16B In the example, the calculation process performed by the AI ​​chip is as follows: (1) The AI ​​chip reads the data to be processed (Input) from the DDR; (2) The AI ​​chip uses the PipeLine corresponding to the depth sub-map or direct sub-map as the basic unit for calculation, where the calculation results of multiple groups of PipeLine of at least one depth sub-map are converged to the intermediate buffer, and the PipeLine of the direct sub-map reads and writes the on-chip buffer or the intermediate buffer, completing the calculation OP by OP; (3) After completing the entire NN calculation, the result is saved to the DDR. Fig. 16B Shown by Fig.16A An example of the computational form of the second-order subnetwork obtained by deep subgraph segmentation in . Fig. 16B It can be seen that Fig.16AThe deep subnetwork in is divided into a first second-order subnetwork (deep subnetwork) including nodes 1, 2 and 3, and a second second-order subnetwork (direct subnetwork) including nodes 4 and 5. The input data of the first second-order subnetwork is divided into two parts, that is, the first second-order subnetwork corresponds to 2 groups of PipeLine, and the output data of the first second-order subnetwork (that is, the output data of node 3) is stored in the intermediate buffer, and the second second-order subnetwork reads data from the intermediate buffer. Among them, node 1 reads data from DDR or reads data from the on-chip buffer, and node 5 writes the output data to the on-chip buffer. If node 1 is the first node in the neural network graph to perform a processing operation, node 1 reads data from DDR, otherwise, it reads data from the on-chip buffer. Exemplarily, the neural network graph segmentation device uses the output of node 3 as Fig.16A The deep subnetwork in is split by the splitting point to obtain a second-order subnetwork including nodes 1 to 3 and a second-order subnetwork including nodes 4 and 5. The second-order subnetwork including nodes 4 and 5 and the direct subnetwork including node 6 are merged to obtain a new direct subnetwork including nodes 4 to 6. Fig.16A and Fig. 16B It can be seen that the processing operations implemented by the multiple second-order subgraphs obtained by cutting any deep sub-network are the same as the processing operations implemented by any deep sub-network.

[0283] Due to the introduction of a readable and writable intermediate buffer (Middle Buffer) in the calculation process, Fig. 16B The processing flow executed by each node in is as follows: 1) Read 1 / 2Tensor input (i.e. 1 / 2 input data), after calculation by node 1, node 2, and node 3, store the calculation result of node 3 (i.e. 1 / 2 calculation result) in the intermediate buffer, where the input comes from DDR or on-chip buffer, and address reading and writing are completed in the on-chip buffer; 2) Read the subsequent 1 / 2Tensor (i.e. subsequent 1 / 2 input data), after calculation by node 1, node 2, and node 3, store the calculation result of node 3 (i.e. 1 / 2 calculation result) in the intermediate buffer; 3) Read the complete Tensor (i.e. the complete calculation result output by node 3) from the intermediate buffer, after calculation by node 4, node 5, and node 6, write the calculation result to DDR or on-chip buffer.

[0284] Theoretically, the number of data segments that are split when the nodes included in the second-order subgraph perform processing operations should be less than the number of data segments that are split when the nodes included in the first-order subgraph perform processing operations. Therefore, the amount of data that is repeatedly loaded during the AI ​​chip calculation process can be reduced overall. Fig.16A and Fig. 16BTake the example in for illustration, the second-order sub-graphs OP1-OP3 are depth sub-graphs, the second-order sub-graphs OP4-OP5 are direct sub-graphs, and the number of data segments is reduced to two. As the depth of the depth sub-graph is deeper, the overlapping part of the segmented data will be larger, the additional amount of calculation will be more, and a certain amount of computing performance will be lost. Therefore, the solution provided in the embodiment of the present application can greatly reduce the amount of calculation of the overlapping part by re-optimizing the depth sub-graph with a depth not less than the first threshold to obtain a second-order sub-graph, and use the relatively moderate read and write performance of the intermediate buffer to compensate for the read and write weaknesses of the external storage, thereby ultimately improving the overall computing performance.

[0285] In order to reduce the load of the on-chip buffer storage, this application introduces a Concat node optimization solution. The structural diagram of the neural network graph segmentation device is shown in FIG. Fig.18 The neural network graph segmentation device also includes a Concat optimization module for eliminating Concat nodes in the neural network graph. Fig.18 The connection optimization module 1801 in the figure is a Concat optimization module. The function of the Concat node in the neural network graph is to splice the output data of at least two nodes in the neural network graph and transmit the obtained output data to one or more nodes in the neural network graph. The Concat node optimization scheme is described as follows: the data splicing effect of the Concat node is achieved by directly mapping the output address of the input operator of the Concat node to the output address of the corresponding Concat node. The Concat node optimization scheme is reflected in the form of the neural network. The change can be as follows Fig.19 As shown, the Concat node can be eliminated in the neural network structure, that is, there is no need to map the Concat node into the corresponding AI chip instructions. Fig.19 The example includes 6 nodes, namely node 1 to node 6, where node 5 is a Concat node. The function of node 5 is to concatenate the output data of nodes 1 to node 4 into one output data and output it to node 6. After the Concat node optimization, the output data of nodes 1 to node 4 can be directly output to node 6, thus eliminating the Concat node. The Concat node optimization scheme is to use the address of the output data of the nodes connected to the Concat node (the output data of these nodes are the input data of the Concat node) in the on-chip buffer as the output data of the Concat node. Fig. 20 A schematic diagram of a Concat node optimizing on-chip storage provided in an embodiment of the present application. Fig. 20 As shown in FIG. 1 , the output addresses of OP1, OP2, OP3, and OP4 (i.e., the addresses corresponding to output data 1 to output data 4) all correspond to the local output address of the Concat node. OP1, OP2, OP3, and OP4 are Fig. 20In other words, the output addresses of OP1, OP2, OP3, and OP4 are connected end to end to form the output address of the Conca node. After completing the calculation of OP1, OP2, OP3, and OP4, you can directly enter the calculation of OP6 (node ​​6) without performing the Concat node operation, such as OP5 (node ​​5).

[0286] In practical applications, in the process of allocating the address of the on-chip buffer for the internal input and output data of the benchmark subgraph, if the function of a node in the benchmark subgraph is to transmit the target data obtained by splicing the output data of at least two nodes in the benchmark subgraph to one or more nodes in the benchmark subgraph, the address of the output data of the at least two nodes in the on-chip buffer will be used as the address of the output data of the node.

[0287] The above solution can achieve the following goals: 1) reduce the on-chip storage space usage and improve the whole network integration benefits; 2) avoid the movement of input data of the Concat node, which can further improve the performance of the neural network whole network calculation.

[0288] In order to further reduce the occupancy of the on-chip buffer storage space and reduce the amount of calculation, the present application provides a DetectionOutput node optimization solution based on the above-mentioned embodiment. Fig.21 As shown, the neural network graph segmentation device also includes a DetectionOutput optimization module, that is, Fig.21 The discovery output optimization module 2101 is found in. Fig. 22 Schematic diagram of the changes in the neural network structure before and after the optimization of the DetectionOutput node provided in the embodiment of the present application. Fig. 22 In the figure, the neural network including nodes 1 to 5 on the left is the neural network before optimization, and the neural network on the right is the neural network after optimization. Among them, node 4 is the DetectionOutput node.

[0289] by Fig. 22 As an example, the optimization scheme for the DetectionOutput node is as follows: the operator of the DetectionOutput node (node ​​4) has three inputs (i.e., the outputs of nodes 1 to 3), but one of the inputs (the output of node 2) is only related to the structural dimension of the neural network and has nothing to do with the input of the neural network. Therefore, the neural network graph segmentation device can calculate the input offline in advance and store it in the offline model. Using the DetectionOutput node optimization scheme, at least one input of the DetectionOutput node does not need to be calculated online, but only needs to be loaded. Therefore, the occupancy of the on-chip storage is reduced to a certain extent. The optimized neural network structure diagram changes as shown below. Fig. 22 As shown, there is no need to calculate the result through node 2, but the result can be directly loaded from the external memory.

[0290] In practical applications, before segmenting the neural network graph, the above DetectionOutput node optimization solution can be used to process the neural network graph.

[0291] In the embodiments of the present application, the Concat node optimization scheme and the DetectionOutput node optimization scheme in the neural network can be introduced into the embodiments separately or simultaneously. Fig.23 A schematic diagram of the structure of another neural network graph segmentation device provided in an embodiment of the present application. Fig.23 As shown, in Figure 4 Based on the neural network graph segmentation device in the embodiment, a detection output optimization module 2301 and a connection optimization module 2302 are added. The output optimization module 2301 is used to implement the DetectionOutput node optimization solution. The connection optimization module 2302 is used to implement the Concat node optimization solution.

[0292] The above embodiments describe the method of segmenting a neural network graph to obtain a deep subgraph and a direct subgraph, and mapping the two types of subgraphs obtained by segmentation into AI chip instructions in the form of a unified Pipeline. A neural network corresponds to a set of AI chip instructions in a computer device, that is, running this set of AI chip instructions implements the function of the neural network. The method in the above embodiment can be used to adjust a set of AI chip instructions corresponding to the neural network to another set of AI chip instructions. Compared with the set of AI chip instructions before the adjustment, the adjusted set of AI chip instructions implements the same processing operations, but the number of accesses to the external memory is greatly reduced and the power consumption is also greatly reduced. In practical applications, computer devices such as servers and terminals can use AI chip instructions corresponding to deep subgraphs and direct subgraphs to implement processing operations. The following describes the method of executing the target task using the deep subgraph (corresponding to the deep subnetwork) and the direct subgraph (corresponding to the direct subnetwork) obtained by segmenting the neural network graph (corresponding to the neural network).

[0293] Fig.24 A flow chart of a prediction method provided in an embodiment of the present application. Fig.24 As shown, the method may include:

[0294] 2401. The data processing device obtains original input data.

[0295] The original input data includes one or more computer-processable signals. The multiple computer-processable signals include: at least one of: a voice signal, a text signal or an image signal. Optionally, the original input data is image data, such as a face image. The data processing device may be a terminal, a server or other computer equipment. The data processing device obtains the original input data by the data processing device from other devices, such as a server, through a communication interface; it may also be that the data processing device acquires the original input data through a camera to acquire image data; it may also be that the audio device acquires voice data to acquire the original input data.

[0296] 2402. The data processing device inputs the original input data into the neural network for prediction processing to obtain a prediction result.

[0297] The prediction processing may be face detection, face feature point detection, image enhancement, image super-resolution processing, natural speech text processing, image semantic segmentation, etc., which can be realized by a neural network. Accordingly, the prediction result may be a face detection result, a face feature point detection result, an enhanced image, an image after super-resolution processing, a natural language text processing result, an image semantic segmentation result, etc., which can be realized by a neural network.

[0298] The prediction processing includes: inputting at least two groups of data obtained by splitting the first input data into the deep sub-network for processing, the deep sub-network is included in the neural network and includes some nodes in the neural network, each node represents a computing unit in the neural network, and the multiple nodes included in the deep sub-network exchange data by reading and writing on-chip buffers; the first input data is obtained in the process of inputting the original input data into the neural network for prediction processing. A specific example of this process can be referred to Figure 2 , input data 0, input data 1 and input data 3 are three sets of data obtained by splitting the first input data; the deep sub-network (i.e. Figure 2The subnetwork including nodes 1 and 2 in the deep subnetwork) first processes input data 0 and stores the obtained output data 20 in a buffer; then, processes input data 1 and stores the obtained output data 21 in a buffer; finally, processes input data 3 and stores the obtained output data 22 in a buffer. It can be understood that each of the multiple nodes included in the deep subnetwork performs at least two processing operations in the process of processing the first input data. Optionally, the storage space required for the first input data is greater than the storage space available in the on-chip buffer, and the storage space required for each of the at least two groups of data is not greater than the storage space available in the on-chip buffer. It can be understood that when the storage space required for the first input data is greater than the available storage space of the on-chip buffer, the first input data is split into at least two groups of data and processed successively, so that the storage space required for each group of data is not greater than the available storage space of the on-chip buffer, and access to the external memory can be avoided.

[0299] The prediction processing also includes: the direct sub-network processes the second input data as a whole, the direct sub-network is included in the neural network and includes some nodes in the neural network, and the second input data is obtained in the process of inputting the original input data into the neural network for prediction processing. A specific example of this process can be referred to Figure 2 , direct subnetwork (i.e. Figure 2 The subnetwork including node 3 and node 4 in the direct subnetwork processes input data 0, input data 1 and input data 3 as a whole. Since the direct subnetwork processes its input data as a whole, at least one node in the direct subnetwork performs a processing operation during the processing of the second input data. Optionally, the storage space required for the second input data is not greater than the storage space available in the on-chip buffer, so that the direct subnetwork only needs to occupy the on-chip buffer when processing the second input data.

[0300] It can be understood that, whether it is a deep subnetwork or a direct subnetwork, in the process of processing their respective input data, there is no need to access the external memory. In some embodiments, the original input data is input into the neural network for prediction processing, and the prediction result can be understood as: the original input data is input into the deep subnetwork and the direct subnetwork obtained by splitting the neural network for processing. In fact, the deep subnetwork and the direct subnetwork are only abstract concepts. The deep subnetwork and the direct subnetwork correspond to two different forms of processing flows in the process of the neural network performing prediction processing on the original input data. In other words, the deep subnetwork implements a part of the processing operations of the neural network, and the direct subnetwork also implements a part of the processing operations of the neural network. It can be understood that each deep subnetwork and each direct subnetwork obtained by splitting the neural network are used to implement a part of the operation of the neural network, and these deep subnetworks and direct subnetworks can realize the functions of the neural network.

[0301] In some embodiments, the prediction processing further includes: the third second-order subnetwork processes the fourth input data to obtain second intermediate data; the second intermediate data is stored in an intermediate buffer, and the intermediate buffer is not the on-chip buffer; the fourth second-order subnetwork processes the second intermediate data obtained from the intermediate buffer to obtain fourth output data; the third second-order subnetwork and the fourth second-order subnetwork are both included in the neural network, the nodes included in the third second-order subnetwork and the nodes included in the fourth second-order subnetwork are different, and the fourth input data is obtained in the process of inputting the original input data into the neural network for prediction processing. The multiple nodes included in the third second-order subnetwork exchange data by reading and writing the on-chip buffer, and the multiple nodes included in the fourth second-order subnetwork exchange data by reading and writing the on-chip buffer. Optionally, the number of nodes included in the third second-order subnetwork and the number of nodes included in the fourth second-order subnetwork are both less than the first threshold. Exemplarily, the intermediate buffer is an off-chip buffer with a read and write speed lower than the on-chip buffer, and the read and write performance of the intermediate buffer is better than that of an external memory, such as DDR. An example of the third second-order subnetwork is Fig. 16B A deep subnetwork including nodes 1, 2 and 3, an example of the fourth second-order subnetwork is Fig. 16B The direct subnetwork includes nodes 4, 5 and 6. In these embodiments, the data processing device includes an on-chip buffer, an intermediate buffer and an external memory. When executing the prediction processing, the data processing device does not need to access the external memory, and the prediction processing can be realized by reading and writing the on-chip buffer and the intermediate buffer, which can reduce power consumption.

[0302] In some embodiments, the data processing device may have two working modes. The first working mode is that the data processing device only reads and writes the on-chip buffer when inputting the original input data into the neural network for predictive processing. The second working mode is that the data processing device reads and writes the on-chip buffer and the intermediate buffer when inputting the original input data into the neural network for predictive processing. Optionally, the data processing device may store two AI chip executable files. When the data processing device executes one AI chip executable file, it adopts the first working mode for predictive processing. When the data processing device executes another AI chip executable file, it adopts the second working mode for predictive processing. The data processing device can switch the working mode according to the control instruction from the user. It should be understood that the first working mode has lower power consumption than the second working mode, and the second working mode has higher computing efficiency than the first working mode. The two working modes have different advantages, and the user can set the corresponding working mode according to actual needs. For example, if the data processing device adopts the first working mode, an example of a processing flow executed by the data processing device is Fig.16A If the data processing device adopts the second working mode, an example of a processing flow executed by the data processing device is Fig. 16B The processing flow in .

[0303] 2403. The data processing device outputs the prediction result.

[0304] Since the neural network does not need to access the external memory during the prediction process of the original input data, the power consumption of the data processing device can be effectively reduced. It can be seen that the data processing device can run many prediction methods such as face detection and face feature point detection in ultra-low power consumption in scenarios such as screen off.

[0305] In an embodiment of the present application, in the process of inputting the original input data into the neural network for predictive processing, a deep sub-network is used to perform the processing tasks of the neural network. Since the deep sub-network does not need to access the external memory in the process of processing its input data, the number of times the external memory is accessed can be greatly reduced, and power consumption can be reduced.

[0306] Fig.25 Another neural network graph segmentation device provided in the embodiment of the present application. Fig.25As shown, the device includes: a memory 2501 and a processor 2502; the memory 2501 is used to store codes; the processor 2502 reads the codes stored in the memory to perform the following operations: obtaining a neural network graph, the neural network graph is used to represent a neural network, the neural network graph includes multiple nodes, each node represents a computing unit in the neural network; the neural network graph is divided to obtain a deep subgraph; the deep subgraph is used to represent a deep subnetwork, the deep subnetwork includes multiple nodes, each of which performs data exchange by reading and writing an on-chip buffer, the deep subnetwork is used to successively process at least two groups of data obtained by splitting first input data to obtain first output data, the first input data is the input data of the deep subnetwork, and the first input data includes one or more computer-processable signals. The processor 2502 is used to implement Figure 4 The processor 2502 is used to implement the function of the connection optimization module 2302.

[0307] In an optional manner, the processor 2502 is further used to split the neural network graph to obtain a direct subgraph; the direct subgraph is used to represent a direct subnetwork, and each of the multiple nodes included in the direct subnetwork interacts with each other by reading and writing the on-chip buffer; the direct subnetwork is used to process the second input data as a whole to obtain second output data, and the second input data is the input data of the direct subnetwork.

[0308] In an optional manner, the processor 2502 is specifically configured to obtain a first reference subgraph, the first reference subgraph including a first node and a second node, the first node being a node to be assigned in the current neural network graph, and the second node being a next node of the first node in the neural network graph; adding a third node to the first reference subgraph to obtain a second reference subgraph, the third node being a next node of the second node in the neural network graph, and the second reference subgraph being used to process third input data; the device further includes:

[0309] The on-chip address manager 2403 is used to allocate the address of the on-chip buffer to the second reference sub-graph during the process of the second reference sub-graph processing the third input data;

[0310] The processor 2502 is specifically configured to use the second reference subgraph as the depth subgraph when the address of the on-chip buffer is successfully allocated to the second reference subgraph and the third node belongs to the termination node; the processor is specifically configured to add a fourth node to the second reference subgraph to obtain a fourth reference subgraph when the address of the on-chip buffer is successfully allocated to the second reference subgraph and the third node does not belong to the termination node, wherein the fourth node is the next node of the third node in the neural network graph. The on-chip address manager 2403 is configured to implement Figure 4 The function of the on-chip address management module 404. The on-chip address manager 2403 can be integrated with the processor 2502 and can be an independent device.

[0311] In an optional manner, the processor 2502 is further configured to use the first reference sub-image as the direct sub-image if the address of the on-chip buffer is not successfully allocated to the second reference sub-image.

[0312] In an optional manner, the on-chip address manager 2403 is specifically configured to allocate the address of the on-chip buffer to the second reference sub-graph during the process in which the second reference sub-graph processes the third input data as a whole;

[0313] Processor 2502 is specifically used to allocate the address of the on-chip buffer to the second reference sub-graph when the on-chip address manager fails to successfully allocate the address of the on-chip buffer to the second reference sub-graph while the second reference sub-graph successively processes at least two groups of data obtained by splitting the third input data.

[0314] In an optional manner, the processor 2502 is further used to generate a target instruction corresponding to the depth sub-map, the target instruction is used to execute a target sub-task, the neural network is used to execute the target task, and the target sub-task is a part of the target task.

[0315] Fig.26 A structural diagram of a data processing device is provided for an embodiment of the present application, such as Fig.26As shown, the data processor device includes: a memory 2601 and a processor 2602; the memory is used to store code and original input data; the processor reads the code stored in the memory to perform the following operations: obtain the original input data, the original input data includes one or more computer-processable signals; input the original input data into the neural network for prediction processing to obtain a prediction result; the prediction processing includes: inputting at least two groups of data obtained by splitting the first input data into a deep sub-network for processing in sequence, the deep sub-network is included in the neural network and includes some nodes in the neural network, each node represents a computing unit in the neural network, and each of the multiple nodes included in the deep sub-network exchanges data by reading and writing on-chip buffers; the first input data is obtained in the process of inputting the original input data into the neural network for prediction processing; outputting the prediction result.

[0316] Processor 2602 is used to implement Fig.24 The method of steps 2401 to 2403 and the equivalent alternative method of steps 2401 to 2403.

[0317] In an optional manner, the processor 2602 is also used for the third second-order sub-network to process the fourth input data to obtain second intermediate data; the second intermediate data is stored in an intermediate buffer, and the intermediate buffer is not the on-chip buffer; the fourth second-order sub-network processes the second intermediate data obtained from the intermediate buffer to obtain fourth output data; the third second-order sub-network and the fourth second-order sub-network are both included in the neural network, the nodes included in the third second-order sub-network are different from the nodes included in the fourth second-order sub-network, and the fourth input data is obtained in the process of inputting the original input data into the neural network for prediction processing.

[0318] In an optional manner, each of the multiple nodes included in the third second-order subnetwork interacts with each other by reading and writing the on-chip buffer, and each of the multiple nodes included in the fourth second-order subnetwork interacts with each other by reading and writing the on-chip buffer.

[0319] In the embodiment of the present application, the memory 2501 and the memory 2601 can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory can store an operating system and other applications. The processor 2502 and the processor 2602 can use a general central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a GPU or one or more integrated circuits to execute related programs to implement the functions required to be performed by the units in the neural network graph segmentation device and / or the data processing device of the embodiment of the present application, or to perform the neural network graph segmentation method and / or prediction method provided in the method embodiment of the present application. The processor can also be an integrated circuit chip with signal processing capabilities. In the implementation process, the various steps of the method provided by the present application can be completed by the hardware integrated logic circuit in the processor or the instructions in the form of software. The above-mentioned processor can also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc.

[0320] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes software program instructions. When the program instructions are executed by a processor in a data processing device, the neural network graph segmentation method and / or data processing method in the aforementioned embodiment are implemented.

[0321] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0322] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A neural network segmentation method, characterized in that: Obtaining a neural network graph, the neural network graph is used to represent a neural network, the neural network graph includes a plurality of nodes, each node represents a computing unit in the neural network; Splitting the neural network graph to obtain deep subgraphs; The deep subgraph is used to characterize the deep subnetwork, wherein each of the multiple nodes included in the deep subnetwork performs data exchange by reading and writing an on-chip buffer, and the deep subnetwork is used to process at least two groups of data obtained by splitting first input data to obtain first output data, wherein the first input data is input data of the deep subnetwork, and the first input data includes one or more computer-processable signals; After the neural network graph is segmented to obtain depth subgraphs, the method further includes: When the number of nodes included in the depth subgraph is not less than a first threshold, splitting the depth subgraph to obtain a first second-order subgraph and a second second-order subgraph; The first second-order subgraph is used to characterize the first second-order subnetwork, and the second second-order subgraph is used to characterize the second second-order subnetwork. The first second-order subnetwork and the second second-order subnetwork are both included in the deep subnetwork. The first second-order subnetwork and the second second-order subnetwork include different nodes. The output data of the first second-order subnetwork is stored in an intermediate buffer, and the second second-order subnetwork is used to read the output data from the intermediate buffer. The intermediate buffer is an off-chip buffer with a lower reading and writing speed than the on-chip buffer, and the reading and writing speed of the intermediate buffer is higher than that of the external memory.

2. The neural network segmentation method according to claim 1, characterized in that: The method further comprises: The neural network graph is divided to obtain a direct subgraph; the direct subgraph is used to represent a direct subnetwork, and each of the multiple nodes included in the direct subnetwork interacts with each other by reading and writing the on-chip buffer; the direct subnetwork is used to process the second input data as a whole to obtain the second output data, and the second input data is the input data of the direct subnetwork.

3. A system, characterized in that: include: An artificial intelligence (AI) chip and an intermediate buffer connected to the AI ​​chip, the AI ​​chip including an on-chip buffer and a computing unit; The system is configured to execute the method as claimed in any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by the system according to claim 3, the system executes the method according to any one of claims 1 to 2.

Images