A data processing method, system on chip, electronic device and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a data processing method, a system on chip, electronic equipment and a medium. The data processing method comprises the following steps: determining data processing resources required by each node in a neural network, and dividing the nodes in the neural network into multiple independent node unit groups according to data dependency relationships of the nodes; determining an execution sequence of each node in the neural network based on the data processing resources required by each node and the multiple independent node unit groups; and controlling the neural network to execute each node based on the execution sequence, so as to process to-be-processed data. The data processing method can allocate the data processing resources required by each node to each node, and can control the neural network to execute each node based on the execution sequence, thereby reducing the waiting time caused by resource allocation, avoiding waste of resources, and improving the calculation speed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, a system-on-a-chip, an electronic device, and a medium. Background Technology

[0002] With the rapid development of artificial intelligence (AI), neural networks are being used more and more widely in the field of AI. It can be understood that neural networks process data by having each node (or operator) perform corresponding operations. The output data of some nodes in a neural network can serve as input data for other nodes; each node can only perform its corresponding operation after receiving input data, meaning there are data dependencies between nodes.

[0003] Currently, neural networks operate sequentially according to the connection order of their nodes. During operation, some nodes, despite having sufficient resources, are forced to wait due to the execution order, resulting in longer execution times. Furthermore, electronic devices allocate the same number of computing and memory units to each node using a uniform allocation method. However, the types of operations performed by each node in a neural network differ. For example, computationally sensitive operations require multiple arithmetic operations on a single data unit after data is read, leading to longer computation times and necessitating more computing resources, such as the common convolution operator, to accelerate computation. Conversely, memory-sensitive operations require only a few arithmetic operations on a single data unit after data is read, resulting in lower computing unit requirements but greater access to memory. Therefore, the demands of each node for computing and memory units are not entirely consistent. Allocating the same number of computing and memory units to nodes performing different operations using the same method leads to prolonged resource allocation times, excessive resources for some nodes (resulting in resource waste), and insufficient resources for others (resulting in resource shortages), ultimately extending execution times. Summary of the Invention

[0004] This application provides a data processing method, a system-on-a-chip, an electronic device, and a medium.

[0005] In a first aspect, embodiments of this application provide a data processing method applied to an electronic device, comprising: determining the data processing resources required by each node in a neural network, and dividing the nodes in the neural network into multiple independent node unit groups according to the data dependencies of each node, wherein each independent node unit group includes multiple independent node units, and there are no data dependencies between the independent node units in the same independent node unit group; determining the execution order of each node in the neural network based on the data processing resources required by each node and the multiple independent node unit groups; and controlling the neural network to execute each node according to the execution order to process the data to be processed.

[0006] It is understandable that nodes without data dependencies are nodes whose execution order can be arbitrarily set and will not affect the final data processing result of the neural network, while nodes with data dependencies are nodes whose execution order changes will affect the final data processing result of the neural network.

[0007] Data processing resources include computing units and memory units. A computing unit can be understood as a central processing unit (CPU) resource of a certain specification or the computing unit (Processor Element, PE) of some AI accelerators, or as a set task processing time. A memory unit can be understood as a memory resource (memory space) of a certain specification, or a set memory transfer time. For example, moving data from external storage with large capacity and slow access speed (such as Double Data Rate Synchronous Dynamic Random Access Memory (DDR)) to on-chip resources with small capacity but fast access speed (such as Static Random-Access Memory (SRAM)) is a memory unit.

[0008] In this embodiment, resources can be allocated to each node according to the resource allocation requirements of each node in the neural network. This can avoid situations where some nodes have too many resources, resulting in resource waste, or some nodes have too few resources, resulting in resource shortage. Furthermore, multiple independent node unit groups can be parallelized to adjust the execution order of each node in the neural network, thereby reducing waiting time and ultimately reducing the overall data processing time.

[0009] In one possible implementation, determining the execution order of nodes in a neural network based on the data processing resources required by each node and multiple independent node unit groups includes: determining a set of execution orders for each node in the neural network based on the data processing resources required by each node and multiple independent node unit groups; determining the first execution order with the shortest execution time in the set of execution orders; when the data processing time corresponding to the first execution order is lower than the data processing time corresponding to the default execution order of each node in the neural network, the first execution order is used as the execution order of each node in the neural network; when the data processing time corresponding to the first execution order is not lower than the data processing time corresponding to the default execution order of each node in the neural network, the default execution order of each node in the neural network is used as the execution order of each node in the neural network.

[0010] It is understandable that for a relatively simple neural network, the data processing time corresponding to the first execution order may not be less than the data processing time corresponding to the default execution order of each node in the neural network; for a relatively complex neural network, the data processing time corresponding to the first execution order is generally less than the data processing time corresponding to the default execution order of each node in the neural network.

[0011] This application embodiment can obtain an execution order set by parallelizing multiple independent node units, and select the execution order with the shortest data processing time as the first execution order; then compare the data processing time of the first execution order with the data processing time of the default execution order, and select the execution order with the shortest data processing time as the execution order of each node of the neural network, thus avoiding the unintentional selection of the execution order with the longer data processing time as the execution order of each node of the neural network.

[0012] In one possible implementation, controlling the neural network to execute each node according to the execution order to process the data to be processed includes: allocating the data processing resources required by each node according to the execution order; and controlling the neural network to process the data to be processed based on the data processing resources required by each node.

[0013] It is understood that the data processing resources required by each node in a neural network are not necessarily the same. However, the embodiments of this application can allocate the data processing resources required by each node according to the execution order, so as to avoid the situation where some nodes have too many resources, resulting in resource waste, and some nodes have too few resources, resulting in resource shortage, thereby improving the execution efficiency of each node in the neural network.

[0014] In one possible implementation, an independent node unit comprises one or more nodes.

[0015] In one possible implementation, data processing resources include computing resources and memory resources.

[0016] It is understood that computing resources are used to perform calculations on the data to be processed. During the data processing, the data to be processed can be moved from the slow storage unit to the fast storage unit. The computing resources are equivalent to the computing unit in the embodiment of this application, and the memory resources are equivalent to the memory unit in the embodiment of this application.

[0017] In one possible implementation, the memory resources required by the neural network are partitioned based on the data processing resources required by each node to obtain multiple types of memory resource fragments; the operation type of each node and the number of memory resource fragments required are determined; and memory resource fragments of the corresponding type and number are allocated to each node based on the operation type of each node and the number of memory resource fragments required.

[0018] In this embodiment of the application, by partitioning memory resources, the utilization rate of memory resources can be improved, avoiding the waste of resources caused by allocating too much memory resources to nodes of the neural network (such as computation-sensitive operators) and the shortage of resources caused by insufficient memory resources allocated to nodes of the neural network (such as memory-sensitive operators).

[0019] In one possible implementation, multiple types of memory resource slices include memory slices for convolutional operators and memory slices for fully connected operators; the operation types of each node include convolutional operations and fully connected operations.

[0020] In one possible implementation, the operation type of each node is determined; each node is divided based on its operation type to obtain multiple types of computational operation arrays; and computational resources are allocated to the multiple types of computational operation arrays.

[0021] In this embodiment of the application, by dividing each node, the utilization rate of memory resources can be improved, avoiding the waste of resources due to excessive memory resources or the shortage of resources due to insufficient memory resources.

[0022] In one possible implementation, multiple types of computational operation arrays include computationally intensive operators for convolution operation arrays and operators for matrix multiplication arrays.

[0023] It can be understood that computationally intensive operators used for convolution operation arrays and operators used for matrix multiplication arrays can be computation nodes in a neural network that needs to perform operations. Memory resources are used to hold the data to be processed, and computational resources are used to perform calculations on the data to be processed.

[0024] In one possible implementation, the control neural network executes each node in the execution order to process the data to be processed, including: storing the data processing resource information required by each node of the neural network into a time configuration table; allocating corresponding data processing resources to each node according to the execution order based on the data processing resource information stored in the time configuration table; and controlling the neural network to process the data to be processed based on the data processing resources required by each node.

[0025] It is understandable that data processing resource information includes computational load and data access / storage load; by storing data processing resource information in a time configuration table, a basis is provided for allocating corresponding data processing resources to each node.

[0026] In one possible implementation, determining the data processing resources required by each node in the neural network includes: obtaining the data processing resources required by each node of the neural network based on a mode controller; allocating data processing resources to each node of the neural network according to the execution order through the mode controller; and controlling the neural network to process the data to be processed based on the data processing resources required by each node.

[0027] It is understandable that data processing resources include computing resources and memory resources.

[0028] Secondly, embodiments of this application provide a system-on-a-chip (SoC), which includes a first determining module, a second determining module, and a processing module. The first determining module is used to determine the data processing resources required by each node in a neural network, and to divide the nodes in the neural network into multiple independent node unit groups based on the data dependencies of each node. Each independent node unit group includes multiple independent node units, and there are no data dependencies between the independent node units within the same independent node unit group. The second determining module is used to determine the execution order of each node in the neural network based on the data processing resources required by each node and the multiple independent node unit groups. The processing module is used to control the neural network to execute each node according to the execution order to process the data to be processed.

[0029] It is understandable that nodes without data dependencies are nodes whose execution order can be arbitrarily set and will not affect the final data processing result of the neural network, while nodes with data dependencies are nodes whose execution order changes will affect the final data processing result of the neural network.

[0030] Data processing resources include computing units and memory units. A computing unit can be understood as a CPU resource of a certain specification, or a PE of some AI accelerators, or a set task processing time. A memory unit can be understood as a memory resource (memory space) of a certain specification, or a set memory transfer time, such as moving data from a large-capacity, slow-access external storage (such as DDR) to a small-capacity, fast-access on-chip resource (such as SRAM), which is a memory unit.

[0031] In this embodiment, resources can be allocated to each node according to the resource allocation requirements of each node in the neural network. This can avoid situations where some nodes have too many resources, resulting in resource waste, or some nodes have too few resources, resulting in resource shortage. Furthermore, multiple independent node unit groups can be parallelized to adjust the execution order of each node in the neural network, thereby reducing waiting time and ultimately reducing the overall data processing time.

[0032] Thirdly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to implement any of the data processing methods provided by the first aspect and various possible implementations of the first aspect.

[0033] Fourthly, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for implementing any of the data processing methods provided by the first aspect and various possible implementations of the first aspect. Attached Figure Description

[0034] Figure 1 A schematic diagram of a topology is shown;

[0035] Figure 2 A schematic diagram of another topology is shown;

[0036] Figures 3a-3b According to an embodiment of this application, a schematic diagram of the execution time of the computing unit and memory unit during data processing is shown;

[0037] Figure 4 According to an embodiment of this application, a flowchart of a data processing method is shown;

[0038] Figure 5 A schematic diagram of a neural network structure is shown;

[0039] Figure 6 According to an embodiment of this application, a schematic diagram of a method for obtaining a computing module is shown;

[0040] Figure 7According to an embodiment of this application, a schematic diagram of a computing module structure is shown;

[0041] Figure 8 According to an embodiment of this application, a schematic diagram of a method for reconstructing a computing module is shown;

[0042] Figure 9 According to an embodiment of this application, a schematic diagram of the structure of an electronic device is shown. Detailed Implementation

[0043] The illustrative embodiments of this application include, but are not limited to, a data processing method, a system-on-a-chip, an electronic device, and a medium.

[0044] It is understood that the technical solutions of this application are applicable to any electronic device capable of running neural network models, such as, but not limited to, mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), or dedicated cameras (such as SLR cameras and point-and-shoot cameras). The embodiments of this application do not impose any restrictions on the specific type of electronic device.

[0045] The following section first describes the process by which electronic devices process data through neural networks in some embodiments.

[0046] For example, such as Figure 1 As shown, Figure 1 This is a topological diagram of a neural network. It should be noted that... Figure 1 The labels of each node are consistent with the execution order and allocation order; among them, electronic devices are processed through... Figure 1 The data processing process of the topology diagram shown is as follows: The electronic device first acquires the data to be processed and inputs it into node 1 in the topology diagram for calculation. After node 1 obtains the first calculation result, it simultaneously inputs the first calculation result into nodes 2, 3, and 4. Node 2 obtains the second calculation result based on the first calculation result and inputs the second calculation result into node 4. Then, node 3 obtains the third calculation result based on the first calculation result and inputs the third calculation result into node 4. Finally, node 4 obtains the fourth calculation result based on the first, second, and third calculation results and outputs it. That is, the default execution order is: node 1 → node 2 → node 3 → node 4.

[0047] It is understandable that in the above data processing process, even if node 1 has output the calculation result to node 3, node 3 still needs to wait for node 2 to complete the calculation before it can execute the corresponding operation. However, there is no data dependency between node 3 and node 2. If the default execution order is followed, the processing time will be increased and the processing efficiency will be reduced.

[0048] During the aforementioned data processing, the electronic device allocates resources to each node, specifically a corresponding number of computing units and memory units, enabling each node to process data based on these resources and obtain computational results. Each computing unit can be understood as a CPU resource of a certain specification, or a PE (Processing Equipment) of an AI accelerator, or a set task processing time. Each memory unit can be understood as a set memory resource (memory space), or a set memory transfer time.

[0049] It is understood that the above topology diagram represents the original structure of a neural network. To meet more complex computational needs, in some embodiments, Neural Architecture Search (NAS) can be used to refine the structure, for example, the one described above. Figure 1 The topological structure graph in the text is expanded by using node combinations, such as subgraph A, subgraph B(1), subgraph B(2), subgraph C(1), and subgraph C(2), to replace it. Figure 1 Construct the edges between nodes in the middle. Figure 2 Neural networks (graph forests) are used to perform more complex data processing.

[0050] For example, in Figure 2In the structure shown in the lower right figure, the execution order of the electronic device controlling each node's operation is as follows: The electronic device inputs the data to be processed into node 1. Node 1 processes the data to be processed and obtains the first calculation result. Then, node 1 simultaneously inputs the first calculation result into subgraphs B(1), C(1), and C(2). After subgraph B(1) completes its calculation, it obtains the fifth calculation result and inputs it into node 2. After subgraph C(2) completes its calculation, it obtains the sixth calculation result and inputs it into node 4. After subgraph C(1) completes its calculation, it obtains the seventh calculation result and inputs it into node 3. Then, node 2 obtains the second calculation result based on the fifth calculation result. As a result, the second calculation result is input into subgraph A; then node 3 obtains the third calculation result based on the seventh calculation result and inputs the third calculation result into subgraph B(2); then subgraph A obtains the eighth calculation result based on the second calculation result and inputs the eighth calculation result into node 4; then subgraph B(2) obtains the ninth calculation result based on the third calculation result and inputs the ninth calculation result into node 4; finally, node 4 obtains the fourth calculation result based on the sixth calculation result, the eighth calculation result, and the ninth calculation result and outputs it; that is, the default execution order is: node 1 → subgraph B(1) → subgraph C(2) → node C(1) → node 2 → node 3 → subgraph A → subgraph B(2) → node 4.

[0051] For the complex neural network described above, following the default execution order would significantly increase data processing time and reduce processing efficiency; furthermore, if electronic devices allocate the same resources to each node in the complex neural network, it would also cause a serious imbalance in resources.

[0052] To address the aforementioned problems, this application proposes a data processing method applied to an electronic device, which includes:

[0053] The neural network to be allocated resources is obtained. Each node of the neural network is traversed to determine the resources (i.e., the required computing units and memory units) and data dependencies of each node in the computation graph of the neural network. Then, based on the data dependencies of each node, multiple sets of mutually independent node units are determined. Each node unit can be a single node or a combination of multiple nodes. The execution order of each node is determined based on the multiple sets of mutually independent node units and the calculated computing time and memory transfer time of each node. Each node is controlled to perform operations in the above execution order. During the execution of operations, resources are allocated to each node according to the resources required by each node.

[0054] It is understandable that each group of independent node units can be a group of node units that are independent of each other, have no data dependency relationship, and whose data processing result will not be affected even if their order is changed. For example, for the above... Figure 1In the structure, nodes 2 and 3 have no data dependency relationship, so executing node 2 or node 3 first will not affect the subsequent data processing results.

[0055] In other words, for each group of independent node units, the execution order of each node unit can be arbitrarily set without affecting the data processing results of the neural network. However, since each node needs to move data to memory before the PE (Process Execution Environment) can perform calculations on the moved data, a long waiting time will occur when some nodes require a long memory moving time. This application can reduce the waiting time by adjusting the execution order of each node unit in each group of independent node units, thereby reducing the overall data processing time.

[0056] For example, the execution order of each node can be determined by performing all possible sorting on each group of independent node units to obtain a set of execution orders for each node. Then, the execution order with the shortest data processing time in the set of execution orders is taken as the first order. The execution time of the first order is compared with the execution time of the system default order. If the execution time of the first order is lower than the execution time of the system default order, the first order is taken as the execution order of each node. If the execution time of the first order is not lower than the execution time of the system default order, the system default order is taken as the execution order of each node. Here, the system default order is the execution order of each node determined according to the connection order of each node in the neural network.

[0057] The following is a pair Figure 2 Using the data calculation process performed by each node in the topology diagram shown as an example, we will introduce the data processing time required for different execution orders.

[0058] Figure 3a The diagram illustrates the time taken to perform calculations according to the default execution order. It can be understood that the PE (Execution Provider) can only perform calculations based on the transferred data after the data has been moved to memory. Figure 3aAs shown, after node 1 obtains memory, it moves the data to be processed to memory, and then uses the PE obtained by node 1 to perform calculations on the data to be processed to obtain the first calculation result; then it moves the first calculation result to memory obtained by subgraph B(1), and uses the PE obtained by subgraph B(1) to calculate the first calculation result to obtain the fifth calculation result; then it moves the first calculation result to memory obtained by subgraph C(2), and uses the PE obtained by subgraph C(2) to calculate the first calculation result to obtain the sixth calculation result; then it moves the first calculation result to memory obtained by subgraph C(1), and uses the PE obtained by subgraph C(1) to calculate the first calculation result to obtain the seventh calculation result; finally, it moves the fifth calculation result to memory obtained by node 2, and uses the PE obtained by node 2 to calculate the fifth calculation result to obtain the sixth calculation result to obtain the seventh calculation result to obtain the seventh calculation result to obtain the seventh calculation result to obtain the seventh calculation result to obtain the eighth calculation result to obtain the ninth calculation result to obtain the seventh calculation result to obtain the ninth calculation result to obtain the tenth calculation result to obtain the t- ... The PE obtained calculates the fifth calculation result to obtain the second calculation result; the seventh calculation result is moved to the memory obtained by node 3, and the PE obtained by node 3 calculates the seventh calculation result to obtain the third calculation result; the first calculation result is moved to the memory obtained by subgraph A, and the PE obtained by subgraph A calculates the second calculation result to obtain the eighth calculation result; the third calculation result is moved to the memory obtained by subgraph B(2), and the PE obtained by subgraph B(2) calculates the third calculation result to obtain the ninth calculation result; the sixth calculation result, the eighth calculation result, and the ninth calculation result are moved to the memory obtained by node 4, and the PE obtained by node 4 calculates the sixth calculation result, the eighth calculation result, and the ninth calculation result to obtain the fourth calculation result and output it.

[0059] In the above process, for subgraph C(1), it is necessary to wait for subgraph B(1) to finish processing before it can perform the operation. That is, subgraph C(1) needs to wait for subgraph B(1) to finish processing (the time for moving data to memory + the calculation time of PE) before it can perform the operation, which results in a long total data processing time. Similarly, node 2 also needs to wait for subgraph C(1) to finish processing before it can perform the operation; node 3 needs to wait for node 2 to finish processing before it can perform the operation; subgraph A needs to wait for node 3 to finish processing before it can perform the operation; and subgraph B(2) needs to wait for subgraph A to finish processing before it can perform the operation. All of these will cause the problem of a long total data processing time.

[0060] Based on the scheme of this application, new execution orders can be obtained by parallelizing mutually independent node units, for example, Figure 3bAs shown, after node 1 obtains memory, it moves the data to be processed to the memory of node 1, and then uses the PE obtained by node 1 to perform calculations on the data to be processed to obtain the first calculation result; then it moves the first calculation result to the memory obtained by subgraph B(1), and uses the PE obtained by subgraph B(1) to calculate the first calculation result to obtain the fifth calculation result; then it moves the first calculation result to the memory obtained by subgraph C(1), and uses the PE obtained by subgraph C(1) to calculate the first calculation result to obtain the seventh calculation result; then it moves the seventh calculation result to the memory obtained by node 3, and uses the PE obtained by node 3 to calculate the seventh calculation result to obtain the third calculation result; finally, it moves the fifth calculation result to the memory obtained by node 2, and uses the PE obtained by node 2 to calculate the fifth calculation result to obtain ... The fifth calculation result is calculated to obtain the second calculation result; the second calculation result is moved to the memory obtained in subgraph A, and the second calculation result is calculated through the PE obtained in subgraph A to obtain the eighth calculation result; the first calculation result is moved to the memory obtained in subgraph C(2), and the first calculation result is calculated through the PE obtained in subgraph C(2) to obtain the sixth calculation result; the third calculation result is moved to the memory obtained in subgraph B(2), and the third calculation result is calculated through the PE obtained in subgraph B(2) to obtain the ninth calculation result; the sixth calculation result, the eighth calculation result, and the ninth calculation result are moved to the memory obtained in node 4, and the sixth calculation result, the eighth calculation result, and the ninth calculation result are calculated through the PE obtained in node 4 to obtain the fourth calculation result and output it.

[0061] Therefore, the method provided in this application embodiment can significantly reduce the waiting time between independent node units by parallelizing independent node units, thereby reducing the overall data processing time and improving the efficiency of data processing.

[0062] It is understood that the resources to be allocated may include a number of computing units (e.g., PEs) and memory units (e.g., memory). As mentioned earlier, each computing unit can be understood as a certain specification of CPU resources, or some PEs of AI accelerators, or a set computing time. Each memory unit can be understood as a set specification of memory resources (memory space), or a set memory transfer time.

[0063] It is understood that in some embodiments, the resources to be allocated to each node in the neural network can be determined based on a resource allocation algorithm. This can avoid situations where some nodes have too many resources, resulting in resource waste, or some nodes have too few resources, resulting in resource shortage. The resource allocation algorithm can be a maximum rate algorithm, a round-robin scheduling algorithm, a proportional fairness algorithm, etc.

[0064] It is understood that the data processing method mentioned in this application can be used in various scenarios where electronic devices with neural networks are deployed. The data processing method is described in detail below, such as... Figure 4 The diagram illustrates a data processing method that can be executed by an electronic device and includes the following steps:

[0065] 401: Neural network for acquiring resources to be allocated.

[0066] It is understood that the neural network mentioned in the embodiments of this application can be a mesh-structured neural network, such as... Figure 5 As shown, node 1 in the lower left corner is the input node, and node 2 in the upper right corner is the output node. The direction of the arrow indicates the flow of data. The output data of some nodes between node 1 and node 2 can serve as input data for other nodes. The structure of each node is unordered, supporting arbitrary connections. For example, data from node 1 flows to nodes 3 and 4; node 3 receives the input data from node 1, performs calculations, and inputs the results to nodes 5 and 6; node 4 receives the input data from node 1, performs calculations, and inputs the results to nodes 6 and 7; the nodes between node 1 and node 2 can all be operated on in the same way, which will not be elaborated further here.

[0067] 402: Determine the resources to be allocated and the data dependencies of each node in the computation graph of the neural network.

[0068] In this embodiment, the required resources for each node in the computation graph of the neural network can be determined by a resource allocation algorithm, and the required resources for each node in the neural network can be stored in [the relevant database]. Figure 6 The time-based configuration shown allocates resources to each node according to the resources required by each node, thereby achieving the purpose of reconfiguring the computing module. The computing module consists of the resources to be allocated, including PE and memory.

[0069] Based on the different resources required for each node, the operation types of each node can be divided into computation-sensitive operations and memory-sensitive operations. Computation-sensitive operations include convolution (Conv) operations, while memory-sensitive operations include fully-connected (FC) operations, matrix operations without shared data, and pointwise convolution operations. Computation-sensitive operations require more physical resources (PEs), while memory-sensitive operations require more memory.

[0070] It is understood that in the embodiments of this application, when the execution order of each node and the resources to be allocated to each node are determined, and each node starts to execute the operation, the mode controller can obtain the PE and memory to be allocated to the neural network from the reconfigurable PE array according to the resources to be allocated to each node stored in the time-based configuration, thereby realizing the reconstruction of the computing modules of each node.

[0071] The following examples illustrate the computation modules of the mesh structure in some embodiments. Figure 7 A schematic diagram of the computing module's structure is shown. (For example...) Figure 7 As shown, the mesh-structured computing module reads the data to be processed from the Dynamic Random Access Memory (DRAM), performs calculations on the data, obtains the result, and returns it to the DRAM. The computing module includes memory and PE. The memory includes instruction memory for carrying instructions, parameter memory for carrying parameters, and activation memory for carrying other data. The PE includes PE memory for accessing data transferred from the memory, computer lines for calculation, and inner memory for storing computer lines.

[0072] It is understandable that the above Figure 7 The ratio of PE to memory in the computing module shown is fixed, so the computing module obtained by each node is also fixed. However, since the types of operations that each node needs to perform are different, the requirements of each node for PE and memory are not completely consistent. Allocating the same PE and memory to nodes that perform different operations in the same way will cause some nodes to have too many resources, resulting in resource waste.

[0073] The method provided in this application can allocate resources to each node in the neural network according to the resource allocation requirements of each node. For example, by... Figure 8The mode controller shown retrieves the required PE and memory for each node in the neural network from the Reconfigurable PE array based on the resource allocation information stored in the time-based configuration. This allows for the reconfiguration of the computation modules of each node, thus allocating appropriate resources to each node and avoiding situations where some nodes have too many resources, leading to resource waste, or some nodes have too few resources, leading to resource insufficiency.

[0074] Among them, such as Figure 8 As shown, before constructing the computing modules of each node by obtaining the corresponding number of PEs and memory according to the resource allocation requirements of each node through the mode controller, the memory used for each node of the neural network is divided into memory banks according to the resource allocation requirements and operation types of each node. Each node is further divided into computationally intensive operators for convolutional operation arrays and operators for matrix multiplication arrays. Memory banks include convolutional memory banks and fully connected memory banks (MM memory banks). Convolutional memory banks are used to carry computationally intensive operators for convolutional arrays, i.e., computationally sensitive operations. MM memory banks are used to carry operators for matrix multiplication arrays, i.e., memory-sensitive operations. Computationally sensitive operations include convolutional operations, and memory-sensitive operations include fully connected (FC) operations, matrix operations without data sharing, and pointwise convolution operations.

[0075] For example, for Figure 1 The topology diagram is shown below. Assume the determined resource allocation requirements are as follows: Node 1 requires 5 PEs and 2 memory (computation-sensitive operation); Node 2 requires 10 PEs and 3 memory (computation-sensitive operation); Node 3 requires 5 PEs and 20 memory (memory-sensitive operation); Node 4 requires 1 PE and 20 memory (memory-sensitive operation). First, [the following is a partial translation of the original text, which is incomplete and requires further context]. Figure 1 The resources required by each node are stored in the time-based configuration. Then, it is determined that 21 PEs, 5 Conv Memory banks, and 40 MM memory banks are needed, and the required resources are allocated accordingly. Figure 1The memory in the topology structure diagram is divided into 5 Conv Memory banks and 40 MM memory banks, and the allocated resources are stored in the Reconfigurable PE array; when the operation starts, the mode controller obtains 5 PEs and 2 memories (Conv Memory banks) from the Reconfigurable PE array and allocates them to node 1 in sequence according to the resources required by each node stored in the time based configuration, obtains 10 PEs and 3 memories (Conv Memory banks) and allocates them to node 2; obtains 5 PEs and 20 memories (MM memory banks) and allocates them to node 3; obtains 1 PE and 20 memories (MM memory banks) and allocates them to node 4, thus realizing the reconfigurable computing module.

[0076] It can be understood that due to the differences in the memory requirements of CONV operations and FC operations, the memory is segmented into Memory banks, and Memory banks can be further divided into Conv banks and FC banks; for FC operations, since there is no data reuse, the memory requirements for FC operations are relatively large, and the number of banks (i.e., FC banks) used is the number of parallel computations. The remaining part of the Memory banks can be allocated to Conv banks. For CONV operations, since there is data reuse, the memory requirements are relatively small. It can be understood that for CONV operations, assuming there are a total of S columns of data, each column is reused K times, and S < K, then K time cycles (clock cycles) are required, and K - S columns are reused.

[0077] Thus, by the mode controller allocating resources to each node, the reconfiguration of the computing module is realized, optimizing the neural network at the hardware level, enabling the ratio of PEs to memory in the computing module to be adjusted according to the operation types of each node, and avoiding resource waste.

[0078] 403: Determine multiple groups of mutually independent node units according to the data dependency relationships of each node in the neural network.

[0079] For example, as Figure 2 shown in the topology structure diagram, first determine Figure 2There are data dependencies between node 1 and subgraph B(1), subgraph B(1) and node 2, node 2 and subgraph A, subgraph A and node 4, node 1 and subgraph C(2), subgraph C(2) and node 4, node 1 and subgraph C(1), subgraph C(1) and node 3, node 3 and subgraph B(2), and subgraph B(2) and node 4. Based on the above data dependencies, it can be determined that subgraph B(1), node 2, subgraph A and subgraph C(2) and subgraph C(1), node 3 and subgraph B(2) are mutually independent node units.

[0080] 404: The execution order of each node is determined based on multiple sets of independent node units and the calculated resources to be allocated.

[0081] In this embodiment, a greedy search algorithm can be used to sort each group of independent node units based on multiple sets of independent node units and the calculated computation time and memory transfer time of each node, to obtain the execution order set corresponding to each node. Then, the execution order with the shortest data processing time in the execution order set is taken as the first order. The execution time of the first order is compared with the execution time of the system default order. When the execution time of the first order is lower than the execution time of the system default order, the first order is taken as the execution order of each node. When the execution time of the first order is not lower than the execution time of the system default order, the system default order is taken as the execution order of each node. The system default order is the execution order of each node determined according to the connection order of each node in the neural network.

[0082] The following is based on Figure 2 Using the topology diagram shown as an example, the data processing method in this application is explained.

[0083] Assume the resources to be allocated are as follows: Node 1 needs 5 PEs and 2 memory; Node 2 needs 10 PEs and 3 memory; Node 3 needs 5 PEs and 20 memory; Node 4 needs 1 PE and 20 memory; Subgraph A needs 10 PEs and 5 memory; Subgraph B(1) needs 20 PEs and 4 memory; Subgraph B(2) needs 20 PEs and 4 memory; Subgraph C(1) needs 5 PEs and 15 memory; Subgraph C(2) needs 5 PEs and 15 memory. Subgraph B(1), Node 2, Subgraph A and Subgraph C(2), as well as Subgraph C(1), Node 3 and Subgraph B(2) are independent node units that can be parallelized.

[0084] Furthermore, based on mutually independent and parallelizable node units, parallel branches can be determined to include branch a, branch b, and branch c. The execution order of branch a is as follows: node 1 → subgraph B (1) → node 2 → subgraph A → node 4; the execution order of branch b is as follows: node 1 → subgraph C (2) → node 4; and the execution order of branch c is as follows: node 1 → subgraph C (1) → node 3 → subgraph B (2) → node 4. The execution order set corresponding to each node is obtained by using a greedy search algorithm based on the computation time and memory transfer time of the resources to be allocated to each node. The obtained execution order set includes:

[0085] ① Node 1 → Subgraph B(1) → Subgraph C(1) → Node 3 → Node 2 → Subgraph A → Subgraph C(2) → Subgraph B(2) → Node 4;

[0086] ② Node 1 → Subgraph B(1) → Subgraph C(1) → Node 3 → Node 2 → Subgraph A → Subgraph C(2) → Subgraph B(2) → Node 4;

[0087] ③ Node 1 → Subgraph B(1) → Subgraph C(1) → Node 3 → Node 2 → Subgraph A → Subgraph C(2) → Subgraph B(2) → Node 4, etc.

[0088] Then, based on the allocation order of the execution order set, the first order is determined as: Node 1 → Subgraph B(1) → Subgraph C(1) → Node 3 → Node 2 → Subgraph A → Subgraph C(2) → Subgraph B(2) → Node 4.

[0089] Furthermore, it can be determined based on the first order. Figure 2 The execution time of the topology diagram shown can be as follows: Figure 3b As shown, the execution time in the default order is as follows: Figure 3a As shown, the data processing method of this application can reduce the waiting time of subgraph C(1), node 2, node 3, subgraph A, and subgraph B(2), significantly reducing the processing time and improving the data processing efficiency of the neural network.

[0090] It is understandable that the methods for determining the execution time of the first order and the default order are as described above, and will not be repeated here.

[0091] The following is based on Figure 5 The data processing method in this application is illustrated using the neural network shown as an example.

[0092] From the above analysis, it can be seen that, Figure 5The input and output data of node 3 are consistent with those of node 4, and the input and output data of node 5 are consistent with those of node 6 and node 7. Therefore, in this embodiment, the data can be divided diagonally. Figure 5 The nodes in the algorithm are configured such that nodes on the same diagonal have the same operation type, input data, and output data. This means that the data of nodes on the same diagonal is independent, while the data of adjacent nodes on different diagonals is dependent, thus enabling parallel processing. Figure 5 Units on the same diagonal.

[0093] It's understandable, if we follow... Figure 5 The default execution order is used for calculation, because Figure 5 The neural network consists of 36 units, thus requiring 36 time slices. However, if the method provided in this application embodiment is followed, i.e., the operation of each node on the diagonal is performed in parallel, only 11 time slices are needed. Therefore, the method provided in this application embodiment can significantly reduce the time slices required by the neural network and reduce the allocation time of the neural network. In other words, the method provided in this application embodiment significantly reduces the resources required by the neural network and improves the execution efficiency of the neural network while ensuring the normal operation of the neural network.

[0094] The following is based on Figure 1 Using the topology diagram shown as an example, the data processing method in this application is explained.

[0095] like Figure 1 As shown, the electronic device first acquires the data to be processed and inputs it into node 1 in the topology diagram for calculation. After obtaining the first calculation result, node 1 simultaneously inputs the first calculation result into nodes 2, 3, and 4. Node 2 obtains the second calculation result based on the first calculation result and inputs it into node 4. Then, node 3 obtains the third calculation result based on the first calculation result and inputs it into node 4. Finally, node 4 obtains the fourth calculation result based on the first, second, and third calculation results and outputs it. That is, the default execution order is: node 1 → node 2 → node 3 → node 4.

[0096] for Figure 1 The structure shown does not affect the final computation time whether node 2 is computed first and then node 3, or vice versa. This is because node 4 requires data from nodes 1, 2, and 3 to be computed. Therefore, adjusting the execution order does not reduce the execution time, and parallel processing is not necessary. Figure 1 Nodes 2 and 3 in the structure shown are... Figure 1 When the structure shown performs operations, resources are allocated to each node according to the default execution order.

[0097] Therefore, the data processing method mentioned in the embodiments of this application optimizes the neural network at the software level by using node units in the parallel neural network that do not have data dependencies.

[0098] 405: When each node performs a computation, control each node to perform the computation in the order of execution, and allocate resources to each node according to the resource allocation requirements of each node.

[0099] It is understood that in some embodiments, the mode controller of the neural network can obtain the corresponding number of computing units and memory units based on the operation type of each node, thereby realizing the reconstruction of the computing module. This allows the proportion of computing units and memory units in the computing module to be adjusted according to the operation type of each node, thus avoiding resource waste.

[0100] For example, for Figure 1 In the topology diagram, assuming the determined resource allocation requirements are: Node 1 needs 5 PEs and 2 memory; Node 2 needs 10 PEs and 3 memory; Node 3 needs 5 PEs and 20 memory; Node 4 needs 1 PE and 20 memory; First, ... Figure 1 The resources required by each node are stored in the time-based configuration. When the operation starts, the mode controller, based on the resource allocation requirements of each node stored in the time-based configuration, sequentially obtains 5 PEs and 2 memory from the Reconfigurable PE array and allocates them to node 1; obtains 10 PEs and 3 memory and allocates them to node 2; obtains 5 PEs and 20 memory and allocates them to node 3; and obtains 1 PE and 20 memory and allocates them to node 4, thereby realizing the reconfiguration of the computing module.

[0101] For example, for Figure 2In the topology diagram, assuming the determined resources to be allocated are: Node 1 needs 5 PEs and 2 memory; Node 2 needs 10 PEs and 3 memory; Node 3 needs 5 PEs and 20 memory; Node 4 needs 1 PE and 20 memory; Subgraph A needs 10 PEs and 5 memory; Subgraph B(1) needs 20 PEs and 4 memory; Subgraph B(2) needs 20 PEs and 4 memory; Subgraph C(1) needs 5 PEs and 15 memory; Subgraph C(2) needs 5 PEs and 15 memory. When the operation starts, the mode controller, based on the resources to be allocated for each node stored in the time-based configuration, selects from the Reconfigurable PEs... In the array, 5 PEs and 2 memory are allocated to node 1 in sequence; 20 PEs and 4 memory are allocated to subgraph B(1); 5 PEs and 15 memory are allocated to subgraph C(1); 5 PEs and 20 memory are allocated to node 3; 10 PEs and 3 memory are allocated to node 2; 10 PEs and 5 memory are allocated to subgraph A; 5 PEs and 15 memory are allocated to subgraph C(2); 20 PEs and 4 memory are allocated to subgraph B(2); and 1 PE and 20 memory are allocated to node 4.

[0102] The method provided in this application embodiment can parallelize nodes in the computation graph of a neural network that have no data dependencies, effectively shortening the execution time of each node. Furthermore, it can allocate appropriate resources to each node according to the operation type of different nodes, avoiding situations where some nodes have too many resources, resulting in resource waste, or some nodes have too few resources, resulting in resource insufficiency.

[0103] This application provides a system-on-a-chip (SoC) comprising a first determining module, a second determining module, and a processing module. The first determining module is configured to determine the data processing resources required by each node in a neural network, and to divide the nodes in the neural network into multiple independent node unit groups based on the data dependencies between nodes. Each independent node unit group includes multiple independent node units, and there are no data dependencies between the independent node units within the same independent node unit group. The second determining module is configured to determine the execution order of each node in the neural network based on the data processing resources required by each node and the multiple independent node unit groups. The processing module is configured to control the neural network to execute each node according to the execution order to process the data to be processed.

[0104] like Figure 9As shown, the electronic device 10 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, buttons 101, and a display screen 102, etc.

[0105] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0106] Processor 110 may include one or more processing units, such as processing modules or circuits of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Micro-programmed Control Unit (MCU), Artificial Intelligence (AI) processor, or Field Programmable Gate Array (FPGA). Different processing units may be independent devices or integrated into one or more processors. Processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in processor 110 is a cache memory 180.

[0107] It is understood that the data processing method in this embodiment can be executed by the processor 110 of the corresponding electronic device. The power module 140 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component is used to manage the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module is used to receive charging input from a charger; the power management module is used to connect the power supply and the processor 110. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 110, display screen 102, camera 170, and wireless communication module 120, etc.

[0108] The mobile communication module 130 may include, but is not limited to, antennas, power amplifiers, filters, and low-noise amplifiers (LNAs). The mobile communication module 130 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the electronic device 10. The mobile communication module 130 can receive electromagnetic waves via the antenna, filter and amplify the received electromagnetic waves, and then transmit them to a modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some functional modules of the mobile communication module 130 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 130 and at least some modules of the processor 110 may be housed in the same device.

[0109] The wireless communication module 120 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 120 can provide solutions for wireless communication applications on the electronic device 10, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The electronic device 10 can communicate with networks and other devices through wireless communication technologies.

[0110] It is understood that, in this embodiment of the application, when the electronic device is a receiving device, it can receive video and audio data from other electronic devices in the recording group, as well as the recording content tags and time stamp information corresponding to each data, through the wireless communication module. And when the electronic device is a transmitting device, it can send video and audio data, as well as the recording content tags and time stamp information corresponding to each data, to other electronic devices in the recording group through the wireless communication module.

[0111] In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the electronic device 10 may also be located in the same module.

[0112] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0113] The sensor module 190 may include proximity sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0114] It is understood that the ambient light sensor in the embodiments of this application can be used to acquire illumination status information and send the illumination status information to the processor.

[0115] Audio module 150 is used to convert digital audio information into analog audio signal output, or to convert analog audio input into digital audio signal. Audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, audio module 150 may be located in processor 110, or some functional modules of audio module 150 may be located in processor 110. In some embodiments, audio module 150 may include a speaker, earpiece, microphone, and headphone jack. Camera 170 is used to capture still images or videos. An object generates an optical image through the lens and projects it onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to image signal processing (ISP) to convert it into a digital image signal. Electronic device 10 can implement shooting functions through ISP, camera 170, video codec, graphics processing unit (GPU), display screen 102, and application processor.

[0116] Interface module 160 includes an external memory interface, a universal serial bus (USB) interface, and a subscriber identification module (SIM) card interface. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of electronic device 10. The external memory card communicates with processor 110 through the external memory interface to perform data storage. The USB interface is used for communication between electronic device 10 and other electronic devices. The SIM card interface is used to communicate with the SIM card installed in electronic device 1010, for example, to read or write phone numbers stored in the SIM card.

[0117] In some embodiments, the electronic device 10 further includes buttons 101, a motor, and indicators. The buttons 101 may include volume buttons, a power button, etc. The motor is used to generate a vibration effect in the electronic device 10, for example, vibrating when the user's electronic device 10 is called to prompt the user to answer the call. The indicators may include laser indicators, radio frequency indicators, LED indicators, etc.

[0118] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0119] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0120] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used to implement the program code when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language. In some cases, the disclosed embodiments can be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments can also be implemented as instructions carried or stored thereon on one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, the instructions can be distributed via a network or through other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc-read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0121] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0122] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0123] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. While this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art will understand that various changes in form and detail can be made without departing from the spirit and scope of this application.

Claims

1. A data processing method applied to electronic devices, characterized in that, include: The data processing resources required for each node in the neural network are determined, and the nodes in the neural network are divided into multiple independent node unit groups according to the data dependencies of each node. Each independent node unit group includes multiple independent node units, and there are no data dependencies between the independent node units in the same independent node unit group. Based on the data processing resources required by each node and the multiple independent node unit groups, the execution order of each node in the neural network is determined; wherein, determining the execution order of each node in the neural network based on the data processing resources required by each node and the multiple independent node unit groups includes: determining a set of execution orders for each node in the neural network based on the data processing resources required by each node and the multiple independent node unit groups; determining a first execution order with the shortest execution time in the set of execution orders; when the data processing time corresponding to the first execution order is lower than the data processing time corresponding to the default execution order of each node in the neural network, the first execution order is used as the execution order of each node in the neural network; when the data processing time corresponding to the first execution order is not lower than the data processing time corresponding to the default execution order of each node in the neural network, the default execution order of each node in the neural network is used as the execution order of each node in the neural network. The neural network is controlled to execute each node according to the execution order in order to process the data to be processed.

2. The data processing method according to claim 1, characterized in that, The process of controlling the neural network to execute each node according to the execution order to process the data to be processed includes: Allocate the data processing resources required by each node according to the execution order; The neural network is controlled to process the data to be processed based on the data processing resources required by each node.

3. The data processing method according to claim 1, characterized in that, The independent node unit includes one node or multiple nodes.

4. The data processing method according to claim 1, characterized in that, The data processing resources include computing resources and memory resources.

5. The data processing method according to claim 4, characterized in that, Based on the data processing resources required by each node, the memory resources required by the neural network are divided into multiple types of memory resource slices. The control of the neural network to execute each node according to the execution order to process the data to be processed includes: Determine the operation type and the number of memory resource slices required for each node; Based on the operation type of each node and the number of memory resource slices required, allocate memory resource slices of the corresponding type and the corresponding number to each node.

6. The data processing method according to claim 5, characterized in that, The various types of memory resource slices include memory slices for convolutional operators and memory slices for fully connected operators; The operation types of each node include convolution operations and fully connected operations.

7. The data processing method according to claim 4, characterized in that, The control of the neural network to execute each node according to the execution order to process the data to be processed includes: Determine the operation type of each node; Based on the operation type of each node, the nodes are divided into multiple types of computational operation arrays; Allocate computing resources to the various types of computing operation arrays.

8. The data processing method according to claim 7, characterized in that, The various types of computational operation arrays include computationally intensive operators for convolution operation arrays and operators for matrix multiplication arrays.

9. The data processing method according to claim 1, characterized in that, The control of the neural network to execute each node according to the execution order to process the data to be processed includes: Store the data processing resource information required by each node of the neural network in a time configuration table; Based on the data processing resource information stored in the time configuration table, data processing resources are allocated to each node in accordance with the execution order. The neural network is controlled to process the data to be processed based on the data processing resources required by each node.

10. The data processing method according to claim 1, characterized in that, The determination of the data processing resources required for each node in the neural network includes: The data processing resources required by each node of the neural network are obtained based on the pattern controller. The process of controlling the neural network to execute each node according to the execution order to process the data to be processed includes: The mode controller allocates data processing resources to each node of the neural network according to the execution order. The neural network is controlled to process the data to be processed based on the data processing resources required by each node.

11. A system-on-a-chip, characterized in that, The system-on-a-chip includes a first determining module, a second determining module, and a processing module; The first determining module is used to determine the data processing resources required by each node in the neural network, and to divide the nodes in the neural network into multiple independent node unit groups according to the data dependency relationship of each node, wherein each independent node unit group includes multiple independent node units, and there is no data dependency relationship between the independent node units in the same independent node unit group. The second determining module is used to determine the execution order of each node in the neural network based on the data processing resources required by each node and the plurality of independent node unit groups; wherein, determining the execution order of each node in the neural network based on the data processing resources required by each node and the plurality of independent node unit groups includes: determining a set of execution orders for each node in the neural network based on the data processing resources required by each node and the plurality of independent node unit groups; determining a first execution order with the shortest execution time in the set of execution orders; when the data processing time corresponding to the first execution order is lower than the data processing time corresponding to the default execution order of each node in the neural network, the first execution order is used as the execution order of each node in the neural network; when the data processing time corresponding to the first execution order is not lower than the data processing time corresponding to the default execution order of each node in the neural network, the default execution order of each node in the neural network is used as the execution order of each node in the neural network. The processing module is used to control the neural network to execute each node according to the execution order in order to process the data to be processed.

12. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the data processing method as described in any one of claims 1 to 10.

13. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of the electronic device, wherein the processor is one of the one or more processors of the electronic device for performing the data processing method as described in any one of claims 1 to 10.

Images