Method and computing device for determining tensor information of deep learning model

Abstract

A computing device and a method for determining tensor information of a deep learning model are disclosed. The computing device includes a memory and a scheduling unit, the memory stores instructions, and the scheduling unit reads instructions to execute: for a deep learning model that does not provide complete tensor information, determine at least one processable subgraph; execute the deep learning model with virtual tensor data as the input of the deep learning model, and in the process, record the input tensor data and output tensor data of the processable subgraph, the virtual tensor data is constructed according to the input tensor information of the deep learning model; infer the tensor information of each node of the processable subgraph according to the input tensor data and output tensor data of the processable subgraph. This method is applicable to deep learning models with arbitrary model structures, and is particularly applicable to obtaining tensor information of nodes that cannot obtain tensor information through the software platform or user-defined nodes.

Description

Technical Field

[0001] The present disclosure relates to the field of neural networks, and more specifically, to a method and computing device for determining tensor information of a deep learning model. Background Art

[0002] In the field of neural networks, deep learning models generally go through two stages: training and inference. In the training stage, the weight parameters of the deep learning model are continuously adjusted according to the training results. In the inference stage, the system will perform some optimization operations on the deep learning model during the compilation and execution stages to improve the inference performance of the deep learning model.

[0003] However, these optimization operations generally require the deep learning model to provide complete tensor information. The complete tensor information includes the input and output tensor information of each node of the deep learning model. However, in some cases, the tensor information provided by the deep learning model is not complete, and it is impossible to obtain complete tensor information through other means. For example, some large and complex models generally do not provide complete tensor information, and it is not easy to obtain complete tensor information based on existing mechanisms. Summary of the invention

[0004] Based on this, the purpose of the present disclosure is to provide a method and computing device for determining tensor information of a deep learning model.

[0005] The present disclosure provides a computing device, including a memory and a scheduling unit, wherein the memory stores instructions, and the scheduling unit reads the instructions to execute:

[0006] For a deep learning model that does not provide complete tensor information, determine at least one processable subgraph, where the processable subgraph consists of a plurality of consecutive nodes of the deep learning model and can be processed by the designated acceleration unit;

[0007] Executing the deep learning model using virtual tensor data as input of the deep learning model, and in the process, recording input tensor data and output tensor data of the processable subgraph, wherein the virtual tensor data is constructed according to input tensor information of the deep learning model;

[0008] The tensor information of each node of the processable subgraph is inferred according to the input tensor data and the output tensor data of the processable subgraph.

[0009] Optionally, determining a plurality of processable subgraphs comprises:

[0010] For the designated acceleration unit, the deep learning model is divided into a plurality of subgraphs using nodes that cannot be processed by the designated acceleration unit as separators;

[0011] Among the multiple sub-graphs, some sub-graphs are screened out, and the remaining sub-graphs are used as at least one processable sub-graph.

[0012] Optionally, the filtering out some subgraphs includes:

[0013] Screening out subgraphs for which tensor information of each node can be obtained via the software platform of the deep learning model; and / or

[0014] The subgraphs that cannot be optimized are screened out, where the subgraphs that cannot be optimized are subgraphs whose calculation optimization effect on the specified acceleration unit is less than a set threshold.

[0015] Optionally, the calculating tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph comprises:

[0016] Determining input tensor data of input boundary nodes of the processable subgraph according to the input tensor data of the processable subgraph;

[0017] For the processable subgraph, based on the specific function of each node and the input tensor data of the input boundary nodes of the processable subgraph, tensor information of each node is calculated one by one starting from the input boundary node to the output boundary node.

[0018] Optionally, the calculating tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph further includes:

[0019] Compare whether the output tensor information of the output boundary nodes of the processable subgraph is consistent with the output tensor data of the processable subgraph to determine whether the inference process is correct.

[0020] Optionally, the calculating tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph further includes:

[0021] When the output tensor information of the output boundary nodes of the processable subgraph does not match the output tensor data of the processable subgraph, reverse calculation is performed based on the output tensor data of the processable subgraph to correct the tensor information of each node of the processable subgraph.

[0022] Optionally, the calculating tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph comprises:

[0023] If the processable subgraph contains unrecognizable nodes, perform the following steps:

[0024] Determining input tensor data of input boundary nodes and output tensor data of output boundary nodes of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph;

[0025] Starting from the input boundary node of the processable subgraph to the node before the unrecognizable node, inferring the tensor information of each node one by one;

[0026] Starting from the output boundary node to the node after the unrecognizable node, reversely infer the tensor information of each node one by one;

[0027] The tensor information of the unrecognizable node is determined according to the tensor information of the node before the unrecognizable node and the tensor information of the node after the unrecognizable node.

[0028] Optionally, the method further includes: using the tensor information of each node to perform compile-time tuning on the deep learning model.

[0029] Optionally, when executing the deep learning model, the designated acceleration unit uses the tensor information of each node to identify the executable instructions of the deep learning model and distributes them to corresponding hardware operators.

[0030] Optionally, the tensor information of each node of the processable subgraph is stored as attribute data of each node of the processable subgraph.

[0031] Optionally, the tensor information of each node includes: the shape and type of the input tensor and output tensor of each node.

[0032] In a second aspect, an embodiment of the present disclosure provides a method for determining tensor information of a deep learning model, including:

[0033] For a deep learning model that does not provide complete tensor information, determine a plurality of processable subgraphs, wherein the processable subgraphs are composed of a plurality of consecutive nodes of the deep learning model;

[0034] Executing the deep learning model using virtual tensor data as input of the deep learning model, and in the process, recording input boundary tensor data and output data of the processable subgraph, wherein the virtual tensor data is constructed according to input tensor information of the deep learning model;

[0035] The tensor information of each node of the processable subgraph is inferred according to the input boundary tensor data and the output data of the processable subgraph.

[0036] Optionally, determining at least one processable subgraph comprises:

[0037] For the designated acceleration unit, the deep learning model is divided into a plurality of subgraphs using nodes that cannot be processed by the designated acceleration unit as separators;

[0038] Among the multiple sub-graphs, some sub-graphs are screened out, and the remaining sub-graphs are used as at least one processable sub-graph.

[0039] Optionally, the filtering out some subgraphs includes:

[0040] Screening out subgraphs for which tensor information of each node can be obtained via the software platform of the deep learning model; and / or

[0041] The subgraphs that cannot be optimized are screened out, where the subgraphs that cannot be optimized are subgraphs whose calculation optimization effect on the specified acceleration unit is less than a set threshold.

[0042] Optionally, the calculating tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph comprises:

[0043] Determining input tensor data of input boundary nodes of the processable subgraph according to the input tensor data of the processable subgraph;

[0044] For the processable subgraph, based on the specific function of each node and the input tensor data of the input boundary nodes of the processable subgraph, tensor information of each node is calculated one by one starting from the input boundary node to the output boundary node.

[0045] Optionally, the calculating tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph further includes:

[0046] Compare whether the output tensor information of the output boundary nodes of the processable subgraph is consistent with the output tensor data of the processable subgraph to determine whether the inference process is correct.

[0047] Optionally, the calculating tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph further includes:

[0048] When the output tensor information of the output boundary nodes of the processable subgraph does not match the output tensor data of the processable subgraph, reverse calculation is performed based on the output tensor data of the processable subgraph to correct the tensor information of each node of the processable subgraph.

[0049] Optionally, the calculating tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph comprises:

[0050] If the processable subgraph contains unrecognizable nodes, perform the following steps:

[0051] Determining input tensor data of input boundary nodes and output tensor data of output boundary nodes of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph;

[0052] Starting from the input boundary node of the processable subgraph to the node before the unrecognizable node, inferring the tensor information of each node one by one;

[0053] Starting from the output boundary node to the node after the unrecognizable node, reversely infer the tensor information of each node one by one;

[0054] The tensor information of the unrecognizable node is determined according to the tensor information of the node before the unrecognizable node and the tensor information of the node after the unrecognizable node.

[0055] Optionally, the method further includes: using the tensor information of each node to perform compile-time tuning on the deep learning model.

[0056] Optionally, when executing the deep learning model, the designated acceleration unit uses the tensor information of each node to identify the executable instructions of the deep learning model and distributes them to corresponding hardware operators.

[0057] Optionally, the tensor information of each node of the processable subgraph is stored as attribute data of each node of the processable subgraph.

[0058] Optionally, the tensor information of each node includes: the shape and type of the input tensor and output tensor of each node.

[0059] In a third aspect, an embodiment of the present disclosure provides a data center, comprising the above-mentioned computing device.

[0060] In this embodiment, the deep learning model is executed on the scheduling unit with virtual tensor data, and the input tensor data and output tensor data of each processable subgraph are recorded. These input tensor data and output tensor data are subsequently used to infer the tensor information of each node in each processable subgraph, so as to achieve the purpose of determining the tensor information of the deep learning model as soon as possible. This method can be applied to deep learning models with arbitrary model structures, and can be run on general-purpose devices such as processors, and is particularly suitable for obtaining tensor information that cannot be obtained through software platforms or tensor information of user-defined nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The above and other objects, features and advantages of the present disclosure will become more apparent through the description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0062] Figure 1 A hierarchical structure diagram of a data center to which an embodiment of the present disclosure is applied is shown;

[0063] Figure 2 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0064] Figure 3 is a block diagram of the internal structure of a server in a data center according to an embodiment of the present disclosure;

[0065] Figure 4 It is a control relationship diagram of a central processing unit (CPU) and a neural network acceleration unit (NPU) inside a server according to an embodiment of the present disclosure;

[0066] Figure 5 is an internal structure diagram of an NPU core according to an embodiment of the present disclosure;

[0067] Figure 6 is an architectural diagram of an exemplary software platform for compilation optimization of deep learning models;

[0068] Figure 7 is a flowchart of a method for determining tensor information of a deep learning model according to an embodiment of the present disclosure;

[0069] Figure 8 is a flowchart of a method for determining tensor information of a deep learning model according to another embodiment of the present disclosure;

[0070] Fig. 9 is a flowchart of a method for determining tensor information of a deep learning model according to a third embodiment of the present disclosure;

[0071] Fig.10 is a schematic diagram of a deep learning model containing a subgraph;

[0072] Fig.11 is a schematic diagram of a deep learning model containing multiple subgraphs. DETAILED DESCRIPTION

[0073] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0074] The following terms are used in this article.

[0075] Acceleration unit: also known as neural network acceleration unit, it is a processing unit designed to improve the data processing speed in some special-purpose fields (for example, processing images, processing various operations of neural networks, etc.) where the general-purpose processor is inefficient. It is often used in conjunction with a general-purpose processor CPU, and is controlled by the general-purpose processor to perform processing for specific purposes or specific fields, thereby improving computer processing efficiency in specific purposes or specific fields.

[0076] On-chip memory: Memory that is used exclusively within the main core or sub-core and cannot be shared.

[0077] Command Processor: The command interface between the accelerator unit and the central processing unit that drives the accelerator unit. The command processor receives instructions from the central processing unit for the accelerator unit to execute, and distributes these instructions to the various cores in the accelerator unit for execution. In addition, it is also responsible for the synchronization of the various cores in the accelerator unit.

[0078] Life cycle: An operand is not involved in all the processes in the instruction sequence. The part between its first appearance and the last use in the instruction sequence is the life cycle of the operand. In other words, after the life cycle, it is no longer used and there is no need to keep it in the on-chip memory.

[0079] Neural network: generally refers to artificial neural network (ANN), which is an algorithmic network that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. A classic neural network, which is also the simplest neural network structure, contains three levels: input layer, output layer and middle layer (also called hidden layer). The input layer, output layer and middle layer each include multiple nodes. A node is the smallest processing unit in a neural network. A very complex neural network structure can be formed through the extensive interconnection of a large number of simple function nodes.

[0080] Neural network model: In a neural network, nodes are mathematized to generate mathematical models of nodes. The mathematical models of a large number of nodes in the neural network constitute the neural network model.

[0081] Deep learning model: The concept of deep learning originates from the study of neural networks, and neural networks with multiple intermediate layers are called deep learning networks. Therefore, in this sense, deep learning models are also a kind of neural network models. Both deep learning models and neural network models must be generated through training. The sample data is input into the designed network structure (that is, the network structure has been determined), feature information is extracted through multiple intermediate layers, and the weight parameters of neurons are continuously corrected based on the output results of the output layer, so that the output results of the output layer tend to the preset results more and more, until the final weight parameters are determined. The trained deep learning model can be truly applied to actual scenarios, and the use of the deep learning model in actual scenarios can also be collected, which in turn optimizes the deep learning model.

[0082] Hardware operator: A hardware module used to execute corresponding instructions in the acceleration unit. When the neural network model is compiled from the source code to obtain an executable instruction sequence, each executable instruction will be assigned to the corresponding hardware operator for execution. There is a corresponding relationship between nodes, executable instructions, and hardware operators. Of course, it is not necessarily a one-to-one relationship, but can also be a many-to-one relationship. For example, multiple nodes are executed in the same hardware operator, or multiple executable instructions are executed by the same hardware operator.

[0083] Subgraph: A group of nodes consisting of multiple consecutive nodes of a deep learning model.

[0084] Input boundary nodes and output boundary nodes: The input boundary node of each subgraph refers to the first node used to receive the input tensor data of the subgraph and process it. The first node can be one or more nodes. The output boundary node of each subgraph refers to the last node that outputs the output tensor data of the subgraph. The last node can be one or more nodes.

[0085] Figure 1 A hierarchical structure diagram of a data center is shown as a scenario in which an embodiment of the present disclosure is applied.

[0086] Data centers are a globally collaborative network of specific devices used to transmit, accelerate, display, calculate, and store data information on the Internet network infrastructure. In the future, data centers will also become competitive assets for enterprises. With the widespread application of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. As an important technology of artificial intelligence, neural networks have been widely used in data center big data analysis and calculation.

[0087] In traditional large data centers, the network structure is usually Figure 1 The three-layer structure shown is a hierarchical inter-networking model. This model includes the following three layers:

[0088] Access Layer 103: Sometimes also called the edge layer, including access switches 130 and servers 140 connected to the access switches. Each server 140 is a processing and storage entity in the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140. Access switches 130 are used to connect these servers to the switches in the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are usually located at the top of the rack, so they are also called top of rack switches, which are physically connected to the servers.

[0089] Aggregation Layer 102: sometimes also called distribution layer, including aggregation switches 120. Each aggregation switch 120 connects to multiple access switches and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0090] Core Layer 101: includes core switch 110. Core switch 110 provides high-speed forwarding for packets entering and leaving the data center and provides connectivity for multiple aggregation layers. The network of the entire data center is divided into an L3 layer routing network and an L2 layer routing network. The core switch 110 generally provides a flexible L3 layer routing network for the network of the entire data center.

[0091] Generally, the aggregation switch 120 is the demarcation point between the L2 and L3 layer routing networks. The network below the aggregation switch 120 is the L2 network, and the network above is the L3 network. Each group of aggregation switches manages a delivery point (POD), and each POD has an independent VLAN network. The server does not need to modify the IP address and default gateway when migrating within the POD, because one POD corresponds to one L2 broadcast domain.

[0092] The spanning tree protocol (STP) is usually used between the aggregation switch 120 and the access switch 130. STP makes only one aggregation switch 120 available for a VLAN network, and other aggregation switches 120 are used only when a failure occurs (dashed lines in the above figure). In other words, horizontal expansion cannot be achieved at the aggregation layer, because even if multiple aggregation switches 120 are added, only one is still working.

[0093] Figure 2 Shows Figure 1 The physical connection of the various components in a hierarchical data center. Figure 2As shown, one core switch 110 is connected to multiple aggregation switches 120 , one aggregation switch 120 is connected to multiple access switches 130 , and one access switch 130 is connected to multiple servers 140 .

[0094] server

[0095] Since the server 140 is the real computing device in the data center, Figure 3 The structure block diagram of the internal structure of a server 140 is shown. The server 140 includes a memory 210, a central processing unit (CPU) 220 and various acceleration units connected by a bus. These acceleration units include a neural network acceleration unit (NPU) 230, a data transfer unit (DTU) 260, a graphics processing unit (GPU, not shown), an application-specific integrated circuit (ASIC, not shown) and a field programmable gate array (FPGA, not shown).

[0096] The architecture design of traditional processors makes the control unit and storage unit occupy a large part of the space in the architecture, while the computing unit occupies insufficient space. Therefore, it is very effective in logic control, but not efficient enough in large-scale parallel computing. Therefore, various specialized acceleration units have been developed to more effectively improve the computing speed for calculations of different functions and different fields. The acceleration unit proposed in the present disclosure can be any of them, and these acceleration units are introduced below.

[0097] Neural network acceleration unit (NPU) 230: It is a processing unit that uses a data-driven parallel computing architecture to process a large number of operations (such as convolution, pooling, etc.) of each neural network node. Since the data and intermediate results in the large number of operations (such as convolution, pooling, etc.) of each neural network node are closely related in the entire computing process and will be used frequently, using the existing CPU architecture, since the memory capacity in the CPU core is very small, a large number of external core memories must be accessed frequently, resulting in inefficient processing. With the use of NPU, each core has an on-chip memory with a storage capacity suitable for neural network calculations, avoiding frequent access to the memory outside the core, which can greatly improve processing efficiency and computing performance.

[0098] Data transfer unit (DTU) 260: It is a wireless terminal device specially used to convert serial port data into IP data or convert IP data into serial port data for transmission through wireless communication network. The main function of DTU is to transmit the data of remote device back to the background center wirelessly. At the front end, DTU and the customer's device are connected through an interface. After DTU is powered on, it first registers to the mobile GPRS network, and then establishes a socket connection with the background center set in DTU. The background center serves as the server of the socket connection, and DTU is the client of the socket connection. Therefore, DTU and background software are used together. After the connection is established, the front-end device and the background center can perform wireless data transmission through DTU.

[0099] Graphics Processing Unit (GPU): A microprocessor that specializes in image and graphics-related computing. GPUs have exploited the shortcoming of too little space for computing units in CPUs, and use a large number of computing units dedicated to graphics computing, which reduces the graphics card's dependence on the CPU and takes on some of the computationally intensive graphics and image processing tasks that the CPU originally undertook.

[0100] Application-specific integrated circuit (ASIC): refers to an integrated circuit designed and manufactured to meet the needs of specific users and specific electronic systems. Since this integrated circuit is customized according to user requirements, its structure is often adapted to specific user requirements.

[0101] Field Programmable Gate Array (FPGA): It is a product further developed on the basis of programmable devices such as PAL and GAL. It appears as a semi-custom circuit in the field of Application Specific Integrated Circuit (ASIC), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices.

[0102] Although the acceleration unit has the advantage of much higher execution efficiency than ordinary processors for specific applications or fields, it is also subject to the control of the scheduling unit 220. Taking the NPU as an example, the memory 210 stores various deep learning models, including the neurons and weight data of the neurons in these models. These deep learning models are called when needed. Figure 3A scheduling unit 220 is deployed to an acceleration unit 230. Specifically, the scheduling unit 220 can inform the acceleration unit 230 of the storage location of the deep learning model of the acceleration unit 230 in the memory 210 in the form of instructions. The acceleration unit 230 can then address these locations and store the instructions to be executed in its on-chip memory. The scheduling unit 220 can also send the instructions to be executed of the acceleration unit 230 to the acceleration unit 230 in the form of instructions, and the acceleration unit 230 receives the instructions and stores them in the on-chip memory. Similarly, the acceleration unit 230 can also obtain input data in the above manner. The acceleration unit 230 obtains the instructions to be executed and the input data for inference calculation. The weight parameters of the node can be included in the instruction sequence of the deep learning model, and the acceleration unit 230 is taken out from the memory 210 together. Of course, the weight parameters of the node can also be stored independently, and the acceleration unit 230 can be taken out from the memory 210 when needed. Here, the scheduling unit 220 can be understood as a hardware unit with scheduling and control capabilities, which can generally be a hardware unit such as a central processing unit (CPU), a microcontroller, or a microprocessor.

[0103] Internal structure of the scheduling unit and acceleration unit

[0104] Combine the following Figure 4 The internal structure diagram of the scheduling unit and the acceleration unit specifically explains how the scheduling unit controls the acceleration unit to work.

[0105] like Figure 4 As shown, the scheduling unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0106] The instruction fetch unit 223 is used to transfer the instruction to be executed from the memory 210 to the instruction register (which may be Figure 4 A register in the register file 229 shown is used to store instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0107] After fetching the instruction, the dispatch unit 220 enters the instruction decoding stage, and the instruction decoding unit 224 decodes the fetched instruction according to the predetermined instruction format to obtain the operand acquisition information required by the fetched instruction, thereby preparing for the operation of the instruction execution unit 225. The operand acquisition information, for example, points to an immediate number, a register, or other software / hardware that can provide a source operand.

[0108] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions so as to efficiently distribute each instruction to different instruction executing units 226 , thereby making it possible to operate multiple instructions in parallel.

[0109] After the instruction transmitting unit 225 transmits the instruction to the instruction executing unit 226, the instruction executing unit 226 starts to execute the instruction. However, if the instruction executing unit 226 determines that the instruction should be executed by the acceleration unit, it forwards it to the corresponding acceleration unit for execution. For example, if the instruction is an instruction for neural network inference, the instruction executing unit 226 no longer executes the instruction, but sends the instruction to the acceleration unit 230 via the bus for execution by the acceleration unit 230.

[0110] The acceleration unit 230 includes multiple cores 236 ( Figure 4 Four cores are shown, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0111] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus. According to different mechanisms, the bus channel 231 may include a PCIE channel 232, an I2C channel 233, and a JTAG channel 234.

[0112] PCIE, or PCI-Express, is a high-speed serial computer expansion bus standard proposed by Intel in 2001 to replace the old PCI, PCI-X and AGP bus standards. PCIE is a high-speed serial point-to-point dual-channel high-bandwidth transmission. The connected devices are allocated exclusive channel bandwidth and do not share bus bandwidth. It mainly supports active power management, error reporting, end-to-end reliability transmission, hot plugging and quality of service and other functions. Its main advantage is its high data transmission rate, and it also has considerable development potential. At present, most PCIE buses are PCIE GEN3, but the disclosed embodiment can also use PCIE GEN4, that is, a bus channel that complies with the PCI-Express4.0 standard.

[0113] I2C channel 233 is a simple, bidirectional, two-wire synchronous serial bus channel developed by Philips. It only requires two wires to transmit information between devices connected to the bus.

[0114] JTAG is the abbreviation of Joint Test Action Group, which is the common name of the IEEE standard 1149.1 called Standard Test Access Port and Boundary Scan Architecture. This standard is used to verify the functionality of printed circuit boards produced by design and testing. In 1990, JTAG was formally standardized by IEEE document 1149.1-1990, and in 1994, a supplementary document was added to describe the boundary scan description language (BSDL). Since then, this standard has been widely adopted by electronic companies around the world. Boundary scan has almost become synonymous with JTAG. JTAG channel 234 is a bus channel that follows this standard.

[0115] The direct memory access (DMA) mechanism 235 is a function provided by some computer bus architectures, which enables data to be written directly from an attached device (such as an external memory) to the on-chip memory of the acceleration unit 230. Compared with the method in which all data transmissions between devices must pass through the command processor 237, this method greatly improves the efficiency of data access. It is precisely because of this mechanism that the core of the acceleration unit 230 can directly access the memory 210, read the parameters in the deep learning model (such as the weight parameters of each node), etc., which greatly improves the efficiency of data access. Although the direct memory access mechanism 235 is shown in the figure as being located between the processor 237 and the bus channel 231, the design of the acceleration unit 230 is not limited to this. In some hardware designs, each NPU core 236 can include a direct memory access mechanism 235, so that the NPU core 236 does not need to go through the command processor 237, but directly reads data from the attached device and writes it to the on-chip memory of the acceleration unit 230.

[0116] The command processor 237 assigns the instructions sent by the scheduling unit 220 to the acceleration unit 230 to the core 236 for execution. The instruction execution unit 226 sends the instructions to be executed that need to be executed by the acceleration unit 230 to the acceleration unit 230, or the instruction execution unit 226 informs the storage location of the instructions to be executed on the memory 210. After the sequence of instructions to be executed enters from the bus channel 231, it is cached in the command processor 237, and the command processor 237 selects the core 236 and assigns the instruction sequence to it for execution. The instructions to be executed come from the compiled deep learning model. It should be understood that the sequence of instructions to be executed may include instructions to be executed in the scheduling unit 220 and instructions to be executed that need to be executed in the acceleration unit 230.

[0117] NPU core

[0118] Figure 5 Detailed description of the internal structure of an NPU core according to an embodiment of the present disclosure.

[0119] In one embodiment, Figure 5 As shown, the NPU core 236 includes a tensor engine 310, a pooling operator 320, a convolution operator 330, a relu operator 380, a sequencer 350, an instruction cache 340, an on-chip memory 360, and a constant buffer 370. Among them, the tensor engine 310, the pooling operator 320, the convolution operator 330 and the activation operation operator 380 are all classified as hardware operators. The hardware operator is a hardware module in the acceleration unit that is actually used to perform various operations. Some hardware operators are not shown in the figure.

[0120] The instruction sequence assigned by the command processor 237 to the NPU core 236 first enters the instruction cache 340 cache. Then, the sequencer 350 fetches instructions from the instruction cache 340 in a first-in-first-out order and assigns them to various hardware operators for execution according to the nature of the instructions. The tensor engine 310 is responsible for processing tensor-related operations in the deep learning model. The pooling operator 320 is responsible for processing the pooling operation in the deep learning model. The convolution operator 330 is responsible for the convolution operation in the deep learning model. The activation operation operator 380 is used to perform the operation corresponding to the activation function in the deep learning model. The sequencer 350 decides to assign the instruction to each hardware operator for execution based on the nature of the operation such as convolution, matrix multiplication or pooling.

[0121] The on-chip memory 360 is an in-core memory that stores the weight parameters in the deep learning model and the input and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters (for example, hyperparameters in the deep learning model) in addition to the weight parameters in the deep learning model. As described above, in the process of the scheduling unit 220 pre-configuring the deep learning model in the acceleration unit 230, the scheduling unit 220 can send the location of the parameters in the model in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weights of the nodes and other parameters (such as hyperparameters). For weights, the acceleration unit 230 will take them out from the corresponding position of the memory 210 when needed and put them in the on-chip memory 360. For other parameters, the acceleration unit 230 will take them out from the corresponding position of the memory 210 when needed and put them in the constant buffer 370. In addition, when the instruction to actually start inference is assigned to the core 236 for execution by the command processor 237, the input parameters in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. In addition, after the tensor engine 310 and the pooling operator 320 perform convolution or pooling operations, various intermediate results obtained are also stored in the on-chip memory 360 .

[0122] Software platform for deep learning models

[0123] The improvement of deep learning models requires not only the support of the above hardware layer, but also the continuous improvement of the software layer and algorithm layer. Only by combining the best hardware support at the bottom layer and the best deep learning algorithm structure above can a powerful computing engine be delivered.

[0124] Figure 6 The software platform of an exemplary deep learning model is shown. The software platform is deployed on the terminal device of the R&D personnel, and provides various environments suitable for the deep learning model, including the editing environment, the compiling environment, the quantization operation environment, and the simulation operation environment. The R&D personnel can implement a series of operations such as code editing, code compilation, parameter adjustment, quantization, simulation operation (i.e., running the model on the CPU) of the deep learning model based on the platform.

[0125] As shown in the figure, from top to bottom, the architecture diagram includes an application layer 401, a framework layer 402, and a function layer 403.

[0126] The application layer 401 is the application of deep learning models in specific scenarios, such as vision 405, natural language 406, recommendation 407, etc. These applications are built using this architecture, and can also call the architecture to provide a running interface in the application to obtain reasoning capabilities in the application.

[0127] The framework layer 402 integrates open source platforms such as TensorFlow 408, MXNet 409, and Caffe 410, and provides operator libraries and tools to continue to optimize and improve various algorithms. TensorFlow 408 is a symbolic mathematical system based on data flow programming, which is widely used in the programming implementation of various machine learning algorithms. MXNet 409 is the deep learning library selected by Amazon. Caffe 410, the full name of Convolutional Architecture for Fast Feature Embedding, is a deep learning framework that combines expressiveness, speed, and modular thinking.

[0128] The functional layer 403 includes a compilation stack 403 and a running stack 404. The compilation stack 403 is used to convert (converter) 411, quantize (quantization) 412, optimize (optimization) 413 and compile (compile) 414 various models. Conversion 411 is to provide the internal data of the model to be converted into an intermediate (IR) format. Quantization 412 is to convert the weight parameters in the deep learning model and the input parameters to the deep learning model from high-precision data types to low-precision data types. Optimization 413 includes operations such as fusion of operators inside the model and multi-model optimization linking. Compilation 414 is to optimize the model according to the acceleration unit to generate a binary model that can be recognized by the hardware. The running stack 404 includes a running API 415, an execution manager 416, a user mode driver 417 and a kernel mode driver 418. The execution manager 416 performs resource allocation and batch scheduling. The optimized running API 415 is used to provide various interfaces that can be called at runtime. The user mode driver 417 is used to provide hardware commands and resource scheduling in kernel mode. The kernel mode driver 418 is used to provide task scheduling and hardware control in kernel mode.

[0129] It should be understood that Figure 6 What is shown is a general software platform. There are also some software platforms on the market that are dedicated to building specific types of deep learning models. Of course, the embodiments of the present disclosure do not rely on a specific software platform.

[0130] Tensor data and tensor information referred to in this article

[0131] Conceptually, a tensor is a generalization of a scalar, vector, and matrix. A scalar is data with only size but no direction, that is, a number. A vector is data with both size and direction, that is, a string of numbers. A matrix is ​​data obtained by combining rows and quantities of several vectors, that is, a pile of numbers. A tensor is data arranged in any number of dimensions, that is, multiple piles of numbers. In a broad sense, tensors include scalars, vectors, and matrices, that is, scalar data is a zero-dimensional tensor, vectors are one-dimensional tensors, and matrices are two-dimensional tensors. Tensors can be represented by two indicators: shape and type. The shape of a tensor defines the number of values ​​in each dimension, and the type of a tensor defines the data type in each dimension.

[0132] The following is a specific example to further illustrate the type and shape of tensors.

[0133] 1. Value 1 is a 0-dimensional tensor with a shape of [] and a type of 8-bit unsigned integer;

[0134] 2. [1,2,3] is a 1-dimensional tensor with a shape of [3], indicating that the tensor has 3 values ​​in the first dimension and the type is an 8-bit unsigned integer;

[0135] 3. [[1,2],[3,4]] is a 2D tensor with a shape of [2,2], which means that the tensor has two data in the first dimension and two data in the second dimension, and the type is an 8-bit unsigned integer;

[0136] 4. [[[1,2],[3,4],[5,6]],[[1,2],[3,4],[5,6]]] is a three-dimensional vector with a shape of [2,3,2], indicating that the tensor has two data in the first dimension, three data in the second dimension, and two data in the third dimension, and its type is an 8-bit unsigned integer.

[0137] The data types of tensors include but are not limited to the following data types: 32-bit floating point numbers, 64-bit floating point numbers, 64-bit signed integers, 32-bit signed integers, 16-bit signed integers, 8-bit signed integers, 8-bit unsigned integers, variable-length byte arrays, Boolean types, complex numbers consisting of two 32-bit floating point numbers, 32-bit signed integers, 8-bit signed integers, and 8-bit unsigned integers.

[0138] For the convenience of explanation, in this article, data related to tensors are referred to as tensor data. Tensor data can be either a single tensor or a sequence of tensors. The extended name terms are: virtual tensor data, input tensor data, output tensor data. The shape and / or type of a tensor is referred to as tensor information. The extended name terms are: input tensor information, output tensor information. When only tensor information is mentioned, it includes input tensor information and output tensor information. Input tensor data and output tensor data can be either input tensor data and output tensor data of a deep learning model, or input tensor data and output tensor data of each node or each subgraph. Input tensor information and output tensor information can be either input tensor information and output tensor information of a deep learning model, or input tensor information and output tensor information of each node or each subgraph.

[0139] Existing technology tensor information determination strategy

[0140] Generally speaking, the compiler will use the tensor information of the deep learning model to perform some optimization operations on the deep learning model during the compilation process to accelerate the execution of the model structure on the acceleration unit. In the prior art, the compiler can obtain the tensor information of the deep learning model through the following ways: first, the deep learning model will provide the tensor information of some or all nodes; second, the tensor information of some or all nodes is obtained through the functions provided by the deep learning model; third, through Figure 6The framework function provided by the software platform shown in the figure obtains the tensor information of some or all nodes of the deep learning model; the fourth type is that if the node is a standard node, such as conv, the tensor information of the node is obtained based on the node specification. Similarly, the acceleration unit also performs some optimization operations on the deep learning model according to the tensor information of the deep learning model during execution. However, the above method cannot ensure that the compiler can obtain the tensor information of each node, especially for deep learning models with large and complex model structures.

[0141] The determination strategy of the disclosed tensor information

[0142] Figure 7 1 is a flow chart of a method for determining tensor information of a deep learning model according to an embodiment of the present disclosure. The method is applied at the algorithm level, and more specifically, the method is executed by the scheduling unit 220 before compiling and optimizing the deep learning model. The method specifically includes the following steps.

[0143] Step S701, for a deep learning model that does not provide complete tensor information, determine at least one processable subgraph. Deep learning models that do not provide complete tensor information refer to those deep learning models that do not provide tensor information for all nodes. A subgraph is a group of nodes obtained by combining multiple consecutive nodes of a deep learning model. A processable subgraph refers to a subgraph that can be processed by a specified acceleration unit. The deep learning model can be regarded as a large static full graph, and then for a specified acceleration unit, at least one processable subgraph is obtained on this static full graph.

[0144] Step S702, execute the deep learning model with the virtual tensor data as the input of the deep learning model, and in the process, record the input tensor data and output tensor data of each processable subgraph. This step first obtains the input tensor information of the deep learning model, and then constructs a virtual tensor data according to the input tensor information of the deep learning model, so that the tensor information of the virtual tensor data is consistent with the input tensor information of the deep learning model, and executes the deep learning model on the scheduling unit with the virtual tensor data as the input of the deep learning model, and records the input tensor data and output tensor data of each processable subgraph.

[0145] Step S703, calculate the tensor information of each node of each processable subgraph according to the input tensor data and output tensor data of each processable subgraph. When the input tensor data of each processable subgraph is known, the input tensor information of each processable subgraph can be obtained, and the input tensor data of each processable subgraph is also the input tensor data of the input boundary node of the processable subgraph, so the input tensor information of the input boundary node of each processable subgraph is the same as the input tensor information of the processable subgraph, thereby obtaining the input tensor information of the input boundary node of each processable subgraph, and then judging the output tensor information of the input boundary node according to the specific function of the input boundary data and the input tensor information, repeating the above steps to judge the input tensor information and output tensor information of each subsequent node, and so on, until the output boundary node. Finally, the input and output tensor information of each node is obtained.

[0146] In this embodiment, the deep learning model is executed on the scheduling unit using virtual tensor data as the input of the deep learning model, and the input tensor data and output tensor data of each processable subgraph are recorded. These input tensor data and output tensor data are subsequently used to infer the tensor information of each node in each processable subgraph, so as to achieve the purpose of determining the tensor information of the deep learning model as soon as possible. This method can be applied to deep learning models with arbitrary model structures, and can be run on general-purpose devices such as processors, and is particularly suitable for obtaining tensor information that cannot be obtained through software platforms or tensor information of user-defined nodes.

[0147] Figure 8 It is a flowchart of a method for determining tensor information of a deep learning model according to another embodiment of the present disclosure, which specifically includes the following steps.

[0148] Step S801, for a deep learning model that does not provide complete tensor information, the deep learning model is divided into multiple subgraphs using nodes that cannot be processed by the specified acceleration unit as separators. This step first determines the nodes in the deep learning model that cannot be processed by the specified acceleration unit, uses the nodes that cannot be processed by the specified acceleration unit as separators, and uses the continuous nodes between the two separators as a subgraph, thereby obtaining multiple subgraphs.

[0149] Step S802, among the multiple subgraphs, some subgraphs are screened out, and the remaining subgraphs are used as processable subgraphs. This step screens out some subgraphs from the multiple subgraphs obtained in step S801. The screening method can be selected arbitrarily. For example, subgraphs whose tensor information of each node can be obtained through the software platform of the deep learning model are screened out. For another example, subgraphs whose calculation optimization effect is not outstanding and do not need to be executed on the specified acceleration unit are screened out. Since they do not need to be executed on the specified acceleration unit, there is naturally no need to know the tensor information of each node. As a specific implementation method, for example, according to the compiler feedback information, subgraphs whose optimization effect is less than the set threshold can be found, and these subgraphs can be screened out in this step as subgraphs that cannot be optimized subsequently.

[0150] Step S803, execute the deep learning model with the virtual tensor data as the input of the deep learning model, and in this process, record the input tensor data and output tensor data of each processable subgraph. This step first obtains the input tensor information of the deep learning model, and then arbitrarily constructs a virtual tensor data according to the input tensor information of the deep learning model. The virtual tensor data can be a single tensor or a tensor sequence. The deep learning model is executed on the scheduling unit with the virtual tensor data as the input of the deep learning model, and the input tensor data and output tensor data of each processable subgraph are recorded.

[0151] Step S804, determining the input tensor data and output tensor data of the input boundary node of each processable subgraph of the deep learning model according to the input tensor data and output tensor data of each processable subgraph. The input tensor data of each processable subgraph is the input data of the input boundary node of the subgraph, and the output tensor data of each processable subgraph is the output tensor data of the output boundary node of the subgraph, thereby obtaining the input tensor data of the input boundary node of each processable subgraph and the output tensor data of the output boundary node.

[0152] Step S805, determine whether all processable sub-graphs have been processed. If all sub-graphs have been processed, the loop ends, otherwise, the next sub-graph is processed.

[0153] Step S806, for each processable subgraph, based on the specific functions of each node and the input tensor data of the input boundary nodes of the processable subgraph, starting from the input boundary node to the output boundary node, the tensor information of each node is deduced one by one. Specifically, the input tensor data of the processable subgraph is the input tensor data of the input boundary nodes of the processable subgraph, thereby obtaining the input tensor information of the input boundary nodes, and then judging the output tensor information of the input boundary nodes according to the specific functions of the input boundary nodes and the input tensor information of the input boundary nodes, repeating the above steps to obtain the subsequent input tensor information and output tensor information, and so on, until the input tensor information and output tensor information of the output boundary nodes are deduced. Finally, the input and output tensor information of each node is obtained.

[0154] Step S807, determine whether the tensor information of the output boundary node of the processable subgraph is consistent with the output data of the output boundary node of the processable subgraph. This step first obtains the output tensor information based on the output tensor data of the processable subgraph, and then checks whether it is consistent with the tensor information of the output boundary node obtained by the above-mentioned inference step. If they are consistent, it means that the above-mentioned inference process is correct. If they are inconsistent, it means that the above-mentioned inference process is wrong. In the case that the inference process is wrong, step S808 can be optionally executed. If the inference process is correct, jump to step S805.

[0155] Step S808, reverse calculation is performed based on the output data of the output boundary nodes of the processable subgraph to correct the tensor information of each node calculated by the above steps. This step is to perform reverse calculation based on the specific function of the output boundary node and the output tensor data of the output boundary node, and correct the tensor information of each node calculated by the above steps according to the calculation result.

[0156] This embodiment uses nodes that cannot be processed by the designated acceleration unit as separators to divide the deep learning model into multiple sub-graphs, then screens out unimportant sub-graphs and only infers the tensor information of each node for the remaining sub-graphs, thereby reducing the number of sub-graphs that need to be calculated. This can reduce the input and output tensor data that needs to be stored in the subsequent calculation, thereby reducing memory usage, and speeding up the acquisition of tensor information of key nodes of the deep learning model.

[0157] Fig. 9 Flow chart of the method for determining tensor information of a deep learning model of the third embodiment of the present disclosure. The method includes the following steps S901-S909. Among them, steps S901-S905 are the same as the above steps S801 to S805, and are not repeated here. Steps S906-S909 are described in detail below.

[0158] Step S906, determine whether the current processable subgraph contains unrecognizable nodes. If yes, execute step S907, if not, execute the above steps S906-S908. An unrecognizable node is a node whose specific function and tensor information cannot be known through the existing technology. An unrecognizable node is generally a user-defined node, and the tensor information of the node cannot be obtained through other means such as software platforms.

[0159] In the case where the current processable subgraph contains an unrecognizable node, step 907 to step S909 first calculates the input tensor information and output tensor information of each node one by one from the input boundary node of the current processable subgraph to the node before the unrecognizable node. Specifically, according to the input tensor data of the processable subgraph, that is, the input tensor data of the input boundary node of the processable subgraph, the input tensor information of the input boundary node is determined, and then the output tensor information of the input boundary node is determined according to the specific function of the input boundary node and the input tensor information of the input boundary node. Repeat the above steps to determine the input tensor information and output tensor information of each subsequent node, and so on, until the node before the unrecognizable node. Finally, the input and output tensor information of each node from the input boundary node to the node before the unrecognizable node is obtained. Then, from the output boundary node to the node after the unrecognizable node, the tensor information of each node is reversely calculated one by one. Specifically, according to the output tensor data of the processable subgraph, that is, the output tensor data of the output boundary node of the processable subgraph, the output tensor information of the output boundary node is determined, and then the input tensor information of the output boundary node is determined according to the specific function of the output boundary node and the output tensor information of the output boundary node, and the input tensor information of each previous node is determined repeatedly according to the above steps, and so on until the node after the unrecognizable node. Finally, according to the output tensor information of the node before the unrecognizable node and the input tensor information of the node after it.

[0160] This embodiment uses unrecognizable nodes as separators to divide the processable subgraph into multiple computable subgraphs, and then respectively computes the tensor information of each node of the computable subgraph, and then obtains the tensor information of the unrecognizable node based on the two nodes adjacent to the unrecognizable node, thereby achieving the purpose of determining the tensor information of each node.

[0161] Below Fig.10 and 11 The figures further illustrate the embodiments of the present disclosure.

[0162] like Fig.10 As shown in the figure above, the deep learning model 10 in the left figure includes optimizable sub- Fig.11 ,son Fig.11The deep learning model 10 is executed at least once on the scheduling unit using a tensor data TensorData as the input of the deep learning model, and then the sub Fig.11 In this example, since TensorData is also a child Fig.11 The input tensor data of Fig.11 The input tensor data is recorded as TensorData, and the input tensor data is recorded as OutputTensor. Fig.11 In terms of TensorData, it is also a sub Fig.11 The input tensor data of the input boundary node Conv, according to TensorData can obtain the tensor information of TensorData, then the child Fig.11 The input tensor information of the input boundary node Conv is determined, and then the output tensor information of conv is inferred according to the specific function of the node conv. The output tensor data of conv is the input tensor data of BatchNorm, so the input tensor information of BatchNorm can be determined, and then the output tensor information of BatchNorm can be inferred according to the specific function of BatchNorm, and so on, until the output tensor information of the last node FC is inferred. In general, the output tensor information is the same as the sub- Fig.11 The tensor information of the output tensor data OutputTensor is consistent. If the two are inconsistent, it means that the previous calculation is wrong. You can reversely deduce the sub- Fig.11 The tensor information of each node in the forward inference is finally corrected according to the tensor information in the reverse inference. If the above steps still cannot determine the tensor information of each node, the deep learning model can be further divided into smaller subgraphs to determine the tensor information of each node.

[0163] Fig.11Three optimizable subgraphs 12-13 are divided in it. Nodes outside the subgraph, such as ToFloat (convert to floating point data) and Tensor Array Stack (tensor data stack), are nodes that cannot be executed on the acceleration unit, so they are not placed in the optimizable subgraph. After dividing the subgraph, the virtual tensor data TensorData is used as the input execution model of the deep learning model 1000, and the input tensor data and output tensor data of each subgraph 12-14 are recorded during the execution process. The figure shows the recorded input tensor data InputData and output tensor data OutputData of subgraph 12. The following takes subgraph 12 as an example to illustrate the inference process. InputData is the input tensor data of sub in subgraph 12. The input tensor information of sub is obtained according to InpuData. The function of sub is tensor subtraction, and then the output tensor information of sub is obtained. For example, the input tensor information obtained from InpuData is of shape [2,3,2] and of type unsigned integer. Then the output tensor information of sub is also: of shape [2,3,2] and of type unsigned integer. Then the input tensor information and output tensor information of conv are deduced, and so on, until the input tensor information and output tensor information of the output boundary node Flatter are obtained. Since outputData is the output tensor data of the output boundary node Flatter, the data is compared with the output tensor information of Flatter deduced above to determine whether the above deduction process is correct. If it is incorrect, the tensor information of each node in subgraph 12 can be reversely deduced based on outputData, and finally the tensor information of each node deduced according to the forward deduction is corrected according to the reverse deduced tensor information. If the above steps still cannot determine the tensor information of each node, you can continue to divide the deep learning model into smaller subgraphs to determine the tensor information of each node.

[0164] It should be understood that, generally, we can determine the tensor information of most nodes through various methods provided by the prior art, and for the remaining nodes, we can derive the tensor information of each node in each subgraph through the embodiments of the present disclosure.

[0165] Optionally, the tensor information of each node finally obtained can be saved in the deep learning model. For example, the tensor information of each node is saved in the deep learning model as the attribute data of each node. When the deep learning model is in the compilation stage, the compiler uses the tensor information of each node to perform compilation-time tuning on the deep learning model, for example, the order of instructions can be adjusted. The compiler compiles the source code of the deep learning model to obtain an executable instruction sequence. These tensor information can be saved in the executable instruction sequence. When the designated acceleration unit reads the executable instruction sequence containing these tensor information from the memory, the command processor of the designated acceleration unit can refer to the tensor information to distribute the executable instructions to different NPU cores, and the NPU core can also refer to these tensor information to distribute the executable instructions to different hardware operators.

[0166] The method for determining tensor information of a tensor deep learning model provided by the embodiments of the present disclosure can be executed on any computing device with general processing capabilities, and therefore has a certain degree of versatility. The implementation of this method depends only on the hardware architecture of the computing device, and has no association with the hardware architecture of the execution model. The model processed on the computing device can be used for various hardware devices that execute the model, for example, for data centers, for artificial intelligence (AI) acceleration units, for graphics processing units (GPUs), for Internet of Things devices that can execute deep learning models, for embedded devices, and so on. The model processed on the computing device can perform many tasks including natural language processing, speech recognition, biometric recognition, target detection, network defense, knowledge work assistance, and so on, and has currently achieved good application results in many fields.

[0167] Therefore, compared with the prior art, the embodiments of the present disclosure not only have practical value, but also have a wide range of application scenarios.

[0168] The commercial value of the disclosed embodiments

[0169] As we all know, deep learning models currently have a wide range of successful application scenarios, and each application scenario has strong commercial value. Taking the field of face recognition as an example, by collecting video surveillance through cameras, recognizing facial images through neural network models, and comparing facial images with faces stored in the cloud, criminals in surveillance videos can be identified. In the field of speech recognition, speech recognition is performed through neural network models to achieve simultaneous interpretation. These application scenarios can bring huge economic benefits. However, this also requires R&D personnel to be able to respond quickly to various application needs and solve problems. The purpose of the disclosed embodiment is to help R&D personnel quickly determine the tensor information of each node, especially the key nodes, without the need or inability to understand the detailed content of each node of the deep learning model, and then perform subsequent work such as optimization and compilation based on the tensor information.

[0170] Those skilled in the art will appreciate that the present disclosure may be implemented as systems, methods, and computer program products. Therefore, the present disclosure may be specifically implemented in the following forms, namely complete hardware, complete software (including firmware, resident software, microcode), and may also be implemented in the form of a combination of software and hardware. In addition, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0171] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media include: an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by a processing unit, a device or a device, or used in combination with the program.

[0172] Computer readable signal media may include data signals propagated in baseband or as part of a chopped wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any other suitable combination. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction system, apparatus, or device.

[0173] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., and any suitable combination of the foregoing.

[0174] The computer program code for executing the disclosed embodiments may be written in one or more programming languages ​​or combinations. The programming languages ​​include object-oriented programming languages, such as JAVA, C++, and may also include conventional procedural programming languages, such as C. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0175] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A computing device comprising a memory and a scheduling unit, wherein the memory stores instructions and the scheduling unit reads the instructions to execute: For a deep learning model that does not provide complete tensor information, determine at least one processable subgraph, where the processable subgraph consists of a plurality of consecutive nodes of the deep learning model and can be processed by a designated acceleration unit; Executing the deep learning model using virtual tensor data as input of the deep learning model, and in the process, recording input tensor data and output tensor data of the processable subgraph, wherein the virtual tensor data is constructed according to input tensor information of the deep learning model; Inferring tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph; Among them, the designated acceleration unit is a unit designed to accelerate the data processing speed of the deep learning model. When executing the deep learning model, the designated acceleration unit uses the tensor information of each node to identify the executable instructions of the deep learning model and distributes them to the corresponding hardware operators. The deep learning model is used to perform speech recognition tasks.

2. The computing device according to claim 1, wherein: The determining of at least one processable subgraph comprises: Dividing the deep learning model into a plurality of subgraphs using nodes that cannot be processed by the designated acceleration unit as separators; Among the multiple sub-graphs, some sub-graphs are screened out, and the remaining sub-graphs are used as at least one processable sub-graph.

3. The computing device according to claim 2, wherein: The screening out of some subgraphs comprises: Screening out subgraphs for which tensor information of each node can be obtained via the software platform of the deep learning model; and / or The subgraphs that cannot be optimized are screened out, where the subgraphs that cannot be optimized are subgraphs whose calculation optimization effect on the specified acceleration unit is less than a set threshold.

4. The computing device according to claim 1, wherein: The step of calculating the tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph comprises: Determining input tensor data of input boundary nodes of the processable subgraph according to the input tensor data of the processable subgraph; For the processable subgraph, based on the specific function of each node and the input tensor data of the input boundary nodes of the processable subgraph, tensor information of each node is calculated one by one starting from the input boundary node to the output boundary node.

5. The computing device according to claim 4, wherein: The step of calculating the tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph further includes: Compare whether the output tensor information of the output boundary nodes of the processable subgraph is consistent with the output tensor data of the processable subgraph to determine whether the inference process is correct.

6. The computing device according to claim 4, wherein: The step of calculating the tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph further includes: When the output tensor information of the output boundary nodes of the processable subgraph does not match the output tensor data of the processable subgraph, reverse calculation is performed based on the output tensor data of the processable subgraph to correct the tensor information of each node of the processable subgraph.

7. The computing device according to claim 1, wherein: The step of calculating the tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph comprises: If the processable subgraph contains unrecognizable nodes, perform the following steps: Determining input tensor data of input boundary nodes and output tensor data of output boundary nodes of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph; Starting from the input boundary node of the processable subgraph to the node before the unrecognizable node, inferring the tensor information of each node one by one; Starting from the output boundary node to the node after the unrecognizable node, reversely infer the tensor information of each node one by one; The tensor information of the unrecognizable node is determined according to the tensor information of the node before the unrecognizable node and the tensor information of the node after the unrecognizable node.

8. The computing device according to claim 1, wherein: It also includes: using the tensor information of each node to perform compile-time tuning on the deep learning model.

9. The computing device of claim 1, wherein: The tensor information of each node of the processable subgraph is stored as attribute data of each node of the processable subgraph.

10. The computing device according to any one of claims 1 to 9, wherein: The tensor information of each node includes: the shape and type of the input tensor data and output tensor data of each node.

11. A method for determining tensor information of a deep learning model, comprising: For the deep learning model that does not provide complete tensor information, determine at least one processable subgraph, where the processable subgraph consists of multiple consecutive nodes of the deep learning model and can be processed by a designated acceleration unit; Executing the deep learning model using virtual tensor data as input of the deep learning model, and in the process, recording input tensor data and output tensor data of the processable subgraph, wherein the virtual tensor data is constructed according to input tensor information of the deep learning model; Inferring tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph; Among them, the designated acceleration unit is a unit designed to accelerate the data processing speed of the deep learning model. When executing the deep learning model, the designated acceleration unit uses the tensor information of each node to identify the executable instructions of the deep learning model and distributes them to the corresponding hardware operators. The deep learning model is used to perform speech recognition tasks.

12. The method according to claim 11, wherein: The determining of at least one processable subgraph comprises: Dividing the deep learning model into a plurality of subgraphs using nodes that cannot be processed by the designated acceleration unit as separators; Among the multiple sub-graphs, some sub-graphs are screened out, and the remaining sub-graphs are used as at least one processable sub-graph.

13. The method according to claim 12, wherein: The screening out of some subgraphs comprises: Screening out subgraphs for which tensor information of each node can be obtained via the software platform of the deep learning model; and / or The subgraphs that cannot be optimized are screened out, where the subgraphs that cannot be optimized are subgraphs whose calculation optimization effect on the specified acceleration unit is less than a set threshold.

14. The method according to claim 11, wherein: The step of calculating the tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph comprises: Determining input tensor data of input boundary nodes of the processable subgraph according to the input tensor data of the processable subgraph; For the processable subgraph, based on the specific function of each node and the input tensor data of the input boundary nodes of the processable subgraph, tensor information of each node is calculated one by one starting from the input boundary node to the output boundary node.

15. The method according to claim 14, wherein: The step of calculating the tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph further includes: Compare whether the output tensor information of the output boundary nodes of the processable subgraph is consistent with the output tensor data of the processable subgraph to determine whether the inference process is correct.

16. The method according to claim 14, wherein: The step of calculating the tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph further includes: When the output tensor information of the output boundary nodes of the processable subgraph does not match the output tensor data of the processable subgraph, reverse calculation is performed based on the output tensor data of the processable subgraph to correct the tensor information of each node of the processable subgraph.

17. The method according to claim 11, wherein: The step of calculating the tensor information of each node of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph comprises: If the processable subgraph contains unrecognizable nodes, perform the following steps: Determining input tensor data of input boundary nodes and output tensor data of output boundary nodes of the processable subgraph according to the input tensor data and the output tensor data of the processable subgraph; Starting from the input boundary node of the processable subgraph to the node before the unrecognizable node, inferring the tensor information of each node one by one; Starting from the output boundary node to the node after the unrecognizable node, reversely infer the tensor information of each node one by one; The tensor information of the unrecognizable node is determined according to the tensor information of the node before the unrecognizable node and the tensor information of the node after the unrecognizable node.

18. The method according to claim 11, wherein: It also includes: using the tensor information of each node to perform compile-time tuning on the deep learning model.

19. The method according to claim 11, wherein: The tensor information of each node of the processable subgraph is stored as attribute data of each node of the processable subgraph.

20. The method according to any one of claims 11 to 19, wherein: The tensor information of each node includes: the shape and type of the input tensor and output tensor of each node.

21. A data center, comprising the computing device according to any one of claims 1 to 10.

Images