Processing unit, acceleration unit, related device and method

Abstract

The present disclosure provides a processing unit, an acceleration unit, and related devices and methods. The processing unit includes: an instruction fetch unit; an instruction decoding unit; an instruction execution unit, which is used to implement: generating acceleration unit model instructions and execution plans based on multiple decoded deep network model instructions and compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores; obtaining the acceleration unit model instructions and execution plans for decoded inference commands, obtaining the load status of the physical core in the acceleration unit, mapping the virtual core to the physical core according to the load status, and loading the acceleration unit model instructions to the physical core according to the execution plan and the mapping. The embodiments of the present disclosure improve the efficiency of running multiple deep neural network models in application scenarios where multiple deep neural network models need to be run, and save bandwidth.

Description

Technical Field

[0001] The present disclosure relates to the field of chips, and more specifically, to a processing unit, an acceleration unit, and related devices and methods. Background Art

[0002] In the application of deep neural networks, some application scenarios are more complex and require the deployment of multiple different types of deep neural network models. Sometimes the same deep neural network model is used to train different data types to generate multiple deep neural network models for different purposes. These models have different weights and can be regarded as different deep neural network models in computation. For such applications that require running many deep neural network models to complete a task, the prior art requires multiple deep neural network models to be executed in turn. That is, after executing one model, another model is loaded. When the number of batches executed is small, the rotation between deep neural network models in this method leads to a significant increase in bandwidth requirements. Summary of the invention

[0003] In view of this, the present disclosure aims to improve the efficiency of running multiple deep neural network models in application scenarios where multiple deep neural network models need to be run, and save bandwidth.

[0004] According to one aspect of the present disclosure, there is provided a processing unit, comprising:

[0005] an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit;

[0006] an instruction decoding unit for decoding the retrieved computer instructions;

[0007] An instruction execution unit is used to execute the decoded computer instructions to achieve:

[0008] Generate an acceleration unit model instruction and an execution plan based on the decoded multiple deep network model instructions and the compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores;

[0009] Obtain the acceleration unit model instructions and execution plan for the decoded inference command, obtain the load status of the physical core in the acceleration unit, map the virtual core to the physical core according to the load status, and load the acceleration unit model instructions to the physical core according to the execution plan and the mapping.

[0010] Optionally, the compilation parameter instruction includes at least the number of cores to be used and the dependency relationship of the multiple deep network models; the generation of acceleration unit model instructions and an execution plan based on the multiple deep network model instructions and the compilation parameter instructions includes: generating acceleration unit model instructions corresponding to each deep network model instruction; determining the execution batches of the multiple deep network models according to the dependency relationship and the number of cores to be used; allocating the deep network models of the same batch to the multiple virtual cores according to the storage occupancy and computational amount of the deep network models of the same batch; generating the execution plan according to the determined execution batches of the multiple deep network models and the allocation of the deep network models of the same batch to the multiple virtual cores.

[0011] Optionally, allocating the same batch of deep network models to the multiple virtual cores according to the storage occupancy and computing amount of the same batch of deep network models includes:

[0012] Determine the weight storage occupancy of the deep network model according to the deep network model instruction;

[0013] According to the deep network model instructions, determine the tensor storage occupancy of the input tensor, intermediate tensor, and output tensor of the deep network model;

[0014] Determine the computational amount of the deep network model according to the deep network model instruction;

[0015] The deep network models of the same batch are allocated to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network models and the computational complexity of the deep network models.

[0016] Optionally, the deep network models of the same batch are distributed to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network models, and the computational amount of the deep network models, in a manner such that the sum is balanced on the allocated virtual cores and the computational amount is balanced on the allocated virtual cores.

[0017] Optionally, the generation of acceleration unit model instructions and execution plans based on the decoded multiple deep network model instructions and compilation parameter instructions is executed in response to receiving the decoded multiple deep network model instructions and compilation parameter instructions, and the generated acceleration unit model instructions and execution plans are stored in a memory outside the processing unit; the acceleration unit model instructions and execution plans for the decoded inference commands are obtained from the memory in response to receiving the decoded inference commands.

[0018] Optionally, the generation of the acceleration unit model instructions and the execution plan based on the decoded multiple deep network model instructions and the compilation parameter instructions is performed in response to simultaneously receiving the decoded multiple deep network model instructions, the compilation parameter instructions and the inference commands.

[0019] Optionally, if M virtual cores are allocated to the acceleration unit model instructions in the execution plan, the mapping of the virtual cores to physical cores according to the load conditions includes: selecting M physical cores with the smallest load conditions as the physical cores to which the acceleration unit model instructions are loaded.

[0020] Optionally, after loading the acceleration unit model instructions into the physical core according to the execution plan and the mapping, the instruction execution unit executes the decoded computer instructions to achieve:

[0021] Loading input tensors used by the inference command into physical cores according to the execution plan and the mapping;

[0022] A calculation result calculated by the physical core according to the input tensor and the weight in the acceleration unit model instruction is received from the physical core.

[0023] According to one aspect of the present disclosure, there is provided a processing unit, comprising:

[0024] an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit;

[0025] an instruction decoding unit for decoding the retrieved computer instructions;

[0026] An instruction execution unit is used to execute the decoded computer instructions to generate acceleration unit model instructions and an execution plan based on the decoded multiple deep network model instructions and compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores.

[0027] Optionally, the compilation parameter instruction includes at least the number of cores to be used and the dependency relationship of the multiple deep network models; the generation of acceleration unit model instructions and an execution plan based on the multiple deep network model instructions and the compilation parameter instructions includes: generating acceleration unit model instructions corresponding to each deep network model instruction; determining the execution batches of the multiple deep network models according to the dependency relationship and the number of cores to be used; allocating the deep network models of the same batch to the multiple virtual cores according to the storage occupancy and computational amount of the deep network models of the same batch; generating the execution plan according to the determined execution batches of the multiple deep network models and the allocation of the deep network models of the same batch to the multiple virtual cores.

[0028] Optionally, allocating the same batch of deep network models to the multiple virtual cores according to the storage occupancy and computing amount of the same batch of deep network models includes:

[0029] Determine the weight storage occupancy of the deep network model according to the deep network model instruction;

[0030] According to the deep network model instructions, determine the tensor storage occupancy of the input tensor, intermediate tensor, and output tensor of the deep network model;

[0031] Determine the computational amount of the deep network model according to the deep network model instruction;

[0032] The deep network models of the same batch are allocated to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network models and the computational complexity of the deep network models.

[0033] Optionally, the deep network models of the same batch are distributed to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network models, and the computational amount of the deep network models, in a manner such that the sum is balanced on the allocated virtual cores and the computational amount is balanced on the allocated virtual cores.

[0034] According to one aspect of the present disclosure, there is provided an acceleration unit, comprising:

[0035] Multiple physical cores; and

[0036] A command processor, configured to, in response to receiving an inference command, execute:

[0037] Acquire an acceleration unit model instruction and an execution plan for the inference command, wherein the execution plan reflects the execution order of the acceleration unit model instruction and the allocation of the acceleration unit model instruction to multiple virtual cores;

[0038] Obtaining the load status of the physical core in the acceleration unit;

[0039] Mapping the virtual core to the physical core according to the load condition;

[0040] The acceleration unit model instructions are loaded into the physical core according to the execution plan and the mapping.

[0041] Optionally, loading the acceleration unit model instruction into the physical core according to the execution plan and the mapping includes:

[0042] Determining the virtual cores allocated to the acceleration unit model instructions according to the execution plan;

[0043] The acceleration unit model instruction is loaded into the physical core to which the determined virtual core is mapped.

[0044] Optionally, if M virtual cores are allocated to the acceleration unit model instructions in the execution plan, the mapping of the virtual cores to physical cores according to the load conditions includes: selecting M physical cores with the smallest load conditions as the physical cores to which the acceleration unit model instructions are loaded.

[0045] Optionally, after loading the acceleration unit model instructions into the physical core according to the execution plan and the mapping, the command processor is further configured to:

[0046] Loading input tensors used by the inference command into physical cores according to the execution plan and the mapping;

[0047] A calculation result calculated by the physical core according to the input tensor and the weight in the acceleration unit model instruction is received from the physical core.

[0048] According to one aspect of the present disclosure, there is provided a processing method, comprising:

[0049] Generate an acceleration unit model instruction and an execution plan based on multiple deep network model instructions and compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores;

[0050] Obtain acceleration unit model instructions and an execution plan for inference commands, obtain the load status of the physical core in the acceleration unit, map the virtual core to the physical core according to the load status, and load the acceleration unit model instructions to the physical core according to the execution plan and the mapping.

[0051] According to one aspect of the present disclosure, there is provided a processing method, comprising:

[0052] Receive multiple deep network model instructions and compilation parameters;

[0053] Based on the multiple deep network model instructions and compilation parameter instructions, an acceleration unit model instruction and an execution plan are generated, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores.

[0054] According to one aspect of the present disclosure, a method for loading an acceleration unit model instruction in an acceleration unit is provided, comprising:

[0055] Obtaining acceleration unit model instructions and an execution plan for inference commands, wherein the execution plan reflects the execution order of the acceleration unit model instructions and the allocation of the acceleration unit model instructions to multiple virtual cores;

[0056] Obtaining the load status of the physical core in the acceleration unit;

[0057] Mapping the virtual core to the physical core in the acceleration unit according to the load condition;

[0058] The acceleration unit model instructions are loaded into the physical core according to the execution plan and the mapping.

[0059] According to one aspect of the present disclosure, there is provided a computing device, comprising:

[0060] A processing unit as described above;

[0061] A memory, used to store the acceleration unit model instructions and execution plan generated by the processing unit;

[0062] The acceleration unit.

[0063] According to one aspect of the present disclosure, there is provided a computing device, comprising:

[0064] A processing unit as described above;

[0065] An acceleration unit as described above.

[0066] The memory is used to store the acceleration unit model instructions and execution plan generated by the processing unit.

[0067] According to one aspect of the present disclosure, a system on chip is provided, comprising the processing unit as described above.

[0068] According to one aspect of the present disclosure, a system on chip is provided, comprising the acceleration unit as described above.

[0069] According to one aspect of the present disclosure, a data center is provided, comprising the computing device as described above.

[0070] In the disclosed embodiment, in an application scenario where multiple deep neural network models need to be run, acceleration unit model instructions and an execution plan are generated based on multiple deep network model instructions and compilation parameter instructions. The execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores. Afterwards, the virtual core is mapped to the physical core based on the load condition of the physical core in the acceleration unit. In this way, the acceleration unit model instructions in the execution plan with the same execution order in the same batch can be run in parallel on multiple physical cores at the same time without switching models, which greatly improves the efficiency of running multiple deep neural network models and saves bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The above and other objects, features and advantages of the present disclosure will become more apparent through the description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0072] Figure 1 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0073] Figure 2 is an internal structure diagram of a server in a data center according to an embodiment of the present disclosure;

[0074] Figure 3 is an internal structure diagram of a processing unit and an acceleration unit in a server according to an embodiment of the present disclosure;

[0075] Figure 4 is an internal structure diagram of an acceleration unit physical core according to an embodiment of the present disclosure;

[0076] Figure 5 is a schematic diagram of allocation of multiple acceleration unit model instructions on multiple physical cores according to an embodiment of the present disclosure;

[0077] Fig. 6A It is a schematic diagram of an implementation process according to an embodiment of the present disclosure.

[0078] Figure 6B A schematic diagram of a process of mapping acceleration unit model instructions to virtual cores or even physical cores according to an embodiment of the present disclosure is shown.

[0079] Figure 7 is a flowchart of a processing method executed by a processing unit according to an embodiment of the present disclosure.

[0080] Figure 8 is a flowchart of a processing method executed by a processing unit according to another embodiment of the present disclosure.

[0081] Fig. 9 yes Figure 8 A flowchart of a method for executing a load acceleration unit model instruction corresponding to an acceleration unit in an embodiment. DETAILED DESCRIPTION

[0082] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0083] The following terms are used in this article.

[0084] Acceleration unit: A processing unit designed to improve the data processing speed in some special-purpose fields (for example, processing images, processing various operations of deep network models, etc.) where the efficiency of traditional processing units is low. The acceleration unit is also called an artificial intelligence (AI) processing unit, including a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and dedicated intelligent acceleration hardware (for example, a neural network processor NPU). The present disclosure is applicable to NPU scenarios, but the embodiments of the present disclosure use a general compiled custom interface, and the acceleration unit hardware implementation can use the above-mentioned CPU, GPU, GPGPU, etc.

[0085] Processing unit: A unit that performs traditional processing (not for complex operations such as image processing and various deep network models) in the server of the data center. In addition to performing these non-image processing and various deep network models and other complex operations, the processing unit also assumes the function of scheduling the acceleration unit and assigning the acceleration unit the tasks it needs to undertake. The processing unit can take the form of a processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and so on.

[0086] Deep network model: Deep network model is a new research direction in the field of machine learning (ML). It is introduced into machine learning to make it closer to its original goal - artificial intelligence. The deep network model learns the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images and sounds. The format of deep network models varies depending on the model framework they rely on, and can be divided into different types of formats, such as tensorflow, pytorch, mxnet, etc.

[0087] Deep network model instructions: instructions written for executing the above-mentioned deep network model, which are coded representations of the deep network model, are generally input into the processing unit by an administrator user through a model instruction input interface. The model instruction input interface is, for example, a human-computer interaction interface, in which instructions for executing the deep network model can be input.

[0088] Compilation parameters: When generating acceleration unit model instructions and execution plans, other parameters are required in addition to deep network model instructions. For example, when generating an execution plan, you need to know the dependencies between the deep network models. If the execution of a deep network model depends on the execution result of another deep network model, in the execution plan, the deep network model must be executed after the execution of the other deep network model. In addition, when generating an execution plan, you also need to know the number of cores to be used. This is because if the number of multiple deep network models that can be executed in parallel exceeds the number of cores to be used, they can only be executed in batches. For example, there are a total of 8 cores, 4 cores are planned to be used, and there are now 6 deep network models that can be executed in parallel. Due to the limitation of the number of cores to be used, only 4 can be allocated to the 4 cores for execution first. After execution, 2 can be allocated to 2 of the cores for execution.

[0089] Compilation parameter instruction: an instruction for passing the above compilation parameters. It is a coded representation of the compilation parameters, which is generally input into the processing unit by the administrator user through the above model instruction input interface. That is, the deep network model instruction is input in a specified position of the interface, and the compilation parameter instruction is input in another specified position.

[0090] Acceleration unit model: The above-mentioned deep network model can only be recognized by the instruction set of the processing unit 220 so that it can be executed on the processing unit 220, but cannot be recognized by the instruction set of the acceleration unit 230 so that it can be executed on the acceleration unit 230. Therefore, it must be compiled into instructions that can be recognized by the instruction set of the acceleration unit 230 before it can be executed on the acceleration unit 230. The model that can be recognized by the instruction set of the acceleration unit 230 after the above-mentioned deep network model is compiled is called the acceleration unit model. It should also be pointed out that the model usually said actually contains the code and data samples that implement the corresponding algorithm. Code exists in many forms, such as source code (such as Java, Python implementation) and executable code (binary code). The source code can only be recognized and executed after it is compiled to obtain executable code. Therefore, the above-mentioned conversion of the deep network model into the acceleration unit model, so that it is recognized by the instruction set of the acceleration unit 230, refers to modifying part of the source code of the deep network model and compiling it into executable code, and then the executable code can be recognized by the instruction set of the acceleration unit 230.

[0091] Acceleration unit model instructions: instructions used to execute the above-mentioned acceleration unit model, that is, the coded representation of the acceleration unit model.

[0092] Execution plan: The above acceleration unit model instructions can only reflect the acceleration unit model itself, but cannot reflect the execution order between them and which core executes each. The code that reflects the execution order between acceleration unit models and the virtual cores allocated to the acceleration unit models is called an execution plan.

[0093] Physical core: The core contained in the acceleration unit that can actually execute the acceleration program. Generally, the acceleration unit contains multiple physical cores, which can execute in parallel and share an on-chip memory.

[0094] Virtual cores: When specifying an execution plan, the inference command has not been received, the actual inference of the acceleration unit has not started, and the actual physical cores to be allocated cannot be determined. It is necessary to use virtual cores instead of physical cores in the execution plan to complete the allocation of acceleration unit model instructions to cores. These cores used to replace physical cores in the execution plan are called virtual cores. Virtual cores must eventually be mapped to physical cores.

[0095] Inference: After the deep network model is trained, the actual input data is input into the deep network model, and the process of the deep network model giving the output result is called inference, which is relative to the training process.

[0096] Inference command: An instruction used to implement inference, which specifies the operations, input tensors, etc. in inference.

[0097] Load status: whether the physical core is being used for inference, the amount of workload it has undertaken, etc.

[0098] Execution batch: In the embodiment of the present disclosure, the acceleration unit model can be executed in batches according to the execution plan. Multiple acceleration unit models in each batch can be executed in parallel. The execution batch refers to the serial number of the batch to which the acceleration unit model is assigned.

[0099] Storage occupancy: The amount of storage space in the physical core occupied by the deep network model weight data, and the input, output, and various intermediate quantities during the execution process.

[0100] Computational capacity: The amount of computing power consumed by the deep network model when performing deep network operations.

[0101] Application environment of the present disclosure

[0102] The embodiment of the present disclosure proposes a solution for improving the efficiency of running multiple deep neural network models when multiple deep neural network models need to be run in an application scenario. The solution is relatively general and can be used for any device in an application scenario that needs to run multiple deep neural network models. The following description will mainly be based on the data center as the application scenario. Those skilled in the art should understand that the embodiment of the present disclosure can also be applied to other application scenarios.

[0103] Data Center

[0104] Data centers are a globally collaborative network of specific devices used to transmit, accelerate, display, calculate, and store data information on the Internet network infrastructure. In the future, data centers will also become competitive assets for enterprises. With the widespread application of data centers, artificial intelligence and other technologies are increasingly being applied to data centers. Deep learning, as an important technology of artificial intelligence, has been widely used in data center big data analysis and computing.

[0105] In traditional large data centers, the network structure is usually as follows Figure 1 As shown in the figure, it is the hierarchical inter-networking model. This model includes the following parts:

[0106] Server 140 : Each server 140 is a processing and storage entity of the data center. The processing and storage of a large amount of data in the data center are completed by these servers 140 .

[0107] Access switch 130: Access switch 130 is used to connect servers 140 to the switch in the data center. One access switch 130 connects to multiple servers 140. Access switches 130 are usually located at the top of the rack, so they are also called top of rack switches, and they are physically connected to the servers.

[0108] Aggregation switch 120: Each aggregation switch 120 connects to multiple access switches 130 and provides other services, such as firewall, intrusion detection, network analysis, etc.

[0109] Core switch 110: The core switch 110 provides high-speed forwarding for packets entering and leaving the data center, and provides connectivity for the aggregation switch 120. The network of the entire data center is divided into an L3 layer routing network and an L2 layer routing network. The core switch 110 generally provides a flexible L3 layer routing network for the network of the entire data center.

[0110] Generally, the aggregation switch 120 is the demarcation point between the L2 and L3 layer routing networks. The network below the aggregation switch 120 is the L2 network, and the network above is the L3 network. Each group of aggregation switches manages a delivery point (POD), and each POD has an independent VLAN network. The server does not need to modify the IP address and default gateway when migrating within the POD, because one POD corresponds to one L2 broadcast domain.

[0111] The spanning tree protocol (STP) is usually used between the aggregation switch 120 and the access switch 130. STP makes only one aggregation layer switch 120 available for a VLAN network, and other aggregation switches 120 are used only when a failure occurs (dashed lines in the above figure). In other words, horizontal expansion cannot be achieved at the aggregation switch 120 level, because even if multiple aggregation switches 120 are added, only one is still working.

[0112] server

[0113] Since the server 140 is the real processing device in the data center, Figure 2 The structural block diagram of the internal structure of a server 140 is shown. The server 140 includes a memory 210 connected by a bus, a processing unit cluster 270 and an acceleration unit cluster 280. The processing unit cluster 270 includes a plurality of processing units 220. The acceleration unit cluster 280 includes a plurality of acceleration units 230. The acceleration unit 230 is a processing unit designed to improve the data processing speed in a special purpose field. The acceleration unit is also called an artificial intelligence (AI) processing unit, including a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a dedicated intelligent acceleration hardware (e.g., a neural network processor NPU). The embodiment of the present disclosure is applicable to the NPU scenario, but due to the use of a general compilation custom interface, CPU, GPU, GPGPU, etc. can also be used in hardware. The processing unit is a processing unit that schedules the acceleration unit and allocates the sequence of instructions to be executed to each acceleration unit. It can take the form of a processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.

[0114] The architecture design of the traditional processing unit makes the control unit and storage unit occupy a large part of the space in the architecture, while the space occupied by the computing unit is insufficient. Therefore, it is very effective in terms of logic control, but not efficient enough in terms of large-scale parallel computing. Therefore, various special acceleration units have been developed to more effectively improve the processing speed of calculations for different functions and different fields. The acceleration unit proposed in the present invention is a processing unit dedicated to accelerating the computing processing speed of deep learning models. It is a processing unit that uses a data-driven parallel computing architecture to process a large number of operations (such as convolution, pooling, etc.) of each deep learning model node. Since the data and intermediate results in the large number of operations (such as convolution, pooling, etc.) of each deep learning model node are closely related in the entire computing process, they will be frequently used. With the existing processing unit architecture, since the memory capacity in the core of the processing unit is very small, it is necessary to frequently access the external core memory in large quantities, resulting in inefficient processing. By using this acceleration unit specifically designed to accelerate the computing speed of deep learning models, since each core has on-chip memory with a storage capacity suitable for deep learning model calculations, frequent access to memory outside the core can be avoided, which can greatly improve processing efficiency and computing performance.

[0115] The acceleration unit 230 must accept the scheduling of the processing unit 220. Figure 2 As shown, the memory 210 stores various deep learning models, including the nodes of these models and the weight data of the nodes. These deep learning models are used when needed. Figure 2 A processing unit 220 is deployed to an acceleration unit 230. That is, the processing unit 220 can send the address of the parameters in the model (such as the weights of each node) in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually uses the deep learning model for calculation, it will directly address these parameters in the memory 210 according to the addresses of these parameters (such as weights) in the memory 210, and temporarily store them in its on-chip memory. When the acceleration unit 230 actually uses the deep learning model for calculation, the processing unit 220 will also send the input of the model to the acceleration unit 230 in the form of instructions, and temporarily store them in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform inference calculations based on these inputs and the parameters in the model (such as weights).

[0116] Internal structure of the processing unit and acceleration unit

[0117] Combine the following Figure 3 The internal structure diagram of the processing unit 220 and the acceleration unit 230 specifically illustrates how the processing unit 220 schedules the acceleration unit 230 to work.

[0118] like Figure 3As shown, the processing unit 220 includes multiple processor cores 222 and a cache 221 shared by the multiple processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction executing unit 226.

[0119] The instruction fetch unit 223 is used to transfer the instruction to be executed from the memory 210 to the instruction register (which may be Figure 3 A register in the register file 229 shown is used to store instructions) and receives the next instruction fetch address or calculates the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example, increasing or decreasing the address according to the instruction length.

[0120] After fetching the instruction, the processing unit 220 enters the instruction decoding stage, and the instruction decoding unit 224 decodes the fetched instruction according to the predetermined instruction format to obtain the operand acquisition information required by the fetched instruction, thereby preparing for the operation of the instruction issuing unit 225. The operand acquisition information, for example, points to an immediate number, a register or other software / hardware that can provide a source operand.

[0121] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction executing unit 226 and is used for scheduling and controlling instructions so as to efficiently distribute each instruction to different instruction executing units 226 , thereby making it possible to operate multiple instructions in parallel.

[0122] After the instruction emission unit 225 emits the instruction to the instruction execution unit 226, the instruction execution unit 226 starts to execute the instruction. However, if the instruction execution unit 226 determines that the instruction should be executed by the acceleration unit, it forwards it to the corresponding acceleration unit for execution. For example, if the instruction is an instruction for deep learning model inference, the instruction execution unit 226 no longer executes the instruction, but sends the instruction to the acceleration unit 230 via the bus for execution by the acceleration unit 230.

[0123] Although the disclosed embodiment is used in NPU scenarios, due to the use of a universal compiled custom interface, Figure 3The acceleration unit 30 shown is not limited to NPU, but can also be GPGPU. GPGPU is a general-purpose graphics processor, which is a graphics processor that uses graphics processing tasks to calculate general-purpose computing tasks originally processed by the central processing unit. These general-purpose calculations often have nothing to do with graphics processing. Due to the powerful parallel processing capabilities and programmable pipelines of modern graphics processors, stream processors can process non-graphic data. Especially in the face of single instruction stream multiple data streams (SIMD), and the amount of data processing operations is much greater than the needs of data scheduling and transmission, the general-purpose graphics processor greatly surpasses the traditional central processing unit application in performance, so it can play an accelerating role and belongs to the acceleration unit 30 like the NPU. Even if the calculation of the NPU scenario is performed, it can also play a role. In addition, the acceleration unit 30 can also be a CPU, GPU, FPGA, ASIC, etc.

[0124] The acceleration unit 30 includes multiple physical cores 236 ( Figure 3 Four physical cores are shown, but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of physical cores 236), a command processor 237, a direct storage access mechanism 235, and a bus channel 231.

[0125] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus.

[0126] The direct memory access (DMA) mechanism 235 is a function provided by some computer bus architectures, which enables data to be written directly from an attached device to the memory of the computer motherboard. Compared with the method in which all data transmissions between devices must pass through the processing unit, this method greatly improves the efficiency of data access. It is precisely because of this mechanism that the core of the acceleration unit 230 can directly access the memory 210, read the parameters in the deep learning model (such as the weight of each node), etc., which greatly improves the efficiency of data access.

[0127] The command processor 237 assigns the instructions sent from the processing unit 220 to the acceleration unit 230 to be executed by the physical core 236. The instruction execution unit 226 sends the to-be-executed instruction sequence that needs to be executed by the acceleration unit 230 to the acceleration unit 230. After the to-be-executed instruction sequence enters from the bus channel 231, it is cached in the command processor 237, and the command processor 237 selects the physical core 236 and assigns the instruction sequence to it for execution. In addition, the command processor 237 is also responsible for the synchronization operation between the physical cores 236.

[0128] Acceleration Unit Physics Core

[0129] Figure 4It is a diagram of the internal structure of the physical core of the acceleration unit according to one embodiment of the present disclosure.

[0130] In one embodiment, Figure 4 As shown, the acceleration unit physical core 236 includes a tensor engine 310 , a pooling engine 320 , a memory copy engine 330 , a sequencer 350 , an instruction cache 340 , an on-chip memory 360 , and a constant buffer 370 .

[0131] The instruction sequence assigned by the command processor 237 to the acceleration unit physical core 236 first enters the instruction cache 340 cache. Then, the sequencer 350 fetches instructions from the instruction cache 340 in a first-in-first-out order, and assigns them to the tensor engine 310 or the pooling engine 320 for execution according to the nature of the instructions. The tensor engine 310 is responsible for processing related operations such as convolution and matrix multiplication in the deep learning model. The pooling engine 320 is responsible for processing the pooling operation in the deep learning model. The memory copy engine 330 is a unit dedicated to processing data copying, where the data copy includes copying some data from the on-chip memory 360 to the memory shared by each physical core 236, or the on-chip memory 360 of other physical cores 236 due to the possibility of overflow of the on-chip memory 360. The sequencer 350 decides to assign the instruction to the tensor engine 310, the pooling engine 320, or the memory copy engine 330 according to the nature of the operation such as convolution, matrix multiplication, pooling, or data copy.

[0132] The on-chip memory 360 is an in-core memory that stores weight parameters in the deep learning model, as well as input parameters and various intermediate results when the deep learning model is actually used. The constant buffer 370 is a buffer that stores other constant parameters (for example, hyperparameters in the neural network model) in addition to weight parameters in the deep learning model. As described above, in the process of the processing unit 220 pre-configuring the deep learning model in the acceleration unit 230, the processing unit 220 sends the address of the parameters in the model in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weights of the nodes and other parameters (such as hyperparameters). For weights, the acceleration unit 230 takes it out from the corresponding position of the memory 210 when the actual deep learning model is operated, and puts it in the on-chip memory 360. For other parameters, the acceleration unit 230 takes it out from the corresponding position of the memory 210 when the actual deep learning model is operated, and puts it in the constant buffer 370. In addition, when the instruction to actually start inference is assigned to the core 236 for execution by the command processor 237, the input parameters in the instruction (input to the deep learning model) are also stored in the on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0133] Principles of the embodiments of the present disclosure

[0134] In the prior art, for application scenarios that require the deployment of multiple different deep neural network models, these multiple deep neural network models need to be executed in turn. That is, after executing one model, another model is loaded. The rotation between deep neural network models leads to a significant increase in bandwidth requirements. The inventors of the present disclosure consider that, if Figure 3 As shown, the acceleration unit 230 has multiple physical cores 236, and different physical cores 236 can actually run different deep neural network models in parallel to improve the operating efficiency of the model and reduce bandwidth requirements. However, different deep neural network models in the same application may not necessarily be executed in parallel at the same time. Some of them have dependencies, and the previous deep neural network model must be executed before the next deep neural network model can be executed. Therefore, while generating the acceleration unit model instructions based on the decoded multiple deep network model instructions and the compilation parameter instructions, an execution plan is also generated, and the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores. Each deep network model corresponds to an acceleration unit model instruction. In the execution plan, it is executed in batches according to the dependency relationship. In the same batch, it can be allocated to multiple virtual cores for parallel execution. When the inference command is actually executed, the virtual core is mapped to the physical core, so that for each acceleration unit model instruction, the corresponding physical core is allocated, and the model instruction is loaded to the corresponding physical core for execution. Since in this scheme, the acceleration unit model instructions of the same batch can be run on multiple physical cores in parallel, the model operation efficiency is greatly improved and the bandwidth is reduced.

[0135] Fig. 6A The principle of the overall implementation of the embodiment of the present disclosure is described in more detail. Assuming that n deep network models need to be run in the same application, these n deep network models correspond to n deep network model instructions 401, that is, deep network model instructions AN. In the embodiment of the present disclosure, first, the acceleration unit model instructions 406 and the execution plan 410 are generated according to these n deep network model instructions AN and the compilation parameter instructions 402 from the user. The difference between the deep network model instructions 401 and the acceleration unit model instructions 406 is that the deep network model instructions 401 can only be recognized by the instruction set of the processing unit 220 and can be executed on the processing unit 220, but cannot be recognized by the instruction set of the acceleration unit 230 and can be executed on the acceleration unit 230. Therefore, it must be compiled into instructions that can be recognized by the instruction set of the acceleration unit 230 before it can be executed on the acceleration unit 230. Therefore, it must be compiled into acceleration unit model instructions 406.

[0136] The compilation parameter instruction 402 is an instruction for passing the compilation parameters, which is generally input by the administrator user through the above-mentioned model instruction input interface. Compilation parameters are other parameters required in addition to the deep network model instructions when generating acceleration unit model instructions and execution plans. For example, when generating an execution plan, it is necessary to know the dependencies of each deep network model. In addition, when generating an execution plan, it is also necessary to know the number of cores to be used. These dependencies and the number of cores are all compilation parameters. When the deep network model instruction 401 is compiled into an acceleration unit model instruction 406 that can be executed by the acceleration unit 230, an execution plan is also generated based on the model dependencies and the number of cores to be used in the compilation parameter instruction 402. If a deep network model depends on the execution result of another deep network model when it is executed, in the execution plan, the acceleration unit model corresponding to the deep network model must be executed in the batch after the acceleration unit model corresponding to the other deep network model is executed. In addition, if it is determined based on the dependencies between deep network models that m acceleration unit model instructions can be executed in parallel, but n cores are required, and m is greater than n, in this case, in the execution plan, the m acceleration unit model instructions can only be divided into different batches and executed by n cores.

[0137] The execution plan only maps the acceleration unit model instruction to the virtual core, not to the physical core 236, and at this time the inference command 404 has not yet started to execute. After the inference command 404 is executed, the virtual core is mapped to the physical core 236 according to the actual load status of the physical core 236, and the acceleration unit model instruction is loaded to the mapped physical core 236 to perform physical core mapping and load the acceleration unit model instruction to the physical core module 420. The physical core 236 executes to obtain the corresponding operation result 405, which is returned to the user interface through the physical core mapping and loading of the acceleration unit model instruction to the physical core modules 420 and 410, and is displayed to the user through the user interface.

[0138] Figure 6BThe mapping relationship between the acceleration unit model instruction 406 and the virtual core 408, as well as the mapping relationship between the virtual core 408 and the physical core 236 are more clearly shown. The virtual core 408 does not really exist, but is an abstract representation of the physical core 236 in the acceleration unit 230, wherein one virtual core 408 represents an uncertain physical core 236. For example, there are 4 physical cores 236 in the acceleration unit 230, and there are also 4 virtual cores 408, and the 4 virtual cores 408 are represented by their respective identifiers, but each virtual core 408 is not uniquely bound to a physical core 236, but can represent any physical core 236 when the acceleration unit model instruction is assigned to the virtual core 408. The mapping between the virtual core 408 and the physical core 236 is started only when the inference command is actually executed, after which the virtual core 408 is bound to the mapped physical core 236. When the corresponding acceleration unit model instruction 406 is executed in the physical core 236, the mapping between the virtual core 408 and the physical core 236 is released. The identifier of the virtual core 408 can begin to refer to any physical core 236 again, until the virtual core 408 is mapped to the physical core 236 again in the next batch in the execution plan, thereby being bound to one of the physical cores 236 again, and so on.

[0139] Figure 6B The acceleration unit model instruction 406 in is generated according to the deep network model instruction 401, and has a one-to-one correspondence with the deep network model instruction 401, but when allocated to the virtual core 408, it may be divided into multiple parts, one part of which is mapped to one virtual core 408, and the other part is mapped to other virtual cores 408. As will be described in detail below, the acceleration unit model instructions 406 corresponding to the deep network models of the same batch can be allocated to the multiple virtual cores 408 according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network model, and the calculation amount of the deep network model. When allocating the acceleration unit model instructions 406 corresponding to the deep network models of the same batch to the multiple virtual cores, it can be done in a manner that the sum is balanced on the allocated virtual cores and the calculation amount is balanced on the allocated virtual cores. In this way, it is possible to divide an acceleration unit model instruction 406 into multiple parts and allocate each to a virtual core 408 for the above-mentioned balance requirement. As Figure 6B In the acceleration unit model instruction A, it is divided into A1 part and A2 part, among which A1 part is allocated to Figure 6B The first virtual core 408 at the top, A2 is allocated to Figure 6B The second virtual core 408 from the top; the acceleration unit model instruction B is divided into a B1 part and a B2 part, wherein the B1 part is allocated to Figure 6B From the top, the second virtual core 408, part B2 is allocated to Figure 6BThe third virtual core 408 from the top; and so on. After the inference command 404 is executed, the virtual core 408 begins to be mapped to the physical core 236 according to the actual load status of the physical core 236. It is uncertain which physical core 236 the virtual core 408 is mapped to, and it is determined according to the actual load status of each physical core 236 at that time. Figure 6B As shown, the top virtual core 408 in the figure is mapped to the second physical core 236 from the top, the second virtual core 408 from the top is mapped to the fourth physical core 236 from the top, the third virtual core 408 from the top is mapped to the first physical core 236 from the top, and the fourth virtual core 408 from the top is mapped to the third physical core 236 from the top.

[0140] In one embodiment, the acceleration unit model instruction and execution plan generation 410, the physical core mapping and loading of the acceleration unit model instruction to the physical core module 420 can all be performed by Figure 3 The instruction execution unit 226 in the processing unit 220 executes, and the command processor 237 of the acceleration unit 230 only needs to load and execute the relevant acceleration unit model instructions according to the physical core 236 that has been allocated by the instruction execution unit 226 in the processing unit 220. In this embodiment, the basic process of the present disclosure is executed in the processing unit 220. In the following description, this embodiment is referred to as the first embodiment.

[0141] In another embodiment, the acceleration unit model instruction and execution plan generation 410 may be performed by Figure 3 The instruction execution unit 226 in the processing unit 220 is executed, and the physical core mapping and loading acceleration unit model instructions to the physical core 420 can be performed by Figure 3 The command processor 237 of the acceleration unit 230 in the processing unit 220 is executed. That is, the instruction execution unit 226 in the processing unit 220 generates the acceleration unit model instruction and the execution plan based on the decoded multiple deep network model instructions and the compilation parameter instructions. When a specific reasoning command is received, the command processor 237 of the acceleration unit 230 executes the physical core mapping and loading the acceleration unit model instruction to the physical core module 420, and loads the relevant acceleration unit model instruction to the assigned physical core 236 for execution. In this embodiment, the generation of the acceleration unit model instruction and the execution plan is executed by the processing unit 220, while the mapping of the virtual core to the physical core and the loading of the acceleration unit model to the physical core during actual reasoning are executed by the acceleration unit 230. In the following description, this embodiment is referred to as the second embodiment.

[0142] It should be noted that if Figure 5 As shown, the generated acceleration unit model instruction is not in a one-to-one correspondence with the assigned physical core 236. An acceleration unit model instruction can be divided into multiple parts and finally executed by different physical cores 236 (the reason is combined with Figure 6B Explained). For example Figure 5 The acceleration unit model instruction A is divided into three parts A1-3, which are respectively deployed on three physical cores 236 for execution. A physical core can also run more than one acceleration unit model instruction at the same time, for example Figure 5 Each physical core 236 runs corresponding portions of more than one acceleration unit model instructions.

[0143] The first and second embodiments described above are described in detail below.

[0144] First embodiment (generating acceleration unit model instructions and execution plan, mapping virtual cores to physical cores are all done by the processor (Executed by the processing unit 220)

[0145] First, the processing unit 220 receives multiple deep network model instructions and compilation parameter instructions input by the user through the user interface. The user interface can be a human-computer interaction interface, on which there are positions for inputting multiple deep network model instructions and positions for inputting compilation parameter instructions. Then, the user inputs multiple deep network model instructions at the above-mentioned position for inputting multiple deep network model instructions, and inputs compilation parameter instructions at the above-mentioned position for inputting compilation parameter instructions. Multiple deep network model instructions correspond to multiple deep network models to be executed in the same application scenario, and are the result of the multiple deep network models being represented by program code. Compilation parameters are other parameters required in addition to deep network model instructions when generating acceleration unit model instructions and execution plans, which include at least the number of cores to be used and the dependencies of the multiple deep network models.

[0146] After multiple deep network model instructions and compilation parameter instructions are input through the user interface, they are first stored in the memory 210. Then, the instruction fetching unit 223 in the processing unit 220 retrieves the multiple deep network model instructions and compilation parameter instructions from the memory 210. The instruction decoding unit 224 decodes the retrieved multiple deep network model instructions and compilation parameter instructions. The instruction execution unit 226 generates an acceleration unit model instruction and an execution plan based on the decoded multiple deep network model instructions and compilation parameter instructions. Generating the acceleration unit model instruction based on the decoded multiple deep network model instructions is achieved by compiling the multiple deep network model instructions. As described above, the deep network model instruction 401 can only be recognized by the instruction set of the processing unit 220 and can be executed on the processing unit 220, but cannot be recognized by the instruction set of the acceleration unit 230 and can be executed on the acceleration unit 230, so it must be compiled into an acceleration unit model instruction. This compilation method already exists.

[0147] The execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores.

[0148] When determining the execution plan, first, according to the dependency relationship in the compilation parameter instruction and the number of cores to be used, the execution batches of the multiple deep network models are determined. If the execution of a deep network model depends on the execution result of another deep network model, in the execution plan, the deep network model is to be executed after the execution of the other deep network model. In this way, a dependency tree can be drawn according to the dependency relationship of each deep network model in the compilation parameter instruction. If the execution of a deep network model depends on the execution result of another deep network model, the deep network model is drawn in the next layer of the other deep network model in the tree, and the dependency relationship is represented by an arrow from the other deep network model to the deep network model. In this way, after the dependency tree is drawn, the deep network models of the same layer in the tree can be executed simultaneously, but whether they can be executed simultaneously depends on the number of cores to be used. For example, there are 8 cores in total, 4 cores are intended to be used, and there are 6 deep network models in the same layer of the tree. Due to the limitation of the number of cores to be used, they can only be executed in two batches. For example, three deep network models are assigned to the core for execution in the previous batch, and three deep network models are assigned to the core for execution in the next batch.

[0149] The above process only determines the batches in the execution plan and the acceleration unit models corresponding to which deep network models are executed in each batch, but how to allocate them to the cores in each batch has not yet been resolved. Since the actual reasoning process has not yet begun, there is no need to allocate the model to the physical core, and it can be allocated to the virtual core first. After the inference command is actually received, the virtual core is mapped to the physical core. In the embodiment of the present disclosure, for the same batch, the deep network models of the same batch can be allocated to the multiple virtual cores according to the storage occupancy and computational complexity of the deep network models of the same batch.

[0150] The storage usage and computational effort of each deep network model can be determined based on the deep network model instructions.

[0151] The deep network model has multiple layers, each layer has multiple operation nodes, each operation node has a weight, which can be operated according to the output of each operation node of the previous layer of the operation node and the weight of the operation node, and the operation result is passed to the operation node in the next layer as input. This is explained below using a convolutional neural network as an example. The convolutional neural network includes multiple layers (convolutional layer, pooling layer, etc.), and the features output by the operation node of each convolutional layer are transmitted to the operation node of the next convolutional layer as the input received by the operation node of the next convolutional layer. Each layer performs matrix multiplication and summation calculation through the convolution kernel to obtain a feature map. The convolution kernel is a two-dimensional matrix, and each value in the two-dimensional matrix can be regarded as a weight parameter. Since the convolutional neural network has multiple layers, there are multiple weight parameters in the convolution kernel of each layer, so the number of weight parameters is huge. These weights need to be stored in the on-chip memory 360 of the physical core 236. Corresponding to the weight parameters, the number of input tensors in the input tensor matrix multiplied by the convolution kernel is also huge, and they are also stored in the on-chip memory of the physical core 236. Moreover, the features output by the operation nodes of each layer must be stored in the on-chip memory of the physical core 236 before being output to the operation nodes of the next layer as input. In addition, the output tensors obtained after the deep network model operation are also stored in the on-chip memory of the physical core 236. By parsing the statements of the deep network model instructions, the storage occupied by the weights, input tensors, intermediate tensors, and output tensors required by the statement can be determined from each statement. The weight storage occupancy required by each statement of the deep network model instruction is added to obtain the weight storage occupancy of the deep network model. The input tensors, intermediate tensors, and output tensors required by each statement of the deep network model instruction are added to obtain the tensor storage occupancy of the input tensors, intermediate tensors, and output tensors of the deep network model.

[0152] By parsing the statements of the deep network model instruction, the computational amount of each statement can be determined, which is the processing capacity of the physical core 236 that the statement needs to occupy. The computational amount of the deep network model is obtained by adding up the computational amount required for each statement of the deep network model instruction, which is the total processing capacity of the physical core 236 that the execution of the entire deep network model needs to occupy.

[0153] Then, the deep network models of the same batch are allocated to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network model, as well as the computational amount of the deep network model. The allocation can be performed in a manner that the sum is balanced on the allocated virtual cores and the computational amount is balanced on the allocated virtual cores. It should be noted that in order to balance the sum of the storage occupancy and the computational amount on each virtual core, a deep network model can also be divided into multiple parts and allocated to multiple virtual cores for execution. For example, if the sum of the storage occupancy and the computational amount occupied by a certain deep network model are too large, it can be divided into multiple parts, each of which is allocated to a virtual core. In this way, the corresponding acceleration unit model may be divided into multiple parts, each of which is on a physical core. Figure 5 As shown, the acceleration unit model instruction A is divided into parts A1-3, which are finally assigned to three different physical cores 236 for execution.

[0154] For example, if the number of cores to be used is 4 and there are 3 deep network models in a batch, these 3 deep network models need to be allocated to 4 virtual cores. The total storage usage of these 3 deep network models is 6MB, 3.5MB, and 2MB, and the computational load is 4, 2, and 2. The first deep network model can be evenly divided into two parts, A and B, and allocated to 2 virtual cores, while the remaining 2 deep network models are each allocated one virtual core.

[0155] In this way, the execution plan can be generated according to the determined execution batches of the multiple deep network models and the allocation of the multiple deep network models to the multiple virtual cores. As described above, the execution plan is used to reflect the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores. Each acceleration unit model instruction corresponds to each deep network model. Therefore, by determining the execution batch of each deep network model, the execution order of each acceleration unit model instruction is also determined. By determining the allocation of each batch of deep network models to the multiple virtual cores, the allocation of the acceleration unit model instructions to the multiple virtual cores is also determined.

[0156] Since the inference command used for actual inference has not been received at this time, the generated acceleration unit model instructions and execution plan will not be used immediately, but will be stored in the memory 210 and retrieved from the memory 210 for use after the actual inference command is received.

[0157] Inference means that after the deep network model is trained, the actual input tensor is input into the deep network model, and the operation nodes of each layer of the deep network model perform operations according to the weight data, and the output results are output to the operation nodes of the next layer as outputs, and finally the entire deep network model gives the final inference result. The inference command is an instruction for implementing inference, which specifies the inference operation and the input data (input tensor) used. After the inference command is input, it is stored in the memory 210. The instruction fetch unit 223 in the processing unit 220 retrieves the inference command from the memory 210. The instruction decoding unit 224 decodes the retrieved inference command. The instruction execution unit 226 determines the acceleration unit model instructions and execution plans stored in the memory 210 for the inference command from the inference command, and obtains these acceleration unit model instructions and execution plans from the memory 210. Since the execution plan only indicates the allocation of the acceleration unit model instructions to the virtual core, and has not been implemented to the physical core, it is necessary to map the virtual core to the physical core.

[0158] At this time, the instruction execution unit 226 in the processing unit 220 needs to obtain the load status of each physical core 236 in the acceleration unit 230. The load status refers to whether the physical core is being used for reasoning, the amount of workload undertaken, etc. The workload can be represented by the amount of storage occupied by the tasks undertaken, or the number of tasks undertaken. The load status of each physical core 236 can be obtained by the instruction execution unit 226 of the processing unit 220 sending a query request to the command processor 237 of the acceleration unit 230. Since the command processor 237 is a component that sends tasks to the physical core 236, it knows the load status of each physical core 236. Therefore, the command processor 237 can send a response to the instruction execution unit 226, and the response contains the load status of each physical core 236.

[0159] If M virtual cores are allocated to the acceleration unit model instructions in the execution plan, the mapping of the virtual cores to physical cores according to the load conditions includes: selecting M physical cores with the smallest load conditions as the physical cores to which the acceleration unit model instructions are loaded.

[0160] The number of cores M to be used in the compilation parameter instruction is less than or equal to the number of physical cores in the acceleration unit. When mapping the virtual cores to the physical cores, it is only necessary to sort the load conditions in each physical core from low to high, select the physical cores ranked in the top M positions, and map the virtual cores to these selected physical cores one by one. For example, the physical cores can be sorted from low to high according to the number of acceleration unit model instructions currently being executed, the physical cores ranked in the top M positions can be selected, and the virtual cores can be mapped to these selected physical cores one by one; or the physical cores can be sorted from low to high according to the currently occupied storage capacity, the physical cores ranked in the top M positions can be selected, and the virtual cores can be mapped to these selected physical cores one by one. Since the existing loads of these physical cores are relatively small, allocating the acceleration unit model instructions to these physical cores for operation will help to balance the loads of all physical cores in the acceleration unit.

[0161] Next, the instruction execution unit 226 loads the acceleration unit model instructions into the physical core according to the execution plan and the mapping. That is, referring to the execution plan, the first batch of the execution plan is obtained, the virtual cores to which the acceleration unit model instructions in the first batch are mapped are obtained, and then according to the mapping between these virtual cores and the physical cores, the acceleration unit model instructions in the first batch are loaded into the corresponding physical cores. When loading into the corresponding physical core, it is loaded through the command processor 237. Next, referring to the execution plan, the second batch of the execution plan is obtained, the virtual cores to which the acceleration unit model instructions in the second batch are mapped are obtained, and then according to the mapping between these virtual cores and the physical cores, the acceleration unit model instructions in the second batch are loaded into the corresponding physical cores. And so on, until all batches in the execution plan are executed.

[0162] Next, the instruction execution unit 226 loads the input tensor used by the inference command to the corresponding physical core through the command processor 237 according to the execution plan and the mapping. In this way, the corresponding physical core obtains the acceleration unit model instruction used for the inference command and the input tensor, and can calculate the operation result according to the input tensor and the weight in the acceleration unit model instruction, and returns it to the instruction execution unit 226 through the command processor 237, and is transmitted to the user interface for presentation to the user.

[0163] In the above process, the generation of acceleration unit model instructions and execution plans based on multiple deep network model instructions and compilation parameter instructions is performed in advance before the actual reasoning command is received. In this way, the acceleration unit model instructions and execution plans can be generated and stored in the memory 210 in advance. When the actual reasoning command is received, it is only necessary to map the virtual core to the physical core, thereby greatly reducing the reasoning running time after receiving the reasoning command and improving the reasoning efficiency. Of course, it is also possible not to generate the acceleration unit model instructions and execution plans based on multiple deep network model instructions and compilation parameter instructions in advance, but to wait until the reasoning command is received before starting to generate the acceleration unit model instructions and execution plans, and mapping the virtual core to the physical core. In this case, the generation of the acceleration unit model instructions and execution plans based on the decoded multiple deep network model instructions and compilation parameter instructions is executed in response to the simultaneous reception of the decoded multiple deep network model instructions, compilation parameter instructions and reasoning commands. Its advantage is that the generated acceleration unit model instructions and execution plans do not need to be stored in the memory 210, and can be directly allocated to the allocated physical core, saving the resources of the memory 210.

[0164] Second embodiment (generating acceleration unit model instructions and execution plan executed by processing unit 220, virtual core to physical core The mapping of the core is performed by the acceleration unit 230)

[0165] The difference between this embodiment and the first embodiment is that after the instruction execution unit 226 receives the inference command decoded by the instruction decoding unit 224, it does not perform subsequent virtual core to physical core mapping processing, but sends it to the command processor 237 in the acceleration unit 230, and the command processor 237 obtains the acceleration unit model instruction and execution plan for the inference command from the memory 210, obtains the load status of the physical core in the acceleration unit, performs the mapping of the virtual core to the physical core according to the load status, and loads the acceleration unit model instruction to the physical core according to the execution plan and the mapping. The method in which the command processor 237 obtains the load status of the physical core in the acceleration unit, performs the mapping of the virtual core to the physical core according to the load status, and loads the acceleration unit model instruction to the physical core according to the execution plan and the mapping is the same as that of the instruction execution unit 226 in the first embodiment, except that the execution subject is different, so it is not repeated.

[0166] Business Value

[0167] The disclosed embodiment runs in an application scenario in a data center. After application testing that requires multiple deep network models, it was found that the computing speed was increased by more than 1 times compared to the traditional solution of sequentially executing multiple deep network models, and the bandwidth occupancy was only 40% of the original, which has extremely high commercial value.

[0168] like Figure 7As shown, in the case where the generation of the acceleration unit model instruction and the execution plan, and the mapping of the virtual core to the physical core are all performed by the processing unit 220, the processing method on the processing unit 220 side includes:

[0169] Step 510: Generate acceleration unit model instructions and an execution plan based on multiple deep network model instructions and compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores;

[0170] Step 520, obtain the acceleration unit model instructions and execution plan for the inference command, obtain the load status of the physical core in the acceleration unit, map the virtual core to the physical core according to the load status, and load the acceleration unit model instructions to the physical core according to the execution plan and the mapping.

[0171] The implementation details of the above processing method have been involved in the detailed description of the first embodiment, and are only described from the perspective of the method here, so they are not repeated here.

[0172] like Figure 8 As shown, when the generation of the acceleration unit model instruction and the execution plan are executed by the processing unit 220, and the mapping of the virtual core to the physical core is executed by the acceleration unit 230, the processing method on the processing unit 220 side includes:

[0173] Step 610: receiving multiple deep network model instructions and compilation parameters;

[0174] Step 620: Generate acceleration unit model instructions and an execution plan based on the multiple deep network model instructions and compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores.

[0175] like Fig. 9 As shown, the corresponding method for loading the acceleration unit model instruction on the acceleration unit 230 side includes:

[0176] Step 710: Obtain acceleration unit model instructions and an execution plan for inference commands, wherein the execution plan reflects the execution order of the acceleration unit model instructions and the allocation of each acceleration unit model instruction to multiple virtual cores;

[0177] Step 720: Obtain the load status of the physical core in the acceleration unit;

[0178] Step 730: Map the virtual core to the physical core in the acceleration unit according to the load condition;

[0179] Step 740: Load the acceleration unit model instructions into the physical core according to the execution plan and the mapping.

[0180] The implementation details of the processing method on the processing unit 220 side and the method for loading the acceleration unit model instruction on the acceleration unit 230 side have been involved in the detailed description of the second embodiment. Here, they are only described from the perspective of the method, so they will not be repeated.

[0181] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the method embodiment, since it is basically similar to the method described in the device and system embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of other embodiments.

[0182] It should be understood that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0183] It should be understood that an element described in singular form herein or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown as separate herein may be combined into a single module or element, and modules or elements described or shown as single herein may be split into multiple modules or elements.

[0184] It should also be understood that the terms and expressions used herein are for description only, and one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or parts thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, changes and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. A processing unit, comprising: an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; An instruction execution unit is used to execute the decoded computer instructions to achieve: Generate an acceleration unit model instruction and an execution plan based on the decoded multiple deep network model instructions and compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores, wherein the deep network model instruction is a coded representation of the deep network model, the acceleration unit model instruction is a coded representation of the acceleration unit model, and the acceleration unit model is a model that can be recognized by the instruction set of the acceleration unit after the deep network model is compiled; the virtual core is allocated the acceleration unit model instruction or a part of the acceleration unit model instruction to achieve balanced allocation; Obtain the acceleration unit model instructions and execution plan for the decoded inference command, obtain the load status of the physical core in the acceleration unit, map the virtual core to the physical core according to the load status, and load the acceleration unit model instructions to the physical core according to the execution plan and the mapping.

2. The processing unit according to claim 1, wherein: The compilation parameter instruction includes at least the number of cores to be used and the dependency relationship of the multiple deep network models; The generating the acceleration unit model instruction and the execution plan based on the multiple deep network model instructions and the compilation parameter instructions includes: Generate acceleration unit model instructions corresponding to each deep network model instruction; Determining execution batches of the multiple deep network models according to the dependency relationship and the number of cores to be used; Allocating the deep network models of the same batch to the multiple virtual cores according to the storage occupancy and computing amount of the deep network models of the same batch; The execution plan is generated according to the determined execution batches of the multiple deep network models and the allocation of the deep network models of the same batch to the multiple virtual cores.

3. The processing unit according to claim 2, wherein: The allocating the same batch of deep network models to the multiple virtual cores according to the storage occupancy and computing amount of the same batch of deep network models includes: Determine the weight storage occupancy of the deep network model according to the deep network model instruction; According to the deep network model instructions, determine the tensor storage occupancy of the input tensor, intermediate tensor, and output tensor of the deep network model; Determine the computational amount of the deep network model according to the deep network model instruction; The deep network models of the same batch are allocated to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network models and the computational complexity of the deep network models.

4. The processing unit according to claim 3, wherein: The deep network models of the same batch are distributed to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network models and the computational amount of the deep network models, and this is done in a manner such that the sum is balanced on the allocated virtual cores and the computational amount is balanced on the allocated virtual cores.

5. The processing unit according to claim 1, wherein: The generating of the acceleration unit model instruction and the execution plan based on the decoded multiple deep network model instructions and the compilation parameter instruction is executed in response to receiving the decoded multiple deep network model instructions and the compilation parameter instruction, and the generated acceleration unit model instruction and the execution plan are stored in a memory outside the processing unit; The acceleration unit model instructions and execution plan for the decoded inference command are obtained from the memory in response to receiving the decoded inference command.

6. The processing unit according to claim 1, wherein: The generation of acceleration unit model instructions and execution plans based on the decoded multiple deep network model instructions and compilation parameter instructions is executed in response to simultaneously receiving the decoded multiple deep network model instructions, compilation parameter instructions and inference commands.

7. The processing unit according to claim 1, wherein: If M virtual cores are allocated to the acceleration unit model instruction in the execution plan, mapping the virtual cores to physical cores according to the load condition includes: The M physical cores with the smallest load conditions are selected as the physical cores to which the acceleration unit model instructions are loaded.

8. The processing unit according to claim 1, wherein: After loading the acceleration unit model instructions into the physical core according to the execution plan and the mapping, the instruction execution unit executes the decoded computer instructions to achieve: Loading the input tensor for the inference command into the physical core according to the execution plan and the mapping; A calculation result calculated by the physical core according to the input tensor and the weight in the acceleration unit model instruction is received from the physical core.

9. A processing unit, comprising: an instruction fetch unit for retrieving computer instructions from a memory external to the processing unit; an instruction decoding unit for decoding the retrieved computer instructions; An instruction execution unit is used to execute the decoded computer instructions to generate acceleration unit model instructions and an execution plan based on the decoded multiple deep network model instructions and compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores, wherein the deep network model instruction is a coded representation of the deep network model, the acceleration unit model instruction is a coded representation of the acceleration unit model, and the acceleration unit model is a model that can be recognized by the instruction set of the acceleration unit after the deep network model is compiled; the virtual core is allocated the acceleration unit model instruction or a part of the acceleration unit model instruction to achieve balanced allocation.

10. The processing unit according to claim 9, wherein: The compilation parameter instruction includes at least the number of cores to be used and the dependency relationship of the multiple deep network models; The generating the acceleration unit model instruction and the execution plan based on the multiple deep network model instructions and the compilation parameter instructions includes: Generate acceleration unit model instructions corresponding to each deep network model instruction; Determining execution batches of the multiple deep network models according to the dependency relationship and the number of cores to be used; Allocating the deep network models of the same batch to the multiple virtual cores according to the storage occupancy and computing amount of the deep network models of the same batch; The execution plan is generated according to the determined execution batches of the multiple deep network models and the allocation of the deep network models of the same batch to the multiple virtual cores.

11. The processing unit according to claim 10, wherein: The allocating the same batch of deep network models to the multiple virtual cores according to the storage occupancy and computing amount of the same batch of deep network models includes: Determine the weight storage occupancy of the deep network model according to the deep network model instruction; According to the deep network model instructions, determine the tensor storage occupancy of the input tensor, intermediate tensor, and output tensor of the deep network model; Determine the computational amount of the deep network model according to the deep network model instruction; The deep network models of the same batch are allocated to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network models and the computational complexity of the deep network models.

12. The processing unit according to claim 11, wherein: The deep network models of the same batch are distributed to the multiple virtual cores according to the sum of the weight storage occupancy and the tensor storage occupancy of the deep network models and the computational amount of the deep network models, and this is done in a manner such that the sum is balanced on the allocated virtual cores and the computational amount is balanced on the allocated virtual cores.

13. An acceleration unit, comprising: Multiple physical cores; and A command processor, configured to, in response to receiving an inference command, execute: Obtaining an acceleration unit model instruction and an execution plan for the inference command, wherein the execution plan reflects the execution order of the acceleration unit model instruction and the allocation of the acceleration unit model instruction to multiple virtual cores, wherein the deep network model instruction is a coded representation of the deep network model, the acceleration unit model instruction is a coded representation of the acceleration unit model, and the acceleration unit model is a model that can be recognized by the instruction set of the acceleration unit after the deep network model is compiled; the virtual core is allocated the acceleration unit model instruction or a part of the acceleration unit model instruction to achieve balanced allocation; Obtaining the load status of the physical core in the acceleration unit; Mapping the virtual core to the physical core according to the load condition; The acceleration unit model instructions are loaded into the physical core according to the execution plan and the mapping.

14. The acceleration unit according to claim 13, wherein: The step of loading the acceleration unit model instruction into the physical core according to the execution plan and the mapping includes: Determining the virtual cores allocated to the acceleration unit model instructions according to the execution plan; The acceleration unit model instructions are loaded into the physical core to which the determined virtual core is mapped.

15. The acceleration unit according to claim 13, wherein: If M virtual cores are allocated to the acceleration unit model instruction in the execution plan, mapping the virtual cores to physical cores according to the load condition includes: The M physical cores with the smallest load conditions are selected as the physical cores to which the acceleration unit model instructions are loaded.

16. The acceleration unit according to claim 13, wherein: After loading the acceleration unit model instructions into the physical core according to the execution plan and the mapping, the command processor is further used to: Loading input tensors used by the inference command into physical cores according to the execution plan and the mapping; A calculation result calculated by the physical core according to the input tensor and the weight in the acceleration unit model instruction is received from the physical core.

17. A processing method comprising: Generate an acceleration unit model instruction and an execution plan based on multiple deep network model instructions and compilation parameter instructions, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores, wherein the deep network model instruction is a coded representation of the deep network model, the acceleration unit model instruction is a coded representation of the acceleration unit model, and the acceleration unit model is a model that can be recognized by the instruction set of the acceleration unit after the deep network model is compiled; the virtual core is allocated the acceleration unit model instruction or a part of the acceleration unit model instruction to achieve balanced allocation; Obtain acceleration unit model instructions and an execution plan for inference commands, obtain the load status of the physical core in the acceleration unit, map the virtual core to the physical core according to the load status, and load the acceleration unit model instructions to the physical core according to the execution plan and the mapping.

18. A processing method comprising: Receive multiple deep network model instructions and compilation parameters; Based on the multiple deep network model instructions and compilation parameter instructions, an acceleration unit model instruction and an execution plan are generated, wherein the execution plan reflects the execution order of each acceleration unit model instruction and the allocation of each acceleration unit model instruction to multiple virtual cores, wherein the deep network model instruction is a coded representation of the deep network model, the acceleration unit model instruction is a coded representation of the acceleration unit model, and the acceleration unit model is a model that can be recognized by the instruction set of the acceleration unit after the deep network model is compiled; the virtual core is allocated the acceleration unit model instruction or a part of the acceleration unit model instruction to achieve balanced allocation.

19. A method for loading an acceleration unit model instruction in an acceleration unit, comprising: Obtaining an acceleration unit model instruction and an execution plan for an inference command, wherein the execution plan reflects the execution order of the acceleration unit model instruction and the allocation of the acceleration unit model instruction to multiple virtual cores, wherein the deep network model instruction is a coded representation of the deep network model, the acceleration unit model instruction is a coded representation of the acceleration unit model, and the acceleration unit model is a model that can be recognized by the instruction set of the acceleration unit after the deep network model is compiled; the virtual core is allocated the acceleration unit model instruction or a part of the acceleration unit model instruction to achieve balanced allocation; Obtaining the load status of the physical core in the acceleration unit; Mapping the virtual core to the physical core in the acceleration unit according to the load condition; The acceleration unit model instructions are loaded into the physical core according to the execution plan and the mapping.

20. A computing device comprising: The processing unit according to any one of claims 1 to 8; A memory, used to store the acceleration unit model instructions and execution plan generated by the processing unit; The acceleration unit.

21. A computing device comprising: A processing unit according to any one of claims 9 to 12; An acceleration unit according to any one of claims 13 to 16; The memory is used to store the acceleration unit model instructions and execution plan generated by the processing unit.

22. A system on chip comprising a processing unit according to any one of claims 1-12.

23. A system on chip comprising the acceleration unit according to any one of claims 13-16.

24. A data center comprising a computing device according to claim 20 or 21.

Images