Task Scheduling Method, Device and Terminal Device of Neural Network Processor

Abstract

This application is applicable to the field of computer technology, and provides a task scheduling method, device and terminal device for a neural network processor. The method includes: determining a task scheduling method according to the number of execution notifications of operator task blocks in the current queue, where the execution notification carries a task descriptor of the operator task block, and then determining an operator task block to be executed according to the task scheduling method, and then controlling direct memory access to transfer the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit according to the task descriptor of the operator task block to be executed. Thus, when the transfer of the task descriptor of the operator operation task in the operator task block to be executed is completed, sub-tasks of the operator operation task in the operator task block to be executed are executed according to the task descriptor of the operator operation task in the operator task block to be executed, thereby improving the execution efficiency of the operator task.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and particularly relates to a task scheduling method, device, terminal device, and computer-readable storage medium for a neural network processor. Background Art

[0002] A neural-network processing unit (NPU) is a processor specifically applied to network application data packets. It adopts an architecture of "data-driven parallel computing" and can be used to process massive multimedia data such as videos and images. When the main control processor (Advanced RISC Machines, ARM) issues an operator task of a neural network model to the NPU, the NPU executes the operator operation task of the neural network model and notifies the main control processor. Among them, the task descriptor of the operator operation task of the neural network model is stored on a Double Data Rate (DDR) synchronous dynamic random access memory. The task descriptor of the operator operation task needs to be transferred to the data storage unit (Data Memory, DM) of the control processing unit (Control Unit, CU) inside the NPU by Direct Memory Access (DMA). Then, according to the address information of the code segment of the operator operation in the task descriptor of the operator operation task, the operator code segment of the neural network model is transferred from Direct Memory Access (DMA) to the program storage unit (Program Memory, PM) of the control processing unit (Control Unit, CU) inside the NPU, and then the operator task of the neural network model is executed.

[0003] In the related art, the scheduling of the operator operation tasks of the neural network model adopts serial scheduling. Only after the previous operator operation task of the neural network model is completed can the next operator operation task of the neural network model be executed. Therefore, for multiple operator operation tasks, the scheduling method of the operator operation tasks results in low execution efficiency of the operator operation tasks. Summary of the Invention

[0004] The task scheduling method, device, and terminal device for a neural network processor provided by the embodiments of this application can solve the problem that for multiple operator tasks, the scheduling method of the operator tasks results in low execution efficiency of the operator tasks.

[0005] In a first aspect, the embodiments of this application provide a task scheduling method for a neural network processor, including:

[0006] Determine a task scheduling method according to the number of execution notifications of operator task blocks in the current queue, where the execution notification carries a task descriptor of an operator task block; determine a to-be-executed operator task block according to the task scheduling method; according to the task descriptor of the to-be-executed operator task block, control direct memory access to transfer the task descriptor of the operator operation task in the to-be-executed operator task block from the main control memory to its own data storage unit; in the case where the transfer of the task descriptor of the operator operation task in the to-be-executed operator task block is completed, execute the subtasks of the operator operation task in the to-be-executed operator task block according to the task descriptor of the operator operation task in the to-be-executed operator task block.

[0007] In a second aspect, an embodiment of the present application provides a task scheduling device for a neural network processor, including:

[0008] A scheduling method confirmation module, configured to determine a task scheduling method according to the number of execution notifications of operator task blocks in the current queue, where the execution notification carries a task descriptor of an operator task block; a to-be-executed operator task block confirmation module, configured to determine a to-be-executed operator task block according to the task scheduling method; a descriptor transfer module, configured to control direct memory access to transfer the task descriptor of the operator operation task in the to-be-executed operator task block from the main control memory to its own data storage unit according to the task descriptor of the to-be-executed operator task block; an operator operation task execution module, configured to execute the subtasks of the operator operation task in the to-be-executed operator task block according to the task descriptor of the operator operation task in the to-be-executed operator task block in the case where the transfer of the task descriptor of the operator operation task in the to-be-executed operator task block is completed.

[0009] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, a neural network processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the task scheduling method of the neural network processor according to any one of the first aspects is implemented.

[0010] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the task scheduling method of the neural network processor according to any one of the first aspects is implemented.

[0011] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a terminal device, enables the terminal device to execute the task scheduling method of the neural network processor according to any one of the first aspects.

[0012] It is understandable that the beneficial effects of the second to fifth aspects described above can be referred to the relevant descriptions in the first aspect, and will not be elaborated here.

[0013] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: by determining the task scheduling method according to the number of execution notifications of operator task blocks in the current queue, where the execution notification carries the task descriptor of the operator task block, and then determining the operator task block to be executed according to the task scheduling method, and further controlling the direct memory access to transfer the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit according to the task descriptor of the operator task block to be executed. Thus, when the transfer of the task descriptor of the operator operation task in the operator task block to be executed is completed, the subtasks of the operator operation task in the operator task block to be executed are executed according to the task descriptor of the operator operation task in the operator task block to be executed. Therefore, by determining the task scheduling method according to the number of execution notifications of operator task blocks in the current queue, determining the operator task block to be executed according to the task scheduling method, and then executing the subtasks of the operator operation task in the operator task block to be executed according to the task descriptor of the operator task block to be executed and the task descriptor of the operator operation task in the operator task block to be executed, the execution efficiency of the operator operation task is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0015] Figure 1 is a structural diagram of a neural network processor provided by an embodiment of the present application;

[0016] Figure 2 is a schematic flowchart of a task scheduling method of a neural network processor provided by an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of the task execution order in the serial scheduling mode provided by another embodiment of the present application;

[0018] Figure 4 is a schematic diagram of the task execution order in the parallel scheduling mode provided by another embodiment of the present application;

[0019] Figure 5 is a schematic structural diagram of a task descriptor provided by another embodiment of the present application;

[0020] Figure 6It is a schematic diagram of the execution process of the operation operators op_ave and op_binary provided by another embodiment of the present application;

[0021] Figure 7 It is a schematic diagram of the structure of the task descriptors corresponding to the operation operators op_ave and op_binary provided by another embodiment of the present application;

[0022] Figure 8 It is a schematic diagram of the structure of the task scheduling device of the neural network processor provided by an embodiment of the present application;

[0023] Figure 9 It is a schematic diagram of the structure of the terminal device provided by an embodiment of the present application. Detailed implementation manners

[0024] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obstructing the description of the present application.

[0025] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0026] It should also be understood that the term " / and" as used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] As used in the specification and appended claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" according to the context.

[0028] In addition, in the description of the specification and appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0030] It should be understood that the magnitudes of the sequence numbers of the steps in this embodiment do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0031] In the related art, the scheduling of the operator operation tasks of the neural network model adopts serial scheduling. The operator operation task of the next neural network model can only be executed after the operator operation task of the previous neural network model is completed. Therefore, for multiple operator operation tasks, the scheduling method of the operator operation tasks results in low execution efficiency of the operator operation tasks.

[0032] This application provides a task scheduling method for a neural network processor. By determining the task scheduling method according to the number of execution notifications of the operator task blocks in the current queue, where the execution notification carries the task descriptor of the operator task block, and then determining the operator task block to be executed according to the task scheduling method, and then according to the task descriptor of the operator task block to be executed, controlling the direct memory access to move the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit. Thus, when the transfer of the task descriptor of the operator operation task in the operator task block to be executed is completed, according to the task descriptor of the operator operation task in the operator task block to be executed, the subtasks of the operator operation task in the operator task block to be executed are executed. Therefore, by determining the task scheduling method according to the number of execution notifications of the operator task blocks in the current queue, determining the operator task block to be executed according to the task scheduling method, and then executing the subtasks of the operator operation task in the operator task block to be executed according to the task descriptor of the operator task block to be executed and the task descriptor of the operator operation task in the operator task block to be executed, the execution efficiency of the operator task is improved.

[0033] The task scheduling method of the neural network processor provided by this application, and the execution entity can be the neural network processor of the terminal device. To illustrate the technical solution of this application, this method takes the neural network processor of the terminal device as the execution entity as an example to illustrate the implementation process of the task scheduling method of the neural network processor provided by the embodiments of this application.

[0034] In one embodiment, as Figure 1 shown in the structural diagram of the neural network processor, the neural network processor includes a control processing unit (Control Unit, CU), a control program storage unit (Program Memory, PM), a data storage unit (Control Memory, CM), a neural network processing unit (Neural Unit, NU), a neural network data storage unit (Data Memory, DM), and a DMA.

[0035] Among them, the control processing unit is used to control the neural network processing unit or the DMA to execute according to the execution notification sent by the main control processor.

[0036] Among them, the control program storage unit can be used to store code segments of operator operations including operator tasks, etc.

[0037] Among them, the data storage unit can be used to store data such as task descriptors including operator operation tasks.

[0038] Among them, the neural network processing unit can be used to execute operator operation tasks.

[0039] Among them, the neural network data storage unit can be a storage unit that stores the input / output data and weight parameters of the operator operation task when the neural network processing unit executes the operator operation task.

[0040] In one embodiment, referring to Figure 2 , a task scheduling method for a neural network processor is provided. By way of example and not limitation, as Figure 2 shown, the task scheduling method of this neural network processor may include the following steps:

[0041] Step 10:1, determine the task scheduling method according to the number of execution notifications of operator task blocks in the current queue, where the execution notification carries the task descriptor of the operator task block.

[0042] Among them, the queue can be a queue that caches the execution notifications of operator task blocks that need to be executed. By caching the execution notifications of operator task blocks that need to be executed in the queue and waiting, after the neural network processor takes away the execution notifications of operator task blocks from the queue, it then executes the corresponding tasks according to the task descriptors of the operator task blocks carried by the execution notifications.

[0043] Among them, the operator task block may be a control information block for performing the operator operation task of the neural network model.

[0044] Among them, the execution notification of the operator task block may be the execution notification of the operator task block sent by the main control processor of the terminal device through the communication unit.

[0045] It should be understood that the main control processor may be an ARM microprocessor (Advanced RISC Machines).

[0046] In a scenario, the operator operation task of the neural network model is compiled using the neural network processor tool chain. The main control processor loads the operator operation task onto the main control memory. Among them, the main control memory may be DDR. The main control processor prepares the input and / or output parameters of the operator operation task, and then configures the task descriptor of the operator operation task and writes it into the operator task block. The main control processor also configures the task descriptor of the operator task block according to the operator task block. The main control processor sends the execution notification of the operator task block to the neural network processor through the communication unit (MailBox). When sending the execution notification of the operator task block, the task descriptor of the operator task block is carried at the same time, and then the neural network processor is notified to process the operator task block. The neural network processor executes the corresponding task according to the task descriptor of the operator task block.

[0047] Among them, the number of execution notifications of the operator task block may be the number or count of the execution notifications of the operator task blocks inserted in the current queue.

[0048] Among them, the task scheduling method may be the scheduling method for executing the operator task blocks in the current queue. The task scheduling method may include serial scheduling and parallel scheduling, etc., and there may also be other scheduling methods.

[0049] It should be understood that when the number of execution notifications of the operator task blocks in the current queue is different, the corresponding task scheduling methods may be different or the same.

[0050] Furthermore, the corresponding task scheduling method can be determined as serial scheduling or parallel scheduling according to the number of execution notifications of the operator task blocks in the current queue. That is, in a possible implementation, step 101 includes:

[0051] When the number of execution notifications of the operator task blocks in the current queue is one, determine the task scheduling method as serial scheduling; when the number of execution notifications of the operator task blocks in the current queue is more than one, determine the task scheduling method as parallel scheduling.

[0052] Among them, serial scheduling can be an execution notice that only takes one operator task block from the current queue at a time, executes the operator task block corresponding to the execution notice, and after all subtasks of the operator tasks in the operator task block are completed, then determines the task scheduling method according to the number of execution notices of the operator task blocks in the current queue, and takes the execution notice of the operator task block from the current queue again according to the task scheduling method and executes it.

[0053] It should be understood that taking away can be understood as dequeuing, which can be an operation of deleting the execution notice of the operator task block from the current queue.

[0054] Among them, parallel scheduling can be taking away multiple operator task blocks to be executed from the current queue at a time, and after all subtasks of the operator task blocks in the multiple operator task blocks to be executed are completed, then determines the task scheduling method according to the number of execution notices of the operator task blocks in the current queue, and takes the execution notice of the operator task block from the current queue again according to the task scheduling method and executes it.

[0055] Step 102: Determine the operator task block to be executed according to the task scheduling method.

[0056] Among them, the operator task block to be executed can be the operator task block corresponding to the currently taken execution notice.

[0057] It should be understood that for different task scheduling methods, the number of operator task blocks to be executed can be different or the same.

[0058] Further, in the case of serial scheduling, the operator task block corresponding to the execution notice in the current queue can be determined as the operator task block to be executed. In the case of parallel scheduling, at least two operator task blocks corresponding to the execution notices in the current queue are used as the operator task blocks to be executed. That is, in a possible embodiment, step 102 includes:

[0059] In the case where the task scheduling method is serial scheduling, the operator task block corresponding to the execution notice in the current queue is used as the operator task block to be executed; in the case where the task scheduling method is parallel scheduling, at least two operator task blocks corresponding to the execution notices in the current queue are used as the operator task blocks to be executed.

[0060] It should be understood that the task scheduling method being serial scheduling can be determined when the number of execution notices in the current queue is one. Therefore, in the case where the task scheduling method is serial scheduling, usually there is exactly one execution notice in the current queue, and the operator task block corresponding to the execution notice in this queue is the operator task block to be executed.

[0061] It should be noted that when the task scheduling method is serial scheduling, after step 101 to obtain the number of execution notifications in the current queue, until step 102 to determine the operator task block to be executed, if a new execution notification is inserted into the current queue before that, the operator task block corresponding to the new execution notification is not regarded as the operator task block to be executed.

[0062] It should be understood that when the task scheduling method is parallel scheduling, according to the preset parallel quantity, the corresponding number of execution notifications can be taken, and the operator task blocks corresponding to the taken execution notifications are regarded as the operator task blocks to be executed.

[0063] It should be understood that the preset parallel quantity can be two, three, four, etc., and can be set according to the actual situation and scheduling requirements.

[0064] In one scenario, taking the preset parallel quantity as two as an example, when the task scheduling method is parallel scheduling, two execution notifications are taken at a time, which are the execution notifications of operator task block A and operator task block B respectively. Correspondingly, operator task block A and operator task block B are regarded as the operator task blocks to be executed.

[0065] Step 103, according to the task descriptor of the operator task block to be executed, control the direct memory access to move the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit.

[0066] It should be understood that according to the execution process of the operator operation task of the execution neural network model, the main control processor can be divided into the execution task of the operator task block and the operator operation task inside the operator task block. Among them, the execution task of the operator task block can include: the task descriptor transfer subtask, the task descriptor transfer end query subtask, and the operator task execution subtask. The operator operation task inside the operator task block can include: the task descriptor parsing subtask, the code segment transfer subtask, the code segment transfer end query subtask, and the operator operation execution subtask.

[0067] Among them, the task descriptor of the operator task block to be executed can be a control information block including important elements of the execution tasks of at least one operator task block.

[0068] It should be understood that the task descriptor of the operator task block to be executed may include the control information of each subtask in the execution task of the operator task block (i.e., the task descriptor transfer subtask, the task descriptor transfer end query subtask, and the operator task block execution subtask), such as: the address of the code segment of each subtask in the execution task of the operator task block and the parameters required to execute the code segment, etc. After obtaining the task descriptor of the operator task block, the neural network processor can control its own processing unit to execute each subtask in the execution task of the operator task block based on the task descriptor of the operator task block.

[0069] Among them, the task descriptor transfer subtask may be a task of transferring the task descriptor of the operator operation task.

[0070] It should be understood that the control information of the task descriptor transfer subtask may be the address of the code segment for transferring the task descriptor of the operator operation task and the parameters required to execute the code segment, etc., for example: the address for storing the task descriptor transfer code, the parameter address, and the source address and destination address of the task descriptor of the operator operation task. Among them, the address of the code segment is the address for storing the task descriptor transfer code and the parameter address, and the parameters required to execute the code segment are the source address and destination address of the task descriptor of the operator operation task.

[0071] Among them, the source address of the task descriptor of the operator operation task may be the position where the task descriptor of the operator operation task is stored before transfer.

[0072] Among them, the destination address of the task descriptor of the operator operation task may be the position where the task descriptor of the operator operation task is stored after transfer.

[0073] Among them, the task descriptor of the operator operation task may be a control information block including the important elements of the operator operation tasks within at least one operator task block.

[0074] Among them, the task descriptor transfer end query subtask may be a task of querying whether the task descriptor transfer is completed.

[0075] Among them, the operator task block execution subtask may be a task of executing the operator operation tasks within the operator task block.

[0076] It should be understood that inside the operator task block is the operator operation task, and the operator operation task is divided into, according to the execution process: the task descriptor parsing subtask, the code segment transfer subtask, the code segment transfer end query subtask, and the operator operation execution subtask. Thus, when the operator task block execution subtask starts, it correspondingly executes the operator operation task inside the operator task block. Based on this, when the operator task block execution subtask starts, it sequentially executes the task descriptor parsing subtask, the code segment transfer subtask, the code segment transfer end query subtask, and the operator operation execution subtask.

[0077] It should be understood that the control information of the task descriptor transfer end query subtask and the operator task block execution subtask also includes the address of the code segment corresponding to the task descriptor transfer end query subtask and the operator task block execution subtask, and the parameters required to execute this code segment, etc., which will not be exemplified here.

[0078] Among them, the task descriptor of the operator operation task can be a control information block including the important elements of at least one operator operation task inside the operator task block.

[0079] It should be understood that the task descriptor of the operator operation task can include the control information of each subtask (i.e., the task descriptor parsing subtask, the code segment transfer subtask, the code segment transfer end query subtask, and the operator operation execution subtask) in the operator operation task inside the operator task block, such as: the address of the code segment of each subtask in the operator operation task inside the operator task block and the parameters required to execute this code segment, etc.

[0080] Among them, the task descriptor parsing subtask can be a task that parses the task descriptor of the operator operation task, and parses the operator code segment information from the task descriptor of the operator operation task.

[0081] Among them, the code segment transfer subtask can be a task that transfers the code segment of the operator operation. It can determine the source address and destination address of the code segment of the operator operation according to the operator operation code segment information parsed in the task descriptor parsing subtask, and then transfer the code segment of the operator operation from the source address to the destination address.

[0082] Among them, the source address of the operator code segment is generally in the main control memory of the terminal device, and the destination address can be in the control program storage unit of the neural network processor.

[0083] Among them, the code segment transfer end query subtask can be a task that queries whether the code segment transfer is completed.

[0084] Among them, the operator operation execution subtask can be a task that executes the code segment of the operator operation.

[0085] It should be understood that when the operator task block executes a subtask, the operator operation task in the operator task block is executed. That is, when the operator task block executes a subtask, the task descriptor parsing subtask in the operator operation task is correspondingly executed.

[0086] Among them, Direct Memory Access (DMA) can copy data from one address space to another address space.

[0087] Furthermore, when the operator task block to be executed includes one operator task block, the task descriptor of the operator operation task in this operator task block is carried; when the operator task block to be executed includes at least two operator task blocks, the task descriptors of the operator operation tasks in each operator task block are carried in sequence. That is, in a possible embodiment, step 103 includes:

[0088] When the operator task block to be executed includes one operator task block, according to the task descriptor of the operator task block to be executed, control the direct memory access to carry the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit; when the operator task block to be executed includes at least two operator task blocks, according to the task descriptor of the operator task block to be executed, control the direct memory access to carry the task descriptors of the operator operation tasks in each operator task block from the main control memory to its own data storage unit in sequence.

[0089] It should be understood that the task scheduling method is serial scheduling, and correspondingly, the operator task block to be executed includes one operator task block.

[0090] Among them, when the operator task block to be executed includes one operator task block, the processing control unit of the neural network processor directly initiates to the direct memory access to carry the task descriptor of the operator operation task in this operator task block. The direct memory access correspondingly executes the task descriptor carrying subtask, and then the processing control unit of the neural network processor executes the task descriptor carrying end query subtask to query whether the direct memory access has finished carrying. When the processing control unit of the neural network processor queries that the task descriptor carrying has ended, it controls the operator task block execution subtask to start.

[0091] In a scenario, such as Figure 3Schematic diagram of the task execution order under the serial scheduling method shown. The task scheduling method is serial scheduling. First, the operator task block A is determined as the operator task block to be executed. After the processing control unit of the neural network processor initiates the task of transferring the task descriptor of the operator operation task in the operator task block A to the direct memory access, the processing control unit of the neural network processor executes the task descriptor transfer end query subtask. After the task descriptor transfer is completed, the operator task block execution subtask of the operator operation task in the operator task block A is executed. After the execution is completed, the operator task block B is determined as the operator task block to be executed. After the processing control unit of the neural network processor initiates the task of transferring the task descriptor of the operator operation task in the operator task block B to the direct memory access, the processing control unit of the neural network processor executes the task descriptor transfer end query subtask. After the task descriptor transfer is completed, the operator task block execution subtask of the operator operation task in the operator task block B is executed.

[0092] It should be understood that the task scheduling method is parallel scheduling. Correspondingly, the operator task block to be executed includes at least two operator task blocks.

[0093] Among them, in the case where the operator task block to be executed includes at least two operator task blocks, it is necessary to transfer the task descriptors of the operator operation tasks in each operator task block. The processing control unit of the neural network processor can sequentially control the direct memory access to transfer the corresponding task descriptors of the operator operation tasks according to the writing order of the execution notifications of each operator task block.

[0094] In one scenario, taking the operator task block to be executed including the operator task block A and the operator task block B as an example, the execution notification of the operator task block A is first written into the queue, and the execution notification of the operator task block B is written after the execution notification of the operator task block A is written. The processing control unit of the neural network processor can sequentially initiate the task descriptor transfer subtasks of the operator task block A and the operator task block B to the direct memory access. The direct memory access sequentially executes the task descriptor transfer subtasks of the operator task block A and the operator task block B, so as to sequentially transfer the task descriptors of the operator operation tasks in the operator task block A and the operator task block B from the main control memory to its own data storage unit.

[0095] It should be understood that when the processing control unit of the neural network processor initiates a task descriptor transfer subtask to the direct memory access, it will occupy resources. After initiating a task descriptor transfer subtask to the direct memory access, the processing control unit of the neural network processor can continue to process the tasks that need to be processed. According to the tasks that need to be executed corresponding to the task scheduling method, it can execute the code segment transfer end query subtask, or initiate another task descriptor transfer subtask, etc.

[0096] It should be understood that the main control processor sends an execution notice for the operator task block in order to enable the neural network processor to finally execute the code segment of the operator operation task in the operator task block and output the corresponding result. Therefore, before executing the code segment of the operator operation task in the operator task block, it is necessary to sequentially execute the task descriptor transfer subtask, the task descriptor transfer end query subtask, the operator task block execution subtask, the task descriptor parsing subtask, the code segment transfer subtask, the code segment transfer end query subtask, and the operator operation execution subtask. When the task descriptor transfer subtask or the code segment transfer subtask is executed, it is necessary to wait for the transfer to end before entering the next subtask. If multiple operator task blocks need to be processed, they are processed in a serial scheduling manner, and only one operator task block is processed at a time. After the processing control unit of the neural network processor initiates the transfer of the task descriptor or code segment of the operator task block to the direct memory access, it only executes the query on whether the transfer of the operator task block is completed, and can only execute the next subtask until the transfer is found to be completed, wasting the resources of the neural network processor and having a low task processing efficiency. Therefore, when multiple operator task blocks need to be processed, they can be processed in a parallel scheduling manner. By using the time waiting for the transfer by taking multiple operator task blocks as the operator task blocks to be executed, first controlling the transfer of multiple operator task blocks, and then sequentially executing the query on whether the transfer of the operator task blocks is completed, the probability of query idling is reduced, and the scheduling efficiency is greatly improved.

[0097] Among them, the processing control unit of the neural network processor controls the direct memory access to execute the task descriptor transfer subtasks of operator task block A and operator task block B, carrying the source address and destination address of the task descriptors of the operator operation tasks in operator task block A and operator task block B. Furthermore, the direct memory access copies the task descriptors of the operator operation tasks from the source to the destination address.

[0098] It should be understood that the direct memory access first transfers the task descriptors of the operator operation tasks in operator task block A, and after the transfer is completed, it then transfers the task descriptors of the operator operation tasks in operator task block B.

[0099] It should be understood that the processing control unit of the neural network processor also needs to execute the task descriptor transfer end query subtask, and after the task descriptor transfer is completed, it then executes the operator task block execution subtask. Therefore, as Figure 4Schematic diagram of the task execution order under the parallel scheduling method shown. After the processing control unit of the neural network processor can sequentially initiate the task descriptor transfer subtasks of operator task block A and operator task block B to the direct memory access, it then executes the task descriptor transfer end query subtask of operator task block A. When it is queried that the task descriptor transfer of the operator operation task in operator task block A is completed, the processing control unit of the neural network processor controls the start of the operator task block execution subtask of operator task block A. After the processing control unit of the neural network processor controls the start of the operator task block execution subtask of operator task block A, it then executes the task descriptor transfer end query subtask of operator task block B. When it is queried that the task descriptor transfer of the operator operation task in operator task block B is completed, the processing control unit of the neural network processor controls the start of the operator task block execution subtask of operator task block B.

[0100] Furthermore, the generation of the task descriptor can be generated in the main control processor. In a possible embodiment, the method for the main control processor to generate the task descriptor of the task includes:

[0101] According to the execution process of the task, each subtask is divided, where the task is an operator task block or an operator operation task; according to the function call relationship of each subtask, the address of the code segment and the parameter address of each subtask are determined; using the address of the code segment and the parameter address of each subtask, the task descriptor control word of each subtask is respectively generated; according to the task descriptor control word and parameters of each subtask, the task descriptor of the task is generated by writing into the task descriptor.

[0102] Among them, the execution process of the task can be the processing process of the task when the task is executed.

[0103] In an example, taking the execution process of the operator task block as an example, it is necessary to first transfer the task descriptor of the operator operation task, and then query whether the task descriptor transfer is completed. Only after the task descriptor transfer is completed can the corresponding operator operation task be executed. Therefore, the execution task of the operator task block can be divided into a task descriptor transfer subtask, a task descriptor transfer end query subtask, and an operator task block execution subtask.

[0104] In an example, taking the execution process of the operator operation task as an example, it is necessary to first parse the task descriptor of the operator operation task, then transfer the code segment of the operator operation, then query whether the code segment transfer is completed, and only after the code segment transfer is completed can the code segment of the operator operation be executed to complete the operator operation. Therefore, the operator operation task is divided into: a task descriptor parsing subtask, a code segment transfer subtask, a code segment transfer end query subtask, and an operator operation execution subtask.

[0105] Among them, the function call of a subtask may include passing the parameters of the subtask and a part of the code segment to another part.

[0106] In a possible embodiment, the main control processor stores the addresses of the code segments of each subtask and the parameter addresses.

[0107] It should be understood that the main control processor can determine the addresses of the code segment and the parameter addresses of the subtask according to the function call of the subtask.

[0108] Among them, the task descriptor control word may be a control character describing the subtask.

[0109] Among them, the address of the code segment may be the storage location information of the code segment of the subtask.

[0110] Among them, the parameter address may be the storage location of the parameters of the subtask.

[0111] Among them, the parameters of the subtask may be the parameters required when executing the subtask.

[0112] In a possible embodiment, the code segments of common subtasks (such as: task descriptor transfer subtask, task descriptor transfer end query subtask, operator task execution subtask, task descriptor parsing subtask, code segment transfer subtask, code segment transfer end query subtask) are placed in the low address of the NPU PM space and uniformly stored in the static image of NPU Firmware. For the code segments of the operator operations of the neural network model stored in the DDR, a code segment transfer subtask can be added to copy the code segments of the operator operations to the PM for execution. If the size of the PM space does not exceed 1MB, then only 18 bits are needed to describe the address information of the tasks on the PM, and the maximum space of the task descriptor of the CM does not exceed 64KB. Therefore, 14 bits are used to describe the parameter address information of the operator operations in the CM. Finally, 4 bytes can be used to describe the control information of the operator operations. Among them, the encoding of the control information of the operator operations can be encoded with reference to the compressed control information encoding table in Table 1.

[0113] Table 1 Compressed Control Information Encoding Table

[0114]

[0115] Suppose it is necessary to execute the function PM_DMA_Task_Start at the PM address 0x1000, and the parameters of this function are stored at the position 0x2000 in the Task Block of the CM. Then the descriptor control word generated by the ARM for this function call is 0x10002000, and the NPU parses it as a call to the function code segment at PM[0x4000], and the function passing parameter is CM[BLOCK_ADDR + 0x8000].

[0116] Further, when the task type of the subtask is a transfer task, the parameters of the subtask are compressed and then written into the task descriptor. That is, in a possible embodiment, when the task type of the subtask is a transfer task, the parameters of the subtask are compressed according to the storage space to which the parameters of the subtask belong, and the compressed parameters are written into the task descriptor.

[0117] Among them, the transfer task can be a task that performs a DMA transfer operation, such as: the task descriptor transfer subtask, the code segment transfer subtask, and so on.

[0118] Among them, the parameters corresponding to the subtask with the task type of transfer task can be information such as the source address, target address, and transfer length during DMA transfer.

[0119] Among them, the storage space can be the DDR space, the shared (Shared) RAM space, the PM space, the CM space, and so on.

[0120] Among them, according to the storage space to which the parameters of the subtask belong, it can be the space to which the source address and target address during DMA transfer belong.

[0121] It should be understood that for the source address and target address during DMA transfer, if the general absolute address addressing is adopted, then each address requires 64 bits (bit), occupying a large amount of CM space. Here, we stipulate the address range of the NPU's DMA transfer (PM transfer, CM transfer) and compress the parameters of the task type of transfer task.

[0122] In a possible embodiment, the sizes of the PM space and CM space of the NPU do not exceed 1MB, the code segment for the operator operations of the NPU stored in the DDR is limited within 64MB. A first identifier can be defined to represent the starting address of the NPU's code segment in the DDR. For example, the first identifier is: DDR_CODE_BASE. The SharedRAM for storing the task descriptors of the NPU operator operation tasks is limited to 256KB. A second identifier can be defined to represent the starting address of the task descriptors of the NPU operator operation tasks in the SharedRAM. For example, the second identifier is: SHARED_RAM_TASK_BASE. With 4-byte (Byte) alignment, the parameter definition for the task type of the transfer task is 64-bit unsigned integer (uint64). Referring to Table 2, various data types are predefined to represent the source address, destination address, transfer length, etc. during DMA transfer. Then, the parameters of the operator operation are encoded according to the parameter encoding table in Table 2.

[0123] Table 2 Parameter Encoding Table

[0124]

[0125] In an example, the parameter of the subtask with the task type of the transfer task is: 0x4A00008800010000. This parameter corresponds to SRC_TYPE being 1, SRC_ADDR being 0xA0000, DST_TYPE being 2, DST_ADDR being 0x8000, and COPY_LEN being 0x1000, which is interpreted as a DMA transfer from the source address of SHARED_RAM_TASK_BASE + 0xA0000 * 4 to CM_BASE + 0x8000 * 4, with a transfer length of 0x1000 bytes.

[0126] It should be understood that the content to be described in the parameters of the subtask is: when transferring from the source address of SHARED_RAM_TASK_BASE + 0xA0000 * 4 to CM_BASE + 0x8000 * 4 with a transfer length of 0x1000 bytes, before parameter compression: the source address is 64-bit, the destination address is 64-bit, and the transfer length is 16-bit, which altogether requires 144 bits. After compression according to the storage space to which the parameters of the subtask belong, the compressed parameters only need to occupy 64 bits.

[0127] It should be understood that in the case where the task type of the subtask is the transfer task, the parameters of the subtask are compressed according to the storage space to which the parameters of the subtask belong, and the compressed parameters are written into the task descriptor, which reduces the space occupation and can also improve the addressing efficiency.

[0128] In a possible embodiment, the data stored in the task descriptor includes: the number information of subtasks, the parameters of each subtask, and the task descriptor control word of each subtask. Among them, the task descriptor control word includes the address of the code segment and the parameter address.

[0129] In a possible embodiment, the way for the main control processor to write the task descriptor control word and parameters of each subtask into the task descriptor includes:

[0130] Writing the preset bits starting from the task descriptor as the number information of subtasks; determining the initial current address decreasing position and the initial current address increasing position of the task descriptor according to the length of the task descriptor and the starting preset bits; when adding a subtask to the task descriptor, adding the parameters of the subtask from the low address direction of the task descriptor according to the current address decreasing position of the task descriptor; updating the current address decreasing position of the task descriptor according to the number of parameters of the added subtask and the size of the parameter type; adding the address of the code segment and the parameter address of the subtask from the current address increasing position to the high address direction according to the current address increasing position of the task descriptor; updating the current address increasing position of the task descriptor according to the parameter address of the subtask; updating the number information of subtasks in the task descriptor.

[0131] It should be understood that the task descriptor can include the description task descriptor control words and parameters of multiple subtasks, so as to realize the scheduling of multiple tasks combined together.

[0132] Among them, the preset bits can be determined according to the space occupied by the number information of subtasks or actual needs, such as 4 bits (byte).

[0133] As an example, as Figure 5 shown in the structural schematic diagram of the task descriptor, the starting 4 bytes are used to write the number information of the subtasks, and the addresses of the code segments, parameter addresses, and parameters of subtask 1 and subtask 2 are written in this task descriptor.

[0134] Among them, the initial current address decreasing position of the task descriptor can be the tail of the task descriptor as the initial current address decreasing position. For example, if the length of the task descriptor is 4KB, the initial current address decreasing position of the task descriptor can point to the tail of the task descriptor.

[0135] Among them, the initial current address increasing position of the task descriptor can be the next position of the starting preset bits. For example, if the starting preset bits are 4 bytes, the initial current address increasing position can point to the 4-byte position of the task descriptor.

[0136] Among them, updating the number information of subtasks in the task descriptor can be, for example, adding a subtask, and the number information of subtasks is incremented by 1, and so on.

[0137] In a possible embodiment, when the remaining space length of the task descriptor is not enough to add a subtask, then it is necessary to reapply for the space of the task descriptor for adding subtasks.

[0138] It should be understood that the main control processor configures the task descriptor in the SharedRAM space, and the NPU CU Firmware needs to copy the task descriptor to the NPU CM space through DMA for use. There is an offset of the parameter address of the subtask inside the task descriptor relative to the current task descriptor. Therefore, when the NPU CU Firmware parses the task descriptor, it needs to update the actual parameter position of the subtask of the NPU CU in the CU according to the position of the current task descriptor in the CM.

[0139] The definition of the task descriptor stipulates the parameter data format of the subtask. The parameters for the NPU to execute the subtask directly use the parameter data structure of the task descriptor, which reduces the parameter parsing and copying of the task descriptor compared to the original scheme.

[0140] In a possible embodiment, the parameters in the task descriptor use 32-bit storage space to store a single parameter type and the corresponding value of the parameter.

[0141] In other possible embodiments, the parameters in the task descriptor can also use storage spaces of sizes such as 64 bits and 128 bits to store a single parameter type and the corresponding value of the parameter, which is not limited here.

[0142] As an example, taking the execution form of the parameters of the subtask defined as int(op_func)(int32_t cm_para_pos) as an example, the input / output parameters are stored at the starting position of NPU CM[cm_para_pos], and the parameter content is defined and agreed by the op_func operator, which is equivalent to passing the pointer type of &NPU_CM[cm_para_pos] in C language as an example. The NPU CU Firmware software does not parse the parameters of op_func, but the op_func internally parses the parameters and judges the legality; if op_func runs normally, it returns 0, otherwise it returns an abnormal value.

[0143] In order to support the exception check of the parameter type of the subtask, 32-bit Meta Type+Value can be used to describe a single parameter type and the corresponding value of the parameter. Among them, the parameter type can be used for parameter exception detection in the running state. Among them, the parameter type of the subtask can be encoded with reference to the parameter type encoding table of the subtask in Table 3.

[0144] Table 3 Parameter Type Encoding Table for Sub - tasks

[0145]

[0146] In a possible embodiment, the parameters of a sub - task include a parameter type and a value corresponding to the parameter. Correspondingly, according to the current address decreasing position of the task descriptor, the parameters of the sub - task are incremented from the low - address direction of the task descriptor, including:

[0147] Determine the type of the parameter according to the input - output relationship between the parameter of the sub - task and the parameters of other sub - tasks; determine the value corresponding to the parameter according to the type of the parameter; according to the current address decreasing position of the task descriptor, increment the parameter type and the value corresponding to the parameter of the sub - task from the low - address direction of the task descriptor.

[0148] Among them, if the parameter of a sub - task is the parameter output by a certain sub - task, it can be determined that the parameter of this sub - task is of reference type.

[0149] Among them, if there is no input - output relationship between the parameter of a sub - task and the parameters of other sub - tasks, it can be determined that the parameter of this sub - task is of non - reference type.

[0150] Among them, the type of the parameter can include reference type, non - reference type, and so on.

[0151] In a possible embodiment, determine the value corresponding to the parameter of the sub - task according to the type of the parameter of the sub - task, including:

[0152] When the type of the parameter of the sub - task is of reference type, the value corresponding to the parameter of the sub - task is the address of the parameter being referenced; when the type of the parameter is of non - reference type, the value corresponding to the parameter is the input and / or output value of the sub - task.

[0153] Among them, the address of the parameter being referenced can be the address storing the parameter being referenced.

[0154] It should be understood that if the parameter of this sub - task is the parameter output by a certain sub - task, then the value of the parameter being referenced can be directly obtained from the address of the parameter being referenced as the value required when the sub - task is executed, thus avoiding the inefficiency caused by repeated transfer and the waste of space resources caused by repeated storage.

[0155] As an example, parameter a is at the CM

[50] position, with an int type value of 10000, and parameter b is at the CM

[100] position, with a REF int type and a value of 50, indicating that the value of b references a. When the sub - task parses parameter b and detects it as a REF type, then it will dereference and read the value 10000 at CM

[50] .

[0156] The reference type can solve the problem of parameter passing between subtasks in CM. For example, Figure 6 take the execution flow diagrams of the operator tasks op_ave and op_binary shown as an example. op_ave calculates the average value of the grayscale image of img_1 and stores the output as ave_value, with the type of int8. op_binary calculates the binary image of img_2 based on ave_value and outputs it to img_3. Here, the input of op_binary references the output ave_value of the previous-level op_ave. When the ARM is configured, the referenced value can be calculated. When the NPU CU calls op_binary, it parses the input parameters and dereferences to obtain the actual ave_value. Thus, the structural diagrams of the task descriptors corresponding to the operator tasks op_ave and op_binary are as Figure 7 shown, and the parameters of the operator task op_binary reference the parameters of the operator task op_ave.

[0157] Step 104, when the transfer of the task descriptor of the operator operation task in the to-be-executed operator task block is completed, according to the task descriptor of the operator operation task in the to-be-executed operator task block, execute the subtasks of the operator operation task in the to-be-executed operator task block.

[0158] In a possible embodiment, the processing unit of the neural network processor queries whether the transfer of the query task descriptor is completed by executing the code segment transfer end query subtask. When it is queried that the transfer of the task descriptor of the operator operation task in the to-be-executed operator task block is completed, start the operator task block to execute the subtasks, and then execute the subtasks of the operator operation task in the to-be-executed operator task block according to the task descriptor of the operator operation task in the to-be-executed operator task block.

[0159] It should be understood that starting the operator task block to execute the subtasks corresponds to executing each subtask of the operator operation task, that is, the task descriptor parsing subtask, the code segment transfer subtask, the code segment transfer end query subtask, and the operator operation execution subtask.

[0160] Among them, when the task scheduling method is serial scheduling, there is only one to-be-executed operator task block, and each subtask of the operator operation task in the to-be-executed operator task block can be executed in sequence.

[0161] In one example, when the task scheduling method is parallel scheduling, taking the execution notification of operator task block A being written into the queue first and the execution notification of operator task block B being written into the queue after the execution notification of operator task block A as an example, the processing control unit of the neural network processor sequentially initiates the task descriptor transfer subtasks of operator task block A and operator task block B to the direct memory access according to the writing order of the execution notifications of each operator task block. Correspondingly, the direct memory access executes the task descriptor transfer subtasks of operator task block A and operator task block B according to the initiation order of the processing control unit of the neural network processor. That is, the direct memory access first transfers the task descriptor of operator task block A, and after the transfer is completed, it transfers the task descriptor of operator task block B. Thus, after the processing control unit of the neural network processor sequentially initiates the task descriptor transfer subtasks of operator task block A and operator task block B to the direct memory access, it can first execute the task descriptor transfer end query subtask corresponding to operator task block A. When it is queried that the task descriptor transfer of operator task block A is completed, it starts the operator task execution subtask of operator task block A, and then jumps to execute the task descriptor parsing subtask to parse out the operator code segment information, and then controls the direct memory access to perform the code segment transfer of the operator operation of operator task block A. While waiting for the code segment transfer of the operator operation of operator task block A, if at this time, the task descriptor transfer of operator task block B is completed, it can execute the operator task execution subtask of operator task block B, and then jumps to execute the task descriptor parsing subtask to parse out the operator code segment information, and then controls the direct memory access to perform the code segment transfer of the operator operation of operator task block B. While waiting for the code segment transfer of the operator operation of operator task block B, if the code segment transfer of the operator operation of operator task block A is completed, it executes the code segment of the operator operation of operator task block A. After the execution is completed, if the code segment transfer of the operator operation of operator task block B is completed, it executes the code segment of the operator operation of operator task block B.

[0162] It should be understood that in the case where multiple operator task blocks need to be processed, it can be processed in a parallel scheduling manner. By taking multiple operator task blocks as the operator tasks to be executed, the waiting transfer time can be utilized. First, the transfer of multiple operator task blocks is controlled, and then the query of whether the transfer of each operator task block is completed is sequentially executed, reducing the probability of query idle rotation and greatly improving the scheduling efficiency.

[0163] The task scheduling method of the above neural network processor determines the task scheduling method according to the number of execution notifications of operator task blocks in the current queue, where the execution notification carries the task descriptor of the operator task block. Then, according to the task scheduling method, the operator task block to be executed is determined. Further, according to the task descriptor of the operator task block to be executed, the direct memory access is controlled to move the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit. Thus, when the transfer of the task descriptor of the operator operation task in the operator task block to be executed is completed, the subtasks of the operator operation task in the operator task block to be executed are executed according to the task descriptor of the operator operation task in the operator task block to be executed. Therefore, according to the number of execution notifications of operator task blocks in the current queue, the task scheduling method is determined, the operator task block to be executed is determined according to the task scheduling method, and then the subtasks of the operator operation task in the operator task block to be executed are executed according to the task descriptor of the operator task block to be executed and the task descriptor of the operator operation task in the operator task block to be executed, thereby improving the execution efficiency of the operator operation task.

[0164] It should be understood that although Figure 2 the steps in the flowchart of Figure 2 are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover,

[0165] In one embodiment, as Figure 8 shown, a task scheduling device for a neural network processor is provided. This device can be a software module or a hardware module, or a combination of both to form a part of a computer device. Specifically, the device includes:

[0166] A scheduling method confirmation module 610, configured to determine a task scheduling method according to the number of execution notifications of operator task blocks in the current queue, where the execution notification carries the task descriptor of the operator task block;

[0167] An operator task block to be executed confirmation module 620, configured to determine the operator task block to be executed according to the task scheduling method;

[0168] A descriptor transfer module 630, configured to control direct memory access to transfer the task descriptor of the operator operation task in the to-be-executed operator task block from the main control memory to its own data storage unit according to the task descriptor of the to-be-executed operator task block;

[0169] An operator operation task execution module 640, configured to execute the subtasks of the operator operation task in the to-be-executed operator task block according to the task descriptor of the operator operation task in the to-be-executed operator task block when the transfer of the task descriptor of the operator operation task in the to-be-executed operator task block is completed.

[0170] In one embodiment, the scheduling mode confirmation module 610 is further configured to: determine that the task scheduling mode is serial scheduling when the number of execution notifications of the operator task blocks in the current queue is one; determine that the task scheduling mode is parallel scheduling when the number of execution notifications of the operator task blocks in the current queue is more than one.

[0171] In one embodiment, the to-be-executed operator task block confirmation module 620 is further configured to, when the task scheduling mode is serial scheduling, use the operator task block corresponding to the execution notification in the current queue as the to-be-executed operator task block; when the task scheduling mode is parallel scheduling, use at least two operator task blocks corresponding to the execution notifications in the current queue as the to-be-executed operator task blocks.

[0172] In one embodiment, the descriptor transfer module 630 is further configured to, when the to-be-executed operator task block includes one operator task block, control direct memory access to transfer the task descriptor of the operator operation task in the to-be-executed operator task block from the main control memory to its own data storage unit; when the to-be-executed operator task block includes at least two operator task blocks, control direct memory access to sequentially transfer the task descriptors of the operator operation tasks in each operator task block from the main control memory to its own data storage unit.

[0173] The task scheduling device of the above neural network processor determines the task scheduling method according to the number of execution notifications of operator task blocks in the current queue. Among them, the execution notification carries the task descriptor of the operator task block. Then, according to the task scheduling method, the operator task block to be executed is determined. Further, according to the task descriptor of the operator task block to be executed, the direct memory access is controlled to transfer the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit. Thus, when the transfer of the task descriptor of the operator operation task in the operator task block to be executed is completed, the subtasks of the operator operation task in the operator task block to be executed are executed according to the task descriptor of the operator operation task in the operator task block to be executed. Therefore, according to the number of execution notifications of operator task blocks in the current queue, the task scheduling method is determined. According to the task scheduling method, the operator task block to be executed is determined. Further, according to the task descriptor of the operator task block to be executed and the task descriptor of the operator operation task in the operator task block to be executed, the subtasks of the operator operation task in the operator task block to be executed are executed, thereby improving the execution efficiency of the operator operation task.

[0174] For the specific limitations of the task scheduling device of the neural network processor, reference can be made to the limitations of the task scheduling method of the neural network processor in the above text, which will not be elaborated here. Each module in the task scheduling device of the above neural network processor can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0175] Figure 9 It is a schematic structural diagram of a terminal device provided in an embodiment of the present application. As Figure 9 shown, the terminal device 700 in this embodiment includes: at least one main control processor 710 ( Figure 9 only one is shown here), a double data rate synchronous dynamic random access memory 720, a neural network processor 730, and a computer program 721 stored in the double data rate synchronous dynamic random access memory 720 and operable on the at least one main control processor 710. When the main control processor 710 executes the computer program 721, it controls the neural network processor 730 to implement the steps in the embodiment of the above task scheduling method of the neural network processor.

[0176] The terminal device 700 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a main control processor 710 and a main control memory 720. Those skilled in the art can understand, Figure 9This is only an example of the terminal device 700, which does not limit the terminal device 700. It may include more or fewer components than those shown in the figure, or combine some components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0177] The so-called main control processor 710 may be a central processing unit (CPU), or may be a microcontroller (Advanced RISC Machines, ARM). The main control processor 710 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0178] The double data rate synchronous dynamic random access memory 720 may be an internal storage unit of the terminal device 700 in some embodiments. The double data rate synchronous dynamic random access memory 720 may also be an external storage device of the terminal device 700 in some other embodiments. For example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 700. Further, the double data rate synchronous dynamic random access memory 720 may also include both an internal storage unit and an external storage device of the terminal device 700. The double data rate synchronous dynamic random access memory 720 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program, etc. The double data rate synchronous dynamic random access memory 720 may also be used to temporarily store data that has been output or will be output.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0180] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0181] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0182] In the embodiments provided in this application, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of the above-mentioned module or unit is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0183] The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, in each embodiment of the present application, each functional unit may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0185] If the above integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, all or part of the processes in the above-described embodiment methods of the present application may also be completed by a computer program instructing related hardware. The computer program may be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described method embodiments may be implemented. Among them, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0186] All or part of the processes in the above-described embodiment methods of the present application may also be completed by a computer program product. When the computer program product runs on a terminal device, the terminal device can execute the steps in the above-described method embodiments.

[0187] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A task scheduling method for a neural network processor, characterized in that Including: Determine a task scheduling method according to the number of execution notifications of operator task blocks in the current queue, where the execution notification carries a task descriptor of the operator task block; Determine an operator task block to be executed according to the task scheduling method; According to the task descriptor of the operator task block to be executed, control direct memory access to transfer the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit; the data stored in the task descriptor includes: the number information of subtasks, the parameters of each subtask, and the task descriptor control word of each subtask, where the task descriptor control word includes the address of the code segment and the parameter address; In the case where the transfer of the task descriptor of the operator operation task in the operator task block to be executed is completed, execute the subtasks of the operator operation task in the operator task block to be executed according to the task descriptor of the operator operation task in the operator task block to be executed; The generation method of the task descriptor includes: dividing each subtask according to the execution process of the task, where the task is an operator task block or an operator operation task; determining the address of the code segment and the parameter address of each subtask according to the function call of each subtask; generating the task descriptor control word of each subtask by using the address of the code segment and the parameter address of each subtask; generating the task descriptor of the task according to the task descriptor control word and parameters of each subtask and writing them into the task descriptor; The method of writing the task descriptor control word and parameters of the subtask into the task descriptor includes: using a preset bit at the start of the task descriptor to write the number information of subtasks; determining the initial current address decreasing position and the initial current address increasing position of the task descriptor according to the length of the task descriptor and the preset bit at the start; when adding a subtask to the task descriptor, adding the parameters of the subtask from the low address direction of the task descriptor according to the current address decreasing position of the task descriptor; updating the current address decreasing position of the task descriptor according to the number of parameters of the added subtask and the size of the parameter type; adding the address of the code segment and the parameter address of the subtask from the current address increasing position to the high address direction according to the current address increasing position of the task descriptor; updating the current address increasing position of the task descriptor according to the parameter address of the subtask; updating the number information of subtasks in the task descriptor.

2. The task scheduling method of the neural network processor according to claim 1, characterized in that The determining the task scheduling method according to the number of execution notifications of operator task blocks in the current queue includes: In the case where the number of execution notifications of operator task blocks in the current queue is one, determine that the task scheduling method is serial scheduling; In the case where the number of execution notifications of operator task blocks in the current queue is more than one, determine that the task scheduling method is parallel scheduling.

3. The task scheduling method of the neural network processor according to claim 2, wherein, The determining the operator task block to be executed according to the task scheduling method includes: In the case where the task scheduling method is serial scheduling, use the operator task block corresponding to the execution notification in the current queue as the operator task block to be executed; When the task scheduling method is parallel scheduling, at least two execution notifications in the current queue are used as the operator task blocks to be executed corresponding to the operator task blocks.

4. The task scheduling method of the neural network processor according to claim 3, wherein The controlling the direct memory access to transfer the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit according to the task descriptor of the operator task block to be executed includes: When the operator task block to be executed includes one operator task block, controlling the direct memory access to transfer the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit according to the task descriptor of the operator task block to be executed; When the operator task block to be executed includes at least two operator task blocks, controlling the direct memory access to sequentially transfer the task descriptors of the operator operation tasks in each operator task block from the main control memory to its own data storage unit according to the task descriptor of the operator task block to be executed.

5. The task scheduling method of the neural network processor according to any one of claims 1-4, characterized in that, When the task type of the subtask is a transfer task, compressing the parameters of the subtask according to the storage space to which the parameters of the subtask belong, and writing the compressed parameters into the task descriptor.

6. The task scheduling method of the neural network processor according to any one of claims 1-4, characterized in that, The parameters of the subtask include a parameter type and a value corresponding to the parameter. The adding the parameters of the subtask from the low address direction of the task descriptor according to the decreasing position of the current address of the task descriptor includes: Determining the type of the parameters of the subtask according to the input-output relationship between the parameters of the subtask and the parameters of other subtasks; Determining the value corresponding to the parameters of the subtask according to the type of the parameters of the subtask; Adding the parameter type and the value corresponding to the parameter of the subtask from the low address direction of the task descriptor according to the decreasing position of the current address of the task descriptor.

7. A task scheduling device for a neural network processor, characterized in that, Includes: A scheduling method confirmation module, configured to determine a task scheduling method according to the number of execution notifications of the operator task blocks in the current queue, where the execution notification carries a task descriptor of the operator task block; An operator task block to be executed confirmation module, configured to determine an operator task block to be executed according to the task scheduling method; A descriptor transfer module, configured to control the direct memory access to transfer the task descriptor of the operator operation task in the operator task block to be executed from the main control memory to its own data storage unit according to the task descriptor of the operator task block to be executed; the data stored in the task descriptor includes: the number information of the subtasks, the parameters of each subtask, and the task descriptor control word of each subtask, where the task descriptor control word includes the address of the code segment and the parameter address; An operator operation task execution module, configured to execute the subtasks of the operator operation task in the operator task block to be executed according to the task descriptor of the operator operation task in the operator task block to be executed when the transfer of the task descriptor of the operator operation task in the operator task block to be executed is completed. Among them, the generation method of the task descriptor includes: dividing each sub-task according to the execution process of the task, where the task is an operator task block or an operator operation task; determining the addresses of the code segments and parameter addresses of each sub-task according to the function calls of each sub-task; using the addresses of the code segments and parameter addresses of each sub-task to generate task descriptor control words for each sub-task respectively; generating the task descriptor of the task according to the task descriptor control words and parameters of each sub-task and writing them into the task descriptor. The method of writing the task descriptor control word and parameters of the sub-task into the task descriptor includes: using a preset bit at the start of the task descriptor to write the number information of the sub-tasks; determining the initial current address decreasing position and the initial current address increasing position of the task descriptor according to the length of the task descriptor and the preset bit at the start; when adding a sub-task to the task descriptor, adding the parameters of the sub-task from the low address direction of the task descriptor according to the current address decreasing position of the task descriptor; updating the current address decreasing position of the task descriptor according to the number of parameters of the added sub-task and the size of the parameter type; adding the address of the code segment and the parameter address of the sub-task from the current address increasing position to the high address direction according to the current address increasing position of the task descriptor; updating the current address increasing position of the task descriptor according to the parameter address of the sub-task; updating the number information of the sub-tasks in the task descriptor.

8. A terminal device, comprising a memory, a processor, a neural network processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1 to 6 is implemented.

Images