Task execution method, compilation method, chip and device of artificial intelligence model

Abstract

This application belongs to the field of computer technology and provides a task execution method, compilation method, chip, and device for an artificial intelligence model. The application first requires compiling the tasks to be processed by the artificial intelligence model according to a preset compilation method, resulting in multiple sub-tasks. This allows each task to be processed to receive loading and scheduling from the NPU at a moderately granular scheduling unit. Specifically, this application uses the sub-tasks as the basic scheduling unit of the NPU, thereby achieving resource scheduling at a moderate granularity that balances scheduling complexity and task switching flexibility. When encountering urgent tasks, the task switching cost of the NPU is significantly reduced; thus, in the event of sudden changes in the target application running on the computer device, the response efficiency and resource utilization of the NPU can be effectively improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a task execution method, compilation method, chip and device for an artificial intelligence model. Background Technology

[0002] When a computer device runs application software related to artificial intelligence (AI) technology, the CPU is responsible for sending inference tasks from various AI models to a task queue. The NPU executes model inference according to task priority, completing a series of tasks sequentially. During application software execution, the CPU may temporarily increase the priority of certain tasks in the task queue or insert high-priority tasks, allowing the NPU to handle these urgent tasks first.

[0003] In the current technological stage, the first conventional scheduling strategy uses a complete neural network model as the basic scheduling unit. That is, the NPU will only continue processing the next highest-priority task in the task queue after completing a full model inference task. When faced with urgent tasks, it either abandons the currently executing task and executes the high-priority urgent task, resuming the previously abandoned task after the urgent task is completed; or it continues executing the current task, processing the urgent task only after its completion. Both approaches have significant drawbacks: the former leads to wasted NPU computing power; the latter significantly increases the processing time for urgent tasks.

[0004] The second conventional scheduling strategy uses a single network layer as the scheduling unit. Compared to the first conventional scheduling strategy, the second conventional scheduling strategy responds more quickly during task context switching. However, the second conventional scheduling strategy causes frequent off-chip memory accesses between network layers (off-chip memory access refers to reading and writing data from memory outside the NPU), which reduces NPU performance.

[0005] To improve the response efficiency and resource utilization of the NPU in emergency situations, it is necessary to propose more flexible and efficient scheduling strategies. Summary of the Invention

[0006] This application provides a task execution method, compilation method, chip, and device for an artificial intelligence model, which can solve the technical problem of how to optimize the resource scheduling of computer equipment when an urgent task requiring artificial intelligence model inference occurs during the operation of application software based on artificial intelligence technology.

[0007] In a first aspect, embodiments of this application provide a task execution method for an artificial intelligence model. The task execution method is applied to a chip according to the second aspect below. The chip includes a CPU and an NPU. The chip is deployed on a computer device storing a target application. The CPU is configured to run the target application and place the inference tasks in the target application into a task queue. The NPU is configured to invoke an artificial intelligence model to perform inference on the inference tasks in the task queue.

[0008] The task to be inferred is divided into multiple subtasks according to a preset compilation method. Each subtask contains several consecutive network layers, and the subtasks are arranged in a first order. Each network layer belongs to one of the artificial intelligence models.

[0009] The task execution method includes:

[0010] The NPU invokes the artificial intelligence model at time k and executes each subtask of the first inference task of the target application in a first order to perform task inference on the first inference task.

[0011] When the target application encounters a second inference task at time k+n, the CPU sends an interrupt signal to the NPU, wherein the second inference task has a higher priority than the first inference task in the task queue, and both k and n are positive numbers.

[0012] When the NPU receives the interrupt signal at the (k+n)th time, it interrupts the processing of the first inference task and records the interrupt progress position to execute each subtask of the second inference task.

[0013] After the NPU has completed the execution of each subtask of the second task to be inferred, it returns to the interrupted progress position and executes the unfinished subtasks of the first task to be inferred in the first order to complete the inference of the first task to be inferred.

[0014] The first beneficial effect of this application is that: Firstly, the tasks to be processed by the artificial intelligence model are compiled according to a preset compilation method, resulting in multiple subtasks. This allows each task to be loaded and scheduled by the NPU at a moderately granular scheduling unit. Specifically, this application uses the subtasks as the basic scheduling unit of the NPU, thus achieving resource scheduling at a moderate granularity while balancing scheduling complexity and task switching flexibility. In the event of urgent tasks, the task switching cost of the NPU is significantly reduced; consequently, it can effectively improve the response efficiency and resource utilization of the NPU in the event of sudden changes in the target application running on the computer device.

[0015] For example, when the NPU receives the interrupt signal at the (k+n)th time, it interrupts the processing of the first inference task and records the interrupt progress position, in order to execute each subtask of the second inference task, including:

[0016] When the NPU receives the interrupt signal at the (k+n)th time, it acquires the first subtask of the first inference task in progress and completes the first subtask.

[0017] The NPU determines a second subtask from the first subtask to be inferred, which is located after the first subtask in the task sequence. The starting point of the task progress of the second subtask is used as the interruption progress position of the first subtask to be inferred, so as to interrupt the processing of the first subtask to be inferred.

[0018] The NPU executes each subtask of the second inference task.

[0019] For example, when the NPU receives the interrupt signal at the (k+n)th time, it interrupts the processing of the first inference task and records the interrupt progress position, in order to execute each subtask of the second inference task, including:

[0020] When the NPU receives the interrupt signal at the (k+n)th time, it pauses the currently executed subtask in the first inference task and records the progress of the paused subtask as the interrupt progress position of the first inference task.

[0021] The NPU executes each subtask of the second inference task.

[0022] Secondly, embodiments of this application provide a chip including a CPU and an NPU. The chip is deployed on a computer device storing a target application. The CPU is configured to run the target application and place the inference tasks in the target application into a task queue. The NPU is configured to invoke an artificial intelligence model to perform inference on the inference tasks in the task queue.

[0023] The task to be inferred is divided into multiple subtasks according to a preset compilation method. Each subtask contains several consecutive network layers, and the subtasks are arranged in a first order. Each network layer belongs to one of the artificial intelligence models.

[0024] The NPU is used to invoke the artificial intelligence model at time k and execute each subtask of the first inference task of the target application in the first order to perform task inference on the first inference task.

[0025] The CPU is configured to send an interrupt signal to the NPU when a second inference task appears in the target application at time k+n; the second inference task has a higher priority than the first inference task in the task queue, and both k and n are positive numbers.

[0026] The NPU is used to interrupt the processing of the first inference task when the interrupt signal is received at the (k+n)th time, and to record the interrupt progress position of the first inference task so as to execute each subtask of the second inference task.

[0027] The NPU is also configured to, after completing the execution of each subtask of the second task to be inferred, return to the interrupted progress position and execute the unfinished subtasks in the first task to be inferred in the first order, so as to complete the inference of the first task to be inferred.

[0028] The second beneficial effect of this application is that the chip provided uses subtasks as the basic scheduling unit of the NPU, and thus completes resource scheduling with moderate granularity, which can balance scheduling complexity and task switching flexibility. When encountering urgent tasks, the task switching cost of the NPU is significantly reduced; thus, it can effectively improve the response efficiency and resource utilization of the NPU when there are sudden changes in the target application running on the computer device.

[0029] For example, the NPU includes a command dispatch unit, a computing unit, and on-chip memory;

[0030] The command distribution unit is configured to write the configuration information of each subtask of the task to be inferred into the computing unit and the memory access module.

[0031] The computing unit is used to perform calculations on the subtask based on the configuration information to obtain the subtask processing result.

[0032] The on-chip memory is used to temporarily store the processing results of the subtask.

[0033] For example, the computer device includes off-chip memory; the NPU also includes a memory access module, an interrupt processor, and on-chip registers;

[0034] The interrupt handler is configured to send an interrupt message to the command distribution unit when the second inference task occurs at time k+n.

[0035] The memory access module is used to store the pause information of the first inference task to the on-chip register based on the interrupt information, and write the completed subtask processing results in the on-chip memory to the off-chip memory.

[0036] The on-chip register is used to maintain the pause information of the first inference task.

[0037] For example, the interrupt processor is further configured to send recovery information to the command distribution unit after the execution of each subtask of the second inference task is completed, so that the NPU obtains pause information from the on-chip register and executes the remaining subtasks in the first inference task based on the pause information to complete the inference of the first inference task.

[0038] For example, the on-chip registers include a checkpoint register, an address register, and a status register; the pause information includes at least the interrupt progress position, storage address, and interrupt status of the paused subtask that was interrupted.

[0039] The checkpoint register is used to store the interruption progress position of the paused subtask;

[0040] The address register is used to store the storage address of various types of data of the paused subtask;

[0041] The status register is used to store the interrupt status of the paused subtask.

[0042] Thirdly, embodiments of this application provide a task compilation method for an artificial intelligence model, the compilation method comprising:

[0043] The artificial intelligence model is compiled and divided according to the number of network layers to obtain a pending division result. The pending division result includes L sub-tasks, and each sub-task contains several consecutive network layers.

[0044] Each subtask of the undetermined partitioning result is iteratively processed according to a preset processing method to adjust the number of network layers in each subtask until the performance balance condition is met between each subtask, and the iteration stops to obtain the target partitioning result that meets the performance balance condition. The target partitioning result includes N subtasks; where L and N are both positive integers.

[0045] Each of the subtasks serves as the basic scheduling unit of the NPU to perform inference on the inference task related to the artificial intelligence model.

[0046] The beneficial effect of the third aspect of this application is that the compilation method of this embodiment can compile the tasks to be processed of the artificial intelligence model, so that it has multiple sub-tasks. This allows each task to be processed to receive the loading and scheduling of the NPU in a scheduling unit with appropriate granularity, which can take into account both scheduling complexity and task switching flexibility.

[0047] For example, the preset processing method includes:

[0048] The undetermined partitioning result is simulated to obtain the simulation result;

[0049] The performance indicators of the simulation results are evaluated to obtain the evaluation results;

[0050] The undetermined partitioning results are compiled and adjusted based on the evaluation results so that the performance balance condition is met among each subtask.

[0051] For example, the simulation of the undetermined partitioning result to obtain the simulation result includes:

[0052] The subtask is fed into the first simulation model for transaction-level simulation to simulate the behavior of the subtask in each clock cycle and obtain simulation results.

[0053] The evaluation of the performance metrics of the simulation results to obtain the evaluation results includes:

[0054] The simulation results are fed into a performance evaluator to evaluate the performance metrics of the simulation results and obtain evaluation results; the performance metrics include at least one of the following: the latency of the whole network inference of the artificial intelligence model, the latency of a single subtask, and the on-chip network bandwidth utilization.

[0055] Fourthly, embodiments of this application provide a task execution device, the execution device comprising:

[0056] The inference module is used to enable the NPU to call the artificial intelligence model at time k and execute the subtasks of the first inference task of the target application in a first order to perform task inference on the first inference task.

[0057] The interrupt module is used to cause the CPU to send an interrupt signal to the NPU when a second inference task appears in the target application at time k+n, wherein the second inference task has a higher priority than the first inference task in the task queue.

[0058] The inference switching module is used to enable the NPU to interrupt the processing of the first inference task when it receives the interrupt signal at the (k+n)th time, and to record the interrupt progress position so as to execute each subtask of the second inference task.

[0059] The recovery module is used to enable the NPU to return to the interrupted progress position after completing the execution of each subtask of the second inference task, and to execute the unfinished subtasks in the first inference task in the first order to complete the inference of the first inference task.

[0060] Fifthly, embodiments of this application provide a task compilation apparatus for an artificial intelligence model, the compilation apparatus comprising:

[0061] The first compilation module is used to compile and divide the artificial intelligence model according to the number of network layers to obtain a pending division result. The pending division result includes L sub-tasks, and each sub-task contains several consecutive network layers.

[0062] The second compilation module is used to iteratively process each subtask of the pending partitioning result according to a preset processing method to adjust the number of network layers in each subtask until the performance balance condition is met between each subtask, and then stop the iteration to obtain the target partitioning result that meets the performance balance condition. The target partitioning result includes N subtasks; where L and N are both positive integers.

[0063] Each of the subtasks serves as the basic scheduling unit of the NPU to perform inference on the inference task related to the artificial intelligence model.

[0064] Sixthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the methods described in the first and third aspects to be executed.

[0065] In a seventh aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the methods described in the first and third aspects above.

[0066] Eighthly, embodiments of this application provide a vehicle equipped with a computer device as described in the first and second aspects above. The computer device includes a chip as described in the second aspect above, and when the computer device runs the target application, the chip performs the steps of the method as described in the first aspect above.

[0067] It is understood that the beneficial effects of aspects four through eight above can be found in the relevant descriptions of aspects one through three above, and will not be repeated here. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a schematic flowchart of an embodiment of a task compilation method for an artificial intelligence model provided in this application;

[0070] Figure 2 This is a schematic flowchart of another embodiment of the task compilation method for an artificial intelligence model provided in this application;

[0071] Figure 3 This is an architectural block diagram of the assembly stage and task execution stage provided in an embodiment of this application;

[0072] Figure 4 This is a structural block diagram of a computer device provided in an embodiment of this application;

[0073] Figure 5 This is a schematic flowchart of an embodiment of a task execution method for an artificial intelligence model provided in this application;

[0074] Figure 6 This is a schematic flowchart of another embodiment of the task execution method of an artificial intelligence model provided in this application;

[0075] Figure 7 This is a schematic diagram of the structure of an NPU chip provided in another embodiment of this application;

[0076] Figure 8 This is a schematic diagram of the structure of the task execution device provided in the embodiments of this application;

[0077] Figure 9 This is a schematic diagram of the structure of the task compilation device for the artificial intelligence model provided in the embodiments of this application. Detailed Implementation

[0078] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0079] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0080] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0081] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0082] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0083] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0084] Understandably, the inventors of this application have discovered that in the process of running application software related to artificial intelligence technology, such as in the field of autonomous driving, the CPU of an AI-powered autonomous driving terminal is responsible for sending inference tasks of various neural network models to a task queue. The NPU of the AI-powered autonomous driving terminal executes model inference according to the priority of the tasks to complete a series of tasks, including environmental perception and path planning. When the vehicle encounters an emergency, such as a sudden obstacle, the CPU temporarily increases the priority of the corresponding task in the task queue or inserts a high-priority task to ensure that the NPU handles these emergency tasks first.

[0085] The first conventional scheduling strategy uses a complete neural network model as the basic scheduling unit. This means that the NPU will only process the next highest-priority task in the task queue after completing a full model inference task. In emergency situations, such as when a vehicle suddenly needs to take evasive action, the NPU either abandons the currently executing task and executes the high-priority emergency task, resuming the abandoned task only after the emergency task is completed; or it continues executing the current task, processing the emergency task only after its completion. Both approaches have significant drawbacks: the former leads to wasted NPU computing power; the latter significantly increases the processing time for emergency tasks.

[0086] The second conventional scheduling strategy uses a single network layer as the scheduling unit. Compared to the first scheduling strategy, this strategy responds more quickly when switching task contexts. However, this approach causes frequent off-chip memory accesses between network layers, reducing NPU performance.

[0087] To address the challenges of improving the response efficiency and resource utilization of the NPU in unforeseen circumstances during the operation of current AI-based application software, this application proposes a task execution method, compilation method, chip, and device for an AI model. This optimizes resource scheduling in computer devices, making the scheduling more flexible and efficient. The technical solution of this application is illustrated below through specific embodiments.

[0088] First, refer to Figure 1 This application provides a task compilation method for an artificial intelligence model, such as... Figure 1 As shown, the compilation method mainly includes steps A and B:

[0089] Step A: Compile and divide the artificial intelligence model according to the number of network layers to obtain a pending partition result. The pending partition result includes L sub-task partitions, and each sub-task partition contains several consecutive network layers.

[0090] Step B: Iterate through each sub-task partition of the undetermined partitioning result according to a preset processing method to adjust the number of network layers in each sub-task partition until the performance balance condition is met among each sub-task partition, and stop iterating to obtain the target partitioning result that meets the performance balance condition. The target partitioning result includes N sub-task partitions; where L and N are both positive integers; where each sub-task partition is used as the basic scheduling unit of the NPU to complete the inference of the inference task related to the artificial intelligence model.

[0091] It should be noted that the execution entity of the task compilation method in this application embodiment is a compilation device equipped with application compilation software tools, which is deployed on the side of the relevant R&D personnel (and...). Figure 4 (The computer equipment shown is different). The compilation method for steps A and B above is completed using this compilation equipment to obtain the program representing the target partitioning result; the program representing the target partitioning result is saved to... Figure 4 The memory 21 of the computer device shown is composed of Figure 4 The chip 20 of the computer device shown can be invoked to perform inference on the task to be inferred related to the artificial intelligence model based on the final compilation result (subtask Partition). The performance balance condition can characterize that the computational cost of each subtask (hereinafter, for ease of description, each task is referred to as a Partition) is the same, or the computational cost of each subtask Partition is close. In this embodiment, a neural network model is used as an example of the artificial intelligence model for illustration.

[0092] Specifically, the compilation method proposed in this application can utilize the deterministic characteristics of neural network models to uniformly divide complex (artificial intelligence) network models into multiple computational sub-tasks with similar computational complexity. The determinism of neural network models can be understood as the network layers in each neural network model being statically fixed and unchanging, and the total number of network layers in each neural network model being pre-defined (maintaining the total number of network layers in each neural network model unchanged in the later stages).

[0093] Each subtask partition consists of several consecutive network layers, and also serves as... Figure 4The basic scheduling unit of the NPU on the computer device shown can be the sequential execution of these subtask partitions by the NPU to run the application software normally. This allows the NPU to execute the application at the subtask partition level during the inference task of the neural network model. Compared to the first conventional scheduling strategy, which uses the complete neural network model as the basic scheduling unit (coarse-grained scheduling), and the second conventional scheduling strategy, which uses a single network layer as the scheduling unit (fine-grained scheduling), the embodiments of this application can achieve resource scheduling at a moderate granularity, balancing scheduling complexity and task switching flexibility.

[0094] The preset processing method in step B includes the following sub-steps B1 to B3:

[0095] Sub-step B1: Simulate the undetermined partitioning result to obtain the simulation result;

[0096] Specifically, the subtask Partition is sent to the first simulation model for transaction-level simulation to simulate the behavior of the subtask Partition in each clock cycle and obtain simulation results. In this embodiment, the Cycle Accuracy Level Model (CA Model) can be used as the first simulation model. The CPU can first send the subtask Partition generated by the AI ​​compiler in sub-step A to the Cycle Accuracy Level Model (CA Model) for transaction-level simulation.

[0097] Sub-step B2: Evaluate the performance indicators of the simulation results to obtain the evaluation results;

[0098] Specifically, the simulation results are fed into a performance evaluator to evaluate the performance metrics of the simulation results and obtain evaluation results. The performance metrics include at least one of the following: the latency of the entire network inference of the artificial intelligence model, the latency of a single subtask partition, and the on-chip network bandwidth utilization. In this embodiment, the Evaluator can be used as the performance evaluator.

[0099] Sub-step B3 involves compiling and adjusting the undetermined partitioning results based on the evaluation results, so that each of the subtask partitions satisfies the performance balance condition.

[0100] Specifically, based on the evaluation results, the performance evaluator can use a greedy algorithm to obtain a better network model partitioning strategy. The undetermined partitioning results are then sent back to the AI ​​compiler for a new round of compilation, adjustment, simulation, and evaluation until the optimal target partitioning result is obtained (i.e., the performance balance condition is met among the various subtasks). The CPU then uses the network model with the target partitioning result as a task to be processed and schedules it into the task queue.

[0101] In its specific implementation, the principle of uniformly dividing a complex (artificial intelligence) network model into multiple computationally similar subtasks (Partitions) by utilizing the deterministic characteristics of neural network models is as follows: During the assembly stage, there are M artificial intelligence models (e.g., ... Figure 3 As shown, including Model#1, Model#2, Model#3…Model#M, taking the partitioning process of an artificial intelligence model Model#1 as an example, assuming that Model#1 has a total of five network layers (Layer#1, Layer#2, Layer#3, Layer#4, Layer#5), according to the task compilation method provided in this application, with the aim of achieving performance balance (i.e., achieving the same or similar computational amount for each subtask partition), the artificial intelligence model Model#1 is divided into two partitions (Partition#1 and Partition#2), which are arranged sequentially. Iterative processing is performed according to a preset processing method. When Partition#1 includes consecutive network layers Layer#1 and Layer#2, and Partition#2 includes consecutive network layers Layer#3, Layer#4, and Layer#5, the subtask partitions of Model#1 satisfy the performance balance condition. This process can be repeated sequentially... Figure 3 The M artificial intelligence models shown in the figure are compiled, so that each neural network model in this embodiment of the application can receive the loading and scheduling of the NPU in such a moderately granular scheduling unit.

[0102] The beneficial effects of Embodiment 1 of this application are as follows: it can compile the tasks to be processed of the artificial intelligence model so that it has multiple sub-tasks. This allows each task to be processed to receive loading and scheduling from the NPU in a scheduling unit with appropriate granularity, which can balance scheduling complexity and task switching flexibility. In turn, it can effectively improve the response efficiency and resource utilization of the NPU in the event of sudden changes in the target application running on the computer device.

[0103] Furthermore, based on the above-mentioned task compilation method for artificial intelligence models, this embodiment provides a task execution method for artificial intelligence models and a chip for executing the method; the chip in this embodiment can be a system-on-chip (SoC), including at least a CPU (Central Processing Unit) and an NPU (Neural Processing Unit); such as Figure 4 As shown, the chip (20) in this embodiment is deployed on a computer device storing the target application. Figure 4 The computer device shown also includes a memory 21, which in some embodiments may be an internal storage unit of the computer device, such as a hard disk or RAM. In other embodiments, the memory 21 may also be an external storage device of the computer device, such as Dynamic Random Access Memory (DRAM), Solid State Drive (SSD), or Hard Disk Drive (HDD) provided on the computer device. Furthermore, the memory 21 may include both internal and external storage units of the computer device. The memory 21 is used to store the operating system, application programs, bootloader, data, and other programs. The memory 21 of the computer device stores the target application 22. The CPU (201) is configured to run the target application 22 and place the inference tasks in the target application 22 into a task queue. The NPU (202) is configured to call an artificial intelligence model to infer the inference tasks in the task queue. In this embodiment, according to the compilation method of Embodiment 1 above, the inference tasks of the running target application can be divided into multiple sub-tasks Partition. Each sub-task Partition contains several consecutive network layers, and each sub-task Partition is arranged in a first order. Each network layer belongs to one of the artificial intelligence models.

[0104] refer to Figure 3 The task queue contains multiple tasks to be reasoned about, and each task is reasoned about by an artificial intelligence model (in this embodiment, we still use a "neural network model" as an example). Therefore, each task to be reasoned about corresponds to a "neural network model," and the task queue contains M tasks to be reasoned about, that is, M models (e.g., ...). Figure 3As shown, the model includes Model#1, Model#2, Model#3…Model#M. Each task to be inferred, Model, is divided into multiple subtasks, each of which contains several consecutive network layers. The subtasks are arranged in a first order (Partition#1, Partition#2…, Partition#N). Taking the first task to be inferred, Model#1, in the task queue as an example, Model#1 has five network layers (Layer#1, Layer#2, Layer#3, Layer#4, Layer#5). The AI ​​model Model#1 is divided into two partitions (Partition#1 and Partition#2), which are arranged in order. When Partition#1 includes consecutive network layers Layer#1 and Layer#2, and Partition#2 includes consecutive network layers Layer#3, Layer#4, and Layer#5, the subtask partitions of Model#1 satisfy the performance balance condition.

[0105] Before running the model inference task, the NPU loads the commands corresponding to the subtask partition results generated by the task compilation method in Implementation Example 1 into the NPU's command distribution unit, which then controls the operation of each submodule of the NPU during the model execution process.

[0106] Understandably, the target application can be autonomous driving application software, VR (Virtual Reality) racing game application software, or other application software (APP) that requires the use of an artificial intelligence model. This application embodiment uses autonomous driving application software as the target application software running on the computing device as an example for explanation. Correspondingly, the computer device is described as an in-vehicle terminal. During vehicle operation, the in-vehicle terminal is automatically started and the autonomous driving application software is run. During normal vehicle operation (here, "normal operation" refers to some routine autonomous driving tasks performed by the in-vehicle terminal under the premise that no unexpected situation occurs), a series of routine tasks, including environmental perception and path planning, will be performed. These tasks will be reasoned by the artificial intelligence model according to a certain priority, and the reasoning tasks will be sent to the task queue (job queue) in the order of the above priority. This application embodiment uses "first task to be reasoned" to define the above routine tasks.

[0107] When a vehicle encounters an emergency, such as when the autonomous driving application software detects a sudden obstacle in front of the vehicle, the CPU temporarily increases the priority of the corresponding non-routine task in the task queue or inserts a high-priority task to ensure that the NPU handles these emergency tasks first. This application embodiment uses the term "second task to be inferred" to define the above-mentioned non-routine task or high-priority task. Correspondingly, the system-on-a-chip provided in this application also includes a GPU (Graphics Processing Unit). During the operation of the autonomous driving application software on the computer device, the onboard camera captures images of the area in front of the vehicle. The GPU processes the captured images, and the CPU can use the GPU's image processing results to determine whether there is an unexpected situation in front of the vehicle.

[0108] The chip execution method of this application embodiment runs the corresponding artificial intelligence model to execute the "first inference task". When a "second inference task" with a higher priority than the "first inference task" appears, the execution of the "first inference task" is interrupted, and the "second inference task" is executed instead. The execution of the "second inference task" is resumed only after the "second inference task" is completed. Accordingly, refer to Figure 5 The task execution method of the artificial intelligence model in this application embodiment mainly includes the following steps S10 to S40:

[0109] Step S10: At time k, the NPU invokes the artificial intelligence model and executes each subtask of the first inference task of the target application in a first order to perform task inference on the first inference task.

[0110] Understandably, the k-th moment refers to a specific moment in the current situation. For example, when a vehicle is running an autonomous driving application, during normal vehicle operation, the onboard terminal will execute a series of routine first-stage reasoning tasks, including environmental perception and path planning. These first-stage reasoning tasks are sent to the task queue by the CPU, and the NPU loads the first-stage reasoning tasks to configure the corresponding subtask partitions (see...). Figure 6 This allows the NPU to execute the highest-priority first inference task in the task queue, as referenced. Figure 3 Taking the first task to be inferred, Model#1, in the task queue as an example, the NPU executes the subtasks Partition#1 and Partition#2 of the first task to be inferred, Model#1, in the first order.

[0111] Step S20: When a second inference task appears in the target application at time k+n, the CPU sends an interrupt signal to the NPU; wherein the second inference task has a higher priority than the first inference task in the task queue.

[0112] Understandably, the k+n time represents a future time. For example, when a vehicle encounters an emergency during operation, such as when the autonomous driving application software detects a sudden obstacle in front of the vehicle, the CPU temporarily inserts a non-routine "second task to be inferred" into the task queue, making the "second task to be inferred" have a higher priority than the "first task to be inferred," and the CPU sends an interrupt signal to the NPU. These second tasks to be inferred are sent to the task queue by the CPU and preloaded by the NPU.

[0113] Step S30: When the NPU receives the interrupt signal at the (k+n)th time, it interrupts the processing of the first inference task and records the interrupt progress position, so as to execute each subtask of the second inference task.

[0114] Understandably, in this embodiment, when the NPU runs to a sub-task partition of the first inference task, the NPU's Context Saving Unit (CSU) receives an interrupt signal from the CPU. The CSU is responsible for interrupting the NPU's work on the first inference task (in this embodiment, the NPU either directly abandons the current partition or enters an interrupt after completing the current partition), recording the corresponding interrupt progress position, and simultaneously writing the completed sub-task processing result into the memory 21 (which can be understood as saving the context, see...). Figure 6 The NPU loads the urgent, high-priority "second task to be inferred" and configures the corresponding subtask partitions. The NPU executes each subtask partition of the second task to be inferred in the same way as it executes each subtask partition of the first task to be inferred, until all subtask partitions of the second task to be inferred have been executed.

[0115] Step S40: After the NPU has completed the execution of each subtask Partition of the second task to be inferred, it returns to the interrupted progress position and executes the unfinished subtasks in the first task to be inferred in the first order to complete the inference of the first task to be inferred.

[0116] In this embodiment, after the NPU completes the execution of the second inference task representing the urgent task, the NPU immediately returns to the interrupted progress position of the previously interrupted first inference task (which can be understood as restoring the context, see...). Figure 6 The process resumes execution of the unfinished subtask Partition in the first task to be inferred from the interrupted progress position, so as to continue the normal execution of routine tasks.

[0117] Specifically, there are two ways to execute step 30, which involves interrupting the NPU's operation of the first inference task:

[0118] The first interrupt method involves the NPU entering an interrupt after completing the current subtask partition. Specifically, when the NPU receives an interrupt signal at time k+n, it acquires the first subtask of the first task to be inferred that is currently being executed, and enters an interrupt after completing the first subtask. The NPU then determines the second subtask in the task sequence that follows the first subtask from the first task to be inferred, and uses the starting point of the task progress of the second subtask as the interrupt progress position of the first task to be inferred, thereby interrupting the processing of the first task to be inferred. After the NPU completes the inference of the second task by executing all the subtasks of the second task to be inferred, it returns to the starting point of the task progress of the second subtask of the first task to be inferred, so as to continue processing the remaining unfinished second subtask partitions of the first task to be inferred, thereby resuming the inference of the first task to be inferred.

[0119] The second interruption method involves the NPU pausing the currently executed subtask within the first task to be inferred during the execution of the current subtask partition. The progress of the paused subtask is recorded as the interruption progress position of the first task to be inferred. After the NPU completes the execution of all subtasks of the second task to be inferred, i.e., completes the inference of the second task to be inferred, it returns to the progress of the paused subtask of the first task to be inferred to continue processing the paused subtask partition and resume the inference of the first task to be inferred.

[0120] Because traditional methods use the entire network model as the basic scheduling unit, when faced with an urgent task, they either have to abandon the current task or wait for the current task to complete before executing the urgent task, both of which incur significant task switching costs. Furthermore, traditional methods require the CPU to run corresponding software programs to save and restore the current state, further exacerbating the task switching costs.

[0121] The beneficial effects of Embodiment 2 of this application are as follows: Compared with the conventional solutions described above, this embodiment uses a subtask partition as the basic scheduling unit of the NPU. When encountering an urgent task, whether waiting for the current partition to complete (the first interruption method mentioned above) or abandoning the current partition (the second interruption method mentioned above), the task switching cost is significantly reduced. This effectively improves the response efficiency and resource utilization of the NPU in the event of a sudden interruption in the target application running on the computer device.

[0122] In this embodiment, as Figure 7 As shown, the NPU of the chip 20 includes a command dispatch unit, a computing unit, a memory access module, an on-chip memory, an interrupt processor, and on-chip registers;

[0123] In a specific implementation, the on-chip memory can be a static random-access memory (SRAM) located inside the NPU chip;

[0124] The memory access module can be a direct memory access module, which is the data transfer module of the NPU and is used to transfer data in batches between the computer device's memory 21 and the NPU's on-chip registers (SRAM).

[0125] The computation unit may include a first computation unit and a second computation unit. For example, the first computation unit may be a matrix multiplication processing core processor used to accelerate important operators (matrix multiplication) in deep learning; the second computation unit may be a vector-scalar processing core processing unit that processes vectors and scalars, and is a major computation unit of the NPU used to accelerate other types of operators (vectors and scalars) in deep learning besides matrix multiplication, such as pooling, ReLU, normalization and other operators.

[0126] The command dispatch unit can be a command dispatcher, used to configure the various modules of the NPU; for example, the command unit writes the configuration information of all currently executed partitions generated during the compilation phase into the first computing unit, the second computing unit, and the memory access module (DMA).

[0127] The interrupt handler and on-chip registers can be components within the context saving unit (CSU) of the NPU mentioned in Embodiment 2 of this application;

[0128] The interrupt handler in this embodiment of the application has two functions:

[0129] Function 1: When an urgent task needs to be performed, an interrupt signal (i.e., int, abbreviation for interrupt in the diagram) is generated. Figure 7 (int in) and interrupt handling commands ( Figure 7 The command (cmd) in the interrupt signal is sent to the command dispatcher of the NPU so that the NPU can save the partition information of the currently executing subtask into the on-chip register (in this embodiment, interrupt information is used to summarize interrupt signals and interrupt handling commands).

[0130] Function 2: After the emergency task is completed, a recovery message is sent to the command distribution unit so that the NPU can read the interrupt progress position of the original Partition pause information from the on-chip register and continue to execute the previous sub-task Partition from the interrupt progress position.

[0131] In specific implementations, the on-chip registers in this application embodiment may include three types of registers: checkpoint registers, address registers, and status registers. Among them:

[0132] The checkpoint register is used to store the interrupt progress position of the paused subtask Partition;

[0133] The address register is used to store the storage addresses of various types of data for the paused subtask Partition;

[0134] The status register is used to store the interrupt status of the paused subtask Partition.

[0135] Understandably, the three types of registers each store partial pause information for a subtask partition, so that after the NPU completes the emergency task, it can resume the normal execution of the previously paused subtask partition based on the stored pause information.

[0136] Specifically, during the execution of the steps and tasks involved in Embodiment 2 above, the NPU chip is configured to write the first configuration information of each subtask partition of the first inference task into the computing unit and the memory access module at time k.

[0137] The computing unit is used to perform calculations on each subtask partition of the first reasoning task based on the first configuration information to obtain the subtask processing result of the first reasoning task.

[0138] The on-chip memory (SRAM) is used to temporarily store the subtask processing results of the first inference task.

[0139] The interrupt handler is configured to send interrupt information to the command distribution unit when a second inference task occurs at time k+n.

[0140] The DMA block is used to store the pause information of the first inference task to the on-chip register based on the interrupt information, and write the subtask processing results of the completed first inference task in the on-chip memory (SRAM) into the off-chip memory (DRAM) of the computer device.

[0141] The command dispatcher is configured to write the second configuration information of each subtask partition of the second inference task into the computing unit and the memory access module;

[0142] The computing unit is also used to perform arithmetic processing on each subtask partition of the second reasoning task based on the second configuration information to obtain the subtask processing result of the second reasoning task.

[0143] The on-chip memory (SRAM) is also used to temporarily store the subtask processing results of the second inference task.

[0144] The memory access module (DMA) block is also used to write the subtask processing results of the completed second inference task in the on-chip memory (SRAM) into the off-chip memory (DRAM) of the computer device.

[0145] The interrupt processor is further configured to send recovery information to the command distribution unit after the execution of each subtask partition of the second inference task is completed, so that the NPU obtains the pause information from the on-chip register and executes the remaining subtask partitions in the first inference task based on the pause information to complete the inference of the first inference task.

[0146] The NPU module hardware with a context switching unit proposed in this application, based on the interrupt processor and on-chip registers in the context switching unit, can assist the CPU of the computer device in completing interrupt tasks and improve the efficiency of saving and restoring the context.

[0147] refer to Figure 8 This application also proposes a task execution device, the execution device comprising:

[0148] The inference module 811 is used to enable the NPU to call the artificial intelligence model at time k and execute each subtask of the first inference task of the target application in a first order, so as to perform task inference on the first inference task.

[0149] Interrupt module 812 is used to cause the CPU to send an interrupt signal to the NPU when a second inference task appears in the target application at time k+n, wherein the second inference task has a higher priority than the first inference task in the task queue.

[0150] The inference switching module 813 is used to enable the NPU to interrupt the processing of the first inference task when it receives the interrupt signal at the (k+n)th time, and to record the interrupt progress position so as to execute each subtask of the second inference task.

[0151] The recovery module 814 is used to enable the NPU to return to the interrupted progress position after completing the execution of each subtask of the second task to be inferred, and to execute the unfinished subtasks in the first task to be inferred in the first order, so as to complete the inference of the first task to be inferred.

[0152] Understandably, Figure 8 The corresponding task execution device can be an integrated circuit, installed in the computer device described in Embodiments 1 / 2 above, to cooperate in executing Embodiment 2. Figure 5 The task execution methods involved.

[0153] refer to Figure 9 This application also proposes a task compilation device for an artificial intelligence model, the compilation device comprising:

[0154] The first compilation module 911 is used to compile and divide the artificial intelligence model according to the number of network layers to obtain a pending division result. The pending division result includes L sub-tasks, and each sub-task contains several consecutive network layers.

[0155] The second compilation module 912 is used to iteratively process each subtask of the pending partitioning result according to a preset processing method to adjust the number of network layers in each subtask until the performance balance condition is met among each subtask, and then stop the iteration to obtain the target partitioning result that meets the performance balance condition. The target partitioning result includes N subtasks; where L and N are both positive integers; where each subtask is used as the basic scheduling unit of the NPU to complete the inference of the inference task related to the artificial intelligence model.

[0156] Understandably, Figure 9The corresponding task compilation device for the artificial intelligence model can be an integrated circuit or an application compilation software tool, loaded into the compilation device described in the above embodiments to cooperate in execution. Figure 1 The embodiments involve task compilation methods for artificial intelligence models.

[0157] It should be noted that the information interaction and execution process between the above-mentioned devices / modules / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0158] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0159] This application embodiment also provides a vehicle equipped with a computer device as described in the above embodiment. When the computer device runs the target application, the chip of the computer device executes the steps of the task execution method of the artificial intelligence model and the task compilation method of the artificial intelligence model as described above.

[0160] Accordingly, the computer device carried by the vehicle in this application embodiment is described using an in-vehicle terminal as an example; the in-vehicle terminal is automatically started during the vehicle's operation and runs the autonomous driving application software. During normal driving (here, "normal driving" refers to some routine autonomous driving tasks performed by the in-vehicle terminal under the premise that no unexpected situation occurs), the vehicle will perform a series of routine tasks, including environmental perception and path planning. These tasks are reasoned by the artificial intelligence model according to a certain priority, and the reasoned tasks are sent to the task queue in the order of the above priorities. When the vehicle encounters an emergency, such as when the autonomous driving application software identifies an obstacle suddenly appearing in front of the vehicle, the CPU temporarily increases the priority of the corresponding non-routine tasks in the task queue or inserts high-priority tasks to ensure that the NPU handles these emergency tasks first.

[0161] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0162] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0163] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0165] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0166] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0167] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A task execution method for an artificial intelligence model, characterized in that, The task execution method is applied to a chip, which includes a CPU and an NPU. The chip is deployed on a computer device storing a target application. The CPU is configured to run the target application and place the tasks to be reasoned in the target application into a task queue. The NPU is configured to invoke an artificial intelligence model to perform reasoning on the tasks to be reasoned in the task queue. The inference task is divided into multiple subtasks according to a preset compilation method. Each subtask contains several consecutive network layers, and the subtasks are arranged in a first order. Each network layer belongs to one of the artificial intelligence models. The task execution method includes: The NPU invokes the artificial intelligence model at time k and executes each subtask of the first inference task of the target application in a first order to perform task inference on the first inference task. When the target application encounters a second inference task at time k+n, the CPU sends an interrupt signal to the NPU, wherein the second inference task has a higher priority than the first inference task in the task queue, and both k and n are positive numbers. When the NPU receives the interrupt signal at the (k+n)th time, it interrupts the processing of the first inference task and records the interrupt progress position to execute each subtask of the second inference task. After the NPU has completed the execution of each subtask of the second task to be inferred, it returns to the interrupted progress position and executes the unfinished subtasks of the first task to be inferred in the first order to complete the inference of the first task to be inferred.

2. The task execution method as described in claim 1, characterized in that, When the NPU receives the interrupt signal at time k+n, it interrupts the processing of the first inference task and records the interrupt progress position to execute each subtask of the second inference task, including: When the NPU receives the interrupt signal at the (k+n)th time, it acquires the first subtask of the first inference task in progress and completes the first subtask. The NPU determines a second subtask from the first subtask to be inferred, which is located after the first subtask in the task sequence. The starting point of the task progress of the second subtask is used as the interruption progress position of the first subtask to be inferred, so as to interrupt the processing of the first subtask to be inferred. The NPU executes each subtask of the second inference task.

3. The task execution method as described in claim 1, characterized in that, When the NPU receives the interrupt signal at time k+n, it interrupts the processing of the first inference task and records the interrupt progress position to execute each subtask of the second inference task, including: When the NPU receives the interrupt signal at the (k+n)th time, it pauses the currently executed subtask in the first inference task and records the progress of the paused subtask as the interrupt progress position of the first inference task. The NPU executes each subtask of the second inference task.

4. A chip, characterized in that, The chip includes a CPU and an NPU, and is deployed on a computer device storing a target application. The CPU is configured to run the target application and place the inference tasks in the target application into a task queue. The NPU is configured to invoke an artificial intelligence model to perform inference on the inference tasks in the task queue. The task to be inferred is divided into multiple subtasks according to a preset compilation method. Each subtask contains several consecutive network layers, and the subtasks are arranged in a first order. Each network layer belongs to one of the artificial intelligence models. The NPU is used to invoke the artificial intelligence model at time k and execute each subtask of the first inference task of the target application in the first order to perform task inference on the first inference task. The CPU is configured to send an interrupt signal to the NPU when a second inference task appears in the target application at time k+n; the second inference task has a higher priority than the first inference task in the task queue, and both k and n are positive numbers. The NPU is used to interrupt the processing of the first inference task when the interrupt signal is received at the (k+n)th time, and to record the interrupt progress position of the first inference task so as to execute each subtask of the second inference task. The NPU is also used to return to the interrupted progress position after all subtasks of the second task to be inferred have been executed, and to execute the unfinished subtasks of the first task to be inferred in the first order, so as to complete the inference of the first task to be inferred.

5. The chip as described in claim 4, characterized in that, The NPU includes a command dispatch unit, a computing unit, and on-chip memory; The command distribution unit is configured to write the configuration information of each subtask of the task to be reasoned into the computing unit; The computing unit is used to perform calculations on the subtask based on the configuration information to obtain the subtask processing result. The on-chip memory is used to temporarily store the processing results of the subtask.

6. The chip as described in claim 5, characterized in that, The computer device includes off-chip memory; the NPU also includes a memory access module, an interrupt handler, and on-chip registers; The interrupt handler is configured to send an interrupt message to the command distribution unit when the second inference task occurs at time k+n. The memory access module is used to store the pause information of the first inference task to the on-chip register based on the interrupt information, and write the completed subtask processing results in the on-chip memory to the off-chip memory. The on-chip register is used to maintain the pause information of the first inference task.

7. The chip as described in claim 6, characterized in that, The interrupt processor is further configured to send recovery information to the command distribution unit after all subtasks of the second inference task have been executed, so that the NPU obtains pause information from the on-chip register and executes the remaining subtasks in the first inference task based on the pause information to complete the inference of the first inference task.

8. The chip as described in claim 7, characterized in that, The on-chip registers include a checkpoint register, an address register, and a status register; the pause information includes at least the interrupt progress position, storage address, and interrupt status of the paused subtask that was interrupted. The checkpoint register is used to store the interruption progress position of the paused subtask; The address register is used to store the storage address of various types of data of the paused subtask; The status register is used to store the interrupt status of the paused subtask.

9. A task compilation method for an artificial intelligence model, characterized in that, The compilation method includes: The artificial intelligence model is compiled and divided according to the number of network layers to obtain a pending division result. The pending division result includes L sub-tasks, and each sub-task contains several consecutive network layers. Each subtask of the undetermined partitioning result is iteratively processed according to a preset processing method to adjust the number of network layers in each subtask until the performance balance condition is met between each subtask, and the iteration stops to obtain the target partitioning result that meets the performance balance condition. The target partitioning result includes N subtasks; where L and N are both positive integers. Each of the subtasks serves as the basic scheduling unit of the NPU to perform inference on the inference task related to the artificial intelligence model.

10. The compilation method as described in claim 9, characterized in that, The preset processing method includes: The undetermined partitioning result is simulated to obtain the simulation result; The performance indicators of the simulation results are evaluated to obtain the evaluation results; The undetermined partitioning results are compiled and adjusted based on the evaluation results so that the performance balance condition is met among each subtask.

11. The compilation method as described in claim 10, characterized in that, The simulation of the undetermined partitioning result to obtain the simulation result includes: The subtask is fed into the first simulation model for transaction-level simulation to simulate the behavior of the subtask in each clock cycle and obtain simulation results. The evaluation of the performance metrics of the simulation results to obtain the evaluation results includes: The simulation results are fed into a performance evaluator to evaluate the performance metrics of the simulation results and obtain evaluation results; the performance metrics include at least one of the following: the latency of the whole network inference of the artificial intelligence model, the latency of a single subtask, and the on-chip network bandwidth utilization.

12. A task execution device, characterized in that, The actuator includes: The inference module is used to enable the NPU to call the artificial intelligence model at time k and execute the subtasks of the first inference task of the target application in a first order to perform task inference on the first inference task. The interrupt module is used to cause the CPU to send an interrupt signal to the NPU when a second inference task appears in the target application at time k+n. The second inference task has a higher priority than the first inference task in the task queue, and both k and n are positive numbers. The inference switching module is used to enable the NPU to interrupt the processing of the first inference task when it receives the interrupt signal at the (k+n)th time, and to record the interrupt progress position so as to execute each subtask of the second inference task. The recovery module is used to enable the NPU to return to the interrupted progress position after completing the execution of each subtask of the second inference task, and to execute the unfinished subtasks in the first inference task in the first order to complete the inference of the first inference task.

Images