Intelligent application control system, intelligent application task processing method and electronic equipment

Abstract

The present application is applicable to the field of edge computing technology, and provides an intelligent application control system, an intelligent application task processing method, and an electronic device. The system includes: a first processor and a second processor; the first processor is configured with a component scheduling module and a model scheduling module running in kernel state; the component scheduling module manages the running status of each functional component in the intelligent application, and sends a first task set to the model scheduling module when the functional component is running; the model scheduling module partially or completely merges the multiple target model tasks when there are multiple target model tasks in the first task set, obtains a second task set that meets the batch processing conditions, and sends a model driving instruction to the second processor; the second processor is used to trigger the relevant neural network model to execute the model tasks in the second task set based on the model driving instruction, and feeds back the results of the task processing to the first processor. Processor overhead can be reduced through this application.

Description

Technical Field

[0001] The present application belongs to the field of edge computing technology, and in particular relates to an intelligent application control system, an intelligent application task processing method, and an electronic device. Background Art

[0002] A heterogeneous system includes multiple different types of processors, such as processor 1 and processor 2. Processor 1 can control processor 2. The operating system (such as Linux) distinguishes between kernel state and user state. Processor 2 has a driver in kernel state and an application programming interface (API) in user state. When an intelligent application involving processor 2 calls a functional component in user state to complete the application process, the functional component needs to call the driver of processor 2 through processor 2's API to control processor 2. In this operating mode, each time an intelligent application calls processor 2's API, a switch between user state and kernel state occurs, and this state switch generates processor overhead. Summary of the Invention

[0003] The embodiments of the present application provide an intelligent application control system, an intelligent application task processing method, and an electronic device, which can reduce processor overhead.

[0004] In a first aspect, an embodiment of the present application provides an intelligent application control system, comprising:

[0005] a first processor and a second processor;

[0006] The first processor is configured with: a component scheduling module and a model scheduling module running in kernel state;

[0007] The component scheduling module is used to manage the operating status of each functional component in the intelligent application and send a first task set to the model scheduling module when the functional component is running. The first task set includes: model tasks triggered by the running of the functional component, and the model tasks are tasks that depend on the execution of the neural network model;

[0008] The model scheduling module is configured to: when there are multiple target model tasks in the first task set, partially or completely merge the multiple target model tasks in the first task set to obtain a second task set that meets a batch processing condition, wherein the multiple target model tasks are model tasks corresponding to the same neural network model, and the batch processing condition includes: the execution time of each model task in the second task set does not exceed the execution time limit corresponding to each model task; and send a model driving instruction to the second processor based on the model tasks included in the second task set;

[0009] The second processor is used to: trigger the relevant neural network model to execute the model tasks in the second task set based on the model driving instruction, and feed back the results of task processing to the first processor.

[0010] In an embodiment of the present application, the first processor manages the running status of each functional component in the intelligent application through a component scheduling module running in kernel state, and can send the model task involving the second processor to the second processor through the model scheduling module running in kernel state to trigger the relevant neural network model to execute the model task, and feedback the result of task processing to the first processor. In the control process of the above-mentioned intelligent application, there is no need to call the API of the second processor running in user state, so it can reduce the repeated switching between kernel state and user state and reduce the overhead of the first processor. And when there are multiple target model tasks in the first task set, the model scheduling module can batch process some or all of the target model tasks in the first task set by partially or fully merging the multiple target model tasks in the first task set, thereby reducing bandwidth overhead and improving the utilization rate of the second processor.

[0011] In a second aspect, an embodiment of the present application provides a method for processing intelligent application tasks, which is applied to a first processor, wherein the first processor is configured with: a component scheduling module and a model scheduling module running in kernel state; the component scheduling module is used to manage the operating status of each functional component in the intelligent application;

[0012] The method comprises:

[0013] When the functional component is running, triggering the component scheduling module to send a first task set to the model scheduling module, the first task set including: model tasks triggered by the running of the functional component, the model tasks being tasks that depend on the execution of the neural network model;

[0014] In a case where there are multiple target model tasks in the first task set, triggering the model scheduling module to partially or completely merge the multiple target model tasks in the first task set to obtain a second task set that meets a batch processing condition, wherein the multiple target model tasks are model tasks corresponding to the same neural network model, and the batch processing condition includes: the execution time of each model task in the second task set does not exceed the execution time limit corresponding to each model task;

[0015] The model scheduling module is triggered to send a model-driven instruction to the second processor based on the tasks included in the second task set. The model-driven instruction is used to trigger the second processor to execute the tasks in the second task set and to feedback the results of task processing to the first processor. The second processor is configured with task processing capabilities based on a neural network model.

[0016] In an embodiment of the present application, the first processor manages the operating status of each functional component in the intelligent application through a component scheduling module running in kernel state, which can reduce repeated switching between kernel state and user state, thereby reducing the overhead of the first processor. In addition, when there are multiple target model tasks in the first task set, the first processor triggers the model scheduling module to partially or completely merge the multiple target model tasks in the first task set, and can batch process some or all of the target model tasks in the first task set, thereby reducing bandwidth overhead and improving the utilization of the second processor.

[0017] In a third aspect, an embodiment of the present application provides an intelligent application task processing method, which is applied to a second processor, wherein the second processor is configured with an operator scheduling module;

[0018] The method comprises:

[0019] In a case where the model tasks included in the second task set are split into a plurality of operator tasks based on the model-driven instruction, triggering the operator scheduling module to schedule the plurality of operator tasks to obtain a scheduling strategy for the plurality of operator tasks;

[0020] The operator scheduling module is triggered to trigger the operator of the relevant neural network model to execute the corresponding operator task according to the scheduling strategy, and the result obtained by task processing is fed back to the first processor.

[0021] In an embodiment of the present application, the second processor is configured with an operator scheduling module. When the second processor splits the model tasks contained in the second task set into multiple operator tasks based on the model-driven instructions, it schedules the multiple operator tasks by triggering the operator scheduler, and triggers the operators of the relevant neural network model to execute the corresponding operator tasks according to the obtained scheduling strategy. It can reasonably allocate the resources of the second processor to the operator tasks and improve the utilization rate of the second processor.

[0022] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the electronic device implements the method described in the second aspect or the third aspect above.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method described in the second aspect or the third aspect is implemented.

[0024] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed, enables the method described in the second aspect or the third aspect to be executed.

[0025] It can be understood that the beneficial effects of the fourth to sixth aspects can be found in the relevant descriptions of the first, second and third aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 This is an example diagram of the operation mode of the smart application;

[0028] Figure 2 This is an example diagram of running multiple intelligent applications simultaneously;

[0029] Figure 3 This is a schematic diagram of the structure of the intelligent application control system provided by an embodiment of the present application;

[0030] Figure 4 This is an example diagram of an existing model task in the model queue provided by an embodiment of the present application;

[0031] Figure 5 This embodiment of the present application provides Figure 4 An example diagram of model task arrangement after each model task is scheduled;

[0032] Figure 6 This is an example diagram of the actual execution start and end time and execution time limit of each model task provided in the embodiment of the present application;

[0033] Figure 7 This is an example diagram of an operator task already in the operator queue provided by an embodiment of the present application;

[0034] Figure 8 This is an example diagram of a scheduling strategy for operator tasks in an operator queue provided in an embodiment of the present application;

[0035] Figure 9 This is another structural diagram of the intelligent application control system provided by an embodiment of the present application;

[0036] Figure 10 This is a flow chart of a method for processing intelligent application tasks provided by an embodiment of the present application;

[0037] Figure 11 This is another flowchart of the intelligent application task processing method provided by an embodiment of the present application;

[0038] Figure 12 This is a structural diagram of an intelligent application task processing device provided by an embodiment of the present application;

[0039] Figure 13 This is another structural diagram of the intelligent application task processing device provided in an embodiment of the present application;

[0040] Figure 14 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0042] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0043] It should be understood that a heterogeneous system typically includes a first processor and at least one second processor. The first processor can be understood as the core of the heterogeneous system, responsible for assigning different tasks to the second processors and coordinating the work between the second processors to ensure that the second processors work together to complete complex tasks. As an example and not a limitation, the first processor is a central processing unit (CPU), and the second processor is at least one of a variety of processors such as a neural network processing unit (NPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), and a graphics processing unit (GPU). The NPU is a hardware module specially designed to run neural network models, aiming to improve the processing efficiency and performance of machine learning and deep learning tasks.

[0044] For ease of understanding, this application is described by taking the first processor as a CPU and the second processor as an NPU as an example, but this does not constitute a limitation on the types of the first processor and the second processor.

[0045] Currently, the operating mode of smart applications is as follows Figure 1 As shown, the intelligent application sequentially calls various functional components in user mode, such as component A1, component A2, and component A3. Most of these functional components require calling the NPU. The current method of calling the NPU is to call the NPU driver running in kernel mode through the NPU API running in user mode to control the NPU. Each time the intelligent application calls the NPU API, a state switch between user mode and kernel mode occurs, thereby generating CPU overhead. In order to reduce the CPU overhead generated by state switching, an embodiment of the present application provides an intelligent application control system, which includes a first processor and a second processor. The first processor manages the operating status of each functional component in the intelligent application through a component scheduling module running in kernel mode. Model tasks involving the second processor can be sent to the second processor through the model scheduling module running in kernel mode to trigger the relevant neural network model to execute the model task, and the results of the task processing are fed back to the first processor. During the control process of the above-mentioned intelligent application, there is no need to call the API of the second processor running in user mode, so it can reduce repeated switching between kernel mode and user mode and reduce the overhead of the first processor (for example, reducing CPU overhead). The API of the second processor is a general term for the user-mode program and its interface that controls the operation of the second processor. The API of the second processor can call the driver of the second processor through input / output control (IOCTL) and other methods to control the second processor.

[0046] Figure 1 The operation mode of the intelligent application shown in the figure requires loading the operation instructions and coefficients each time the NPU is called to run the neural network model. The computing unit will not start running until the operation instructions and coefficients are loaded. The computing unit is not running during the process of loading the operation instructions and the system, which reduces the utilization of the NPU. The utilization of the NPU can refer to the efficiency of the use of its computing power units when the NPU performs tasks, usually expressed as a percentage. High utilization means that the NPU is performing neural network calculations most of the time, while low utilization indicates that the computing power units of the NPU may be idle. For example, when the NPU is loading operation instructions and coefficients, the computing power units may be idle. Moreover, with the development of electronic devices, there will be more and more demands to run multiple intelligent applications at the same time, such as the requirement to run multiple perimeter protection, multiple target recognition and other intelligent applications at the same time. Figure 2The figure shows an example of running multiple intelligent applications simultaneously. Running multiple intelligent applications simultaneously not only increases the challenge to NPU utilization, but also increases the pressure on resource overhead such as the CPU and bandwidth. Specifically, the NPU needs to spend more time loading operating instructions and coefficients, and frequent switching between user mode and kernel mode will also significantly increase CPU overhead. An embodiment of the present application provides a method for processing intelligent application tasks. This method is applied to a first processor. The first processor manages the operating status of each functional component in the intelligent application through a component scheduling module running in kernel mode, which can reduce repeated switching between kernel mode and user mode, thereby reducing the overhead of the first processor. Furthermore, when the first processor has multiple target model tasks in a first task set, the model scheduling module is triggered to partially or fully merge the multiple target model tasks in the first task set. This allows batch processing of some or all of the target model tasks in the first task set, thereby reducing bandwidth overhead and improving the utilization of the second processor. The above-mentioned process of partially or fully merging the multiple target model tasks in the first task set is automatically performed by the model scheduling module at the bottom layer, without the need for direct intervention or attention from the intelligent applications, and the intelligent applications remain decoupled from each other.

[0047] As an example and not a limitation, for N concurrently running inference tasks corresponding to the same neural network model, these N inference tasks can be batched by partially or fully merging them. The NPU only needs to load the operation instructions and coefficients once and then perform N inferences, reducing the number of operations of loading operation instructions and coefficients by N-1 times. In this way, the NPU's computing power units or computing units can have more time for calculations, which improves NPU utilization and reduces the number of times operation instructions and coefficients are read from memory, which reduces bandwidth overhead. N is an integer greater than 1.

[0048] Among them, the neural network model is a mathematical model or computational model that imitates the structure and function of a biological neural network (brain), and can be used to perform tasks such as target detection and classification. In edge computing, the neural network model usually runs on the NPU, and the neural network model usually consists of multiple operators, each of which is calculated by a different sub-intellectual property (IP) within the NPU. The NPU sub-IP can be understood as the computing power unit of the NPU. Operation instructions can refer to instructions for performing specific operations. As an example and not a limitation, the above-mentioned operation instructions may include operation instructions such as addition, subtraction, multiplication, and division, and may also include convolution operation instructions, normalization instructions, matrix operation instructions, etc. This application does not limit the specific types of the above-mentioned operation instructions. The above-mentioned coefficients may refer to weight parameters in the neural network model. These weight coefficients are learned during the model training process and are used to determine the connection strength between neurons.

[0049] In addition, the NPU usually includes multiple computing units for processing different operators in the neural network model. In multi-channel intelligent applications, there are usually multiple operator tasks of multiple neural network models waiting to be executed. The embodiment of the present application schedules multiple operator tasks by triggering the operator scheduler, and triggers the operators of the relevant neural network models to execute the corresponding operator tasks according to the obtained scheduling strategy, which can reasonably allocate NPU resources to the operator tasks and improve the utilization rate of the NPU.

[0050] See also Figure 3 , Figure 3 A schematic diagram of the structure of the intelligent application control system provided by the embodiment of the present application is shown. Figure 3 As shown, the intelligent application control system includes: a first processor 301 and a second processor 302.

[0051] The first processor is configured with: a component scheduling module and a model scheduling module running in kernel state;

[0052] The component scheduling module is used to manage the operating status of each functional component in the intelligent application and send a first task set to the model scheduling module when the functional component is running. The first task set includes: model tasks triggered by the running of the functional component, and the model tasks are tasks that depend on the execution of the neural network model;

[0053] The model scheduling module is configured to: when there are multiple target model tasks in the first task set, partially or completely merge the multiple target model tasks in the first task set to obtain a second task set that meets a batch processing condition, wherein the multiple target model tasks are model tasks corresponding to the same neural network model, and the batch processing condition includes: the execution time of each model task in the second task set does not exceed the execution time limit corresponding to each model task; and send a model driving instruction to the second processor based on the model tasks included in the second task set;

[0054] The second processor is used to: trigger the relevant neural network model to execute the model tasks in the second task set based on the model-driven instruction, and feed back the results of the task processing to the first processor.

[0055] Intelligent applications are functions with practical uses, such as target recognition systems and perimeter defense systems. An intelligent application consists of several components, such as those that implement detection and classification. In intelligent applications, these components are typically implemented by neural network models.

[0056] The target recognition system mainly authenticates or identifies the target by analyzing and comparing the visual feature information of the target object. It is widely used in scenarios such as security systems, mobile device unlocking, monitoring systems, and social media tagging.

[0057] A perimeter security system is a protective measure installed around a specific area or facility to prevent unauthorized persons or objects from entering or leaving the area. These measures typically include physical barriers and electronic surveillance systems, providing multi-layered security. Perimeter security systems can be used in a variety of settings, including residential areas, commercial facilities, industrial sites, and specialized bases.

[0058] The functional components in an intelligent application may refer to the functional modules that constitute the intelligent application. For example, a perimeter protection system may include a functional component for human detection, a functional component for face detection, a functional component for classification, and the like.

[0059] The running state of a functional component includes, but is not limited to, the creation, startup, operation, termination, and destruction of the functional component. In this embodiment, each functional component runs in kernel state, and the running state of each functional component can be managed by the component scheduling module running in kernel state. On this basis, model tasks are executed, which can reduce repeated switching between kernel state and user state and reduce CPU overhead. Among them, user state is an execution mode of the operating system. Programs in user state are subject to strict permission restrictions and cannot directly access hardware resources or execute privileged instructions. Kernel state is an execution mode of the operating system. Programs in kernel state have the highest permissions and can access all hardware resources and execute all privileged instructions.

[0060] The first processor can run one intelligent application or multiple intelligent applications at the same time. When running multiple intelligent applications at the same time, the component scheduling module can send the corresponding model task to the model scheduling module when any functional component in the multiple intelligent applications is running.

[0061] The multi-channel intelligent applications may be a variety of different intelligent applications, or multiple copies of the same intelligent application from different sources.

[0062] By way of example and not limitation, electronic devices such as IP cameras (IPCs), network video recorders (NVRs), mobile phones, tablets, and IoT devices can run multiple different smart applications simultaneously, or they can run multiple copies of the same smart application processing different video sources. IPCs are a new generation of cameras that combine traditional cameras with network technology. They can transmit video images to remote locations via the network, allowing remote viewers to view the images through a web browser. NVRs are the storage and forwarding component of a network video surveillance system. NVRs work in conjunction with video encoders or network cameras to record, store, and forward video.

[0063] It should be understood that in the case of multi-channel intelligent applications, there are cases where different intelligent applications use the same neural network model. For example, if 8 perimeter protection systems are running simultaneously in the NVR, the neural network models used by these 8 perimeter protection systems are usually the same.

[0064] It should be understood that the first processor and the second processor in the intelligent application control system can be deployed on the same electronic device or on different devices, and this application is not limited to this. As an example and not a limitation, the first processor is deployed on the electronic device and the second processor is deployed on the server, or the first processor is deployed on one electronic device and the second processor is deployed on another electronic device.

[0065] In one embodiment, the component scheduling module may add the model task to the model queue when each functional component triggers the corresponding model task, and the model scheduling module obtains the first task set from the model queue.

[0066] In one embodiment, the model scheduling module may schedule the model tasks in the model queue each time a new model task is queued in the model queue.

[0067] In one embodiment, the model scheduling module may send model tasks to the second processor in the order in which the model tasks in the second task set are executed. Optionally, multiple model tasks may be sent to the second processor in sequence until the number of sent model tasks reaches a task number threshold for the second processor. The task number threshold is used to limit the number of model tasks that can be executed simultaneously by the second processor.

[0068] In one embodiment, the model scheduling module may send a model driving instruction to the second processor via a driver of the second processor running in kernel mode, the model driving instruction carrying the second task set. The driver of the second processor is a program that controls the operation of the second processor and typically includes functions such as configuring registers, controlling the startup of the second processor, and receiving interrupt signals from the second processor.

[0069] The target model tasks in the aforementioned multiple target model tasks can be model tasks corresponding to different functional components of a single intelligent application, or model tasks corresponding to different functional components of multiple intelligent applications. The model tasks corresponding to functional components are model tasks triggered when the functional components are running.

[0070] As an example and not a limitation, intelligent application 1 includes functional component 11, functional component 12, and functional component 13. The model task corresponding to functional component 11 and the model task corresponding to functional component 13 correspond to the same neural network model 1, and these two functional components can be executed simultaneously. Therefore, when running intelligent application 1, the model task corresponding to functional component 11 and the model task corresponding to functional component 13 can be determined as the target model task. Intelligent application 2 includes functional component 21, functional component 22, and functional component 23. The model task corresponding to functional component 22 and the model task corresponding to functional component 12 in intelligent application 1 correspond to the same neural network model 2. Therefore, when running intelligent application 1 and intelligent application 2 simultaneously, the model task corresponding to functional component 22 and the model task corresponding to functional component 12 can be determined as the target model task.

[0071] After partially or fully merging multiple target model tasks, batch tasks can be obtained. The second task set includes the batch tasks. If the first task set also includes remaining tasks, then it is determined that the second task set also includes the remaining tasks. The remaining tasks may refer to model tasks in the first task set other than the target model tasks being merged.

[0072] like Figure 4 The figure shows an example of existing model tasks in the model queue. Assume that the current time is 7, when Model Task 5 enters the model queue. The model queue has received five model tasks, Model Task 1, Model Task 2, Model Task 3, Model Task 4, and Model Task 5, from the component scheduling module, waiting for execution. Model Task 2 and Model Task 4 correspond to the same neural network model. Therefore, the model scheduling module can merge Model Task 2 and Model Task 4 into a batch while ensuring that the execution time of each model task does not exceed the corresponding execution time limit.

[0073] When there are multiple target model tasks in the first task set, the task merging unit can partially or completely merge the multiple target model tasks in the first task set, so that the second processor can batch process the batch tasks in the second task set, thereby reducing bandwidth overhead, improving the utilization of the second processor, and increasing the number of intelligent applications.

[0074] In one possible implementation, the model scheduling module includes a task scheduling unit, a sequential execution unit, and a target determination unit;

[0075] The task scheduling unit is used to schedule each model task in the first task set based on the priority, execution time limit, estimated running time and queue entry time of each model task in the first task set, obtain a first execution order of each model task in the first task set, and store each model task in the first task set in a model queue;

[0076] The sequential execution unit is configured to: determine a second execution order of the plurality of target model tasks based on the first execution order when there are a plurality of target model tasks in the first task set;

[0077] The target determination unit is used to: merge part or all of the multiple target model tasks based on the second execution order and the estimated running time of the multiple target model tasks.

[0078] The execution time limit of the model task may refer to a limit set on the execution time of the model task during the execution of the model task.

[0079] This embodiment schedules each model task in the first task set based on its priority, execution deadline, estimated running time, and enqueue time, thereby ensuring that important tasks can be executed in a timely manner.

[0080] The first execution order is the execution order of each model task in the first task set. The multiple target model tasks are model tasks corresponding to the same neural network in the first task set. Therefore, the sequential execution unit can determine the execution order of the multiple target model tasks (i.e., the second execution order) based on the first execution order. The target determination unit partially or fully merges the multiple target model tasks based on the second execution order and the estimated running time of the multiple target model tasks, which can ensure that the execution time of each model task in the second task set does not exceed the execution time limit corresponding to each model task.

[0081] As an example and not a limitation, the first execution order of model task 1, model task 2, model task 3, model task 4 and model task 5 is model task 1, model task 5, model task 4, model task 2 and model task 3. If after merging model task 2 and model task 4, it can be guaranteed that the execution time of model task 1, model task 5, batch task and model task 3 does not exceed the corresponding execution time limit, and model task 2 and model task 4 correspond to the model of the same neural network, then model task 2 and model task 4 can be merged. Therefore, the execution order after the model scheduling module schedules the model tasks in the model queue is model task 1, model task 5, batch task and model task 3.

[0082] It should be understood that the estimated runtimes for multiple target model tasks corresponding to the same neural network model are generally the same. For a given neural network model and NPU, the runtime of the neural network model on the NPU can be theoretically calculated, so the estimated runtime can be set based on this theoretical value.

[0083] In one embodiment, before scheduling each model task in the first task set based on the priority, execution deadline, estimated running time and enqueue time of each model task in the first task set, the task scheduling unit may first determine whether there is a model task in the first task set whose execution deadline and the time interval between the current moment are less than the first preset time length. If there is a model task in the first task set whose execution deadline and the time interval between the current moment are less than the first preset time length, the priority of the model task may be increased, that is, as time goes by, the priority of the model task that is close to the execution deadline is gradually increased. Optionally, for any model task, a first preset time length may be set based on the estimated running time of the model task, such as setting the first preset time length to be slightly greater than or equal to the estimated running time of the model task, for example, determining the sum of the estimated running time of the model task and the preset first target value as the first preset time length.

[0084] Optionally, when the first preset duration is greater than the expected running time of the corresponding model task, for any model task that needs to have its priority increased, if the time interval between the execution deadline of the model task and the current moment is less than the first preset duration and is greater than or equal to the expected running time of the model task, the priority of the model task is increased by P levels, where P is an integer greater than zero; if the time interval between the execution deadline of the model task and the current moment is less than the expected running time, the priority of the model task is increased by M levels, where M is an integer greater than P.

[0085] In a possible implementation, the task scheduling unit is specifically configured to:

[0086] The model tasks in the first task set are sorted in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of enqueuing time.

[0087] Here, the descending priority order may refer to the order from high to low priority. The ascending execution time order may refer to the order from earliest to latest execution time. The ascending estimated running time order may refer to the order from shortest to longest estimated running time. The ascending enqueuing time order may refer to the order from earliest to latest enqueuing time.

[0088] Specifically, the task scheduling module can sort the model tasks in the first task set in descending order of priority; in the case where there are multiple model tasks with the same priority in the first task set, the multiple model tasks with the same priority are sorted in ascending order of execution time limit; in the case where there are no multiple model tasks with the same priority in the first task set, it means that the sorting of the model tasks in the first task set can be completed in descending order of priority, and there is no need to sort them in ascending order of execution time limit, ascending order of estimated running time, and ascending order of time of joining the queue. In the case where there are multiple model tasks with the same execution time limit among multiple model tasks with the same priority, the multiple model tasks with the same execution time limit are sorted in ascending order of estimated running time; in the case where there are no multiple model tasks with the same execution time limit among multiple model tasks with the same priority, it means that the sorting of the model tasks in the first task set can be completed in descending order of priority and ascending order of execution time limit, and there is no need to sort them in ascending order of estimated running time and ascending order of time of joining the queue. When there are multiple model tasks with the same execution time limit (that is, multiple model tasks with the same execution time limit existing in multiple model tasks with the same priority) with multiple model tasks with the same estimated running time, the multiple model tasks with the same estimated running time are sorted in ascending order of the time of enqueuing; when there are not multiple model tasks with the same estimated running time among multiple model tasks with the same execution time limit, it means that the sorting of each model task in the first task set can be completed in descending order of priority, ascending order of execution time limit and ascending order of estimated running time, and there is no need to sort in ascending order of enqueuing time.

[0089] This embodiment sorts the model tasks in the first task set in descending order of priority, ascending order of execution deadline, ascending order of estimated running time and ascending order of time of joining the queue, thereby ensuring that important tasks with higher priority, tighter execution deadline, shorter estimated running time and earlier time of joining the queue can be executed in a timely manner.

[0090] It should be understood that when sorting the model tasks in the first task set, the execution order can also be set in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of enqueuing time according to scenario requirements. This application does not limit this.

[0091] In a possible implementation, the target determination unit is specifically configured to:

[0092] The first model task in the second execution order is determined as the first model task, and the target model task next to the first model task in the second execution order is determined as the second model task; the first model task is the target model task that is first in the second execution order;

[0093] Based on the estimated running time of the first model task and the estimated running time of the second model task, determining whether the execution time of the third model task exceeds the corresponding execution time limit; the third model task is a model task that is located after the first model task in the first execution order and is other than the first model task and the second model task;

[0094] When the execution time of the third model task does not exceed the corresponding execution time limit, the first model task and the second model task are merged to obtain a batch task;

[0095] Determine the batch task as the first model task, determine the target model task that is second only to the second model task in the second execution order as the second model task, and return to the steps of executing the estimated running time based on the first model task and the estimated running time of the second model task, and determine whether the execution time of the third model task exceeds the corresponding execution time limit, as well as subsequent steps, until the execution time of the third model task exceeds the corresponding execution time limit or multiple target model tasks are traversed.

[0096] The execution time of the first model task can be determined based on the estimated running time of the model task that precedes the first model task in the first execution order. The execution time of the third model task can be determined based on the execution time of the first model task, the estimated running time of the first model task, and the estimated running time of the second model task, so as to judge whether the execution time of the third model task exceeds the corresponding execution time limit. If the execution time of the third model task does not exceed the corresponding execution time limit, the first model task and the second model task can be merged, and the traversal of the multiple target model tasks can continue. If the execution time of the third model task exceeds the corresponding execution time limit, it is determined that the first model task and the second model task cannot be merged, and the traversal of the multiple target model tasks is stopped.

[0097] In one possible implementation, the model scheduling module further includes a time determination unit;

[0098] The time determination unit is used to determine the execution time of the first model task as the execution time of the batch task.

[0099] This embodiment determines the execution time of the first model task as the execution time of the batch task, so that other model tasks in the batch task can be advanced to be batch processed together with the first model task. Other model tasks refer to target model tasks other than the first model task in the batch task.

[0100] like Figure 5 The shown is Figure 4The following is an example diagram of the model task arrangement after the model tasks in are scheduled. Assume that the current time is 7, at this time, model task 5 enters the model queue, and the model queue has model tasks 1, 2, 3, 4 and 5. Figure 4 If the model tasks 2 and 4 in the example are executed separately, the operation instructions and coefficients of the corresponding neural network models need to be loaded twice. Figure 5 The batch processing task in only needs to load the operation instructions and coefficients once, which saves the loading process of the operation instructions and coefficients of the corresponding neural network model. This can make the running time of the batch processing task less than the total running time of model task 2 and model task 4. Figure 4 The estimated running time of model task 2 and model task 4 is 4 seconds, so the total running time of these two model tasks is 8 seconds. Figure 5 The batch processing task takes 6 seconds to run.

[0101] like Figure 6 The following is an example diagram of the actual execution start and end time and the required execution time limit of each model task after scheduling. Figure 6 The actual execution start time of the model task is the execution time of the model task. Figure 6 It can be seen that the execution time of each model task in the second task set does not exceed the corresponding execution time limit.

[0102] In one embodiment, the first processor is further configured with an application scheduling module. The application scheduling module runs in user mode and is responsible for creating, starting, terminating, and destroying each intelligent application, and does not participate in the scheduling of functional components during the operation of the intelligent application.

[0103] In one possible implementation, the second processor is configured with an operator scheduling module;

[0104] The operator scheduling module is used to: when the model tasks contained in the second task set are split into multiple operator tasks based on the model-driven instructions, schedule the multiple operator tasks, obtain the scheduling strategy of the multiple operator tasks, and trigger the operators of the relevant neural network model to execute the corresponding operator tasks according to the scheduling strategy, and feedback the results of task processing to the first processor.

[0105] The operator scheduling module schedules multiple operator tasks and triggers the operators of the relevant neural network model to execute the corresponding operator tasks according to the obtained scheduling strategy. It can reasonably allocate the resources of the second processor to the operator tasks and improve the utilization rate of the second processor.

[0106] In one possible implementation, the operator scheduling module includes a parallel determination unit, a first processing unit, and a second processing unit;

[0107] The parallel determination unit is configured to: determine parallel execution as a scheduling strategy for the multiple operator task teams when there are multiple operator task teams that need to be executed in parallel among the multiple operator tasks and the computing power units corresponding to the respective operator tasks in the multiple operator task teams are in an idle state; each operator task team includes at least one operator task;

[0108] The first processing unit is configured to: for an operator task team including two or more operator tasks, schedule each operator task in the operator task team based on the priority, execution time limit, estimated running time, and time of entry of each operator task in the operator task team, obtain an execution order of each operator task in the operator task team, determine the execution order of each operator task in the operator task team as a scheduling policy for each operator task in the operator task team, and store each operator task in the operator task team in an operator queue;

[0109] The second processing unit is used to: when there is no operator task queue among the multiple operator tasks and the computing power units corresponding to the multiple operator tasks are in an idle state, schedule the multiple operator tasks based on the priority, execution time limit, estimated running time and entry time of the multiple operator tasks, obtain the execution order of the multiple operator tasks, and determine the execution order of the multiple operator tasks as the scheduling strategy of the multiple operator tasks, and the multiple operator tasks are stored in the operator queue.

[0110] The operators of the neural network model typically run on the computing unit of the second processor, and multiple operator task teams executed in parallel correspond to different computing units. Operator tasks in the same operator task team can be related operator tasks or operator tasks corresponding to the same computing unit.

[0111] The execution time limit of an operator task may refer to a limit set on the execution time of an operator task during the execution of the operator task.

[0112] In one embodiment, before the first processing unit or the second processing unit schedules each operator task based on the priority, execution deadline, expected running time, and enqueue time of each operator task, it can first determine whether there is an operator task among the operator tasks whose execution deadline and the time interval between the current moment are less than the second preset time length. If there is an operator task among the operator tasks whose execution deadline and the time interval between the current moment are less than the second preset time length, the priority of the operator task can be increased, that is, as time goes by, the priority of the operator task approaching the execution deadline is gradually increased. Optionally, for any operator task, a second preset time length can be set based on the expected running time of the operator task, such as setting the second preset time length to be slightly greater than or equal to the expected running time of the operator task, for example, the sum of the expected running time of the operator task and the preset second target value is determined as the second preset time length.

[0113] Optionally, when the second preset duration is greater than the estimated running time of the corresponding operator task, for any operator task that needs to have its priority raised, if the time interval between the execution deadline of the operator task and the current moment is less than the second preset duration and is greater than or equal to the estimated running time of the operator task, the priority of the operator task is raised by L levels, where L is an integer greater than zero; if the time interval between the execution deadline of the operator task and the current moment is less than the estimated running time, the priority of the operator task is raised by S levels, where S is an integer greater than L. This embodiment schedules each operator task according to its priority, execution deadline, estimated running time, and enqueuing time, thereby ensuring that important tasks can be executed in a timely manner. Furthermore, by executing multiple operator task teams in parallel, the total time required to complete the operator tasks can be reduced, thereby improving the utilization rate of the second processor.

[0114] In one embodiment, after splitting each model task into multiple operator tasks, the second processor can add the operator task to the operator queue. The operator scheduling module schedules each operator task in the operator queue and determines the execution order or parallel execution of each operator task based on the priority, execution time limit, expected running time, enqueuing time and idle status of the computing power unit in the second processor.

[0115] In a possible implementation, the first processing unit is specifically configured to: sort the operator tasks in the operator task queue in descending order of priority, ascending order of execution time limit, ascending order of estimated running time, and ascending order of time of entry into the queue;

[0116] The second processing unit is specifically used to sort the multiple operator tasks in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of enqueuing time.

[0117] Specifically, the first processing unit can sort each operator task in the operator task team in descending order of priority; when there are multiple operator tasks with the same priority in the operator task team, the multiple operator tasks with the same priority are sorted in ascending order of execution time limit; when there are not multiple operator tasks with the same priority in the operator task team, it means that the sorting of each operator task in the operator task team can be completed in descending order of priority, and there is no need to sort them in ascending order of execution time limit, ascending order of estimated running time, and ascending order of time of entry into the queue. When there are multiple operator tasks with the same execution time limit among multiple operator tasks with the same priority, the multiple operator tasks with the same execution time limit are sorted in ascending order of estimated running time; when there are not multiple operator tasks with the same execution time limit among multiple operator tasks with the same priority, it means that the sorting of each operator task in the operator task team can be completed in descending order of priority and ascending order of execution time limit, and there is no need to sort them in ascending order of estimated running time and ascending order of time of entry into the queue. When there are multiple operator tasks with the same execution time limit (that is, multiple operator tasks with the same execution time limit in multiple operator tasks with the same priority) with the same estimated running time, the multiple operator tasks with the same estimated running time are sorted in ascending order of the time they were queued; when there are no multiple operator tasks with the same execution time limit with the same estimated running time, it means that the sorting of the operator tasks in the operator task queue can be completed in descending order of priority, ascending order of execution time limit, and ascending order of estimated running time, and there is no need to sort them in ascending order of the time they were queued.

[0118] The second processing unit can sort each operator task among the multiple operator tasks in descending order of priority; in the case where there are multiple operator tasks with the same priority among the multiple operator tasks, the multiple operator tasks with the same priority are sorted in ascending order of execution time limit; in the case where there are not multiple operator tasks with the same priority among the multiple operator tasks, it means that the sorting of each operator task among the multiple operator tasks can be completed in descending order of priority, and there is no need to sort them in ascending order of execution time limit, ascending order of estimated running time, and ascending order of time of joining the queue. In the case where there are multiple operator tasks with the same execution time limit among the multiple operator tasks with the same priority, the multiple operator tasks with the same execution time limit are sorted in ascending order of estimated running time; in the case where there are not multiple operator tasks with the same execution time limit among the multiple operator tasks with the same priority, it means that the sorting of each operator task among the multiple operator tasks can be completed in descending order of priority and ascending order of execution time limit, and there is no need to sort them in ascending order of estimated running time and ascending order of time of joining the queue. When there are multiple operator tasks with the same execution time limit (that is, multiple operator tasks with the same execution time limit in multiple operator tasks with the same priority) with the same estimated running time, the multiple operator tasks with the same estimated running time are sorted in ascending order of the time they were queued; when there are no multiple operator tasks with the same execution time limit with the same estimated running time, it means that the sorting of the multiple operator tasks can be completed in descending order of priority, ascending order of execution time limit, and ascending order of estimated running time, and there is no need to sort them in ascending order of the time they were queued.

[0119] This embodiment sorts the operator tasks in descending order of priority, ascending order of execution deadline, ascending order of estimated running time, and ascending order of time of joining the queue, thereby ensuring that important tasks with higher priority, tighter execution deadline, shorter estimated running time, and earlier time of joining the queue can be executed in a timely manner.

[0120] It should be understood that when the first processing unit sorts the operator tasks in the operator task team and the second processing unit sorts multiple operator tasks, the execution order can also be set in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of entry time according to scenario requirements. This application does not limit this.

[0121] like Figure 7 The following diagram shows an example of an operator task already in the operator queue. Figure 7The operator queue in includes operator task 1, operator task 2, operator task 3, operator task 4 and operator task 5. The operators corresponding to operator task 1, operator task 2 and operator task 4 are all running on computing power unit 1. The operators corresponding to operator task 3 and operator task 5 are all running on computing power unit 2. And computing power unit 1 and computing power unit 2 are in idle state. Therefore, operator task team 1 including operator task 1, operator task 2 and operator task 4 and operator task team 2 including operator task 3 and operator task 5 can be executed in parallel. For operator task team 1, operator task 1 has the highest priority, operator task 2 and operator task 4 have the same priority, but the execution deadline of operator task 2 is earlier than the execution deadline of operator task 4, so as Figure 8 The execution order of each operator task in operator task team 1 is operator task 1, operator task 2 and operator task 4. For operator task team 2, the priority of operator task 5 is higher than that of operator task 3, so Figure 8 The execution order of the operator tasks in the operator task team 2 is shown as operator task 5 and operator task .

[0122] like Figure 9 FIG. 1 is another structural diagram of the intelligent application control system provided by an embodiment of the present application. Figure 9 The multi-level scheduling modules, including the application scheduling module, component scheduling module, model scheduling module, and operator scheduling module, can realize the control of intelligent applications, reduce CPU overhead and bandwidth overhead, improve NPU utilization, and thus increase the number of intelligent applications.

[0123] See also Figure 10 , Figure 10 A flowchart of a method for processing intelligent application tasks provided by an embodiment of the present application is shown. By way of example and not limitation, the method is applied to a first processor configured with: a component scheduling module and a model scheduling module running in kernel mode; the component scheduling module is used to manage the operating status of each functional component in the intelligent application. The method includes the following steps:

[0124] Step 1001: When the functional component is running, the component scheduling module is triggered to send a first task set to the model scheduling module.

[0125] Among them, the first task set includes: model tasks triggered when the functional components are running, and the model tasks are tasks that depend on the execution of the neural network model.

[0126] Step 1002 , when there are multiple target model tasks in the first task set, trigger the model scheduling module to partially or completely merge the multiple target model tasks in the first task set to obtain a second task set that meets the batch processing condition.

[0127] Among them, the multiple target model tasks are model tasks corresponding to the same neural network model, and the batch processing conditions include: the execution time of each model task in the second task set does not exceed the execution time limit corresponding to each model task.

[0128] Step 1003 , triggering the model scheduling module to send a model driving instruction to the second processor based on the tasks included in the second task set, the model driving instruction is used to trigger the second processor to execute the tasks in the second task set and feed back the results of task processing to the first processor.

[0129] The second processor is configured with task processing capabilities based on a neural network model.

[0130] In a possible implementation, before triggering the model scheduling module to partially or completely merge the multiple target model tasks in the first task set, the method further includes:

[0131] The triggering model scheduling module schedules each model task in the first task set based on the priority, execution time limit, estimated running time and queue entry time of each model task in the first task set, obtains a first execution order of each model task in the first task set, and stores each model task in the first task set in the model queue;

[0132] The triggering model scheduling module determines a second execution order of the plurality of target model tasks based on the first execution order when there are a plurality of target model tasks in the first task set;

[0133] The triggering model scheduling module partially or completely merges multiple target model tasks in the first task set, including:

[0134] The trigger model scheduling module partially or completely merges the multiple target model tasks based on the second execution order and the estimated running time of the multiple target model tasks.

[0135] In one possible implementation, the triggering model scheduling module schedules each model task in the first task set based on the priority, execution time limit, estimated running time, and enqueue time of each model task in the first task set, including:

[0136] The trigger model scheduling module sorts the model tasks in the first task set in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of enqueuing time.

[0137] In one possible implementation, the trigger model scheduling module partially or completely merges the multiple target model tasks based on the second execution order and the estimated running times of the multiple target model tasks, including:

[0138] The trigger model scheduling module determines the first model task in the second execution order as the first model task, and determines the target model task next to the first model task in the second execution order as the second model task; the first model task is the target model task that is first in the second execution order;

[0139] The trigger model scheduling module determines whether the execution time of the third model task exceeds the corresponding execution time limit based on the estimated running time of the first model task and the estimated running time of the second model task; the third model task is a model task that is located after the first model task in the first execution order and is other than the first model task and the second model task;

[0140] The triggering model scheduling module merges the first model task with the second model task to obtain a batch task when the execution time of the third model task does not exceed the corresponding execution time limit;

[0141] The trigger model scheduling module determines the batch task as the first model task, determines the target model task that is second only to the second model task in the second execution order as the second model task, and returns the execution step of judging whether the execution time of the third model task exceeds the corresponding execution time limit based on the estimated running time of the first model task and the estimated running time of the second model task, and the subsequent steps, until the execution time of the third model task exceeds the corresponding execution time limit or multiple target model tasks are traversed.

[0142] In a possible implementation, after obtaining the batch processing task, the following steps are further included:

[0143] The execution time of the first model task is determined as the execution time of the batch task.

[0144] In an embodiment of the present application, the first processor manages the operating status of each functional component in the intelligent application through a component scheduling module running in kernel state, which can reduce repeated switching between kernel state and user state, thereby reducing the overhead of the first processor. In addition, when there are multiple target model tasks in the first task set, the first processor triggers the model scheduling module to partially or completely merge the multiple target model tasks in the first task set, and can batch process some or all of the target model tasks in the first task set, thereby reducing bandwidth overhead and improving the utilization of the second processor.

[0145] It should be noted that Figure 10 The method embodiment shown is Figure 2 The function of the first processor in the control system embodiment shown is based on the same concept. For the relevant content of the method embodiment, please refer to the system embodiment part, which will not be repeated here.

[0146] See also Figure 11 , Figure 11 Another flowchart of the intelligent application task processing method provided in an embodiment of the present application is shown. As an example and not a limitation, the method is applied to a second processor configured with an operator scheduling module. The method includes the following steps:

[0147] Step 1101 : When the model task included in the second task set is split into multiple operator tasks based on the model driving instruction, the operator scheduling module is triggered to schedule the multiple operator tasks to obtain scheduling strategies for the multiple operator tasks.

[0148] Step 1102: trigger the operator scheduling module to trigger the operator of the relevant neural network model to execute the corresponding operator task according to the scheduling strategy, and feed back the result of task processing to the first processor.

[0149] In one possible implementation, the operator scheduling module is triggered to schedule multiple operator tasks, and the scheduling strategies for the multiple operator tasks are obtained, including:

[0150] When there are multiple operator task teams that need to be executed in parallel among the multiple operator tasks and the computing power units corresponding to the operator tasks in the multiple operator task teams are in an idle state, the operator scheduling module is triggered to determine parallel execution as a scheduling strategy for the multiple operator task teams; each operator task team includes at least one operator task;

[0151] For an operator task team including more than two operator tasks, the operator scheduling module is triggered to schedule each operator task in the operator task team based on the priority, execution time limit, expected running time and entry time of each operator task in the operator task team, obtain the execution order of each operator task in the operator task team, and determine the execution order of each operator task in the operator task team as the scheduling policy of each operator task in the operator task team. Each operator task in the operator task team is stored in the operator queue;

[0152] When there is no operator task queue among multiple operator tasks and the computing power units corresponding to the multiple operator tasks are in an idle state, the operator scheduling module is triggered to schedule the multiple operator tasks based on the priorities, execution deadlines, estimated running times and queue entry times of the multiple operator tasks, obtain the execution order of the multiple operator tasks, and determine the execution order of the multiple operator tasks as the scheduling policy of the multiple operator tasks. The multiple operator tasks are stored in the operator queue;

[0153] The computing power unit is used to run the operator that executes the corresponding operator task.

[0154] In one possible implementation, the triggering operator scheduling module schedules each operator task in the operator task queue based on its priority, execution deadline, estimated running time, and enqueue time, including:

[0155] The trigger operator scheduling module sorts the operator tasks in the operator task queue in descending order of priority, ascending order of execution time limit, ascending order of estimated running time, and ascending order of enqueuing time;

[0156] The trigger operator scheduling module schedules multiple operator tasks based on their priorities, execution deadlines, estimated running times, and enqueue times, including:

[0157] The trigger operator scheduling module sorts multiple operator tasks in descending order of priority, ascending order of execution time, ascending order of estimated running time, and ascending order of enqueuing time.

[0158] In an embodiment of the present application, the second processor is configured with an operator scheduling module. When the second processor splits the model tasks contained in the second task set into multiple operator tasks based on the model-driven instructions, it schedules the multiple operator tasks by triggering the operator scheduler, and triggers the operators of the relevant neural network model to execute the corresponding operator tasks according to the obtained scheduling strategy. It can reasonably allocate the resources of the second processor to the operator tasks and improve the utilization rate of the second processor.

[0159] It should be noted that Figure 11 The method embodiment shown is Figure 2 The function of the second processor in the control system embodiment shown is based on the same concept. For details of the method embodiment, please refer to the system embodiment part, which will not be repeated here.

[0160] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0161] Corresponding to the above Figure 10 The intelligent application task processing method described in the embodiment, Figure 12 A schematic diagram of the structure of an intelligent application task processing device provided in an embodiment of the present application is shown. This device is applied to a first processor, which is configured with a component scheduling module and a model scheduling module running in kernel mode. The component scheduling module is used to manage the operating status of each functional component in the intelligent application. For ease of illustration, only the portions relevant to the embodiment of the present application are shown.

[0162] Reference Figure 12 , the device comprises:

[0163] A first model triggering module 1201 is configured to trigger the component scheduling module to send a first task set to the model scheduling module when the functional component is running. The first task set includes: model tasks triggered when the functional component is running, and the model tasks are tasks that depend on the execution of the neural network model;

[0164] The second model triggering module 1202 is configured to trigger the model scheduling module to partially or completely merge the multiple target model tasks in the first task set to obtain a second task set that meets a batch processing condition when multiple target model tasks exist in the first task set, wherein the multiple target model tasks are model tasks corresponding to the same neural network model, and the batch processing condition includes: the execution time of each model task in the second task set does not exceed the execution time limit corresponding to each model task;

[0165] The third model triggering module 1203 is used to trigger the model scheduling module to send a model-driven instruction to the second processor based on the tasks included in the second task set. The model-driven instruction is used to trigger the second processor to execute the tasks in the second task set and to feedback the results of task processing to the first processor. The second processor is configured with task processing capabilities based on a neural network model.

[0166] Optionally, the above device further includes:

[0167] a fourth model triggering module, configured to trigger the model scheduling module to schedule each model task in the first task set based on the priority, execution time limit, estimated running time, and enqueue time of each model task in the first task set, thereby obtaining a first execution order for each model task in the first task set, and storing each model task in the first task set in a model queue;

[0168] a fifth model triggering module, configured to trigger the model scheduling module to determine a second execution order of the plurality of target model tasks based on the first execution order when there are a plurality of target model tasks in the first task set;

[0169] The second model triggering module 1202 is specifically used to:

[0170] The trigger model scheduling module partially or completely merges the multiple target model tasks based on the second execution order and the estimated running time of the multiple target model tasks.

[0171] Optionally, the fourth model triggering module is specifically configured to:

[0172] The trigger model scheduling module sorts the model tasks in the first task set in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of enqueuing time.

[0173] Optionally, the second model triggering module 1202 is specifically configured to:

[0174] The trigger model scheduling module determines the first model task in the second execution order as the first model task, and determines the target model task next to the first model task in the second execution order as the second model task; the first model task is the target model task that is first in the second execution order;

[0175] The trigger model scheduling module determines whether the execution time of the third model task exceeds the corresponding execution time limit based on the estimated running time of the first model task and the estimated running time of the second model task; the third model task is a model task that is located after the first model task in the first execution order and is other than the first model task and the second model task;

[0176] The triggering model scheduling module merges the first model task with the second model task to obtain a batch task when the execution time of the third model task does not exceed the corresponding execution time limit;

[0177] The trigger model scheduling module determines the batch task as the first model task, determines the target model task that is second only to the second model task in the second execution order as the second model task, and returns the execution step of judging whether the execution time of the third model task exceeds the corresponding execution time limit based on the estimated running time of the first model task and the estimated running time of the second model task, and the subsequent steps, until the execution time of the third model task exceeds the corresponding execution time limit or multiple target model tasks are traversed.

[0178] Optionally, the second model triggering module 1202 is further configured to:

[0179] The execution time of the first model task is determined as the execution time of the batch task.

[0180] It should be noted that the information interaction and execution process between the above modules are Figure 2 The function of the first processor in the control system embodiment shown is based on the same concept. Its specific functions and technical effects can be found in the system embodiment section and will not be repeated here.

[0181] Corresponding to the above Figure 11 The intelligent application task processing method of the embodiment, Figure 13 Another structural diagram of the intelligent application task processing device provided by an embodiment of the present application is shown, which is applied to a second processor, and the second processor is configured with an operator scheduling module. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0182] Reference Figure 13 , the device comprises:

[0183] The first operator triggering module 1301 is configured to trigger the operator scheduling module to schedule the multiple operator tasks and obtain scheduling policies for the multiple operator tasks when the model task included in the second task set is split into multiple operator tasks based on the model driving instruction.

[0184] The second operator triggering module 1302 is used to trigger the operator scheduling module to trigger the operator of the relevant neural network model to execute the corresponding operator task according to the scheduling strategy, and feed back the result of task processing to the first processor.

[0185] Optionally, the first operator triggering module 1301 includes:

[0186] A first triggering unit is configured to trigger the operator scheduling module to determine parallel execution as a scheduling strategy for the multiple operator task teams when there are multiple operator task teams that need to be executed in parallel among the multiple operator tasks and the computing power units corresponding to the operator tasks in the multiple operator task teams are in an idle state, where each operator task team includes at least one operator task;

[0187] a second triggering unit configured to trigger, for an operator task team including two or more operator tasks, an operator scheduling module to schedule each operator task in the operator task team based on the priority, execution time limit, estimated running time, and time of entry into the team of each operator task in the operator task team, obtain an execution order of each operator task in the operator task team, and determine the execution order of each operator task in the operator task team as a scheduling policy for each operator task in the operator task team; and store each operator task in the operator task team in an operator queue;

[0188] a third triggering unit configured to, when no operator task queue exists among the multiple operator tasks and the computing power units corresponding to the multiple operator tasks are in an idle state, trigger the operator scheduling module to schedule the multiple operator tasks based on the priorities, execution deadlines, estimated running times, and enqueuing times of the multiple operator tasks, obtain an execution order of the multiple operator tasks, and determine the execution order of the multiple operator tasks as a scheduling policy for the multiple operator tasks, and store the multiple operator tasks in an operator queue;

[0189] The computing power unit is used to run the operator that executes the corresponding operator task.

[0190] Optionally, the second trigger unit is specifically configured to:

[0191] The trigger operator scheduling module sorts the operator tasks in the operator task queue in descending order of priority, ascending order of execution time limit, ascending order of estimated running time, and ascending order of enqueuing time;

[0192] Optionally, the third triggering unit is specifically configured to:

[0193] The trigger operator scheduling module sorts multiple operator tasks in descending order of priority, ascending order of execution time, ascending order of estimated running time, and ascending order of enqueuing time.

[0194] It should be noted that the information interaction and execution process between the above modules are Figure 2 The function of the second processor in the control system embodiment shown is based on the same concept. Its specific functions and technical effects can be found in the system embodiment section and will not be repeated here.

[0195] Figure 14 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 14 As shown, the electronic device 14 of this embodiment includes: at least one processor 1400 ( Figure 14 Only one is shown), a memory 1401, and a computer program 1402 stored in the memory 1401 and executable on the at least one processor 1400. When the processor 1400 executes the computer program 1402, the steps of any of the above-mentioned method embodiments are implemented. The electronic device may include, but is not limited to, a processor 1400 and a memory 1401. It will be understood by those skilled in the art that Figure 14 This is merely an example of the electronic device 14 and does not constitute a limitation on the electronic device 14 . The electronic device 14 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0196] The processor 1400 may be a CPU, or other general-purpose processor, NPU, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. A general-purpose processor may be a microprocessor, or any conventional processor. In some embodiments, the memory 1401 may be an internal storage unit of the electronic device 14, such as a hard drive or memory of the electronic device 14. In other embodiments, the memory 1401 may be an external storage device of the electronic device 14, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 1401 may include both the internal storage unit of the electronic device 14 and an external storage device. The memory 1401 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 1401 may also be used to temporarily store data that has been output or is about to be output.

[0197] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of this application.

[0198] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

Claims

1. An intelligent application control system, characterized in that: include: a first processor and a second processor; The first processor is configured with: a component scheduling module and a model scheduling module running in kernel state; The component scheduling module is used to manage the operating status of each functional component in the intelligent application and send a first task set to the model scheduling module when the functional component is running. The first task set includes: model tasks triggered by the running of the functional component, and the model tasks are tasks that depend on the execution of the neural network model; The model scheduling module is configured to: when there are multiple target model tasks in the first task set, partially or completely merge the multiple target model tasks in the first task set to obtain a second task set that meets a batch processing condition, wherein the multiple target model tasks are model tasks corresponding to the same neural network model, and the batch processing condition includes: the execution time of each model task in the second task set does not exceed the execution time limit corresponding to each model task; and send a model driving instruction to the second processor based on the model tasks included in the second task set; The second processor is used to: trigger the relevant neural network model to execute the model tasks in the second task set based on the model driving instruction, and feed back the results of task processing to the first processor.

2. The intelligent application control system according to claim 1, characterized in that: The model scheduling module includes a task scheduling unit, a sequential execution unit and a target determination unit; The task scheduling unit is configured to schedule each model task in the first task set based on the priority, execution time limit, estimated running time, and queue entry time of each model task in the first task set, to obtain a first execution order for each model task in the first task set, wherein each model task in the first task set is stored in a model queue; The sequential execution unit is configured to: determine a second execution order of the plurality of target model tasks based on the first execution order when the plurality of target model tasks exist in the first task set; The target determination unit is configured to: merge part or all of the multiple target model tasks based on the second execution order and the estimated running times of the multiple target model tasks.

3. The intelligent application control system according to claim 2, characterized in that: The task scheduling unit is specifically used for: The model tasks in the first task set are sorted in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of enqueuing time.

4. The intelligent application control system according to claim 2, characterized in that: The target determination unit is specifically configured to: Determine the first model task in the second execution order as the first model task, and determine the target model task that is second only to the first model task in the second execution order as the second model task; the first model task is the target model task that is first in the second execution order; Based on the estimated running time of the first model task and the estimated running time of the second model task, determining whether the execution time of the third model task exceeds the corresponding execution time limit; The third model task is a model task that is located after the first model task in the first execution order and is other than the first model task and the second model task; When the execution time of the third model task does not exceed the corresponding execution time limit, merging the first model task and the second model task to obtain a batch task; Determine the batch task as the first model task, determine the target model task that is second only to the second model task in the second execution order as the second model task, return to the steps of executing the estimated running time based on the first model task and the estimated running time of the second model task, and determine whether the execution time of the third model task exceeds the corresponding execution time limit, as well as subsequent steps, until the execution time of the third model task exceeds the corresponding execution time limit or the multiple target model tasks are traversed.

5. The intelligent application control system according to claim 4, characterized in that: The model scheduling module also includes a time determination unit; The time determination unit is used to determine the execution time of the first model task as the execution time of the batch processing task.

6. The intelligent application control system according to any one of claims 1 to 5, characterized in that: The second processor is configured with an operator scheduling module; The operator scheduling module is used to: when the model tasks included in the second task set are split into multiple operator tasks based on the model-driven instructions, schedule the multiple operator tasks, obtain the scheduling strategy of the multiple operator tasks, trigger the operators of the relevant neural network model to execute the corresponding operator tasks according to the scheduling strategy, and feedback the results of task processing to the first processor.

7. The intelligent application control system according to claim 6, characterized in that: The operator scheduling module includes a parallel determination unit, a first processing unit and a second processing unit; The parallel determination unit is configured to: when there are multiple operator task teams that need to be executed in parallel among the multiple operator tasks and the computing power units corresponding to the respective operator tasks in the multiple operator task teams are in an idle state, determine parallel execution as a scheduling strategy for the multiple operator task teams; each operator task team includes at least one operator task; The first processing unit is configured to: for an operator task team including two or more operator tasks, schedule each operator task in the operator task team based on the priority, execution time limit, estimated running time, and time of entry into the team of each operator task in the operator task team, obtain an execution order of each operator task in the operator task team, and determine the execution order of each operator task in the operator task team as a scheduling policy for each operator task in the operator task team, and store each operator task in the operator task team in an operator queue; The second processing unit is configured to: when the operator task queue does not exist among the multiple operator tasks and the computing power units corresponding to the multiple operator tasks are in an idle state, schedule the multiple operator tasks based on the priorities, execution deadlines, estimated running times, and enqueuing times of the multiple operator tasks to obtain an execution order of the multiple operator tasks, and determine the execution order of the multiple operator tasks as a scheduling policy for the multiple operator tasks, and store the multiple operator tasks in an operator queue; The computing power unit is used to run the operator that executes the corresponding operator task.

8. The intelligent application control system according to claim 7, characterized in that: The first processing unit is specifically configured to sort the operator tasks in the operator task queue in descending order of priority, ascending order of execution time limit, ascending order of estimated running time, and ascending order of time of entry into the queue; The second processing unit is specifically configured to sort the plurality of operator tasks in descending order of priority, ascending order of execution time limit, ascending order of estimated running time, and ascending order of enqueuing time.

9. A method for processing intelligent application tasks, characterized in that: Applied to a first processor, the first processor is configured with: a component scheduling module and a model scheduling module running in kernel state; The component scheduling module is used to manage the operating status of each functional component in the intelligent application; The method comprises: When the functional component is running, triggering the component scheduling module to send a first task set to the model scheduling module, the first task set including: model tasks triggered by the running of the functional component, the model tasks being tasks that depend on the execution of the neural network model; In a case where there are multiple target model tasks in the first task set, triggering the model scheduling module to partially or completely merge the multiple target model tasks in the first task set to obtain a second task set that meets a batch processing condition, wherein the multiple target model tasks are model tasks corresponding to the same neural network model, and the batch processing condition includes: the execution time of each model task in the second task set does not exceed the execution time limit corresponding to each model task; The model scheduling module is triggered to send a model-driven instruction to the second processor based on the tasks included in the second task set. The model-driven instruction is used to trigger the second processor to execute the tasks in the second task set and to feedback the results of task processing to the first processor. The second processor is configured with task processing capabilities based on a neural network model.

10. The method according to claim 9, characterized in that Before triggering the model scheduling module to partially or completely merge the multiple target model tasks in the first task set, the method further includes: triggering the model scheduling module to schedule each model task in the first task set based on the priority, execution time limit, estimated running time, and enqueue time of each model task in the first task set, to obtain a first execution order of each model task in the first task set, and storing each model task in the first task set in a model queue; triggering the model scheduling module to determine a second execution order of the plurality of target model tasks based on the first execution order when there are a plurality of target model tasks in the first task set; The triggering the model scheduling module to partially or completely merge the multiple target model tasks in the first task set includes: The model scheduling module is triggered to partially or completely merge the multiple target model tasks based on the second execution order and the estimated running time of the multiple target model tasks.

11. The method according to claim 10, characterized in that The triggering of the model scheduling module to schedule each model task in the first task set based on the priority, execution time limit, estimated running time and enqueue time of each model task in the first task set includes: The model scheduling module is triggered to sort the model tasks in the first task set in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of enqueuing time.

12. The method according to claim 10, characterized in that The triggering of the model scheduling module to partially or completely merge the multiple target model tasks based on the second execution order and the estimated running time of the multiple target model tasks includes: triggering the model scheduling module to determine the first model task in the second execution order as the first model task, and to determine the target model task that is second only to the first model task in the second execution order as the second model task; the first model task is the target model task that is first in the second execution order; triggering the model scheduling module to determine whether the execution time of a third model task exceeds the corresponding execution time limit based on the estimated running time of the first model task and the estimated running time of the second model task; the third model task is a model task that is located after the first model task in the first execution order and is other than the first model task and the second model task; triggering the model scheduling module to merge the first model task and the second model task to obtain a batch task when the execution time of the third model task does not exceed the corresponding execution time limit; Trigger the model scheduling module to determine the batch task as the first model task, determine the target model task that is second only to the second model task in the second execution order as the second model task, and return to the steps of executing the estimated running time based on the first model task and the estimated running time of the second model task, and judging whether the execution time of the third model task exceeds the corresponding execution time limit, as well as subsequent steps, until the execution time of the third model task exceeds the corresponding execution time limit or the multiple target model tasks are traversed.

13. The method according to claim 12, characterized in that After getting the batch processing task, it also includes: The execution time of the first model task is determined as the execution time of the batch processing task.

14. A method for processing intelligent application tasks, characterized in that: A second processor applied to the intelligent application control system according to any one of claims 1 to 8, wherein the second processor is configured with an operator scheduling module; The method comprises: In a case where the model tasks included in the second task set are split into a plurality of operator tasks based on the model-driven instruction, triggering the operator scheduling module to schedule the plurality of operator tasks to obtain a scheduling strategy for the plurality of operator tasks; The operator scheduling module is triggered to trigger the operator of the relevant neural network model to execute the corresponding operator task according to the scheduling strategy, and the result obtained by task processing is fed back to the first processor.

15. The method according to claim 14, characterized in that Triggering the operator scheduling module to schedule the multiple operator tasks to obtain scheduling strategies for the multiple operator tasks includes: When there are multiple operator task teams that need to be executed in parallel among the multiple operator tasks and the computing power units corresponding to the respective operator tasks in the multiple operator task teams are in an idle state, triggering the operator scheduling module to determine parallel execution as a scheduling strategy for the multiple operator task teams, each operator task team including at least one operator task; For the operator task team including more than two operator tasks, triggering the operator scheduling module to schedule each operator task in the operator task team based on the priority, execution time limit, estimated running time and queue entry time of each operator task in the operator task team, obtain the execution order of each operator task in the operator task team, and determine the execution order of each operator task in the operator task team as the scheduling policy of each operator task in the operator task team, and store each operator task in the operator task team in the operator queue; When the operator task queue does not exist among the multiple operator tasks and the computing power units corresponding to the multiple operator tasks are in an idle state, triggering the operator scheduling module to schedule the multiple operator tasks based on the priorities, execution deadlines, estimated running times, and enqueuing times of the multiple operator tasks, obtain an execution order of the multiple operator tasks, and determine the execution order of the multiple operator tasks as a scheduling policy for the multiple operator tasks, and the multiple operator tasks are stored in an operator queue; The computing power unit is used to run the operator that executes the corresponding operator task.

16. The method according to claim 15, characterized in that The triggering of the operator scheduling module to schedule each operator task in the operator task team based on the priority, execution time limit, estimated running time and queue entry time of each operator task in the operator task team includes: Triggering the operator scheduling module to sort the operator tasks in the operator task queue in descending order of priority, ascending order of execution time limit, ascending order of estimated running time, and ascending order of enqueuing time; Triggering the operator scheduling module to schedule the multiple operator tasks based on the priorities, execution deadlines, estimated running times, and enqueue times of the multiple operator tasks includes: The operator scheduling module is triggered to sort the multiple operator tasks in descending order of priority, ascending order of execution time limit, ascending order of estimated running time and ascending order of enqueuing time.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the method according to any one of claims 9 to 13 or the method according to any one of claims 14 to 16.

Images