Artificial intelligence reasoning task scheduling method and system having real-time performance and determinacy

Abstract

An artificial intelligence (AI) reasoning task scheduling method and system having real-time performance and determinacy, and a storage medium. The method comprises: when receiving a scheduling request for an AI reasoning task, an AI engine assembly allocates the AI reasoning task to a matched neural network model assembly on the basis of a task demand and a time deterministic constraint of the received AI reasoning task; on the basis of a resource demand and the time deterministic constraint of the AI reasoning task, and an actual resource condition of a computing node, in a plurality of operation capsules of the computing node, the matched neural network model assembly schedules a matched AI-specific real-time operation capsule for the AI reasoning task, wherein the AI-specific real-time operation capsule comprises a real-time operation environment and a basic assembly used when the AI reasoning task is executed. The method can reasonably allocate the task, fully use the computing resources, ensure that the neural network model assembly completes output of a reasoning result within a given time period, and meet the time deterministic constraint.

Description

Real-time and deterministic artificial intelligence reasoning task scheduling method and system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on February 19, 2024, application number 202410185400.8, and application name "A New Industrial Operating System, Equipment and Storage Medium".

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on March 16, 2024, application number 202410301524.8, and application name: "Real-time and deterministic artificial intelligence reasoning task scheduling method and system". Technical Field

[0003] The present application relates to the field of industrial control technology, and in particular to a method and system for scheduling artificial intelligence reasoning tasks with real-time and deterministic performance, and a storage medium. Background Art

[0004] In the actual application of existing neural network frameworks, there are often problems with resource utilization efficiency in some scenarios that require real-time performance. Due to the characteristics of parallel computing, existing neural network frameworks may not be able to fully utilize all computing resources when performing calculations, resulting in resource waste or low computing efficiency.

[0005] While existing technologies are capable of implementing the basic functions of neural networks, they face the aforementioned real-time challenges, which limit their application in certain key areas. For example, scenarios like autonomous driving, industrial automation, and financial transactions place extremely high demands on real-time performance, and any computational delay can have serious consequences. Therefore, a new neural network framework is needed to ensure that inference results are output within a given timeframe, meeting the demands of these demanding scenarios. Summary of the Invention

[0006] In view of this, the present application proposes a real-time and deterministic artificial intelligence reasoning task scheduling method and system, equipment, and storage medium, which can reasonably allocate tasks, make full use of computing resources, ensure that the neural network model component completes the output of reasoning results within a given time period, and realize applications in scenarios with high real-time requirements.

[0007] In a first aspect, the present application provides a real-time and deterministic artificial intelligence reasoning task scheduling method, comprising:

[0008] After receiving the scheduling request for the AI ​​reasoning task, the artificial intelligence engine component allocates the AI ​​reasoning task to the matching neural network model component based on the task requirements and time determinism constraints of the AI ​​reasoning task; the artificial intelligence engine component includes neural network model components for different industrial application scenarios;

[0009] The matched neural network model component schedules a matched AI-specific real-time running capsule for the AI ​​reasoning task among multiple running capsules of the computing node based on the resource requirements and time deterministic constraints of the AI ​​reasoning task and the actual resource situation of the computing node. Each of the running capsules is isolated from each other. The running capsule is used to provide an independent running environment for task running. The AI-specific real-time running capsule includes a real-time runtime environment and basic components used when executing the AI ​​reasoning task.

[0010] From the above, the artificial intelligence engine component of the present application assigns the AI ​​reasoning task to the matching neural network model component according to the task requirements and time determinism constraints of the received AI reasoning task, and parses the AI ​​reasoning task through the neural network model component. According to the resource requirements, time determinism constraints and actual resource conditions of each computing node of the AI ​​reasoning task, the matching AI dedicated real-time running capsule is scheduled for the AI ​​reasoning task. The AI ​​dedicated real-time running capsule is pre-deployed with the real-time runtime environment and the basic components used when executing the AI ​​reasoning task. The resources provided by the AI ​​dedicated real-time running capsule enable the neural network model component to complete the output of the reasoning result within a given time period, meet the time determinism constraints of the AI ​​reasoning task, and be suitable for business scenarios with high real-time requirements.

[0011] In a possible implementation of the first aspect, based on the resource requirements and time determinism constraints of the AI ​​reasoning task and the actual resource situation of the computing node, scheduling the AI-dedicated real-time running capsule matched for the AI ​​reasoning task from among multiple running capsules of the computing node includes:

[0012] According to the resource requirements of the AI ​​reasoning task, the configuration information and historical execution data of the currently idle AI-dedicated real-time running capsules on the computing node, the execution time of running the AI ​​reasoning task on the AI-dedicated real-time running capsule is estimated, and among multiple currently idle running capsules, one or more AI-dedicated real-time running capsules are selected to run the AI ​​reasoning task to meet the time determinism constraint of the AI ​​reasoning task.

[0013] Based on the resource requirements of the AI ​​reasoning task, the configuration information of the currently idle AI-dedicated real-time running capsules on the computing node, and historical execution data, the execution time of the AI ​​reasoning task running on the AI-dedicated real-time running capsule is estimated, and one or more AI-dedicated real-time running capsules are selected to execute the AI ​​reasoning task, so that the reasoning result output by the neural network model component meets the time determinism constraint of the AI ​​reasoning task.

[0014] In a possible implementation of the first aspect, the basic components used to execute AI reasoning tasks include one or more of the following in any combination: a linear algebra operator library, a processor instruction set dedicated to AI computing, a computational graph accelerator, and a driver for a hardware acceleration processor.

[0015] As described above, by deploying resources such as the algebraic operator library dedicated to AI reasoning tasks, the processor instruction set dedicated to AI calculations, the computational graph accelerator, and the driver of the hardware acceleration processor in the AI ​​dedicated real-time running capsule, when the neural network model component schedules the AI ​​dedicated real-time running capsule, it can directly use the dedicated resources to perform the calculation of the AI ​​reasoning task, thereby achieving accelerated execution of the AI ​​reasoning task.

[0016] In a possible implementation of the first aspect, the processor instruction set of the AI-dedicated real-time running capsule provides a mechanism for reducing memory access latency.

[0017] From the above, by optimizing the memory allocation strategy, memory resources are reserved for AI reasoning tasks, memory fragmentation is reduced, and memory utilization is improved to accelerate the execution of the AI ​​reasoning tasks.

[0018] In a possible implementation of the first aspect, the AI ​​dedicated real-time running capsule is allocated dedicated processor resources and memory resources.

[0019] As mentioned above, for real-time AI inference tasks, it is necessary to prioritize the computing resources required. Therefore, when allocating resources for AI-dedicated real-time running capsules, processor resources and memory resources dedicated to AI inference tasks can be allocated to ensure resource isolation between different tasks.

[0020] In a possible implementation of the first aspect, the elastic microkernel virtualizes hardware resources into multiple independent hardware resource components; the elastic microkernel statically or dynamically allocates hardware resource components to the AI-dedicated real-time running capsule.

[0021] As mentioned above, the computing node is also equipped with an elastic microkernel. This elastic microkernel uses virtualization technology to virtualize hardware resources into multiple independent hardware resource components. Hardware resource components are statically or dynamically allocated to each running capsule in the computing node to ensure resource isolation between different tasks and prevent real-time tasks from being affected by other non-real-time tasks.

[0022] In a possible implementation of the first aspect, the matched neural network model component stores basic components required by the model and memory requirements of each level of the model, and the method further includes:

[0023] The matched neural network model component selects the AI ​​dedicated real-time running capsule to run the AI ​​reasoning task based on the basic components required by the model and the memory requirements of each level of the model.

[0024] In a possible implementation of the first aspect, the method further includes:

[0025] The AI ​​dedicated real-time running capsule sends the memory requirement information of each layer of the model analyzed by the neural network model component to the memory management component of the elastic microkernel;

[0026] The memory management component selects a memory allocation strategy based on the demand information. The memory allocation strategy includes one or more of the following mechanism combinations provided by the memory management component: fixed-length memory allocation mechanism, hierarchical allocation mechanism, and static memory allocation mechanism.

[0027] From the above, if the various layers or operations of a neural network model have regular memory requirements, a fixed-length memory allocator can be designed to allocate memory blocks of a fixed size or several common sizes to meet the model's regular memory requests. For example, static memory allocation: During the model initialization phase, a sufficiently large memory block is allocated at one time based on the model structure and parameter scale to ensure that all memory requirements during model operation are met within this memory block, thereby avoiding the fragmentation problem caused by dynamic allocation. Fixed-length memory allocator: If the various layers or operations of a model have regular memory requirements, a fixed-length memory allocator can be designed to allocate memory blocks of a fixed size or several common sizes to meet the model's regular memory requests. Hierarchical memory management: A multi-level memory allocation strategy is adopted based on the characteristics of the memory requirements of each part of the model. For example, for fixed-size tensors, a specialized allocator is used; for variable-length sequence data, another allocation strategy is adopted.

[0028] In a possible implementation of the first aspect, based on the resource requirements and time determinism constraints of the AI ​​reasoning task and the actual resource situation of the computing node, scheduling the AI-dedicated real-time running capsule matched for the AI ​​reasoning task from among multiple running capsules of the computing node includes:

[0029] According to the complexity of the AI ​​reasoning task, the AI ​​reasoning task is decomposed into a series of subtasks that can be processed by the neural network model component. According to the resource requirements and time deterministic constraints of each subtask and the actual resource conditions of the computing node, a matching AI dedicated real-time running capsule is scheduled for each subtask.

[0030] As mentioned above, since the AI ​​reasoning task is a complex reasoning process, when the neural network model component reasons on the AI ​​reasoning task, it usually decomposes the AI ​​reasoning task into a series of subtasks based on the complexity of the AI ​​reasoning task. Then, based on the resource requirements and time deterministic constraints of each subtask and the actual resource conditions of the computing node, a matching AI-specific real-time running capsule is scheduled for each subtask, thereby realizing the parallel execution of multiple subtasks and accelerating the reasoning speed.

[0031] In a possible implementation of the first aspect, the further comprising:

[0032] When the AI ​​reasoning task is decomposed into multiple subtasks, and the neural network model component schedules multiple AI-specific real-time running capsules that match the multiple subtasks and are distributed on different computing nodes, the delay jointly determined by the execution delay of each subtask by the multiple AI-specific real-time running capsules and the transmission delay of execution data between each other meets the time determinism constraint of the AI ​​reasoning task.

[0033] From the above, when the neural network model component schedules multiple AI-specific real-time running capsules on multiple computing nodes to execute decomposed subtasks, the delay jointly determined by the execution delay of the subtasks by the multiple AI-specific real-time running capsules and the transmission delay between them must meet the time determinism constraint of the AI ​​reasoning task.

[0034] In a possible implementation of the first aspect, the further comprising:

[0035] The priority of the AI ​​reasoning task is determined according to the time deterministic constraint of the AI ​​reasoning task. The neural network model component uses a priority scheduling algorithm and a real-time scheduling algorithm to schedule an AI-specific real-time running capsule that matches the priority of the AI ​​reasoning task, so that the AI-specific real-time running capsule gives priority to executing the AI ​​reasoning task. The high-priority AI-specific real-time running capsule can obtain the hardware resources required to execute the AI ​​reasoning task first, or can preempt the hardware resources occupied by the low-priority running capsule.

[0036] As described above, the priority of the AI ​​inference task is determined based on its time deterministic constraints, ensuring that the priority of the scheduled AI real-time capsule matches the priority of the AI ​​inference task. This ensures that real-time AI inference tasks receive priority access to hardware resources. Priority matching also enables task preemption, allowing high-priority capsules to preempt the hardware resources of lower-priority capsules when necessary.

[0037] In a possible implementation of the first aspect, when there are multiple AI-dedicated real-time running capsules that meet the resource requirements, a matching AI-dedicated real-time running capsule is scheduled for the AI ​​reasoning task based on load balancing and / or latency.

[0038] Based on the above, the best AI-specific real-time running capsule is scheduled for the AI ​​reasoning task based on load balancing and / or latency among multiple AI-specific real-time running capsules that meet resource requirements, so as to improve the time determinism of the AI ​​reasoning task.

[0039] In a possible implementation of the first aspect, the further comprising:

[0040] Monitor the running status of the matched neural network model component in real time, and adjust the computing resource allocation in the AI ​​dedicated real-time running capsule according to the running status so that the inference result output by the neural network model component meets the time determinism constraint.

[0041] As described above, by monitoring the running status of the neural network model component in real time and adjusting the allocation of computing resources in the scheduled AI-specific real-time running capsule according to the running status, the neural network model component can complete the output of the inference result within a given time period, thus meeting the time determinism constraint of the AI ​​inference task.

[0042] In a possible implementation of the first aspect, the further comprising:

[0043] Monitor the running status of the matched neural network model component in real time, increase the priority of the AI ​​reasoning task according to the running status, and add hardware resources to the matched AI dedicated real-time running capsule so that the reasoning result output by the neural network model component meets the time determinism constraint.

[0044] As described above, by increasing the priority of the AI ​​reasoning task, the AI ​​reasoning task is executed first and the AI ​​dedicated real-time running capsule obtains more hardware resources to support the calculation of the AI ​​reasoning task, so that the reasoning result output by the neural network model component meets the time deterministic constraint.

[0045] In a possible implementation of the first aspect, the real-time monitoring of the running status of the matched neural network model component and adjusting the allocation of computing resources in the AI-dedicated real-time running capsule according to the running status include at least one of the following:

[0046] Monitor the load of each computing node scheduled by the neural network model component in real time. When the performance of a computing node degrades, notify the computing node to create a new AI-specific real-time running capsule to enable the AI ​​reasoning task to run in parallel on multiple AI-specific real-time running capsules, or increase hardware resources for the running AI-specific real-time running capsule;

[0047] monitoring the response time of each computing node scheduled by the neural network model component in real time, and when the response time of a computing node exceeds a set threshold, notifying the computing node to create a new AI-specific real-time running capsule to enable the AI ​​reasoning task to run in parallel on multiple AI-specific real-time running capsules, or to increase hardware resources for the running AI-specific real-time running capsule;

[0048] monitoring the average response time of multiple inference tasks in the neural network model component in real time, and when the average response time exceeds a set threshold, notifying the computing node to create a new AI-specific real-time running capsule to enable the AI ​​inference task to run in parallel on multiple AI-specific real-time running capsules, or to increase hardware resources for the running AI-specific real-time running capsule;

[0049] monitoring the hardware resource utilization of each computing node scheduled by the neural network model component in real time, and when the hardware resources are sufficient, notifying the computing node to create a new AI-specific real-time running capsule to enable the AI ​​reasoning task to run in parallel on multiple AI-specific real-time running capsules, or to increase hardware resources for the running AI-specific real-time running capsule;

[0050] The hardware resource utilization of each computing node scheduled by the neural network model component is monitored in real time. When the hardware resources are insufficient, the AI-specific real-time running capsule on other computing nodes is scheduled for the AI ​​reasoning task.

[0051] In a possible implementation of the first aspect, the further comprising:

[0052] During the process of the AI ​​dedicated real-time running capsule executing calculations, the performance of the AI ​​dedicated real-time running capsule is monitored in real time through the elastic microkernel corresponding to the AI ​​dedicated real-time running capsule. When the performance degrades, the dedicated processor resources and memory resources allocated to the AI ​​dedicated real-time running capsule are increased.

[0053] As described above, by monitoring the performance of the AI-dedicated real-time running capsule in real time and adjusting the processor resources and memory resources allocated to the AI-dedicated real-time running capsule based on the monitoring results, performance fluctuations or resource bottlenecks that may occur during task execution can be addressed.

[0054] In a possible implementation of the first aspect, the further comprising:

[0055] Collect and analyze the execution data of the AI ​​reasoning task in the AI ​​dedicated real-time running capsule, save it to the artificial intelligence engine component, and optimize the AI ​​dedicated real-time running capsule scheduling strategy of the neural network model component and the resource allocation strategy of the AI ​​dedicated real-time running capsule based on the analysis results.

[0056] As mentioned above, by analyzing the execution data of AI-specific real-time running capsules and conducting deep learning and optimization on the AI-specific real-time running capsule scheduling strategy and resource allocation strategy of the neural network model component based on the analysis results, it helps to continuously improve and enhance the output speed of the inference results of the neural network model component.

[0057] In a possible implementation of the first aspect, the running capsule supports one of the following running scenarios: thread, process, real-time container, non-real-time container, real-time virtual machine, non-real-time virtual machine;

[0058] The running capsules include real-time running capsules and non-real-time running capsules, wherein the real-time running capsules include dedicated real-time running capsules for various real-time application tasks.

[0059] As mentioned above, Run Capsules are compatible with a variety of tasks, including those requiring threads, processes, real-time containers, non-real-time containers, real-time virtual machines, or non-real-time virtual machines. This allows them to meet diverse task requirements and improve the efficiency of computing node resources. Run Capsules can also be divided into real-time Run Capsules and non-real-time Run Capsules, used to execute real-time and non-real-time tasks, respectively.

[0060] In a second aspect, the present application provides a real-time and deterministic artificial intelligence reasoning task scheduling system. The artificial intelligence engine component of the platform layer is deployed on the edge management node or in the cloud. The artificial intelligence engine component includes a neural network model component for different industrial application scenarios. Each computing node includes an elastic microkernel and several running capsules. The elastic microkernel is used to flexibly allocate hardware resources to the running capsules. The running capsules are isolated from each other and are used to provide an independent running environment for task execution.

[0061] The artificial intelligence engine component receives AI reasoning tasks from the service layer and assigns the AI ​​reasoning tasks to matching neural network model components based on the task requirements and time determinism constraints of the received AI reasoning tasks.

[0062] The matched neural network model component schedules a matched AI-specific real-time running capsule for the AI ​​computing task from multiple running capsules of the computing node according to the resource requirements and time deterministic constraints of the AI ​​reasoning task and the actual resource situation of the computing node. The AI-specific real-time running capsule includes a real-time runtime environment and basic components used when executing the AI ​​reasoning task.

[0063] In a third aspect, the present application provides a computing device, comprising:

[0064] processor;

[0065] a memory for storing one or more programs;

[0066] When the one or more programs are executed by the processor, the processor implements the above-mentioned real-time and deterministic artificial intelligence reasoning task scheduling method.

[0067] In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, it implements the above-mentioned real-time and deterministic artificial intelligence reasoning task scheduling method.

[0068] These and other aspects of the present application will become more apparent from the following description of the embodiment(s). BRIEF DESCRIPTION OF THE DRAWINGS

[0069] FIG1 is a schematic diagram of the structure of a new industrial operating system provided in an embodiment of the present application.

[0070] Figure 2 is a flowchart of a real-time and deterministic artificial intelligence reasoning task scheduling method provided in an embodiment of the present application.

[0071] FIG3 is a structural diagram of a real-time and deterministic artificial intelligence reasoning task scheduling system provided in an embodiment of the present application.

[0072] FIG4 is a structural diagram of a computing device provided in an embodiment of the present application.

[0073] It should be understood that the sizes and shapes of the blocks in the above structural diagrams are for reference only and should not constitute an exclusive interpretation of the embodiments of this application. The relative positions and inclusion relationships between the blocks presented in the structural diagrams are only schematic representations of the structural relationships between the blocks, and do not limit the physical connection methods of the embodiments of this application. DETAILED DESCRIPTION

[0074] The technical solution provided by this application is further described below with reference to the accompanying drawings and examples. It should be understood that the system structure and business scenarios provided in the examples of this application are mainly for illustrating possible implementation methods of the technical solution of this application and should not be interpreted as the sole limitation of the technical solution of this application. It is known to those skilled in the art that with the evolution of the system structure and the emergence of new business scenarios, the technical solution provided by this application is also applicable to similar technical problems.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of this application. In the event of any inconsistency, the meaning described in this specification or the meaning derived from the contents recorded in this specification shall prevail. In addition, the terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit this application.

[0076] The embodiments of the present application propose a real-time and deterministic artificial intelligence reasoning task scheduling method, system, device, and storage medium, which can reasonably allocate tasks. The method includes: after the artificial intelligence engine component receives a scheduling request for an AI reasoning task, it allocates the AI ​​reasoning task to a matching neural network model component based on the task requirements and time deterministic constraints of the received AI reasoning task; the artificial intelligence engine component includes a neural network model component for different industrial application scenarios; the matched neural network model component schedules a matching AI-specific real-time running capsule for the AI ​​reasoning task from multiple running capsules of the computing node based on the resource requirements and time deterministic constraints of the AI ​​reasoning task and the actual resource conditions of the computing node, wherein each of the running capsules is isolated from each other, and the running capsule is used to provide an independent running environment for task running, and the AI-specific real-time running capsule includes a real-time running environment and basic components used when executing the AI ​​reasoning task.

[0077] The artificial intelligence engine component of the embodiment of the present application assigns the AI ​​reasoning task to a matching neural network model component based on the task requirements and time determinism constraints of the received AI reasoning task, and parses the AI ​​reasoning task through the neural network model component. Based on the resource requirements, time determinism constraints and actual resource conditions of each computing node of the AI ​​reasoning task, a matching AI-specific real-time running capsule is scheduled for the AI ​​reasoning task. The AI-specific real-time running capsule is pre-deployed with a real-time runtime environment and basic components used when executing the AI ​​reasoning task. Through the resources provided by the AI-specific real-time running capsule, the neural network model component can complete the output of the reasoning result within a given time period, meet the time determinism constraints of the AI ​​reasoning task, and be suitable for business scenarios with high real-time requirements.

[0078] The following describes various embodiments of the present application in conjunction with the accompanying drawings. First, the scenarios in which the embodiments of the present application are used are described.

[0079] The various embodiments of this application are used for computing resource scheduling in an industrial operating system. A novel industrial operating system is described below in conjunction with Figure 1. As shown in Figure 1, the structure of the novel industrial operating system provided by the embodiments of this application includes, from bottom to top, a base layer 100, a platform layer 200, and a service layer 300. The various embodiments of this application are designed to operate on the platform layer 200.

[0080] The base layer 100 is deployed on each compute node in the industrial cloud and includes the node's elastic microkernel 110 and several runtime capsules 120. The elastic microkernel 110 allocates hardware resources to the runtime capsules. One possible implementation of the base layer 100 is in the new Rust language, leveraging its features to provide memory safety. The system utilizes a standardized functional component design approach, enabling modular assembly. This allows for flexible, on-demand integration of advanced features to support virtualization and provide enhanced security isolation.

[0081] The elastic microkernel 110 includes componentized hardware resources for allocating componentized hardware resources to each running capsule 120. The elastic microkernel manages the hardware resources of the computing node in a componentized manner, and each componentized hardware resource is a standardized resource component. The elastic microkernel elastically loads and / or deletes each resource component as needed to achieve elastic management of the hardware resources on the computing node. These resources include the CPU core on the chip, the motherboard and / or chip memory, the computing node peripherals, etc. The elastic microkernel is a super-elastic microkernel that can randomly and quickly combine chip resources according to demand. The elastic microkernel is also used to statically and / or dynamically allocate componentized hardware resources to the running capsules on the computing node, achieving elastic allocation of resource components.

[0082] Among them, the running capsule 120 supports one of the following scenarios: thread, process, real-time container, non-real-time container, real-time virtual machine, non-real-time virtual machine, and the running capsule 120 includes a corresponding running environment for tasks in the service components of the new industrial operating system.

[0083] For example, Figure 1 shows two partitioned virtual machines and one non-partitioned container, the real-time runtime environment of one partitioned virtual machine supports real-time application tasks, the high-security runtime environment of one partitioned virtual machine supports high-security application tasks, and the non-real-time runtime environment in the non-partitioned container supports the application tasks of the non-real-time container.

[0084] The running capsule 120 can also be divided into a real-time running capsule and a non-real-time running capsule, which are used to run real-time tasks and non-real-time tasks respectively. The real-time running capsule is a time-critical running capsule 120.

[0085] The platform layer 200 is used to schedule running capsules 120 with matching capabilities on several computing nodes for each task in the service component of the new industrial operating system from the industrial cloud, so that at least the predicted delay of the service component obtained based on the computing delay of the scheduled running capsules 120 and the transmission delay between the scheduled computing nodes meets the deterministic constraints of the service component.

[0086] In a possible implementation of the platform layer, the platform layer 200 includes an orchestrator 210 , a scheduler 220 , and middleware 230 .

[0087] The coordinator 210 is deployed on the computing nodes for management of the industrial cloud in each region, and is used to schedule several computing nodes from the industrial cloud for each task in the service component of the new industrial operating system.

[0088] Scheduler 220 is deployed on a management compute node in each region's industrial cloud, or on each compute node. It schedules a run capsule that matches the capabilities of each service component in service layer 300 from the compute nodes scheduled by orchestrator 210. It also predicts the predicted latency for running the service component based on the computational latency of the scheduled run capsule 120 and the transmission latency between the scheduled compute nodes. Real-time tasks are scheduled to run in real-time run capsules, which are time-critical tasks. Non-real-time tasks are scheduled to run in non-real-time run capsules.

[0089] Middleware 230 manages communication between computing nodes, stores data, and supports the artificial intelligence engine. These components run on runtime capsules 120 on computing nodes dedicated to their respective functions and can be scheduled statically or dynamically. Together with the runtime capsules 120 scheduled by scheduler 220, middleware 230 helps the service components of the new industrial operating system complete their functions.

[0090] The platform layer 200 provides distributed collaboration and deterministic control capabilities based on a distributed coordination framework and deterministic scheduling. This includes: providing elastic resource allocation and scheduling based on the base layer 100, providing distributed deterministic communication capabilities based on the TSN time-sensitive network, and standardized consistency protocols and mechanisms, thereby solving problems such as real-time determinism and data consistency in distributed control and computing. At the same time, the platform layer 200 provides security and privacy protection mechanisms that are applied in the collaboration process to ensure data security and confidentiality. Finally, protocol definition and standardization are key to ensuring collaboration and control between different layers. Through unified protocol specifications, seamless integration and communication between different devices, edge platforms, and cloud platforms can be achieved.

[0091] Platform layer 200 also provides ubiquitous industrial connectivity, addressing interoperability issues across diverse industrial devices, and software-defined control, addressing the need for flexible, on-demand deployment of control systems. Platform layer 200 also rapidly detects the entry of industrial control terminals into the network. Combining platform layer 200 collaborative control with deterministic scheduling and communication, it enables rapid handover of device control across regions.

[0092] The service layer 300 includes an industrial application cloud development kit 310, which is deployed on computing nodes in an industrial cloud with an integrated development environment. This kit is used to develop service components that form various service suites. The industrial application cloud development kit 310 decomposes each service suite into several service components. Each service component is open to users and decomposed into several tasks that can be subscribed to. The service component tasks are essentially dispatched by the platform layer 200 to the runtime capsule 120 for execution.

[0093] The service layer 300 also deploys an industrial control suite 320 and an industrial simulation cloud platform 330. The industrial control suite 320 includes time-critical industrial control components. The tasks of the industrial control components are scheduled by the platform layer 200 to real-time running capsules (such as partition-based real-time containers or real-time virtual machines). The service layer 300 is also used to start related service components in the industrial control suite to control and access industrial actuators or sensors connected to the edge cloud in the industrial cloud through the platform layer 200 and the base layer 100 to complete industrial control.

[0094] The following describes a real-time and deterministic artificial intelligence reasoning task scheduling method provided by an embodiment of the present application in conjunction with Figure 2. This method can be specifically applied to the artificial intelligence engine of the platform layer shown in Figure 1. As shown in Figure 2, the method includes:

[0095] S110: After receiving the scheduling request for the AI ​​reasoning task, the artificial intelligence engine component allocates the AI ​​reasoning task to the matching neural network model component according to the task requirements and time deterministic constraints of the received AI reasoning task.

[0096] Among them, the artificial intelligence engine component is located in the platform layer of the above embodiment. The artificial intelligence engine component includes a neural network model component for different industrial application scenarios. The received AI reasoning task includes the task requirements and time deterministic constraints of the AI ​​reasoning task. The artificial intelligence engine allocates the AI ​​reasoning task to the matching neural network model component according to the task requirements and time deterministic constraints of the AI ​​reasoning task.

[0097] It should be noted that the artificial intelligence engine component may include multiple neural network model components, and each neural network model component has differences in model parameters, model structure, model algorithm, incentive function, loss function, preprocessing method, etc. Such differences lead to certain differences in the resources occupied by the neural network model components and the output time of the reasoning results. Therefore, the embodiment of the present application first needs to determine a matching neural network model component for the AI ​​reasoning task based on the task requirements and time deterministic constraints of the received AI reasoning task, so as to perform the reasoning calculation process of the AI ​​reasoning task through the matching neural network model component.

[0098] S120: The matched neural network model component schedules a matched AI-specific real-time running capsule for the AI ​​reasoning task from among multiple running capsules of the computing node according to the resource requirements and time deterministic constraints of the AI ​​reasoning task and the actual resource situation of the computing node.

[0099] In this embodiment, after receiving the AI ​​inference task, the matching neural network model component can estimate the execution time of the AI ​​inference task on the AI-specific real-time running capsules in each computing node based on the resource requirements of the AI ​​inference task, the configuration information of the currently idle AI-specific real-time running capsules on the computing node, and historical execution data. It can then select one or more AI-specific real-time running capsules to execute the AI ​​inference task, so that the inference results output by the neural network model component meet the time determinism constraints of the AI ​​inference task. When there are multiple AI-specific real-time running capsules that meet the resource requirements of the AI ​​inference task, the matching AI-specific real-time running capsules can be scheduled for the AI ​​inference task based on load balancing and / or latency, thereby improving both the time determinism of the AI ​​inference task and the utilization rate of the AI-specific real-time running capsules.

[0100] In some embodiments, the computing node is located at the base layer of the industrial operating system shown in Figure 1. The computing node also includes an elastic microkernel. The elastic microkernel uses virtualization technology to virtualize hardware resources into multiple independent hardware resource components, and adaptively allocates hardware resource components statically or dynamically to each running capsule, so that each running capsule provides an independent running environment for task execution, ensuring resource isolation between different tasks and preventing real-time tasks from being affected by other non-real-time tasks. Among them, each running capsule supports the following running scenarios: threads, processes, real-time containers, non-real-time containers, real-time virtual machines, and non-real-time virtual machines. Running capsules can also be divided into real-time running capsules and non-real-time running capsules, which are used to execute real-time tasks and non-real-time tasks respectively. The real-time running capsules include dedicated real-time running capsules for various real-time application tasks. In this embodiment, for the above-mentioned real-time AI reasoning tasks, the elastic microkernel allocates dedicated processor resources and memory resources to the AI-specific real-time running capsules, so that when executing the calculation of the AI ​​reasoning task, the computing resources required for the AI ​​reasoning task are prioritized.

[0101] In some embodiments, the above-mentioned AI-dedicated real-time running capsule includes a real-time runtime environment and basic components used when executing AI reasoning tasks, such as a linear algebra operator library, a processor instruction set dedicated to AI calculations, a computational graph accelerator, a driver for a hardware acceleration processor, etc. When the neural network model component schedules the AI-dedicated real-time running capsule, it can directly use the dedicated resources to perform the calculation of the AI ​​reasoning task, thereby achieving accelerated execution of the AI ​​reasoning task.

[0102] The training and inference processes of neural network models involve a large number of linear algebra operations, such as matrix multiplication, vector addition, and matrix-vector multiplication. Encapsulating the linear algebra operator library in the runtime capsules used for AI inference tasks can effectively improve the efficiency of AI inference tasks and accelerate the output of AI inference results.

[0103] The processor instruction set is a collection of instructions supported by the processor that accelerates operations involved in deep learning models, such as matrix multiplication and convolution. These instructions optimize data transfer between memory and the processor, reducing unnecessary data movement and streamlining the computational process. The instruction set must support parallel computing technologies such as multithreading, vectorization, and SIMD (single instruction, multiple data) to enable simultaneous execution of computational tasks across multiple processor cores, providing advanced vector-matrix operations. SIMD technology allows a single instruction to process multiple data elements simultaneously, which is particularly useful in deep learning, as models often require parallel processing of large amounts of data. Furthermore, the processor instruction set provides mechanisms to reduce memory access latency by optimizing memory access, such as through prefetching and caching, to reduce memory latency and increase data access speed. Furthermore, the instruction set provides specific memory access modes tailored to the data structures of deep learning models to further enhance efficiency. Furthermore, since deep learning models often use floating-point numbers for computation, especially when high precision is required, the processor instruction set must also provide efficient support for floating-point operations, including addition, multiplication, division, and exponential operations. At the same time, for some scenarios where accuracy requirements are not so strict, the processor instruction set can also provide fixed-point number support to further reduce computational complexity and power consumption.

[0104] Neural network models are expressed using computational graphs, which visualize all mathematical operations and the dependencies between variables in graphical form. The computational graph optimization accelerator optimizes the graph and reorganizes its execution order through operator fusion and pruning. This reduces unnecessary and duplicate computations and improves memory efficiency, thereby accelerating model training and inference.

[0105] In some embodiments, since the AI ​​reasoning task is a complex reasoning process, the neural network model component may also decompose the AI ​​reasoning task into a series of subtasks based on the complexity of the AI ​​reasoning task when reasoning on the AI ​​reasoning task. Then, based on the resource requirements and time deterministic constraints of each subtask and the actual resource conditions of the computing node, a matching AI-specific real-time running capsule is scheduled for each subtask, thereby achieving parallel execution of multiple subtasks and accelerating the reasoning speed. Specifically, when the AI ​​reasoning task is decomposed into multiple subtasks, and the neural network model component dispatches multiple AI-specific real-time running capsules for the multiple subtasks to be distributed on different computing nodes, the execution delay of each subtask by the multiple AI-specific real-time running capsules and the transmission delay of the execution data between each other jointly determine the delay that satisfies the time deterministic constraints of the AI ​​reasoning task.

[0106] In some embodiments, the neural network model component can also determine the priority of the AI ​​reasoning task based on the time deterministic constraints of the AI ​​reasoning task. The neural network model component uses a priority scheduling algorithm and a real-time scheduling algorithm to schedule an AI-specific real-time running capsule that matches the priority of the AI ​​reasoning task, so that the AI-specific real-time running capsule can execute the AI ​​reasoning task first. The high-priority AI-specific real-time running capsule can obtain the hardware resources required to execute the AI ​​reasoning task first, or can preempt the hardware resources occupied by the low-priority running capsule.

[0107] In some embodiments, the neural network model component stores information about the basic components required by the model and the memory requirements of each model layer. The neural network model component selects an AI-specific real-time run capsule to run the AI ​​reasoning task based on the basic components required by the model and the memory requirements of each model layer. The AI-specific real-time run capsule sends the memory requirements of each model layer analyzed by the neural network model component to the memory management component of the elastic microkernel; the memory management component selects a memory allocation strategy based on the requirement information. The memory allocation strategy can be a combination of one or more of the following mechanisms provided by the memory management component: a fixed-length memory allocation mechanism, a hierarchical allocation mechanism, and a static memory allocation mechanism.

[0108] During the model initialization phase, a static memory allocation method can be used to allocate a sufficiently large memory block at one time based on the model structure and parameter scale, ensuring that all memory requirements during model operation are met in this memory block, thereby avoiding the fragmentation problem caused by dynamic allocation.

[0109] If the various layers or operations of the model have regular memory requirements, a fixed-length memory allocation method can be used to allocate several fixed-size memory blocks to the AI-specific running capsule to meet the regular memory requests of the model.

[0110] Hierarchical memory management divides memory space into multiple levels or regions, each with specific uses and management rules. For example, a layer of memory regions uses a virtual memory management mechanism, combining physical memory with swap files on the hard disk to temporarily swap inactive data out to the hard disk, alleviating physical memory pressure. A layer of memory regions is set up to cache frequently used data, such as recently used data samples and intermediate calculation results, to reduce frequent access to external storage devices. A layer of memory regions is set up to pre-allocate and reuse fixed-size regions for tensors or data structures, effectively reducing memory fragmentation.

[0111] Furthermore, the present invention also provides a method for accelerating the execution of AI reasoning tasks by dynamically scheduling CPU resources through an elastic microkernel:

[0112] The runtime capsules on each compute node (i.e., physical node) are located in the node's adaptive partition, with each adaptive partition configured with a budgeted CPU runtime. When the node's elastic microkernel allocates hardware resources to the runtime capsules, it schedules the adaptive partition's budgeted CPU runtime for the tasks in the runtime capsules of each adaptive partition on the node. When the actual CPU runtime used by any adaptive partition on the node exceeds its budgeted CPU runtime, the remaining CPU runtime of another adaptive partition is allocated to the runtime capsules of other adaptive partitions with AI inference tasks on the node. Specifically, to accelerate the execution of AI inference tasks and meet time determinism constraints, the budgeted CPU runtime of one adaptive partition can be allocated to the AI ​​inference tasks on the dedicated AI real-time runtime capsules. By scheduling the remaining CPU runtime of other adaptive partitions to the adaptive partition (i.e., the high-priority adaptive partition) where the dedicated AI real-time runtime capsules are located, dynamic resource scheduling is achieved for the high-priority adaptive partitions, ensuring that AI inference tasks with time determinism constraints can obtain CPU resources.

[0113] Each adaptive partition can support scenarios such as threads, processes, containers, or virtual machines. Components for each adaptive partition are allocated by the elastic microkernel from resource components within the elastic microkernel according to the adaptive partition's budget ratio. The usage time of these allocated resource components can be dynamically adjusted by the elastic microkernel. In practical scenarios, there may be multiple adaptive partitions, each supporting one of the following scenarios: threads, processes, containers, or virtual machines. The number of threads, processes, containers, or virtual machines in each adaptive partition can vary. The elastic microkernel includes a resource component and a kernel scheduler. The resource component is a virtual component that abstracts computer resources, isolating the adaptive partition from computer resources. The kernel scheduler is connected to the partition scheduler of each adaptive partition, and the kernel scheduler within the elastic microkernel directly schedules the tasks of the adaptive partition. Each partition scheduler of each adaptive partition is connected to the elastic microkernel, and the elastic microkernel schedules the tasks of each adaptive partition based on the data structure of the partition scheduler of each adaptive partition, achieving flexible scheduling of computer resources. The existing technology uses the microkernel to schedule resources for each partition. At this time, the resources of each partition have been statically scheduled, and then the partition scheduler inside the partition performs secondary scheduling, which cannot achieve flexible scheduling of computer resources.

[0114] In some embodiments, when the matched neural network model component performs reasoning on the AI ​​reasoning task, the artificial intelligence engine component can also monitor the running status of the neural network model component in real time, increase the priority of the AI ​​reasoning task according to the running status, and increase hardware resources for the matched AI dedicated real-time running capsule, so that the reasoning result output by the neural network model component meets the time deterministic constraint. Alternatively, the computing resource allocation in the AI ​​dedicated real-time running capsule can be adjusted according to the running status of the neural network model component, so that the reasoning result output by the neural network model component meets the time deterministic constraint. Specifically, the monitoring and adjustment process includes the following methods:

[0115] Monitor the load of each computing node scheduled by the neural network model component in real time. When the performance of a computing node degrades, notify the computing node to create a new AI-specific real-time run capsule to enable AI reasoning tasks to run in parallel on multiple AI-specific real-time run capsules, or increase hardware resources for the running AI-specific real-time run capsules.

[0116] Monitor the response time of each computing node scheduled by the neural network model component in real time. When the response time of a computing node exceeds the set threshold, notify the computing node to create a new AI-specific real-time running capsule to enable the AI ​​reasoning task to run in parallel on multiple AI-specific real-time running capsules, or increase hardware resources for the running AI-specific real-time running capsule.

[0117] Monitor the average response time of multiple inference tasks in the neural network model component in real time. When the average response time exceeds the set threshold, notify the computing node to create a new AI-specific real-time running capsule to enable the AI ​​inference task to run in parallel on multiple AI-specific real-time running capsules, or increase hardware resources for the running AI-specific real-time running capsules.

[0118] Monitor the hardware resource utilization of each computing node scheduled by the neural network model component in real time. When the hardware resources are sufficient, notify the computing node to create a new AI-specific real-time run capsule to enable AI reasoning tasks to run in parallel on multiple AI-specific real-time run capsules, or increase hardware resources for the running AI-specific real-time run capsules.

[0119] Monitor the hardware resource utilization of each computing node scheduled by the neural network model component in real time. When hardware resources are insufficient, schedule AI-specific real-time running capsules on other computing nodes for AI inference tasks.

[0120] In some embodiments, while the AI-dedicated real-time running capsule is executing calculations, the performance of the AI-dedicated real-time running capsule can also be monitored in real time through the elastic microkernel corresponding to the AI-dedicated real-time running capsule. When the performance degrades, the dedicated processor resources and memory resources allocated to the AI-dedicated real-time running capsule are increased to cope with performance fluctuations or resource bottlenecks that may occur during task execution.

[0121] Based on the allocation of computing resources and hardware resources in the above-mentioned AI-specific real-time running capsules, the execution data of AI inference tasks in the AI-specific real-time running capsules can be collected and analyzed, saved to the artificial intelligence engine component, and based on the analysis results, deep learning and optimization are performed on the AI-specific real-time running capsule scheduling strategy and resource allocation strategy of the neural network model component. This helps to continuously improve and enhance the output speed of the inference results of the neural network model component.

[0122] In summary, in the real-time and deterministic artificial intelligence reasoning task scheduling method provided in the embodiment of the present application, the artificial intelligence engine component located at the platform layer assigns the AI ​​reasoning task to the matching neural network model component according to the task requirements and time determinism constraints of the received AI reasoning task, and parses the AI ​​reasoning task through the neural network model component. According to the resource requirements, time determinism constraints and actual resource conditions of each computing node of the AI ​​reasoning task, the matching AI dedicated real-time running capsule is scheduled for the AI ​​reasoning task. The resources provided by the AI ​​dedicated real-time running capsule enable the neural network model component to complete the output of the reasoning result within a given time period, meet the time determinism constraints of the AI ​​reasoning task, and be suitable for business scenarios with high real-time requirements. It has broad application prospects and commercial value.

[0123] As shown in Figure 3, an embodiment of the present application provides an artificial intelligence reasoning task scheduling system with real-time and deterministic performance, which can be used to implement any step of the method for scheduling artificial intelligence reasoning tasks with real-time and deterministic performance shown in Figure 2 above and its optional embodiments. Referring to Figure 3, the system includes an artificial intelligence engine component 410 located at the platform layer and a computing node located at the base layer. The artificial intelligence engine component 410 can be deployed on an edge management node or in the cloud. The artificial intelligence engine component 410 includes a neural network model component for different industrial application scenarios; each computing node includes an elastic microkernel 420 and several running capsules, such as an AI-specific real-time running capsule 430 for executing AI reasoning real-time tasks, a real-time running capsule 440 for executing robot control real-time tasks, and a non-real-time running capsule 450 for executing non-real-time tasks, etc. The elastic microkernel 420 is used to flexibly allocate hardware resources to the running capsules; each running capsule is isolated from each other, and the running capsule is used to provide an independent running environment for task execution;

[0124] Among them, the artificial intelligence engine component 410 receives AI reasoning tasks from the service layer, and allocates the AI ​​reasoning tasks to the matching neural network model components according to the task requirements and time determinism constraints of the received AI reasoning tasks; the matched neural network model components, based on the resource requirements and time determinism constraints of the AI ​​reasoning tasks and the actual resource conditions of the computing nodes, schedule the matching AI-specific real-time running capsule 430 for the AI ​​computing task from multiple running capsules of the computing node. The AI-specific real-time running capsule 430 includes a real-time runtime environment and basic components used when executing AI reasoning tasks.

[0125] In some embodiments, the artificial intelligence engine component 410 can be implemented by the artificial intelligence engine middleware 230 in the industrial operating system shown in Figure 1, the elastic microkernel 420 in the computing node can be implemented by the elastic microkernel 110 in the industrial operating system shown in Figure 1, and each running capsule can be implemented by the running capsule 120 in the industrial operating system shown in Figure 1. It should be noted that during the implementation of the application scenario, the artificial intelligence engine component 410, the elastic microkernel 420, and each running capsule in the embodiment of the present application need to perform corresponding resource deployment and function configuration for the executed tasks.

[0126] It should be understood that the system or module in the embodiment of the present application can be implemented by software, for example, it can be implemented by a computer program or instruction having the above functions, and the corresponding computer program or instruction can be stored in a memory inside the terminal, and the processor reads the corresponding computer program or instruction inside the memory to implement the above functions. Alternatively, the system or module in the embodiment of the present application can also be implemented by hardware. Alternatively, the system or module in the embodiment of the present application can also be implemented by a combination of a processor and a software module.

[0127] It should be understood that the processing details of the system or module in the embodiments of the present application can refer to the relevant descriptions of the embodiments shown in Figures 1-2 and related extended embodiments, and the embodiments of the present application will not be repeated.

[0128] 4 is a structural diagram of a computing device 500 provided in an embodiment of the present application. The computing device 500 includes: a processor 510 , a memory 520 , a communication interface 530 , and a bus 540 .

[0129] It should be understood that the communication interface 530 in the computing device 500 shown in FIG. 4 can be used to communicate with other devices.

[0130] The processor 510 may be connected to a memory 520. The memory 520 may be used to store the program code and data. Therefore, the memory 520 may be a storage unit within the processor 510, an external storage unit independent of the processor 510, or a component including both a storage unit within the processor 510 and an external storage unit independent of the processor 510.

[0131] Optionally, the computing device 500 may further include a bus 540. The memory 520 and the communication interface 530 may be connected to the processor 510 via the bus 540. The bus 540 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. The bus 540 may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, FIG4 shows only one line, but this does not imply that there is only one bus or only one type of bus.

[0132] It should be understood that in the embodiment of the present application, the processor 510 can adopt a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. Alternatively, the processor 510 uses one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiment of the present application.

[0133] The memory 520 may include a read-only memory and a random access memory, and provides instructions and data to the processor 510. A portion of the processor 510 may also include a non-volatile random access memory. For example, the processor 510 may also store information about the device type.

[0134] When the computing device 500 is running, the processor 510 executes the computer execution instructions in the memory 520 to perform the operating steps of the above-mentioned real-time and deterministic artificial intelligence reasoning task scheduling method.

[0135] It should be understood that the computing device 500 according to the embodiment of the present application can correspond to the corresponding subject in executing the method according to each embodiment of the present application, and the above-mentioned other operations and / or functions of each module in the computing device 500 are respectively for implementing the corresponding processes of each method of the present embodiment. For the sake of brevity, they will not be repeated here.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0142] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it is used to execute the above-mentioned real-time and deterministic artificial intelligence reasoning task scheduling method, which includes at least one of the schemes described in the above-mentioned embodiments.

[0143] The computer storage medium of the embodiment of the present application can adopt any combination of one or more computer-readable media.Computer-readable media can be computer-readable signal media or computer-readable storage media.Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof.More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connection with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof.In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0144] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0146] The computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0147] It should be noted that the embodiments described in this application are only a part of the embodiments of this application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the above detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the application for protection, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0148] The words "first, second, third, etc." or module A, module B, module C and other similar terms in the specification and claims are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that the specific order or sequence can be interchanged where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0149] In the above description, the numbers representing the steps involved do not necessarily mean that the steps must be executed. Intermediate steps may also be included or replaced by other steps. If permitted, the order of the previous and next steps may be interchanged or executed simultaneously.

[0150] The term "comprising" as used in the specification and claims should not be construed as limiting to what is listed thereafter; it does not exclude other elements or steps. Thus, it should be interpreted as specifying the presence of the features, integers, steps, or components mentioned, but not excluding the presence or addition of one or more other features, integers, steps, or components, or groups thereof. Thus, the expression "a device comprising means A and B" should not be limited to a device consisting solely of components A and B.

[0151] The term "one embodiment" or "an embodiment" mentioned in this specification means that the specific features, structures, or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing in various places in this specification do not necessarily refer to the same embodiment, but may refer to the same embodiment. In addition, in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0152] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present application has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A real-time and deterministic artificial intelligence reasoning task scheduling method, characterized in that: include: After receiving a scheduling request for an AI reasoning task, the artificial intelligence engine component allocates the AI ​​reasoning task to a matching neural network model component based on the task requirements and time determinism constraints of the AI ​​reasoning task; the artificial intelligence engine component includes neural network model components for different industrial application scenarios; The matched neural network model component schedules a matched AI-specific real-time running capsule for the AI ​​reasoning task among multiple running capsules of the computing node based on the resource requirements and time deterministic constraints of the AI ​​reasoning task and the actual resource situation of the computing node. Each of the running capsules is isolated from each other. The running capsule is used to provide an independent running environment for task running. The AI-specific real-time running capsule includes a real-time runtime environment and basic components used when executing the AI ​​reasoning task.

2. The method according to claim 1, characterized in that Based on the resource requirements and time determinism constraints of the AI ​​reasoning task and the actual resource situation of the computing node, the AI ​​dedicated real-time running capsules matched for scheduling the AI ​​reasoning task among the multiple running capsules of the computing node include: According to the resource requirements of the AI ​​reasoning task, the configuration information and historical execution data of the currently idle AI-dedicated real-time running capsules on the computing node, the execution time of running the AI ​​reasoning task on the AI-dedicated real-time running capsule is estimated, and among multiple currently idle running capsules, one or more AI-dedicated real-time running capsules are selected to run the AI ​​reasoning task to meet the time determinism constraint of the AI ​​reasoning task.

3. The method according to claim 1, characterized in that The basic components used to execute AI reasoning tasks include one or any combination of the following: linear algebra operator library, processor instruction set dedicated to AI computing, computational graph accelerator, and driver for hardware acceleration processor.

4. The method according to claim 3, characterized in that The processor instruction set for the AI-specific real-time capsule operation provides a mechanism for reducing memory access latency.

5. The method according to claim 1, wherein The AI ​​dedicated real-time running capsule is allocated with dedicated processor resources and memory resources.

6. The method according to claim 1, characterized in that The elastic microkernel virtualizes hardware resources into multiple independent hardware resource components; the elastic microkernel statically or dynamically allocates hardware resource components to the AI-specific real-time running capsule.

7. The method according to claim 1, characterized in that The neural network model component stores basic components required by the model and memory requirements of each level of the model. The method further includes: The matched neural network model component selects the AI ​​dedicated real-time running capsule to run the AI ​​reasoning task based on the basic components required by the model and the memory requirements of each level of the model.

8. The method according to claim 7, characterized in that The method further comprises: The AI ​​dedicated real-time operation capsule sends the memory requirement information of each layer of the model provided by the neural network model component to the memory management component of the elastic microkernel; The memory management component selects a memory allocation strategy based on the demand information. The memory allocation strategy includes one or more of the following mechanism combinations provided by the memory management component: fixed-length memory allocation mechanism, hierarchical allocation mechanism, and static memory allocation mechanism.

9. The method according to claim 1, characterized in that Based on the resource requirements and time determinism constraints of the AI ​​reasoning task and the actual resource situation of the computing node, the AI ​​dedicated real-time running capsules matched for scheduling the AI ​​reasoning task among the multiple running capsules of the computing node include: According to the complexity of the AI ​​reasoning task, the AI ​​reasoning task is decomposed into a series of subtasks that can be processed by the neural network model component. According to the resource requirements and time deterministic constraints of each subtask and the actual resource conditions of the computing node, a matching AI dedicated real-time running capsule is scheduled for each subtask.

10. The method according to claim 1, characterized in that Also includes: When the AI ​​reasoning task is decomposed into multiple subtasks, and the neural network model component schedules multiple AI-specific real-time running capsules that match the multiple subtasks and are distributed on different computing nodes, the delay jointly determined by the execution delay of each subtask by the multiple AI-specific real-time running capsules and the transmission delay of execution data between each other meets the time determinism constraint of the AI ​​reasoning task.

11. The method according to claim 1, wherein Also includes: The priority of the AI ​​reasoning task is determined according to the time deterministic constraint of the AI ​​reasoning task. The neural network model component uses a priority scheduling algorithm and a real-time scheduling algorithm to schedule an AI-specific real-time running capsule that matches the priority of the AI ​​reasoning task, so that the AI-specific real-time running capsule gives priority to executing the AI ​​reasoning task. The high-priority AI-specific real-time running capsule can obtain the hardware resources required to execute the AI ​​reasoning task first, or can preempt the hardware resources occupied by the low-priority running capsule.

12. The method according to claim 1, characterized in that When there are multiple AI-specific real-time running capsules that meet the resource requirements, a matching AI-specific real-time running capsule is scheduled for the AI ​​reasoning task based on load balancing and / or latency.

13. The method according to claim 1, wherein Also includes: Monitor the running status of the matched neural network model component in real time, and adjust the computing resource allocation in the AI ​​dedicated real-time running capsule according to the running status so that the inference result output by the neural network model component meets the time determinism constraint.

14. The method according to claim 1, wherein Also includes: Monitor the running status of the matched neural network model component in real time, increase the priority of the AI ​​reasoning task according to the running status, and add hardware resources to the matched AI dedicated real-time running capsule so that the reasoning result output by the neural network model component meets the time determinism constraint.

15. The method according to claim 13, characterized in that The real-time monitoring of the running status of the matched neural network model component and adjusting the computing resource allocation in the AI-dedicated real-time running capsule according to the running status include at least one of the following: Monitor the load of each computing node scheduled by the neural network model component in real time. When the performance of a computing node degrades, notify the computing node to create a new AI-specific real-time running capsule to enable the AI ​​reasoning task to run in parallel on multiple AI-specific real-time running capsules, or increase hardware resources for the running AI-specific real-time running capsule; monitoring the response time of each computing node scheduled by the neural network model component in real time, and when the response time of a computing node exceeds a set threshold, notifying the computing node to create a new AI-specific real-time running capsule to enable the AI ​​reasoning task to run in parallel on multiple AI-specific real-time running capsules, or to increase hardware resources for the running AI-specific real-time running capsule; monitoring the average response time of multiple inference tasks in the neural network model component in real time, and when the average response time exceeds a set threshold, notifying the computing node to create a new AI-specific real-time running capsule to enable the AI ​​inference task to run in parallel on multiple AI-specific real-time running capsules, or to increase hardware resources for the running AI-specific real-time running capsule; monitoring the hardware resource utilization of each computing node scheduled by the neural network model component in real time, and when the hardware resources are sufficient, notifying the computing node to create a new AI-specific real-time running capsule to enable the AI ​​reasoning task to run in parallel on multiple AI-specific real-time running capsules, or to increase hardware resources for the running AI-specific real-time running capsule; The hardware resource utilization of each computing node scheduled by the neural network model component is monitored in real time. When the hardware resources are insufficient, the AI-specific real-time running capsule on other computing nodes is scheduled for the AI ​​reasoning task.

16. The method according to claim 13, characterized in that Also includes: During the process of the AI ​​dedicated real-time running capsule executing calculations, the performance of the AI ​​dedicated real-time running capsule is monitored in real time through the elastic microkernel corresponding to the AI ​​dedicated real-time running capsule. When the performance degrades, the dedicated processor resources and memory resources allocated to the AI ​​dedicated real-time running capsule are increased.

17. The method according to claim 1 or 13, characterized in that Also includes: Collect and analyze the execution data of the AI ​​reasoning task in the AI ​​dedicated real-time running capsule, save it to the artificial intelligence engine component, and optimize the AI ​​dedicated real-time running capsule scheduling strategy of the neural network model component and the resource allocation strategy of the AI ​​dedicated real-time running capsule based on the analysis results.

18. The method according to claim 1, wherein The running capsule supports one of the following running scenarios: thread, process, real-time container, non-real-time container, real-time virtual machine, non-real-time virtual machine; The running capsules include real-time running capsules and non-real-time running capsules, wherein the real-time running capsules include dedicated real-time running capsules for various real-time application tasks.

19. A real-time and deterministic artificial intelligence reasoning task scheduling system, characterized by: The platform layer's artificial intelligence engine components are deployed on edge management nodes or in the cloud. These components include neural network model components for different industrial application scenarios. Each computing node includes an elastic microkernel and several runtime capsules. The elastic microkernel is used to flexibly allocate hardware resources to the runtime capsules. Each running capsule is isolated from each other and is used to provide an independent running environment for task running; The artificial intelligence engine component receives AI reasoning tasks from the service layer and assigns the AI ​​reasoning tasks to matching neural network model components based on the task requirements and time determinism constraints of the received AI reasoning tasks. The matched neural network model component schedules a matched AI-specific real-time running capsule for the AI ​​computing task from multiple running capsules of the computing node according to the resource requirements and time deterministic constraints of the AI ​​reasoning task and the actual resource situation of the computing node. The AI-specific real-time running capsule includes a real-time runtime environment and basic components used when executing the AI ​​reasoning task.

20. A computer-readable storage medium, characterized in that A computer program is stored thereon, which, when executed by a computer, implements a real-time and deterministic artificial intelligence reasoning task scheduling method as described in any one of claims 1 to 18.

Images